Drape — Visual Fashion Retrieval

Upload a photo of an outfit, get the most visually similar products from a 20,269-item fashion catalog — no keyword search, no captioning model. Two parts:

1. Retrieval pipeline (ml/) — YOLOv8 segments each garment out of the uploaded photo, DINOv2 embeds every crop into a 768-d vector that encodes visual appearance (not a class label), and FAISS does exact cosine nearest-neighbour search over the catalog. A neighbour-consensus step then describes the upload in words — using the retrieved items' own metadata, not a second model — and links out to a live AJIO/Myntra search.
2. Serving + UI — a thin FastAPI layer (backend/) over the pipeline, backed by Neon Postgres (product metadata) and Nhost Storage (private, presigned catalog-crop images), and a React + Vite + Tailwind frontend (frontend/). The backend deploys to AWS Lambda (container image, 3 GB memory, CPU inference); the frontend deploys to Render's free static-site tier from one render.yaml Blueprint.

Live demo: drape-web.onrender.com

⚠️ The backend scales to zero

Lambda is billed per-invocation and the container isn't kept warm, so:

• The first search after a while cold-starts — ~30–40s to load YOLOv8 + DINOv2 + the FAISS index into a fresh container. Every search after that is fast (~1.5–2.5s). The frontend's ApiStatus/useApiHealth show a "waking up" state during this window rather than looking broken.
• The catalog is a fixed offline build. 20,269 vectors from the DeepFashion In-shop Clothes Retrieval Benchmark, indexed once via scripts/build_faiss_index.py. Growing it means re-running the offline pipeline and re-baking the Lambda image — there's no live ingestion path.
• Brand and price are synthesised. DeepFashion has neither. They exist to exercise the filter UI and are labelled as fake there; they never reach the external retailer search.

Layout

backend/app/main.py                  lifespan hook — loads YOLOv8/DINOv2/FAISS once at startup, never per-request
backend/app/config.py                every env var the app reads, centralised (pydantic-settings)
backend/app/api/v1/endpoints/        search, products, external_search, health — thin HTTP layer
backend/app/services/search_service.py   orchestrates detect -> embed -> FAISS -> consensus -> results
backend/app/services/product_service.py  /products, /categories, /brands — reads straight from Neon
backend/app/services/product_lookup.py   SQLAlchemy Core queries + row -> JSON normalisation
backend/app/db/client.py             Neon engine (SQLAlchemy + psycopg 3, sync, called in a threadpool)
backend/app/storage/nhost.py         presigned Nhost crop URLs, process-wide TTL cache, parallel fan-out
backend/app/core/                    exceptions, request-id middleware, structlog setup, model registry (/health)
ml/detection/detector.py             YOLOv8-seg wrapper — deepfashion2_yolov8s-seg, 13 garment classes
ml/embeddings/dinov2.py              DINOv2 (facebook/dinov2-base) -> L2-normalised 768-d embedding
ml/indexing/faiss_index.py           IndexFlatIP build/load/search
ml/attributes/consensus.py           neighbour-title consensus -> search phrase (no captioning model)
ml/attributes/colour.py              dominant-colour estimate from the crop
ml/configs/*.yaml                    categories, pipeline, retailers, fashion vocab — read by both
                                      offline scripts/ and the online service (ADR-005 symmetry)
scripts/build_catalog.py             DeepFashion -> dedupe -> validity filters -> catalog.parquet
scripts/detect_and_crop.py           YOLOv8 crop every catalog image once, offline
scripts/generate_embeddings.py       DINOv2 embed every crop once, offline
scripts/build_faiss_index.py         embeddings.npy -> catalog.faiss + id_map.json + manifest.json
scripts/seed_database.py             upload crops to Nhost, upsert rows to Neon
scripts/migrate_from_supabase.py     one-shot Supabase -> Neon/Nhost migration (ADR-014)
scripts/regenerate_crops.py          rebuild crops from a fresh DeepFashion download, keyed to Neon's
                                      existing source_id list — no re-index needed
frontend/src/pages/SearchPage.jsx    upload, cold-start UX, results grid
frontend/src/components/             DetectionOverlay, GarmentTabs, ProductCard, RetailerLinks, ...
frontend/src/lib/api.js              typed fetch wrapper over the FastAPI backend
db/schema.sql                        Neon `products` table
render.yaml                          Render Blueprint — frontend static site only
Dockerfile                           AWS Lambda container image (Lambda Web Adapter, ADR-012)

ml/ is a shared top-level package, imported by both backend/ and scripts/ — the offline indexing pipeline and the live API run byte-identical preprocessing and embedding code, or retrieval quality degrades silently. A FAISS manifest fingerprint check at startup catches drift loudly instead.

Quick start

# Backend (FastAPI) — from repo root
conda create -n drape python=3.11 && conda activate drape
pip install -r requirements.txt -r requirements-dev.txt
pip install -e .                        # makes `ml` importable

cp .env.example .env                    # fill in DATABASE_URL + NHOST_* keys
psql "$DATABASE_URL" -f db/schema.sql   # create the products table on Neon

uvicorn backend.app.main:app --reload   # docs at http://localhost:8000/docs

# Frontend (React / Vite) — from frontend/
npm install
npm run dev                             # http://localhost:5173

curl -X POST localhost:8000/api/v1/search \
     -F "file=@outfit.jpg" -F "max_garments=3" -F "top_k=12"

Endpoints: POST /api/v1/search, GET /api/v1/products/{id}, GET /api/v1/products/{id}/similar, GET /api/v1/categories, GET /api/v1/brands, GET /api/v1/external-search, GET /api/v1/health.

Rebuilding the catalog (offline, needs a GPU-friendly machine)

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128  # match nvidia-smi
pip install -r ml/requirements-train.txt

python scripts/build_catalog.py
python scripts/detect_and_crop.py
python scripts/generate_embeddings.py
python scripts/build_faiss_index.py
python scripts/seed_database.py          # uploads crops to Nhost, upserts rows to Neon

Deploying

Service	Type	Build	Serves
drape-api	AWS Lambda (container image, 3 GB memory)	docker build from Dockerfile → push to ECR → update-function-code	the FastAPI backend; readiness check /api/v1/health
drape-web	Render static site (render.yaml)	npm ci && npm run build in frontend/, publish dist/	the React app; SPA rewrite to /index.html

The backend stays on Lambda rather than Render — DINOv2 + YOLOv8 + Torch + FAISS don't fit Render's memory tiers (ADR-012). VITE_API_BASE_URL is baked into render.yaml at build time. After a Lambda redeploy, CORS_ORIGINS needs the Render origin; after a Render redeploy, VITE_API_BASE_URL needs the Lambda Function URL — both are already wired to each other's current values in this repo.

Model weights (models/weights/*.pt) and the FAISS index (models/index/*) are gitignored and baked directly into the Lambda image — nothing is downloaded at cold start beyond what's already in that layer.

Method notes

Retrieval, not classification. Every catalog image is embedded once, offline, into a space that encodes visual appearance. An uploaded image goes through the identical pipeline at query time, and FAISS finds its nearest neighbours in that space — so the system answers "what looks like this?" instead of "what category is this?", which is the actual question a shopper is asking.

DINOv2 over a fine-tuned classifier. DINOv2's self-supervised embedding space preserves visual similarity as its primary signal, not class separability — the property nearest-neighbour retrieval needs. (ADR-001)

FAISS IndexFlatIP, not an ANN index. Exact cosine search over 20k 768-d vectors runs in well under 10ms on Lambda's CPU — there's no accuracy tradeoff to make at this scale, and no external vector-DB service to run. (ADR-002)

The retailer description is a readout of retrieval, not a second model. If four of the five closest catalog neighbours are titled as bomber jackets, that's evidence the upload is one — combined with a dominant-colour estimate from the crop, that becomes an AJIO/Myntra search URL. Cost: ~2ms, no model, no extra memory — and it can't disagree with the results shown next to it, because it's derived from them. It never scrapes or verifies those pages (the UI says "Search on AJIO", never "in stock" or a price), and below a similarity floor it shows nothing rather than a confidently wrong guess. (ADR-004)

Object storage is file-id addressed, not path-addressed. Nhost assigns its own ids on upload; products.image_url/crop_url hold a deterministic uuid5(source_id) rather than a path, and the online service resolves each to a short-lived presigned URL at read time, cached process-wide (backend/app/storage/nhost.py). DeepFashion's licence forbids public redistribution, so the bucket stays private end to end. (ADR-014)

id_map.json is the only join between FAISS and Postgres. Row i of the FAISS index corresponds to id_map[i], a source_id string, which is the join key into products. Index, id map, and DB row count must all agree — migrating the DB (Supabase → Neon) never required rebuilding the index, because the source_id list was preserved exactly. (ADR-003, ADR-014)

Config is duplicated between ml/configs/*.yaml and backend/app/config.py on purpose. Offline scripts/ read the YAML directly; the online service reads Settings. Keeping both in sync by hand, verified by a FAISS manifest fingerprint check at startup, is what catches an offline/online drift loudly instead of silently degrading retrieval quality. (ADR-005)

Limitations

This is a portfolio project, scoped honestly:

• No custom model training — pretrained DINOv2 and YOLOv8 only.
• Catalog is 20k images, not millions; it cannot be grown without re-running the offline pipeline and rebaking the Lambda image.
• No GPU at inference time; latency reflects Lambda's free-tier CPU.
• No personalisation, no learning from user feedback.
• Retrieval quality is bounded by DeepFashion's domain — studio-shot catalog photos — so everyday phone photos in unusual lighting do worse.
• Brand and price are synthesised, not real catalog data (see the cold-start callout above).
• External retailer links are unverified searches, not confirmed matches — the system constructs a URL, it never checks what's on the other end.

License

MIT