Fashion Item Matcher

Project Background

This system serves content placement: given an influencer's outfit photo, it splits out the fashion items within it, then retrieves the same or highly similar products from a product pool so the upstream can complete product selection and content assembly. It also provides text-to-item selection, content-style matching and pre-placement price validation.

The system is served with FastAPI, uses Elasticsearch to hold both vectors and scalar attributes, uses Kafka to drive incremental sync of products, SKUs, content and rebate flags, and uses multi-node embedding services and a cloud VLM to split images, tag, vectorize and rerank.

My Responsibilities

The project was built by two people. I mainly owned the vector index layer, the ingestion pipeline, price filtering, AI cost governance, data-operation tooling and the external interfaces; ranking fusion, outfit joint selection and the evaluation platform were done collaboratively by the team and are not presented as my individual lead work.

Core Work

1. Migrating the vector-retrieval foundation

The initial version stored vectors in Milvus and used CLIP for encoding, with limited effect on Chinese fashion semantics, and category, price and shop filtering still had to query back to the relational database, so recall and filtering semantics were hard to keep consistent.

I drove a migration to Qwen3-VL-Embedding and Elasticsearch dense_vector: image and text enter the same vector space, and the vector and scalar attributes land in the same document, so one kNN filter completes recall and business filtering together. The write side performs L2 normalization uniformly and uses dot product to reduce the repeated-normalization overhead of high-dimensional queries.

2. A nested multi-vector index

The old index used "one vector per product," keeping only the main item of a set and sharing one vector across multi-color SKUs, so secondary items in a set and exact color matching could not be recalled.

I designed an index structure with a product parent document plus SKU/single-item nested vectors: each single item stores an image vector, an attribute-text vector and item-level tags, so recall results are naturally deduplicated by product. For the issue where a nested kNN parent document's score source is ambiguous, I use inner_hits to select the hit item whose category matches the query, and align the score, image, category and tags consistently into the downstream rerank chain; I also expand the recall categories for sets, so the filtering stage does not miss a target item inside a set.

3. An SKU-granularity vectorization pipeline

I helped build the full encoding chain from SKU data to the vector index:

1. Use an LLM to deduplicate same-item same-color SKUs, failing open on exceptions;
2. Group by image to avoid reprocessing SKUs that share an image;
3. For single items, select an image, crop the subject, apply item-level tagging and encode via the VLM;
4. For sets, split items guided by the title, then tag and encode each item;
5. Use each SKU's own image and color anchor to fix category cross-contamination, color misjudgment and spec-text pollution.

After merging attribute tagging and the product short name into a single VLM call, the related calls at the ingestion stage were roughly halved.

4. Kafka CDC automated ingestion

I built four incremental consumer chains from scratch for products, SKUs, content and rebate-product flags. To handle the message storms produced by upstream bulk updates, I separated poll from business processing, and used a fixed worker pool and a semaphore to limit VLM and embedding concurrency; the same product uses task-level deduplication and pending compensation to ensure "no reburning the model, and no losing the latest state," and I disabled auto offset commit in favor of a manual commit with controllable processing progress.

A nested document can only be overwritten as a whole. To avoid a Kafka update wiping out an operator's manual image change, pinning or soft-delete state, the write reads back the manual-state fields first and merges them, while distinguishing an operator's manual deletion from a system auto-deletion, to prevent a delisted product from being silently restored by an incremental message.

5. Price-filtering system

I converged the platform price range, category rules and the operations exclusion list into a single judgment function, with the platform and category ranges intersected and the exclusion list unioned. The search chain applies the filter at three layers — ES recall, SKU selection and result assembly — and the external batch-validation interface reuses the same rules but uses a different risk semantics for "unknown price."

Price is judged at SKU granularity, with explicit integer price-bucket and decimal exact-match semantics, so the back-office config and the actually-effective rules do not diverge.

6. AI cost metering and governance

I built a dashboard of token details, daily aggregates and cost estimates split by model, stage and source, so the call costs of tagging, image splitting, denoising, image selection, reranking, query understanding and SKU deduplication can be tracked. Based on the metering, I switched high-frequency low-difficulty stages to lightweight models, bringing the related stage cost down to about 1/7 of the original; caching, SKU deduplication and merged calls further reduce repeated consumption.

I also made the statistical boundary explicit: abnormal or canceled calls may not fully reflect the real billing, so a consumer should not mistake the cost table for a source of model success-rate data.

7. Content retrieval and operations tooling

I built the content-style retrieval module from scratch, using a VLM to normalize gender, style, scene, season and tone, combined with vector recall, season-compatibility filtering and a degraded re-search when results are insufficient; I added a multi-grid and face-integrity check, and applied price filtering before random selection.

Alongside it I provided SKU image swap and rollback, listing/delisting and discard reasons, batch refresh with resumable checkpoints, data-sync interfaces, Feishu anomaly alerts, scheduled statistics and a runtime-config page, so index-quality issues can be found and fixed directly by operations.

Key Challenges

• Score attribution: a nested kNN returns the parent document, and the actually-hit item must be aligned from inner_hits, otherwise you may get an end-to-end mismatch of "searching for pants but reranking with the top of a set";
• Manual-state protection: a whole-document overwrite that succeeds but quietly restores a deleted product is a silent failure with no error, so the manual state must be explicitly merged before the write;
• Message storms: a burst of CDC simultaneously occupying the VLM and embedding can drag down the main service, requiring concurrency limits, task deduplication, pending compensation and reliable offsets together;
• Config visibility: environment variables, the database and code defaults form a multi-layer priority, so the states of "overridden, read-only or restart-required" must be visible in the back office, to avoid a save that succeeds but does not actually take effect.

Project Results

• The product pool reached 120k+, maintaining 210k+ SKU vectors and 20k+ content vectors;
• The multi-vector index effectively covers 50k+ products, supporting recall of secondary items in a set and multi-color SKUs, and completed the decommissioning of the single-vector legacy chain;
• After merging tagging and the short name in the ingestion chain, the related VLM calls were roughly halved;
• After switching stages such as denoising, image selection and classification to lightweight models, cost dropped to about 1/7 of the original;
• Content-placement filtering reached a 91.5% pass rate in sampled validation, and responses retain filter details for issue localization.

The figures above are phased project statistics rounded to public-facing granularity; the page does not show internal system names, Topics, keys, interface addresses or specific business figures.

Tech Stack

Python · FastAPI · Elasticsearch · Kafka · Redis · MySQL · Vue 3 · Qwen3-VL-Embedding · VLM · DeepSeek