Recognizing one specific object among thousands of near-twins.
Context
A mobile collecting app identifies an object from a photo taken with a phone: reflections, angle, blur, sometimes a photo of a screen. Thousands of references look alike. A pre-trained image model, used as is, did not tell them apart well enough.
Approach
We fully retrained a compact open image model (MobileCLIP2) so that it produces one embedding per reference, with a loss function that pushes neighboring references apart (ArcFace). Training starts from the official catalog images, transformed to imitate a real photo, and evaluation is done on real photos. A second model, trained on synthetic images, finds the four corners of the object to straighten it. Both models run on the server and directly on the phone.
Training
- Official catalog images
- Transformed to imitate a real photo
- Embedding model MobileCLIP2 · ArcFace
- Versioned embedding index never overwritten
Recognition
- Photo taken with the phone
- Second model: the four corners the object is straightened
- Embedding of the photo
- Closest reference in the index
Both models run on the server and on the phone.
What we learned
-
The default settings made training collapse: we measured it, then fixed it.
-
When confusions persisted, measurement showed that they came from near-identical variants in the catalog and from an index that needed refreshing. We did not retrain the model.
-
Each embedding index is versioned, never overwritten: rolling back takes only a version change.
