When Labels Say No, But Embeddings Say Yes
AI Summary: The article examines whether label‑based relevance metrics unfairly penalize retrievals that are semantically similar but belong to different categories, using CLIP embeddings on a Kaggle dataset of 9,315 images spanning 23 architectural styles. Baseline performance is reported with Precision@1, @5, and @10 measured against exact style labels, revealing a gap between label agreement and visual similarity. Detailed analysis of CLIP’s “mistakes” shows that many label mismatches correspond to buildings sharing roof types, materials, symmetry, or window patterns, indicating meaningful cross‑style similarity. The authors conclude that retrieval evaluation should incorporate graded similarity rather than binary label matching to better reflect model performance.