Memory Pollution in Multi-Product Anomaly Detection: Diagnosis and Resolution

In multi-product industrial inspection, maintaining one memory bank per product yields costs that scale linearly with the number of product types. A shared bank with a fixed memory budget is more practical, but mixing embeddings from different products introduces inter-product interference. We call this memory pollution: nearest-neighbor queries retrieve features from other products, and budget allocations optimized on isolated banks degrade once retrieval is shared. Across 15 MVTec AD products, a per-product oracle allocator underperforms uniform allocation by 1.6 percentage points (pp) at 18 MB, and the wrong-neighbor rate (WNR) reaches 38% at 2.9 MB. We address this with a training-free router based on mean-embedding prototypes that identifies the product before nearest-neighbor search. The router adds 0.06 MB and achieves perfect top-1 accuracy over 30 product types (MVTec AD, VisA, BTAD). With routing, the performance gap across five allocation strategies shrinks to at most 1 pp. Top 1 routing with uniform allocation improves image-level area under the ROC curve (AUROC) from 90.8% to 91.1% at 18 MB. Coreset selection and clustering further provide 16× memory reduction with less than 1 pp AUROC loss. All components are training-free and operate on frozen DINOv3 features.