The article analyzes the population risk of the InfoNCE loss function in contrastive learning, demonstrating that using k negative samples brings the risk within O(1/k) of an expected cross-entropy. This metric quantifies the deviation between softmax similarity search on unseen data and an idealized search based on the positive sample generator.
- The analysis complements existing interpretations of InfoNCE in the k-to-infinity limit, which are typically framed in terms of mutual information and alignment versus uniformity.
- A new continuity bound for the InfoNCE loss is introduced via Gâteaux differentiation to quantify generalization performance.
- This bound preserves the averaging structure over negative samples and includes an inverse temperature parameter tunable to account for algorithmic temperature.
- For Lipschitz embedding functions, the study shows that averaging over k negative samples stabilizes the generalization error as k increases.