Fine-Grained Visual Representations through Sparse Selection of Localized Features

Overview

Vision Transformer (ViT) foundation models have become the standard backbone for visual representation learning, yet the features they learn are coarse-grained in that they describe the input at a single, global level of granularity. This coarseness originates in how each model reduces its input to one global representation - a reduction that, while in principle representative of the entire input, inherently obscures the distinct local spatial or temporal regions that constitute fine-grained visual structure. This dissertation demonstrates that a mechanism for sparsely selecting localized features enables visual encoders to model the fine-grained detail that global representations tend to collapse or smooth away. We study four mechanisms for such selectivity, organized into two paradigms - query-based and non-parametric - and in each case we pair the specific global reduction that produces coarseness with the sparse-selection mechanism that replaces it.

Query-based Selection learns to select localized features through cross-attention between a set of queries and the visual features, across three models. CLIP encoders align a single global visual feature with text, collapsing the distinct visual regions into one vector. β-CLIP instead queries the visual features at multiple textual granularities in order to align localized visual features with their corresponding text (global → multi-granular). For video representations, Hypernetworks predicting Implicit Neural Representation (INR) encode an entire video in a single set of weights, smoothing away per-frame detail. HyperGS instead queries the visual features to predict a set of explicit 2D Gaussian primitives for each frame (global weights → local Gaussians).

Non-Parametric Selection revisits existing parameter-free rules. Averaging patch tokens into a global class token, and frame-level features into a global video-level feature, smooths every region into a single mean. Max-pooling instead sparsely selects peak activations across patches and frames (mean-pooling → max-pooling). And softmax attention spreads weight densely across all tokens when contextualizing them. α-entmax instead applies a data-dependent threshold to sparsely select only the relevant tokens (softmax → α-entmax). Across all four mechanisms, we show that sparse selection of localized features represents fine-grained structure more effectively than the global and implicit representations it replaces.