AnyUp: Universal Feature Upsampling (wimmerth.github.io)

Researchers introduced AnyUp, a universal feature upsampler that converts coarse encoder feature maps to high-resolution features without needing encoder-specific training. Unlike prior learned upsamplers that must be retrained for each backbone (e.g., DINO, CLIP, SigLIP), AnyUp is trained once and at inference can upsample features from any encoder, resolution, or channel dimensionality. This makes it immediately useful for many pixel-level tasks (semantic segmentation, monocular depth, surface normals) and reduces the engineering and compute burden of developing per-backbone upsamplers. AnyUp’s design combines three key ideas: a feature-agnostic layer that maps arbitrary input channels into a canonical local representation via learned per-channel filter bases and normalized aggregation; windowed attention where image pixels act as queries, keys fuse downsampled image and coarse features, and values are coarse feature patches to preserve locality and efficiency; and a crop-based training strategy that supervises upsampling only within random high-resolution crops to avoid costly full-res encoder queries. Training uses a mix of cosine and L2 losses plus self- and input-consistency regularizers to keep semantics aligned with the source feature space. Empirically, AnyUp sets new state-of-the-art upsampled-feature accuracy, preserves original feature semantics (verified with frozen linear probes), and generalizes to unseen encoders and model sizes—offering a practical, performant tool for downstream vision pipelines.