Automatic concept-guided fine-tuning steers vision transformers toward semantic reasoning and improves robustness.
Vision Transformers (ViTs) often degrade under distribution shifts because they rely on spurious correlations, such as background cues, rather than semantically meaningful features. Existing regularization methods, typically relying on simple foreground-background masks, which fail to capture the fine-grained semantic concepts that define an object (e.g., ``long beak'' and ``wings'' for a ``bird''). As a result, these methods provide limited robustness to distribution shifts. To address this limitation, we introduce a novel finetuning framework that steers model reasoning toward concept-level semantics. Our approach optimizes the model's internal relevance maps to align with spatially grounded concept masks. These masks are generated automatically, without manual annotation: class-relevant concepts are first proposed using an LLM-based, label-free method, and then segmented using a VLM. The finetuning objective aligns relevance with these concept regions while simultaneously suppressing focus on spurious background areas. Notably, this process requires only a minimal set of images and uses half of the dataset classes. Extensive experiments on five out-of-distribution benchmarks demonstrate that our method improves robustness across multiple ViT-based models. Furthermore, we show that the resulting relevance maps exhibit stronger alignment with semantic object parts, offering a scalable path toward more robust and interpretable vision models. Finally, we confirm that concept-guided masks provide more effective supervision for model robustness than conventional segmentation maps, supporting our central hypothesis.
Vision Transformers rely on spurious correlations like background textures instead of learning semantic object features, causing poor generalization.
Concept-Guided Fine-Tuning uses LLMs and vision-language models to automatically identify and localize semantic concepts without manual annotation.
CFT improves robustness on out-of-distribution benchmarks using only 1,500 images and half of ImageNet classes, generalizing to unseen classes.
Method requires vision-language grounding models like Grounded-SAM, adding computational overhead to the fine-tuning pipeline.
Validation threshold for concept presence (15%) and coverage (20%) are fixed hyperparameters that may not generalize across all domains.
Maijunxian Wang, Ruisi Wang +54
Rapid progress in video models has largely focused on visual quality, leaving their reasoning capabilities underexplored. Video reasoning grounds intelligence in spatiotemporally consistent visual env...
Xiangyan Qu, Zhenlong Yuan +10
Image Chain-of-Thought (Image-CoT) is a test-time scaling paradigm that improves image generation by extending inference time. Most Image-CoT methods focus on text-to-image (T2I) generation. Unlike T2...
Junyi Zhang, Charles Herrmann +6
Feedforward geometric foundation models achieve strong short-window reconstruction, yet scaling them to minutes-long videos is bottlenecked by quadratic attention complexity or limited effective memor...
Yucheng Wang, Zedong Wang +3
Unified diffusion editors often rely on a fixed, shared backbone for diverse tasks, suffering from task interference and poor adaptation to heterogeneous demands (e.g., local vs global, semantic vs ph...
Yang Cao, Feize Wu +4
Current multi-view indoor 3D object detectors rely on sensor geometry that is costly to obtain (i.e., precisely calibrated multi-view camera poses) to fuse multi-view information into a global scene r...
