arxiv:2601.16211

Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Published on Jul 2

· Submitted by

Geo Ahn on Jul 10

#3 Paper of the day

NAVER

Upvote

Authors:

Geo Ahn ,

Abstract

RCORE addresses object-driven shortcuts in zero-shot compositional action recognition by using co-occurrence prior regularization and temporal order regularization to improve compositional generalization.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Zero-Shot Compositional Action Recognition (ZS-CAR) requires recognizing novel verb-object combinations composed of previously observed primitives. In this work, we tackle a key failure mode: models predict verbs via object-driven shortcuts (i.e., relying on the labeled object class) rather than temporal evidence. We argue that sparse compositional supervision and verb-object learning asymmetry can promote object-driven shortcut learning. Our analysis with proposed diagnostic metrics shows that existing methods overfit to training co-occurrence patterns and underuse temporal verb cues, resulting in weak generalization to unseen compositions. To address object-driven shortcuts, we propose Robust COmpositional REpresentations (RCORE) with two components. Co-occurrence Prior Regularization (CPR) adds explicit supervision for unseen compositions and regularizes the model against frequent co-occurrence priors by treating them as hard negatives. Temporal Order Regularization for Composition (TORC) enforces temporal-order sensitivity to learn temporally grounded verb representations. Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and consequently improves compositional generalization.

View arXiv page View PDF Project page GitHub 8 Add to collection

Community

ahnGeo

Paper author Paper submitter about 19 hours ago

TL;DR: We quantify how object-driven shortcuts sabotage compositional generalization in video understanding.

A model that has seen "Open Window" and "Close Drawer" should be able to recognize "Open Drawer"—an unseen yet plausible verb–object combination. But we found that when existing models fail, it's mostly a verb-collapse phenomenon: they get the object right and the verb wrong. The model simply assumes the verb that is most often paired with that object in training and thus fails to generalize to unseen compositions.

We show that these object-driven shortcuts are not a matter of insufficient model capacity or pretrained knowledge. They stem from two intertwined, structural problems: (i) sparse compositional supervision and (ii) the learning asymmetry between verbs and objects (objects are simply easier to learn than verbs). We prove this empirically, and we further propose a suite of diagnostic tools that measure exactly how far a model overfits to co-occurrence statistics.

Our diagnostic-driven fix—RCORE (Robust COmpositional REpresentations)—directly mitigates both root causes and delivers consistent gains in unseen-composition generalization across multiple benchmarks.