Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers
Abstract
Visual geometry transformers are accelerated through a two-stage token selection framework that reduces computational costs while maintaining performance.
Visual geometry transformers have become powerful architectures for multi-view 3D reconstruction, enabling joint prediction of multiple 3D attributes in a feed-forward manner. However, their computational cost grows quadratically with the input sequence length due to the global attention layers inside these models. This limits both their scalability and efficiency. In this work, we address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention. To achieve effective token selection, we introduce a two-stage framework. First, an inter-frame selection step operates at the frame level to identify frames that should be preserved. Second, an intra-frame selection step further discards more redundant tokens within the selected frames. Our analysis highlights the advantage of a diversity-based strategy for inter-frame selection, which ensures broad coverage of the scene. For intra-frame selection, we show that layer-aware sparsification is necessary, with the selection process guided by the entropy of the global attention pattern. Our approach offers a superior speed-accuracy trade-off compared to existing solutions. Extensive experiments show that it accelerates visual geometry transformers by over 85% for scenes with 500 images while maintaining, or even improving, baseline performance, which hints that how our token selection strategy can play a crucial role in future applications of visual geometry transformers. Our project website is available at https://zsh2000.github.io/good-token-hunting.github.io.
Community
General solution for accelerating visual geometry transformers.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction (2026)
- TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention (2026)
- Geometry-Guided 3D Visual Token Pruning for Video-Language Models (2026)
- GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction (2026)
- PaceVGGT: Pre-Alternating-Attention Token Pruning for Visual Geometry Transformers (2026)
- StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression (2026)
- SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answering (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2605.23892 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper