Open vocabulary semantic segmentation applies the
dense prediction paradigm to the open-vocabulary setting using
the knowledge about the categories contained in the visionlanguage
models like the CLIP model. However, in most existing
segmentation methods based on CLIP, the computational cost of
prompt generation remains constant for each individual image
but might vary across different categories out of which only
some are relevant for that specific image. In this paper, we
study the task of prompt selection as a form of dynamic
allocation of resources and introduce our proposed method,
Uncertainty-Entropy Guided CLIP (UE-CLIP). UE-CLIP uses
the pre-trained CLIP ViT-L/14-336 image encoder and a class
presence head. Prompt selection for the cross-attention is guided
by binary Shannon entropy and a threshold that can be learned,
hence selecting the most relevant and confident prompts. The
selected prompts yield confidence-weighted scene tokens that get
aggregated along with the visual feature after refinement through
ViT-Adapter and decoding through UperNet. On the ADE20K
dataset, UE-CLIP attains CLIP-SP mIoU level (51.89%) and
demonstrates 76.7% reduction of the number of active prompts
from 30 to 7.0 and 0.23× cost compared to 1.00×.

Open-vocabulary semantic segmentation, prompt sparsification, entropy-guided learning, adaptive computation, CLIP, vision-language models, uncertainty estimation, dynamic inference.