Beyond Exact Match: An Empirical Diagnostic Evaluation and Failure Analysis of Open-Weight Vision-Language Models on Regional Visual Datasets
编号:83
访问权限:仅限参会人
更新:2026-11-10 16:42:33 浏览:10次
Online
摘要
Vision-language models (VLMs) have made rapid advances as multimodal systems for image understanding, read ing text, and visual reasoning. Recent architectures such as BLIP-2, Qwen-VL, and Qwen2-VL have advanced multimodal alignment through frozen-modality bridging, broad task gener alization, and dynamic visual tokenization. Yet robustness with respect to text remains limited, particularly in the case of multilingual text, degraded visual documents, and specialized agricultural imagery. Although current benchmarks are wide in scope, they often fail to adequately represent localized scripts, regional document conditions, and detailed field imagery. In this paper, a 180-sample curated diagnostic benchmark covering three categories—Financial Receipts, Public Signage, and Agricultural Plant Pathology—is presented (n = 60 per domain). We evaluate open-weight vision-language models, utilizing Qwen2-VL-7B Instruct alongside Qwen2-VL-2B-Instruct under controlled 4-bit quantization, and compare strict Exact Match (EM) accuracy against Normalized Levenshtein Similarity. Our findings demon strate that strict binary string evaluation frequently penalizes predictions that differ from the target primarily through for matting variations (e.g., currency symbols, commas, and trailing zeros) rather than fundamental visual misinterpretations: while the 7B model attains only 56.67% EM on receipts, it reaches a Mean Levenshtein Similarity of 0.861. Furthermore, we provide an independently validated failure taxonomy on all 127 Exact Match errors, isolating non-Latin Devanagari script breakdown (0.00% EM on pure Hindi signboards) and zero-shot taxonomic abstraction in crop pathology.
关键词
Vision-language models, optical character recog nition, multimodal benchmarking, document understanding, fine grained classification, regional evaluation
稿件作者
Suparna Lahiri
Indira Gandhi Delhi Technical University for Women
Swastha Gupta
Indira Gandhi Delhi Technical University for Women
Srishti Singh
Indira Gandhi Delhi Technical University for Women
发表评论