Beyond Exact Match: An Empirical Diagnostic Evaluation and Failure Analysis of Open-Weight Vision-Language Models on Regional Visual Datasets
编号:83 访问权限:仅限参会人 更新:2026-11-10 16:42:33 浏览:10次 Online

报告开始:2026年11月21日 12:15(Asia/Kolkata)

报告时间:15min

所在会场:[S4] Artificial Intelligence and Intelligent Computing [S4-7] Artificial Intelligence and Intelligent Computing

暂无文件

摘要
Vision-language models (VLMs) have made rapid advances as multimodal systems for image understanding, read ing text, and visual reasoning. Recent architectures such as BLIP-2, Qwen-VL, and Qwen2-VL have advanced multimodal alignment through frozen-modality bridging, broad task gener alization, and dynamic visual tokenization. Yet robustness with respect to text remains limited, particularly in the case of multilingual text, degraded visual documents, and specialized agricultural imagery. Although current benchmarks are wide in scope, they often fail to adequately represent localized scripts, regional document conditions, and detailed field imagery. In this paper, a 180-sample curated diagnostic benchmark covering three categories—Financial Receipts, Public Signage, and Agricultural Plant Pathology—is presented (n = 60 per domain). We evaluate open-weight vision-language models, utilizing Qwen2-VL-7B Instruct alongside Qwen2-VL-2B-Instruct under controlled 4-bit quantization, and compare strict Exact Match (EM) accuracy against Normalized Levenshtein Similarity. Our findings demon strate that strict binary string evaluation frequently penalizes predictions that differ from the target primarily through for matting variations (e.g., currency symbols, commas, and trailing zeros) rather than fundamental visual misinterpretations: while the 7B model attains only 56.67% EM on receipts, it reaches a Mean Levenshtein Similarity of 0.861. Furthermore, we provide an independently validated failure taxonomy on all 127 Exact Match errors, isolating non-Latin Devanagari script breakdown (0.00% EM on pure Hindi signboards) and zero-shot taxonomic abstraction in crop pathology.
关键词
Vision-language models, optical character recog nition, multimodal benchmarking, document understanding, fine grained classification, regional evaluation
报告人
Suparna Lahiri
Student Indira Gandhi Delhi Technical University for Women

稿件作者
Suparna Lahiri Indira Gandhi Delhi Technical University for Women
Swastha Gupta Indira Gandhi Delhi Technical University for Women
Srishti Singh Indira Gandhi Delhi Technical University for Women
发表评论
验证码 看不清楚,更换一张
全部评论
重要日期
  • 会议日期

    11月20日

    2026

    至

    11月22日

    2026

  • 10月31日 2026

    初稿截稿日期

  • 11月07日 2026

    提前注册日期

  • 11月22日 2026

    注册截止日期

主办单位
United Societies of Science
承办单位
Pondicherry University
协办单位
IEEE Section
IEEE Madras Section
移动端
在手机上打开
小程序
打开微信小程序
客服
扫码或点此咨询