报告开始:2026年11月21日 11:45(Asia/Kolkata)
报告时间:15min
所在会场:[S2] Internet of Things and Intelligent Sensing Systems [S2-1] Internet of Things and Intelligent Sensing Systems
暂无文件
Autonomous agents built on large language models (LLMs) can begin executing actions that gradually diverge from the semantic objective a user intended, and this drift may not be visible from the final output alone. We formulate this as semantic intention-action divergence and present ShadowTrace, a cost-aware auditing framework that represents an agent's execution as a trajectory, scores its divergence from the stated intention using sentence embeddings, and applies a calibrated decision threshold, frozen on a held-out calibration split and never re-tuned, to raise a monitoring signal. An optional Cascade stage escalates ambiguous trajectories to a local natural-language-inference (NLI) contradiction estimator and, if still ambiguous, to a locally hosted LLM adjudicator. We evaluate the framework on a TraceSafe-Bench-derived corpus of 2,250 labeled agentic trajectories (1,170 benign, 1,080 malicious) under a leakage-safe, pair-grouped train/calibration/test protocol (1,592/317/341), with a single pre-registered hypothesis tested exactly once on held-out data. The full embedding-plus-NLI configuration performs at essentially chance level on the test split (ROC-AUC 0.4959), while a single pre-registered embedding signal performs modestly above chance (ROC-AUC 0.5496) with a substantially lower false-positive rate but reduced recall — evidence we report as promising but insufficient to establish a production-ready auditing signal. An independently discovered corpus defect bounds achievable classification accuracy on this benchmark at 0.7947 (test split) without bounding ranking quality (ROC-AUC ceiling 0.9987), which rules out the corpus defect as an explanation for the near-chance result. Measured whole-trajectory latency (mean 4507 ms, tail to 85.8 s) motivated a documented architectural decision to run auditing asynchronously relative to agent execution; we report this decision honestly as a design choice, not as a benchmarked live-monitoring result, since the only implemented interface is an offline trajectory-replay demonstration. The Cascade's LLM-adjudication stage is implemented but was not exercised in any reported experiment. We report these findings — a rigorous evaluation protocol, a mostly negative primary result, a disclosed corpus defect, and a partially validated architecture — as an honest account of what a lightweight semantic-divergence auditor can currently establish.
11月20日
2026
11月22日
2026
初稿截稿日期
提前注册日期
注册截止日期
发表评论