An Empirical Study of GPU Concurrency Scaling for Pathology Inference at the Clinical Edge
编号:4
访问权限:仅限参会人
更新:2026-11-10 16:07:25 浏览:9次
Online
摘要
Pathology models deployed at the clinical edge often share one GPU, so they cannot absorb a surge of requests by starting more replicas. What they can change is how many queued images each forward pass carries. We study three policies that set this micro-batch slot count (1-8, applied 10 s after a scale-up request) for a ResNet-50 PathMNIST service on an NVIDIA RTX 4060 Laptop GPU: a CPU-threshold rule, a reactive GPU-utilization rule, and a Prophet forecaster. Under light load (60 runs, at most 42.3 requests/s), every request succeeded and the policies were indistinguishable, with median client p95 latency between 23 and 33 ms (Kruskal-Wallis p=0.89). Under overload (48 valid runs, a 79.3 requests/s base with 116.3 requests/s peaks), Prophet kept a median p95 of 24 ms through a sudden spike, while the CPU and reactive policies reached 685 ms and 622 ms (p=0.025). Bursty and ramped traffic showed no consistent difference, and Prophet was slightly slower under steady overload. A 16-run ablation with a fixed two-slot control explains the spike result: two pinned slots gave 24.5 ms, Prophet 26.9 ms, and the other two policies about 1.4 s (p=0.0045). Prophet won because a conservative capacity estimate kept a second slot active before the spike began, not because it predicted the spike. The reactive rule seldom scaled, as GPU utilization stayed at 37-56% while requests queued. On a single edge GPU, keeping spare slots under heavy load delivered the whole benefit without a forecaster.
关键词
Edge Inference,GPU Micro-Batching,Autoscaling,Time-Series Forecasting,Digital Pathology,PathMNIST
稿件作者
Visshva R
SRM Institute of Science and Technology
Babu K
SRM Institute of Science and Technology *
G Senthil Kumar
SRM Institute of Science and Technology
Ruthwik S
SRM Institute of Science and Technology
Adithyan E
SRM Institute of Science and Technology
Irfan Fidel Farhath V
SRM Institute of Science and Technology
发表评论