Hand landmarks serve as input for gesture recognition, sign language understanding, and touchless human–computer interaction. Yet hand pose estimators are typically evaluated with frame-wise localization metrics such as PCK or MPJPE, which are computed over matched predictions and overlook missed or spurious hands, handedness errors, and temporal instability. To address these limitations, we present VERA-Hand, a benchmark that evaluates estimators using manually verified, frame-level 2D annotations for 10,690 hands across four complementary dimensions (detection, localization, identification, and temporal stability), as well as a joint Pose-F1 metric that requires both correct detection and accurate localization. Evaluating six estimators spanning hand-specific, whole-body, reconstruction-based, and foundation-model paradigms, we find that no estimator dominates: rankings reorder sharply across dimensions. VERA-Hand also exposes failures that localization-only and downstream evaluations may not reveal, including cross-hand overlap in over 85% of frames for a recent foundation model. A post-hoc cleanup case study demonstrates how the benchmark can validate corrective interventions, while stratified analyses show that robustness varies across motion-blur classes and hand regions. We release the verified annotations, protocol, and code.
Ranking of the six evaluated estimators on VERA-Hand across four evaluation axes. Within each axis, estimators are ordered from best to worst, and scores are not directly comparable across axes (Detection and Localization ↑; Identification Error and Temporal Jitter ↓). No estimator ranks first across all four axes.
VERA-Hand is built on the IPN Hand dataset and provides manually verified, frame-level 2D annotations (21 keypoints, MediaPipe topology, with a left/right label) for every visible hand in frames sampled from 52 third-person continuous dynamic-gesture videos. Annotations were initialized with an off-the-shelf estimator and manually corrected in two passes, followed by an independent audit of 3,500 frames in which 7.7% needed corrections, all of which are fixed in the released annotations.
Estimators are evaluated along four axes: detection (is each hand recovered?), localization (where do the keypoints fall?), identification (is the correct hand label assigned?) and temporal stability (how steady are predictions over full videos?), plus Pose-F1, which credits a hand only when it is both detected and accurately localized.
Browse the ground truth next to the predictions of RTMW-wb and Sapiens2-1B (raw, and after cleanup) on 39 benchmark frames. Keypoints are drawn live on the raw frames. Use the arrow keys to move between frames, z / x to toggle the GT / prediction keypoints, and 1–3 to switch predictor (click inside the viewer first). Faces are blurred. Open it full screen.
Comparison across detection (Det-F1), localization (PCK@0.10), the joint Pose-F1@0.10≥0.5, identification (label-swap rate), and temporal stability (Jjerk, with motion energy Ev in parentheses). Best value per column in bold.
| Estimator | Det-F1 ↑ | PCK ↑ | Pose-F1 ↑ | Swap ↓ | Jjerk ↓ (Ev) |
|---|---|---|---|---|---|
| MP-holistic | 0.840 | 0.439 | 0.309 | 0.51% | 1.59 (1.52) |
| MP-hands | 0.898 | 0.573 | 0.389 | 4.41% | 1.69 (1.51) |
| WiLoR | 0.978 | 0.470 | 0.407 | 0.11% | 1.18 (1.38) |
| RTMW-wb | 0.932 | 0.560 | 0.477 | 0.31% | 1.47 (1.38) |
| RTMP-hand | 0.872 | 0.489 | 0.340 | 3.51% | 1.76 (1.43) |
| Sapiens2-1B | 0.666 | 0.491 | 0.361 | 0.03% | 1.30 (1.39) |
Effect of the post-hoc cleanup on all six estimators, before (raw) and after (clean) cleanup. Bold marks the two largest improvements within Det-F1 and Pose-F1.
| Estimator | Cross-hand overlap | Det-F1 | Pose-F1 | |||
|---|---|---|---|---|---|---|
| raw | clean | raw | clean | raw | clean | |
| MP-holistic | 4.82% | 3.32% | 0.840 | 0.888 | 0.309 | 0.325 |
| MP-hands | 0% | 0% | 0.898 | 0.926 | 0.389 | 0.442 |
| WiLoR | 0.07% | 0.03% | 0.978 | 0.987 | 0.407 | 0.419 |
| RTMW-wb | 0.91% | 0.03% | 0.932 | 0.951 | 0.477 | 0.477 |
| RTMP-hand | 0.02% | 0.09% | 0.872 | 0.883 | 0.340 | 0.342 |
| Sapiens2-1B | 85.44% | 0.05% | 0.666 | 0.940 | 0.361 | 0.406 |
Columns, from left to right: GT, MP-hands, WiLoR, RTMW-wb, and Sapiens2-1B. Green dashed rectangles mark detection or identification errors, and green dashed ellipses mark gross localization errors.
Failures are pipeline-dependent: a frame that defeats one estimator is often handled correctly by another, and the errors shown here are invisible to an evaluation restricted to matched predictions. The last row is a no-hand frame in which two pipelines still emit a prediction.
@article{bega2026verahand,
title = {{VERA-Hand}: A Verified Multidimensional Benchmark for 2D Hand Pose Estimation in Third-Person Dynamic Gesture Videos},
author = {Benitez-Garcia, Gibran and Takahashi, Hiroki},
journal = {arXiv preprint arXiv:2610.32517},
year = {2026},
eprint = {2610.32517},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}