HOForce arXiv

HOForceMotion and Force Reconstruction of Hand–Object Interaction Using a Single-View RGB-D Video

Jiahui Wang1,2,*, Wanqing Du1,2,*, Yuanbin Zou2, Zhefeng Xu3, Feng Xu1,†

1 Tsinghua University2 Parity Lab3 Hangzhou Dianzi University

* Equal contribution   † Corresponding author

One hand guides the peg while the other steadies the socket. The right panel replays the object motion using the recovered fingertip forces, shown in orange.

Abstract

Reconstructing hand–object interactions is an important task, and achieving it with a single camera makes large-scale capture practical but challenging due to heavy occlusions. We introduce HOForce to track hands and objects, and further recover interaction forces from single-view RGB-D video and known object properties. For two-hand manipulation, we jointly model fingertip forces of both hands and the contact forces between objects when two objects are presented. The reconstructed forces also improve motion tracking by giving pose constraints in the occluded regions. HOForce recovers hand and object motion more accurately on DexYCB and TACO and produces less hand–object penetration than interaction baselines on DexYCB. On our dataset of fingertip forces recorded with tactile sensors, HOForce reduces normal-force MAE by 24.5% relative to the strongest baseline. The recovered forces reproduce object motion in MuJoCo and transfer to other engines after adaptation.

Method

HOForce first reconstructs motion from RGB-D observations, then estimates contact forces and uses those forces to refine the reconstruction.

Two-stage HOForce pipeline. Depth-guided hand and object initialization is followed by joint motion refinement with nonpenetration constraints. A physics stage alternates fitting forces to observed motion and refining fingertip contacts and poses.
Known object meshes and physical properties let the simulator account for forces exchanged between objects as well as the loads applied by the fingers.

Results

We evaluate the recovered forces against tactile sensors and test whether they reproduce the recorded motion during simulation.

Fingertip force accuracy

Measured and estimated thumb normal force over a 31-second recording. The estimated force tracks the major peaks in the sensor signal.

1.002 N mean absolute error for the normal component across 32 sequences, compared with 1.326 N for the strongest baseline.

The plot shows one example recording.

Replay across simulators

19 of 21 insertions succeed in each target engine after adapting the MuJoCo forces to that engine.

The application points and physical parameters are retained.

Full demo

Five minutes of motion reconstruction, force comparisons, replay across engines, and ablations. The video includes English narration and captions.

Citation

BibTeX
@misc{wang2026hoforcemotionforcereconstruction,
  title = {HOForce: Motion and Force Reconstruction of Hand-Object Interaction Using a Single-View RGB-D Video},
  author = {Jiahui Wang and Wanqing Du and Yuanbin Zou and Zhefeng Xu and Feng Xu},
  year = {2026},
  eprint = {2611.23871},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2611.23871}
}