Chanyoung Ahn

Dexterous manipulation in contact-rich environments relies heavily on haptic feedback. While prior work has primarily leveraged tactile and proprioception sensing, the relative contributions of different haptic modalities to learned manipulation tasks remain poorly understood. We investigate how kinesthesia (force/torque-based), proprioception, and tactile sensing contribute to object property estimation and grasp stability prediction using a four-fingered robotic hand. Through systematic modality ablations on over 5,220 real-world grasps, we find that modality importance is strongly task-dependent. kinesthesia feedback plays a dominant role in property estimation, whereas fingertip contact sensing is critical for stability prediction, with its removal reducing F1 from 0.963 to 0.510. Multimodal haptic fusion consistently achieves the best performance. We further demonstrate a prediction-gated recovery strategy that improves real-robot grasp success from 64.6% to 77.1%. We release our multimodal grasp dataset and show that these findings directly inform observation space design for learning-based manipulation policies.

Dexterous in-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera. We study how to structure the sparse haptic signals already available on multi-fingered hands—proprioception, proximal force/torque, and binary contact—to complement a pretrained visual pose tracker under occlusion. We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation: per-frame refinement, sequential open-loop tracking, and closed-loop manipulation. Our experiments reveal that (i) per-frame evaluation cannot distinguish encoder quality, while sequential tracking amplifies architectural differences by up to 15 times; (ii) the structured encoder learns task-specific cross-modal gating—using vision exclusively for translation and dedicating one attention head to haptic for rotation—without explicit supervision; and (iii) compact finger-level tokenization (4 tokens) outperforms both flat fusion and joint-level representations, which suppress vision through norm dominance. We validate that improved tracking yields higher success in a downstream reorientation task and provide qualitative real-world demonstrations.

Humans are capable of in-hand manipulation without visual feedback, by inferring its pose through haptic feedback. However, in-hand manipulation with multi-fingered robotic hands remains highly challenging due to severe self-occlusion and limited visual accessibility. To address this problem, we propose a vision-free approach that integrates multiple haptic sensing modalities. Specifically, we develop a haptic attention-based pose estimator that captures correlations among kinesthetic, contact, and proprioceptive signals, as well as their temporal dynamics. Experimental results demonstrate that haptic feedback alone enables reliable pose estimation and that contact-rich sensing substantially improves reorientation performance. Our pose estimator achieves average errors of only 4.94 mm in position and 11.6 degrees in orientation during 300 iterations (10 seconds), underscoring the effectiveness of haptic-driven pose estimation for dexterous manipulation.