Chanyoung Ahn

Dexterous manipulation in contact-rich environments relies heavily on haptic feedback. While prior work has primarily leveraged tactile and proprioception sensing, the relative contributions of different haptic modalities to learned manipulation tasks remain poorly understood. We investigate how kinesthesia (force/torque-based), proprioception, and tactile sensing contribute to object property estimation and grasp stability prediction using a four-fingered robotic hand. Through systematic modality ablations on over 5,220 real-world grasps, we find that modality importance is strongly task-dependent. kinesthesia feedback plays a dominant role in property estimation, whereas fingertip contact sensing is critical for stability prediction, with its removal reducing F1 from 0.963 to 0.510. Multimodal haptic fusion consistently achieves the best performance. We further demonstrate a prediction-gated recovery strategy that improves real-robot grasp success from 64.6% to 77.1%. We release our multimodal grasp dataset and show that these findings directly inform observation space design for learning-based manipulation policies.

Dexterous in-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera. We study how to structure the sparse haptic signals already available on multi-fingered hands—proprioception, proximal force/torque, and binary contact—to complement a pretrained visual pose tracker under occlusion. We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation: per-frame refinement, sequential open-loop tracking, and closed-loop manipulation. Our experiments reveal that (i) per-frame evaluation cannot distinguish encoder quality, while sequential tracking amplifies architectural differences by up to 15 times; (ii) the structured encoder learns task-specific cross-modal gating—using vision exclusively for translation and dedicating one attention head to haptic for rotation—without explicit supervision; and (iii) compact finger-level tokenization (4 tokens) outperforms both flat fusion and joint-level representations, which suppress vision through norm dominance. We validate that improved tracking yields higher success in a downstream reorientation task and provide qualitative real-world demonstrations.

Humans are capable of in-hand manipulation without visual feedback, by inferring its pose through haptic feedback. However, in-hand manipulation with multi-fingered robotic hands remains highly challenging due to severe self-occlusion and limited visual accessibility. To address this problem, we propose a vision-free approach that integrates multiple haptic sensing modalities. Specifically, we develop a haptic attention-based pose estimator that captures correlations among kinesthetic, contact, and proprioceptive signals, as well as their temporal dynamics. Experimental results demonstrate that haptic feedback alone enables reliable pose estimation and that contact-rich sensing substantially improves reorientation performance. Our pose estimator achieves average errors of only 4.94 mm in position and 11.6 degrees in orientation during 300 iterations (10 seconds), underscoring the effectiveness of haptic-driven pose estimation for dexterous manipulation.

In-hand manipulation is a key capability for dexterous control, yet it becomes challenging when the mass or center of mass (CoM) of an object is not well known. Such intrinsic properties are difficult to infer precisely through visual sensing alone, which limits the reliability of manipulation strategies. This study investigates how kinesthetic sensing can support in-hand rotational tasks by enabling reinforcement learning (RL) agents to adjust to variations in object dynamics, particularly weight and center of mass. Our method incorporates both proprioceptive signals, such as joint angles, and kinesthetic data, joint forces and torques, captured from sensors embedded in a four-finger robotic hand. To reduce the dimensionality of the input space while retaining the relevant dynamics, we applied Principal Component Analysis (PCA). The resulting policy demonstrates improved adaptability. In the simulation, the manipulation success rates increased by 2.09 and 2.40 times on six and twelve previously unseen CoM configurations, respectively. In addition, kinesthetic detection improves performance 1.52 times in ten known configurations. These findings indicate that kinesthetic feedback contributes substantially to robust and generalizable in-hand manipulation. To access our data and video, please visit our project page: https://cold-young.github.io/kinesthetic_rotation/.

We aim to create a simulated testbed for training and assessing deformable object manipulation skills. This testbed requires tactile sensing to distinguish heterogeneous elasticities of a deformable object to acquire skills. In this work, we introduce a visuotactile testbed, DetactGym, for deformable object manipulation, integrating a novel architecture of tactile sensors leveraging collision cascades. These sensors integrate a diamond-shaped rigid element encased within an external rigid structure. This design overcomes a fundamental limitation of PhysX engine-based simulators: their inability to directly measure contact force on deformable objects. The diamond-shaped element efficiently transmits the contact force to the outer structure during interactions with these objects, ensuring force measurements. Our evaluation focuses on the testbed’s capability, equipped with tactile sensors, to facilitate the learning of lifting heterogeneous deformable objects with minimal deformation through reinforcement learning methods. Our findings highlight the effectiveness of tactile feedback over visual cues in manipulating deformable objects with diverse elasticities, reducing deformation.

Our aim is to create a human-expert manipulation dataset of deformable objects toward learning deformable manipulation. However, the data collection is challenging due to the limited sensing capability against to the high number of degree of freedom and complicate deformation/contact of soft bodies. Further, the self occlusion restricts the observation in grasping. In this work, we introduce a haptic-telemanipulation suite by adopting a haptic glove with a state-of-the-art physics simulator, IsaacGym from NVIDIA. The suite enables users to obtain realistic visual-and-tactile feedback as well as collect any part of object states while teleoperating a robotic gripper. We evaluate the suite by building one hundred demonstrations of dataset given five deformable objects.

We aim to create a simulated testbed for training and assessing deformable object manipulation skills. This testbed requires tactile data to detect occluded deformation to acquire skills. In this work, we introduce a tactile testbed, DetactGym, with custom-built tactile sensors. We implement tactile sensors as a combination of a rigid trigger and a force-reading part to address the IsaacSim simulator’s inability to read contact forces in deformable. The trigger transmits the contact force to the sensor upon interaction with the target deformable. We highlight the effectiveness of tactile information over visual cues in handling deformable objects while minimizing deformation.