Chanyoung Ahn

Humans are capable of in-hand manipulation without visual feedback, by inferring its pose through haptic feedback. However, in-hand manipulation with multi-fingered robotic hands remains highly challenging due to severe self-occlusion and limited visual accessibility. To address this problem, we propose a vision-free approach that integrates multiple haptic sensing modalities. Specifically, we develop a haptic attention-based pose estimator that captures correlations among kinesthetic, contact, and proprioceptive signals, as well as their temporal dynamics. Experimental results demonstrate that haptic feedback alone enables reliable pose estimation and that contact-rich sensing substantially improves reorientation performance. Our pose estimator achieves average errors of only 4.94 mm in position and 11.6 degrees in orientation during 300 iterations (10 seconds), underscoring the effectiveness of haptic-driven pose estimation for dexterous manipulation.

In-hand manipulation is a key capability for dexterous control, yet it becomes challenging when the mass or center of mass (CoM) of an object is not well known. Such intrinsic properties are difficult to infer precisely through visual sensing alone, which limits the reliability of manipulation strategies. This study investigates how kinesthetic sensing can support in-hand rotational tasks by enabling reinforcement learning (RL) agents to adjust to variations in object dynamics, particularly weight and center of mass. Our method incorporates both proprioceptive signals, such as joint angles, and kinesthetic data, joint forces and torques, captured from sensors embedded in a four-finger robotic hand. To reduce the dimensionality of the input space while retaining the relevant dynamics, we applied Principal Component Analysis (PCA). The resulting policy demonstrates improved adaptability. In the simulation, the manipulation success rates increased by 2.09 and 2.40 times on six and twelve previously unseen CoM configurations, respectively. In addition, kinesthetic detection improves performance 1.52 times in ten known configurations. These findings indicate that kinesthetic feedback contributes substantially to robust and generalizable in-hand manipulation. To access our data and video, please visit our project page: https://cold-young.github.io/kinesthetic_rotation/.

We aim to create a simulated testbed for training and assessing deformable object manipulation skills. This testbed requires tactile sensing to distinguish heterogeneous elasticities of a deformable object to acquire skills. In this work, we introduce a visuotactile testbed, DetactGym, for deformable object manipulation, integrating a novel architecture of tactile sensors leveraging collision cascades. These sensors integrate a diamond-shaped rigid element encased within an external rigid structure. This design overcomes a fundamental limitation of PhysX engine-based simulators: their inability to directly measure contact force on deformable objects. The diamond-shaped element efficiently transmits the contact force to the outer structure during interactions with these objects, ensuring force measurements. Our evaluation focuses on the testbed’s capability, equipped with tactile sensors, to facilitate the learning of lifting heterogeneous deformable objects with minimal deformation through reinforcement learning methods. Our findings highlight the effectiveness of tactile feedback over visual cues in manipulating deformable objects with diverse elasticities, reducing deformation.

Our aim is to create a human-expert manipulation dataset of deformable objects toward learning deformable manipulation. However, the data collection is challenging due to the limited sensing capability against to the high number of degree of freedom and complicate deformation/contact of soft bodies. Further, the self occlusion restricts the observation in grasping. In this work, we introduce a haptic-telemanipulation suite by adopting a haptic glove with a state-of-the-art physics simulator, IsaacGym from NVIDIA. The suite enables users to obtain realistic visual-and-tactile feedback as well as collect any part of object states while teleoperating a robotic gripper. We evaluate the suite by building one hundred demonstrations of dataset given five deformable objects.

We aim to create a simulated testbed for training and assessing deformable object manipulation skills. This testbed requires tactile data to detect occluded deformation to acquire skills. In this work, we introduce a tactile testbed, DetactGym, with custom-built tactile sensors. We implement tactile sensors as a combination of a rigid trigger and a force-reading part to address the IsaacSim simulator’s inability to read contact forces in deformable. The trigger transmits the contact force to the sensor upon interaction with the target deformable. We highlight the effectiveness of tactile information over visual cues in handling deformable objects while minimizing deformation.