Effectiveness of Kinesthetic Sensing in In-Hand Rotation of Objects
with an Eccentric Center of Mass

ICRA 2025 Workshop
Handy Moves: Dexterity in Multi-Fingered Hands

Chanyoung Ahn1, Sungwoo Park1, 2, Donghyun Hwang1
1KIST, 2Korea University
Paper Poster Video

TL;DR:

We explore how kinesthetic feedback, such as joint forces and torques,
helps a robotic hand adapt to varying object unvisible properties.
With PCA and RL, it improves performance by up to 2.40 times in unseen objects.

Abstract

In-hand manipulation is a key capability for dexterous control, yet it becomes challenging when the mass or center of mass (CoM) of an object is not well known. Such intrinsic properties are difficult to infer precisely through visual sensing alone, which limits the reliability of manipulation strategies.

This study investigates how kinesthetic sensing can support in-hand rotational tasks by enabling reinforcement learning (RL) agents to adjust to variations in object dynamics, particularly weight and center of mass. Our method incorporates both proprioceptive signals, such as joint angles, and kinesthetic data, joint forces and torques, captured from sensors embedded in a four-finger robotic hand. To reduce the dimensionality of the input space while retaining the relevant dynamics, we applied Principal Component Analysis (PCA). The resulting policy demonstrates improved adaptability.

In the simulation, the manipulation success rates increased by 2.09 and 2.40 times on six and twelve previously unseen CoM configurations, respectively. In addition, kinesthetic detection improves performance 1.52 times in ten known configurations. These findings indicate that kinesthetic feedback contributes substantially to robust and generalizable in-hand manipulation.


Research Aim

This study aims to leverage kinesthetic feedback-specifically, joint forces and torques-
to facilitate in-hand object rotation,
particularly for objects with varying weight and an eccentric center of mass (CoM).


research_aim_1 research_aim_2


To investigate the effectiveness of kinesthetic sensing, our goal is to achieve object rotation toward a downward-facing target within a 5-seconds rollout with Reinformcement Learning (RL) approaches.


research_aim_3

Experimental Setup

We define the in-hand rotation problem as a Markov Decision Process (MDP).
The elements of this MDP are defined as follows:


structure

Formulation as MDP

Action
16-DoF joint position with an exponential moving average for target update smoothly.


ex_set_action

State
joint angle, object pose, goal orientation, delta rotation, fingertip pose,
previous target, and difference of kinesthetic feedback (F/T) (see Table 1)


Reward
alignment with goal, penalty for fall/contact, velocity constraint, success bonus (see Table 2)


ex_set_action

ex_set_state ex_set_reward


Simulation Setup

Simulation: Isaac Sim with 4096 parallel environments
Training: PPO, 40K steps, and five random seeds
Evaluation: 500 rollouts per instance
H/W: Single RTX 4090 GPU
Observation Modality:
1. Only Proprioceptive
2. Proprioceptive + Kinesthesia
3. Proprioceptive + Kinesthesia with PCA



Objects

Train set: 10 eccentric CoM objects
Test set: Six and twelve unknown mass, CoM objects (test)

dataset

Results

This study aims to leverage kinesthetic feedback -specifically, joint forces and torques- to facilitate in-hand object rotation, particularly for objects with varying mass and an eccentric center of mass (CoM).


graph_1

Performance on Objects with Unknown Mass

(a) According to the graph, we observe that kinesthetic data becomes increasingly dominant as the overall mass of the unknown object increases. We hypothesize that this is because heavier objects exert greater influence due to their eccentric centers of mass.



Performance on Objects with Unknown Center of Mass

(b) According to the second graph, the policy using kinesthetic sensing combined with PCA-based state representation achieved the highest overall performance under unknown CoM positions. We observe that the lower the center of mass, the more stable the manipulation becomes.

Test Object

Unknown mass

Test Video

As a result of our experiment, the Kinesthetic w/ PCA policy achieved an average of 6.624 rotations within a 5-second rollout.

Test Object

Unknown position

Test Video

As a result of our experiment, the Kinesthetic w/ PCA policy achieved an average of 6.356 rotations within a 5-second rollout.

Ablation Study


1. Effect of Haptic Modalities

  1. We compared combinations of force/torque (F/T) sensing and fingertip contact signals. While F/T + Torque and F/T + Contact performed similarly overall, the latter showed a slight advantage, indicating the utility of binary contact information at the fingertips in precise manipulation tasks.


  2. Interestingly, adding all sensing modalities (F/T + Torque + Contact) led to a slight decrease in performance. We hypothesize that the increased state dimensionality may have made policy learning more difficult, underscoring the importance of compact and structured representations. In addition, as more sensors were fused, the policy’s output variance slightly increased, highlighting the need for a robust state encoder—further motivating the use of state representation learning (SRL).


ablation_graph

2. Effect of Object Shape

We evaluated the policy’s performance on two object shapes: a cube and a cylinder. The results indicate that the policy performed better on the cylinder, which has a more symmetric shape. In contrast, performance on the cube was noticeably lower, suggesting that asymmetry in object geometry adversely affects in-hand manipulation. While these findings are based on only two shapes, the failure to generalize beyond them implies that incorporating visual or geometric information may be essential for handling more diverse and complex objects.


ablation_graph

Results

research_aim_3

As a result, in the first test case, the "Kinesthetic with PCA" policy outperformed its counterpart by a factor of 2.40, and in the second case, by a factor of 2.09. These findings confirm that kinesthetic feedback contributes significantly to performance improvement in in-hand manipulation tasks involving eccentric objects.

Future Work

Expeirment in Real-World

* Just open-loop rollouts
We did not deploy the policy on a real robotic platform due to the sim-to-real gap. Kinesthetic sensing data can differ significantly between simulation and real-world.

dataset
dataset

BibTeX

@inproceedings{
      ahn2025effectiveness,
      title={Effectiveness of Kinesthetic Sensing in In-Hand Rotation of Objects with an Eccentric Center of Mass},
      author={Chanyoung Ahn and Sungwoo Park and Donghyun Hwang},
      booktitle={ICRA 2025 Workshop "Handy Moves: Dexterity in Multi-Fingered Hands" Paper Submission},
      year={2025},
      url={https://openreview.net/forum?id=65wkjAvIw5}
      }

Affiliations

icra Logo kist Logo Lab Logo