Responsive-ExtendedHand: Adaptive Visuo-Haptic Feedback Recognizing Object Property With RGB-D Camera for Projected Extended Hand

Sato, Yushi; Iwai, Daisuke; Sato, Kosuke

IEEE Access, 2024, 12, 38247-38257

Number of Access:410（2025-07-27 02:46 Counts）

Identifier to cite or link to this item: https://hdl.handle.net/11094/95647

Link to primary information 

File Format Terms of use Size Views Date.Available Description information
IEEEAccess_12_38247 pdf None 3.74 MB 105 2024.05.17  
Creative Commons : Attribution

Item Information

Output File Export EndNote Basic Export Mendeley

Title
Responsive-ExtendedHand: Adaptive Visuo-Haptic Feedback Recognizing Object Property With RGB-D Camera for Projected Extended Hand
Creator
CreatorName
Sato, Yushi
NRID
1000090504837 1000090504837
researchmap permalink
daisukeiwai daisukeiwai
Scopus Author ID
55931917600 55931917600
CreatorName
Iwai, Daisuke
NRID
1000090187188 1000090187188
researchmap permalink
read0076311 read0076311
Scopus Author ID
56192879500 56192879500
CreatorName
Sato, Kosuke
Subject
Augmented reality
body augmentation
deep neural network
pseudo-haptics
Description
Sato Y., Iwai D., Sato K.. Responsive-ExtendedHand: Adaptive Visuo-Haptic Feedback Recognizing Object Property With RGB-D Camera for Projected Extended Hand. IEEE Access 12, 38247 (2024); https://doi.org/10.1109/ACCESS.2024.3375917.
Abstract
The ExtendedHand interface displays computer graphics (CG) hand images in real space from a projector, allowing the user to visually point at and touch real objects that are out of their physical reach. Furthermore, when the projected CG hand (extended hand) touches an object, the user can feel the tactile sensation of the object through pseudo-haptics by giving the extended hand visual effects that emphasize the action. In the previous psychological study, the human operator had to manually assign the location and shape of objects and the intensities of their visual effects in advance in order to emphasize the appropriate visual effect for the object touched by the extended hand. To increase practical feasibility, we propose an adaptive system that utilizes an RGB-D camera and deep neural networks to generate the appropriate visual effects automatically and apply them to the projected extended hand. By employing U-Net to generate the appropriate intensities of the visual effects from the captured color and depth images, the system can estimate the appropriate visual effects for objects without pre-setting them. The user evaluation results showed that the proposed system allowed users to naturally perceive the tactile sensation of objects at a rate of 44%, instead of the manual rate of 49%.
Publisher
Institute of Electrical and Electronics Engineers Inc.
Source Title
IEEE Access
Volume (Issue)
12
Page
38247-38257
Date of Issued
2024-03-15
Language
English
Handle URL
https://hdl.handle.net/11094/95647
eISSN
21693536
Relation.isIdenticalTo
DOI (Publisher Version)
https://doi.org/10.1109/ACCESS.2024.3375917
Access Rights
open access
CopyRight
This article is licensed under a Creative Commons Attribution 4.0 International License.
oaire:version
Version of Record
Category
学術雑誌論文 Journal Article