Imagin-4D: Image-Guided Controllable Interaction Generation (imagin4d.github.io)

IMAGIN-4D has been introduced as an innovative diffusion-based generator for human-object interactions (HOI), crucial for fields like character animation, robotics, and AR/VR. Traditional methods that rely on textual prompts and object geometry often fail to provide distinct and specific interactions due to ambiguous signals. IMAGIN-4D addresses this challenge by using reference images as visual specifications that allow for the nuanced generation of movements and interactions, enabling better control over dynamics such as body poses, object placements, and contact points. The significance of IMAGIN-4D lies in its spatio-temporal conditioning approach, which extracts distinct interaction-state tokens and frame-aware visual cues, allowing different segments of a generated sequence to attend to various image details. This method utilizes role-aware conditioning to balance inputs from text, waypoints, and visual tokens, enhancing the generated motion’s quality. Moreover, to tackle the lack of paired image datasets in HOI generation, a synthetic motion-to-image rendering pipeline was developed along with a new image-adherence metric. Experimental results show that IMAGIN-4D outperforms existing methods in fine-grained control while maintaining the integrity of waypoint-following, opening new avenues for more realistic and responsive interactions in AI-driven applications.