Runway AI, an artificial intelligence (AI) company that focuses on video generation models, announced a new feature on Tuesday. Dubbed Act-One, the new capability is available within the company's latest Gen-3 Alpha large language model (LLM) and is said to accurately capture facial expressions from a source video and then play them back on a character generated by AI in a video. The feature solves a major problem in AI movie generation technology, which turns real people into AI characters without losing realistic expressions.
Runway Act-One capability introduced in Gen-3 Alpha
In a blog post, the AI firm detailed the new video generation capability. Runway stated that the Act-One tool can create live-action and animated content using video and voice performances as inputs. The tool aims to offer expressive character performances in AI-generated videos.
We present the first act. A new way to generate expressive character performances within Gen-3 Alpha using a single driving video and character image. No motion capture or manipulation required.
Learn more about Act-One below.
(1/7) RTq" target="_blank">pic.twitter.com/p1Q8lR8K7G
– Track (@runwayml) HrT" target="_blank">October 22, 2024
AI-generated videos have significantly changed the video content creation process, as people can now generate specific videos using plain language text prompts. However, there are certain limitations that have prevented the adaptation of this technology. One of those limitations is the lack of controls to change a character's expressions in a video or improve their performance in terms of pronunciation of a sentence, gestures, and eye movements.
However, with Act-One, Runway is trying to close that gap. The tool, which only works with the Gen-3 Alpha model, simplifies the facial animation process, which can often be complex and require multi-step workflows. Today, animating these types of characters requires recording videos of an individual from multiple angles, manually manipulating the face, and capturing their facial movement separately.
Runway claims that Act-One replaces the workflow and makes it a two-step process. Users can now record a video of themselves or an actor from a single-point camera, which can also be a smartphone, and select an AI character. Once this is done, the tool is claimed to faithfully capture not only facial expressions but also minor details such as eye movements, micro-expressions and expression style.
Highlighting the scope of this feature, the company stated in the blog post: “The model retains realistic facial expressions and accurately translates performances into characters with different proportions than the actual video. “This versatility opens up new possibilities for inventive character design and animation.”
One of the model's strengths is producing cinematic and realistic results at a large number of camera angles and focal lengths. Allowing him to generate emotional performances with a depth of character previously impossible, opening new avenues for creative expression.
(4/7) x50" target="_blank">pic.twitter.com/JG1Fvj8OUm
— Track (@runwayml) bjI" target="_blank">October 22, 2024
Specifically, while Act-One can be used for animated characters, it can also be used for current action characters in a cinematic sequence. Additionally, the tool can also capture details even if the actor's face angle is different from the AI character's face angle.
This feature is currently being rolled out gradually to all users; However, since it only works with Gen-3 Alpha, those on the free tier will get a limited amount of tokens to generate videos with this tool.
02q">Source link