A Unified Multimodal Director

Bridging Your Imagination with Audio-Video Generation via a Unified Director

View on arXiv

Jiaxu Zhang1,2, Tianshu Hu1,†, Yuan Zhang1, Zenan Li1, Linjie Luo1, Guosheng Lin2,‡, Xin Chen1,†,‡

1 ByteDance Intelligent Creation, 2 Nanyang Technological University

† Project Lead, ‡ Corresponding Authors

Abstract

Existing AI-driven video creation systems typically treat script drafting and key-shot design as two disjoint tasks: the former relies on large language models, while the latter depends on image generation models. We argue that these two tasks should be unified within a single framework, as logical reasoning and imaginative thinking are both fundamental qualities of a film director.

In this work, we propose UniMAGE, a unified director model that bridges user prompts with well-structured scripts, thereby empowering non-experts to produce long-context, multi-shot films by leveraging existing audio-video generation models.

To achieve this, we employ the Mixture-of-Transformers architecture that unifies text and image generation. To further enhance narrative logic and keyframe consistency, we introduce a "first interleaving, then disentangling" training paradigm. Specifically, we first perform Interleaved Concept Learning, which utilizes interleaved text-image data to foster the model's deeper understanding and imaginative interpretation of scripts. We then conduct Disentangled Expert Learning, which decouples script writing from keyframe generation, enabling greater flexibility and creativity in storytelling.

Extensive experiments demonstrate that UniMAGE achieves state-of-the-art performance among open-source models, generating logically coherent video scripts and visually consistent keyframe images.

Video Narratives

<Character1>: A woman with long, dark brown hair, smiling warmly. She wears a dark grey coat and a blue and green patterned scarf. <Character2>: A man with short, dark brown hair, a friendly smile, wearing a dark pinstriped shirt. <Character3>: A woman with long, blonde hair, smiling brightly, wearing a white t-shirt and a silver necklace. <Environment1>: A dusty, dimly lit archaeological dig site, with ancient stone walls and unearthed artifacts, illuminated by work lights and natural light filtering from above. <Environment2>: A vibrant, lush jungle with dense foliage, ancient ruins partially covered by vines, and sunlight dappling through the canopy.

Demo 1 scene 0
Scene 1

<Frame1>: Medium shot: <Character1> is kneeling in the dirt of the archaeological dig site<Environment1>, carefully brushing dust from an ancient, ornate stone tablet. Her expression is focused and intrigued. Around her are various archaeological tools. <Video1>: The shot opens with <Character1> meticulously cleaning the artifact in the archaeological dig site<Environment1>. She gasps softly. <It's beautiful... and unlike anything I've seen before.-> The camera slowly zooms in on the intricate carvings of the tablet. <-Soft, inquisitive orchestral music begins->.

Demo 1 scene 1
Scene 2

<Frame2>: Medium shot: <Character2> stands over <Character1> in the archaeological dig site<Environment1>, holding a notepad and looking at the tablet with a curious, slightly excited expression. <Character1> looks up at him, a question in her eyes. <Video2>: Still in the archaeological dig site<Environment1>, <Character2> leans in, examining the tablet. <-Do you think it's a map? Or a key?-> <Character1> shakes her head, still studying the carvings. <-It feels more than that. It feels... active.-> The camera pans slightly to show more of the ancient stone walls around them. <-The music becomes slightly more mysterious->.

Demo 1 scene 2
Scene 3

<Frame3>: Close-up: <Character3> enters the frame, her eyes wide with wonder as she looks at the tablet held by <Character1>. Her blonde hair frames her face, and her smile is replaced with an expression of awe. <Video3>: <Character3> rushes over to the tablet in the archaeological dig site<Environment1>, her previous bright demeanor now softened by awe. <-Incredible! What do you think it means?-> <Character1> points to a specific symbol. <-I believe it points to something hidden. Something beyond our current understanding.-> <-A subtle, shimmering sound effect accompanies the pointed symbol->.

Demo 1 scene 3
Scene 4

<Frame4>: Long shot: The three characters, <Character1>, <Character2>, and <Character3>, are now in the lush jungle<Environment2>. They are walking cautiously through dense foliage, with <Character1> holding the tablet, looking ahead with determination. Sunlight filters through the canopy. <Video4>: The scene shifts to the vibrant jungle<Environment2>. The trio navigates through the dense undergrowth, <Character1> leading the way, the tablet clutched in her hands. <Character2> scans their surroundings. <-Are you sure this is the right way, Dr. Anya?-> <Character1> nods confidently. <-The tablet's energy readings are strong here.-> <-Jungle ambient sounds, birds chirping, and a sense of adventure in the music->.

Demo 1 scene 4
Scene 5

<Frame5>: Medium shot: <Character3> is carefully pushing aside large, leafy plants in the jungle<Environment2>, revealing a partially overgrown ancient stone archway. Her expression is one of excited discovery. <Video5>: <Character3> pushes aside a curtain of vines in the jungle<Environment2>, revealing a hidden, vine-covered archway. She gasps. <-Look! An entrance!-> <Character1> and <Character2> quickly move to her side, their faces alight with anticipation. <-The music swells with a sense of triumph and discovery->.

Demo 1 scene 5
Scene 6

<Frame6>: Close-up: The three characters, <Character1>, <Character2>, and <Character3>, stand before the archway in the jungle<Environment2>, their faces illuminated by a soft, ethereal glow emanating from the tablet held by <Character1>. Their expressions are a mix of wonder and excitement. <Video6>: As they stand before the archway in the jungle<Environment2>, the tablet in <Character1>'s hands begins to glow more intensely, casting an ethereal light on their faces. <Character2> whispers, <-It's reacting to the archway.-> <Character3> reaches out a hand, almost touching the glowing stone. <-What wonders lie beyond?-> <-The music becomes more magical and suspenseful, with a rising crescendo->.

Demo 1 scene 6
Scene 7

<Frame7>: Medium shot: <Character1> holds the glowing tablet, her eyes wide with a mix of wonder and trepidation, as she stands at the entrance of a newly revealed, dimly lit chamber within the jungle<Environment2>. <Character2> and <Character3> are just behind her, their faces reflecting the tablet's light, their expressions mirroring hers. The entrance is carved into ancient stone, partially covered by vines. <Video7>: The camera pushes slowly into the dimly lit chamber as <Character1>, <Character2>, and <Character3> step inside from the jungle<Environment2>. The tablet in <Character1>'s hands pulses with a soft, ethereal light, illuminating ancient carvings on the chamber walls. <Character1> whispers, <-This isn't just an archway... it's a gateway.-> <Character2> looks around nervously. <-I have a bad feeling about this, Anya.-> <Character3> gasps, pointing to a central pedestal. <-Look!-> <-Mysterious, echoing ambient sounds, and a low, rumbling orchestral hum->.

Demo 1 scene 7
Scene 8

<Frame8>: Long shot: Inside the chamber, the three characters, <Character1>, <Character2>, and <Character3>, are standing before a large, ornate pedestal. On the pedestal rests a larger, identical-looking stone tablet, glowing faintly. The chamber walls are covered in intricate, celestial patterns, illuminated by the tablet's light. The entrance they came through is now blocked by falling debris. <Video8>: The camera pans across the chamber, revealing a grand, circular room with celestial patterns etched into the walls. In the center, on a stone pedestal, rests a larger, identical tablet, glowing with a faint, internal light. As <Character1> approaches it, the tablet in her hand intensifies its glow, and the chamber's entrance suddenly collapses, sealing them inside. <-A loud crash as the entrance seals->. <Character2> shouts, <-The entrance! It's blocked!-> <Character3> clutches <Character1>'s arm. <-What's happening?-> <-The music becomes more urgent and percussive->.

Demo 1 scene 8
Scene 9

<Frame9>: Close-up: The larger tablet on the pedestal in the chamber begins to spin slowly, its glowing symbols shifting and rearranging. <Character1>'s face is illuminated by its intense light, her expression a mix of awe and dawning realization. <Character2> and <Character3> are partially visible in the background, looking on with concern. <Video9>: The camera focuses on the larger tablet on the pedestal. Its surface begins to glow intensely, and the celestial patterns on its surface start to shift and rearrange, forming a complex, star-filled map. <Character1> stares, mesmerized. <-It's... it's a map. Not of Earth, but of the cosmos!-> <Character2> steps closer, his concern mixed with fascination. <-But what does it mean?-> <-A high-pitched, harmonic hum emanates from the tablet, growing in intensity->.

Demo 1 scene 9
Scene 10

<Frame10>: Close-up: <Character1> stands before the spinning, glowing tablet, her face illuminated by its golden light. Her expression shifts from awe to realization. Tiny motes of light swirl around her as if drawn from the tablet itself. Behind her, faint beams of light trace ancient symbols across the chamber walls, revealing constellations that seem to move. <Video10>: The camera slowly pushes in on <Character1> as the glow from the tablet intensifies, bathing her face in warm, radiant light. Her eyes widen, reflecting galaxies forming within the spinning tablet. <Character2> calls out, <-Step back!-> but his voice is drowned out by the rising hum. <Character1> whispers, barely audible, <-It’s… showing me… everything.-> The chamber trembles, dust falling from the ceiling.

Demo 1 scene 10
Scene 11

<Frame11>: Medium shot: As the celestial map on the tablet fully forms, beams of light shoot out from its surface, projecting onto the chamber walls. These beams highlight specific constellations and symbols. <Character1> reaches out a hand towards the light, while <Character2> and <Character3> look up in astonishment, their faces bathed in the ethereal glow. <Video11>: As the celestial map fully forms, beams of light shoot out from the tablet, projecting onto the chamber walls. These beams highlight specific constellations and ancient symbols, causing them to glow in response. <Character1> reaches out, almost touching a beam. <-It's reacting to us!-> <Character3> gasps. <-It's beautiful... but also terrifying.-> <-The harmonic hum intensifies, joined by a chorus of ethereal voices->.

Demo 1 scene 11
Scene 12

<Frame12>: Long shot: The chamber floor begins to glow with the same celestial patterns, and cracks appear, revealing ancient, stone guardians slowly rising from the ground. Their eyes glow with a faint, internal light. <Character1>, <Character2>, and <Character3> stand frozen in fear, looking at the awakening guardians. <Video12>: Suddenly, the chamber floor begins to glow with the same celestial patterns, and cracks spiderweb across the stone. From these cracks, ancient, stone guardians slowly rise, their forms imposing and their eyes glowing with a faint, internal light. <Character1>, <Character2>, and <Character3> are frozen in fear, backing away slowly. <Character2> stammers, <-What... what are those things?-> <-A deep, resonant rumble accompanies the guardians' awakening, followed by a menacing, low-frequency drone->.

<Character1>: A middle-aged man with short, graying hair, wearing glasses, a light blue collared shirt, and a striped tie. He has a serious yet engaging expression, with his right hand raised in a gesturing motion. <Environment1>: A modern lecture hall with rows of seats, a large projection screen behind the speaker, and professional stage lighting. The atmosphere is academic and inspiring. <Environment2>: A dimly lit planetarium dome, filled with swirling projections of stars and galaxies, creating an immersive and awe-inspiring celestial environment.

Demo 2 scene 0
Scene 1

<Frame1>: Close-up: The astrophysicist<Character1> in the lecture hall<Environment1>, his face illuminated by the stage lights, a thoughtful smile playing on his lips as he looks towards the audience. His glasses reflect the light, adding to his scholarly appearance. <Video1>: A close-up on the astrophysicist's<Character1> face in the lecture hall<Environment1>. He leans slightly forward, his voice filled with wonder. <-Imagine, if you will, the sheer vastness... the unimaginable scales at play.-> The camera then pans slowly to show a segment of the eager audience, nodding in appreciation. <-The music swells slightly, maintaining an inspiring tone->.

Demo 2 scene 1
Scene 2

<Frame2>: Medium shot: The astrophysicist<Character1> stands confidently on stage in the lecture hall<Environment1>, gesturing with both hands as he explains a complex concept. The projection screen behind him now displays a vibrant image of a galaxy. <Video2>: The shot returns to a medium shot of the astrophysicist<Character1> in the lecture hall<Environment1>. He points to the projection screen, which now displays a breathtaking image of a distant galaxy. <-This, my friends, is just a glimpse of the grand design.-> He takes a step forward, his voice becoming more emphatic. <-The very fabric of space-time, bending and twisting under the immense gravity of these cosmic behemoths!-> <-Music becomes more dynamic and slightly dramatic->.

Demo 2 scene 2
Scene 3

<Frame3>: Long shot: The astrophysicist<Character1> is a silhouette against the bright projection of a swirling nebula in the planetarium dome<Environment2>. He stands with his arms outstretched, as if embracing the celestial display. The dome above him is a canvas of stars and cosmic dust. <Video3>: The scene transitions to a long shot inside a breathtaking planetarium dome<Environment2>. The astrophysicist<Character1> is silhouetted against a stunning projection of a swirling nebula, his arms wide. His voice, now softer, echoes through the vast space. <-And here, in the planetarium, we are not just observers, but participants in this cosmic dance.-> The camera slowly rotates, showcasing the immersive starry projections. <-Gentle, awe-inspiring ambient music with celestial sounds begins->.

Demo 2 scene 3
Scene 4

<Frame4>: Close-up: The astrophysicist<Character1> in the planetarium dome<Environment2>, his face lit by the ethereal glow of the star projections. His eyes are wide with wonder, and a profound sense of discovery is etched on his features. <Video4>: A close-up on the astrophysicist's<Character1> face in the planetarium dome<Environment2>. His expression is one of pure awe and childlike wonder. <-Every star, every galaxy, tells a story billions of years in the making. And we, are privileged to witness it.-> A single tear of joy might glisten in his eye. The camera slowly pulls back to a medium shot, revealing him looking up at the dome in profound contemplation. <-The music swells to a powerful, hopeful crescendo, then fades slowly->.

Result Gallery

Result 01

A young bakers patisserie is subject to a surprise inspection by urban management officers.

Result 01 storyboard preview
Result 02

A man discovers an ancient map in his attic, leading him on a solitary adventure to uncover a hidden treasure.

Result 02 storyboard preview
Result 03

A kind teacher organizes a special art project for two shy students to help them express themselves and bond.

Result 03 storyboard preview
Result 04

A Qatari diplomat and a Japanese martial arts master work together with an American tourist at a bustling international market.

Result 04 storyboard preview
Result 05

A renowned art critic and a cunning art forger find themselves in a tense cat-and-mouse game during a high-stakes auction for a disputed masterpiece.

Result 05 storyboard preview
Result 06

A young boy, excited about his first magic show, accidentally conjures a playful ghost that creates delightful chaos.

Result 06 storyboard preview
Result 07

A renowned historian delivers an inspiring lecture on ancient civilizations at a public forum, captivating her audience with her knowledge and passion.

Result 07 storyboard preview
Result 08

A young biologist discovers a rare frog species. He and his partners extracted the virus from the frog's body.

Result 08 storyboard preview
Result 09

A jovial professor and a bright student discover a lost ancient artifact, leading them to present their findings to an eager audience.

Result 09 storyboard preview
Result 10

A street performer and a focused scholar cross paths in a bustling city, leading to an unexpected moment of connection.

Result 10 storyboard preview
Result 11

A little boy and his mother traveled through a storybook to a fantasy world.

Result 11 storyboard preview
Result 12

An old fisherman was fishing by the lake and caught a unexpected treasure.

Result 12 storyboard preview
Result 13

A man's cow was sick, so he called a veterinarian to treat it.

Result 13 storyboard preview
Result 14

A seasoned architect and a resourceful street artist discuss art.

Result 14 storyboard preview
Result 15

A lone fisherman on a small boat casts his net into the tranquil sea, hoping for a good catch.

Result 15 storyboard preview
Result 16

A young musician teaches himself to play the guitar and fantasizes about performing on stage.

Result 16 storyboard preview
Result 17

A dedicated archeologist discovers an ancient, forgotten artifact in a dusty library.

Result 17 storyboard preview
Result 18

A nature photographer and a park ranger discover a rare bird species.

Result 18 storyboard preview

Please reference UniMAGE using the BibTeX entry below.

@article{zhang2025bridging,
  title={Bridging Imagination with Audio-Video Generation via a Unified Director},
  author={Zhang, Jiaxu and Hu, Tianshu and Zhang, Yuan and Li, Zenan and Luo, Linjie and Lin, Guosheng and Chen, Xin},
  journal={arXiv:2512.23222},
  year={2025}
}