Text to video model

Field

Artificial Intelligence / Computer Vision

Primary Function

Generating video from text instructions, still images, or existing video extensions

Key Architectures

Diffusion Models, Transformers, Generative Adversarial Networks (GANs), Recurrent Neural Networks (RNNs)

Training Data Source

Synthetic Data and video-text pairs (e.g., YouTube titles and descriptions)

Max Video Length (Sora)

60 seconds

Parameter Count (CogVideo)

9.4 billion

Parameter Count (HunyuanVideo)

13 billion+

2022-01-01

2022-09-29

2022-10-05

2024-02-15

2024-10-01

2024-12-01

30. ^ Jin, Jiayao; Wu, Jianhang; Xu, Zhoucheng; Zhang, Hang; Wang, Yaxin; Yang, Jielong (4 August 2023). "Text to Video: Enhancing Video Generation Using Diffusion Models and Reconstruction Network". 2023 2nd International Conference on Computing, Communication, Perception and Quantum Technology (CCPQT). IEEE. pp. 108–114. doi "Doi (identifier)"):10.1109/CCPQT60491.2023.00024. ISBN "ISBN (identifier)") 979-8-3503-4269-7. [...] There are several architectures that have been used to create text-to-video models. Similar to text-to-image models, these models can be trained using Recurrent Neural Networks (RNNs) such as long short-term memory (LSTM) networks, which has been used for Pixel Transformation Models and Stochastic Video Generation Models, which aid in consistency and realism respectively. An alternative for these include transformer models. Generative adversarial networks (GANs), Variational autoencoders [...] There are different models, including open source models. Chinese-language input CogVideo is the earliest text-to-video model "of 9.4 billion parameters" to be developed, with its demo version of open source codes first presented on GitHub in 2022. That year, Meta Platforms released a partial text-to-video model called "Make-A-Video", and Google's Brain (later Google DeepMind) introduced Imagen Video, a text-to-video model with 3D U-Net.

| Model | Parameters | Created by | Released | --- --- | | HunyuanVideo | 13B+ | Tencent | Dec 2024 | | Mochi (deploy on Modal) | 10B | Genmo | Oct 2024 | | Wan2.2 | 5B and 14B | Alibaba | Jul 2025 | Text-to-video AI models build on text-to-image foundations but add a more difficult dimension: time. Every frame must not only look convincing on its own but also stay coherent across seconds of motion. This shift introduces new failure modes: [...] Text-to-video models introduce a whole new set of unique operational challenges. This means that the choice of a model isn’t as easy as picking the one that looks good. We have to find a model that fits into our existing workflow and hardware budget. ### Things to Think About When Selecting a Model [...] ## Closing Thoughts The text-to-video space is moving at a very fast clip, with new models claiming “state-of-the-art” being released every few weeks. The common message across these models is that there is no single “best” option, but a growing set of trade-offs.

Introducing Sora, our text-to-video model. Sora can generate videos up to a minute long while maintaining visual quality and adherence to the user’s prompt. [...] In addition to being able to generate a video solely from text instructions, the model is able to take an existing still image and generate a video from it, animating the image’s contents with accuracy and attention to small detail. The model can also take an existing video and extend it or fill in missing frames. Learn more in our technical report⁠.

In comparison, the T2V model provides both background and motion features. The intermediate gist-generation step fixes the background style and structure, and the following Text2Filter step forces the synthesized motion to use the text information. These results demonstrate the necessity of both the gist generator and the Text2Filter components in our model. In the following subsections, we intentionally generate videos that do not usually happen in real world. This is to address concerns of [...] • Text-to-video generation with pair information (PT2V): DT2V is extended using the framework of (Reed et al. 2016). The discriminator judges whether the video and text pair are real, synthetic, or a mismatched pair. This is the method in Figure 3(b). We use a linear concatenation for the video and text feature in the discriminator. [...] Our contributions are summarized as follows: (i) By view-ing the gist as an intermediate step, we propose an effective text-to-video generation framework. (ii) We demonstrate that using input text to generate a filter better models dynamic features. (iii) We propose a method to construct a training dataset based on YouTube (www.youtube.com) videos where the video titles and descriptions are used as the accompany-ing text. This allows abundant on-line video data to be used to construct robust and