Luma AI, a San Francisco-based AI startup, has released Dream Machine, a powerful new AI tool that enables users to create stunning, high-quality videos from text and images in a matter of minutes. AI content creators on social media haven't been this excited for a product release since last week 😅.
From the moment OpenAI first showed off the capabilites of Sora, their text-to-video model, earlier this year, the creative community has been eagerly awaiting its release, or hoping for similar alternative to come along. And while we have seen challengers from China like Vidu and Kling, we really haven't had an option that was publicly available–until now.
Here are some fun examples of videos we have seen in the wild generated by Dream Machine:
We are living in the best time in human history!!
— Mantle Hood (@mantlehood) June 12, 2024
😍😍😍😍
To see the strawberry elephant come to life like this.@LumaLabsAI #lumadreammachine #aiart #aivideo pic.twitter.com/PNmzF7oW6G
Image to Video. 🖤 Created with @LumaLabsAI #LumaDreamMachine pic.twitter.com/dsMPPFWN56
— Karoline Georges (@KarolineGeorges) June 13, 2024
Ha! And you thought @LumaLabsAI was just about cool 3D objects. WRONG! It turns out that thousands of Gaussian splats can EASILY be turned into a POWERFUL video generation model.
— Justin Hart (@justin_hart) June 12, 2024
I got early access!
(Envision a teaser trailer for a new streaming channel!) #LumaDreamMachine pic.twitter.com/4HFgwaHuAk
Luma AI has swiftly and quietly become a major player in the consumer AI space. Last February, the company rose to fame for pushing the state-of-the-art with their Realtime NeRF on the web. In January this year, they raised $43B form a16z and launched Genie, a text-to-3D AI model.
Chris McKay
Dream Machine builds on that foundation, and the result is a tool that some have been quick to annoint as the Sora-killer. That may have been premature. In our tests, we found that while it may not be at Sora's level just yet, it does have a few tricks up its sleeve.
For starters, we found that grounding the prompt with an image, fared much better than text-only prompts. Once you play around with it for a while, it quickly becomes apparent that the model doesn't fully understand or will stay faithful to the details in your prompt. However, by starting with an AI-generated image from a state-of-the-art model like Midjourney, you can get more control over the scene composition and dramatically improve the quality of the output. And this is where Dream Machine really shines—at animating images.
With the right image and prompt, you can capture realistic and consistent motion and even complex facial expressions and emotion. Dream Machine artfully layers in expert cinematography, while maintaining character consistency and simulating real-world physical characteristics.
We are LOVING the generations from @LumaLabsAI Dream Machine.
— Curious Refuge (@CuriousRefuge) June 12, 2024
Now it’s time to answer the question, how does it compare to these other big name video generators.
For most of these shots, Luma wins.#lumadreammachine #aifilmmaking #aivideo #filmmaking pic.twitter.com/xP4bPUfH17
That being said, we did notice some limitations, as Luma AI itself has transparently highlighted. These include issues with objects morphing during movement, unrealistic character motions (like a gliding dog instead of a running one), and challenges with generating coherent text within scenes.
It makes sense that Luma AI felt comfortable releasing this product publicly. It is much closer to Pika and Runway than it is to Sora. Unlike Sora, the videos generated by Dream Machine certainly didn't give me that "this could be really bad" uneasiness. Using the tool, it hardly felt like we had crossed some major threshold—though, as Midjourney has shown, this can quickly change.
For now, Luma AI has built a delightful creative playground where enthusiasts can wander and discover artistry at the frontier of AI video generation. In the end, the giveaway is in the name—Dream Machine.