Databricks has announced the acquisition of Lilac AI, a startup that provides a scalable, user-friendly tool for data scientists to search, cluster, and analyze text datasets with a focus on generative AI. The move comes as Databricks aims to strengthen its position in the rapidly evolving AI landscape.
According to Databricks CEO Ali Ghodsi, the company's research team strongly advocated for the acquisition during a recent offsite meeting. "They were already using it and they were very stoked," Ghodsi said. "It was a no brainer, they were interested, we were interested, you know, this was a love marriage."
Lilac AI's technology will be integrated into Databricks' platform to help customers accelerate the development of production-quality generative AI applications using their own enterprise data. The startup's core mission aligns with Databricks' commitment to provide customers with end-to-end GenAI capabilities.
The acquisition is part of Databricks' ongoing strategy to build out its suite of AI tools. In 2023, the company made headlines with its $1.3 billion purchase of generative AI startup MosaicML. Ghodsi emphasized Databricks' position as the go-to vendor for companies looking to train LLMs or AI on their private data. "When it comes to Gen AI, our strategy is very simple," he said. "We are the best vendor out there if you want to train LLMs or AI on your private data."
Lilac AI's technology addresses the challenges of exploring and understanding unstructured text data in the age of GenAI. Traditionally, analyzing such data has been a manual, labor-intensive process that lacks scalability. Lilac AI offers a scalable solution that encourages and facilitates interaction with data through an intuitive user interface and AI-augmented features.
The startup's founders, Daniel Smilkov and Nikhil Thorat, each spent a decade at Google honing their expertise in developing enterprise-scale data quality solutions. Their experience, team, and technology will be a valuable addition to Databricks.
With the integration of Lilac AI's technology, Databricks hopes to make it easier for customers to evaluate and monitor the outputs of their LLMs in a unified platform, as well as prepare datasets for Retrieval-Augmented Generation (RAG), fine-tuning, and pre-training. The company plans to share more details as the integration progresses.