Harvard Law School Library has announced the release of nearly one million public-domain books as a dataset for training AI models. The initiative aims to "level the playing field" for AI researchers and startups by providing access to high-quality, openly available training data.
Key points:
- Harvard's Institutional Data Initiative will release nearly one million public domain books from its library, including works from Shakespeare, Dickens, and Dante
- The collection, digitized through Google Books, spans multiple languages and genres, making it roughly five times larger than the Books3 dataset used to train Meta's Llama
- Microsoft and OpenAI are financially backing the initiative, which aims to lower barriers for AI research and development
Harvard’s Institutional Data Initiative (IDI) marks a significant step toward bridging the gap between knowledge institutions and the AI community. With initial funding from Microsoft and OpenAI, the project is focused on refining and releasing data collections that can support AI development while honoring principles of accessibility and stewardship. Among its first outputs is the release of nearly one million public-domain books, which have been scanned as part of the Google Books project.
This initiative comes at a critical moment in the AI landscape. High-quality training data remains a cornerstone of AI model development, yet access to such data has often been restricted to well-funded tech giants. IDI aims to “level the playing field” by offering a meticulously curated dataset to researchers, startups, and open-source developers. According to IDI’s executive director Greg Leppert, the project aspires to be as transformative as Linux in its ability to democratize technology development.
Beyond the dataset of books, IDI is collaborating with the Boston Public Library to digitize millions of public-domain newspaper articles, tackling complex challenges like extracting accurate text from historical layouts. These efforts highlight the initiative’s broader goal: enabling diverse and culturally representative AI systems by expanding access to previously untapped knowledge collections.
The announcement also underscores the ongoing debate around AI training data and copyright. While lawsuits over the use of copyrighted material in AI training remain unresolved, projects like IDI provide a legally sound alternative. Critics, however, caution that public-domain datasets must replace—not merely supplement—unlicensed data scraping to ensure ethical AI practices.
As Harvard prepares to host a symposium this spring to connect institutional and AI communities, the IDI’s ambitions extend beyond data sharing. The initiative seeks to establish best practices for responsible data use, fostering a collaborative ecosystem where knowledge institutions can continue to thrive in the age of AI. Whether this marks the beginning of a new era in AI training remains to be seen, but it’s clear that IDI’s work is poised to make a lasting impact.