Announcing a New Partnership: California Digital Library, UC Libraries, and Internet Archive’s Archive-It Service

The CDL and the UC Libraries are partnering with Internet Archive’s Archive-It Service. In the coming year, CDL’s Web Archiving Service (WAS) collections and all core infrastructure activities, i.e., crawling, indexing, search, display, and storage, will be transferred to Archive-It. The CDL remains committed to web archiving as a fundamental component of its mission to support the acquisition, preservation and dissemination of content. This new partnership will allow the CDL to meet its mission and goals more efficiently and effectively and provide a robust solution for our stakeholders.

Why now?

Eight years after the release of WAS, we found ourselves at a critical juncture. The constantly changing and ever-increasing complexity of the web poses significant challenges to the current web archiving toolset and requires frequent upgrades to stay ahead. It became clear that there was a significant opportunity cost to maintaining WAS, which would not leave us with the capacity to develop new added-value web archiving services, such as tools for researchers, computational analysis of aggregated archival corpora, or work toward integrating web archives with other format types.

Collaboration is the Solution

In 2014, the CDL held a series of meetings with peer institutions to investigate the possibility of collaborating on web archiving solutions. We ultimately came to the conclusion that running the core web archiving infrastructure is not the best use of our limited resources. Instead, enlisting the services of Archive-It was the most efficient solution because it will permit the CDL and its partners to reallocate their local resources to activities through which they can uniquely add stakeholder value to the baseline function provided by Archive-It.

Thus, the CDL is currently exploring opportunities with Harvard, MIT, Stanford, UCLA, and others to work closely with Archive-It to create an expanded roster of added-value tools and services. Our goals are to define technical needs as well as the organizational structure that can ensure creation of new tools and services and make them broadly available across the community.

Thank you!

Thank you to the many UC and Non-UC archivists and librarians who have partnered with CDL on web archiving activities – we look forward to continuing our collaboration through this next phase of development.

CDL would also like to express sincere thanks to and recognition of CDL staff (past and current) who have worked on WAS, including Stephen Abrams, Trisha Cruse, Rondy Epting-Day, Scott Fisher, Erik Hetzner, John Kunze, Rosalie Lack, Tracy Seneca, Marisa Strong, Ken Weiss, and Perry Willett.