CDL in the Cloud!
The California Digital Library (CDL) provides many services for the scholarly community and the Infrastructure & Application Support (IAS) team is responsible for the technology underlying all of these services. The IAS team recently hit a new milestone as they completed a full migration of CDL programs and services into the Amazon Web Services (AWS) environment. After six years of training, planning, and development, CDL is now quite literally in the cloud!
Prior to 2010, the CDL services were hosted on physical Sun/Solaris servers located in local datacenters. While functional, this environment had a number of drawbacks, so in 2010 John Ober (then IAS manager) started a project to migrate CDL services to a virtualized environment based on VMware technology. This project was still ongoing in 2012 when John Ober retired and Kurt Ewoldsen was hired as the new IAS manager. While the organization had already derived benefits from the virtualization of CDL systems, a number of technical and operational challenges remained, so Kurt proposed a migration to the AWS cloud, even though the VMware migration was not yet complete.
In 2014, after review and approval of the plan, CDL embarked on a project to move the entire technology infrastructure for all services to the AWS platform. The first step was for the IAS team to create a virtual infrastructure within the AWS environment that would serve as a foundation for all of the CDL services to come later. They attended training classes, worked together through self-study, consulted with their account team, and worked through a variety of exercises in an experimental account before determining the final design for their AWS environment. Once the design was agreed upon, the real work got underway. They created an appropriate virtual network and populated it with all of the core services required to support CDL operations (Nagios monitoring and alerting, Puppet automation, Tripwire intrusion detection, NFS home server, LDAP servers, bastion remote access gateways, etc.). In conjunction with creating these new technical solutions, the team had to define and implement the new processes and procedures required to work within the new cloud infrastructure.
The pilot project started in mid-2014 and lasted about 6 months, during which time several representative services from each program area were migrated to the new AWS environment and tested for functionality, performance, and availability. During this phase, processes were revised and refined, and each application was thoroughly tested to ensure the viability of the new platform and validate the initial assumptions and projections of the project plan.
After a successful completion of the pilot project, the main migration effort started in early 2015 with a goal to complete the project that year. While the subsequent work proceeded (mostly) as planned, due to unforeseen requirements and resource constraints the project was actually completed in 18 months. During this time every remaining CDL service was successfully migrated to the new cloud environment and the legacy physical infrastructure was decommissioned. As of May 2016, CDL was entirely in the cloud!
The technical innovation at CDL does not end with this project, but simply starts a new chapter. Now that our services are running on the AWS platform, we have many opportunities to leverage the agility and breadth of services in AWS to provide even more efficient and cost-effective services to our community.
Photo credit: Craig Thompson