Skip to main content

Journal Value Metrics Assessment

March 30, 2010 Author: Jacqueline WilsonCategories:

By Jacqueline Wilson, CDL Senior Associate for Shared Content

In 2008, the UC libraries adopted a new value-based strategy developed by CDL which is now used in negotiations with major publishers.  The strategy involves using objective metrics to compare the value of scholarly journals, establish price containment targets, and identify titles of poor value to consider for potential cancellation. The process is objective and quantifiable and is based both on cost and value with a goal of leading to more sustainable prices overall.

A key aspect of this new methodology is the use of a Weighted Value Algorithm to assess multiple vectors of value for each journal title under review.  Value is assessed in three overall categories:  Utility, Quality, and Cost Effectiveness.  For example, usage statistics contribute to a journal’s Utility score, impact factor contributes to its Quality score, while both cost per use and cost per impact factor contribute to its Cost Effectiveness score.  A composite score is then assigned to each journal to assess its overall value in comparison to other journals in the same broad subject category.  In addition to the weighted value algorithm, many other metrics are compiled and provided to campus librarians by CDL to ensure the richest possible set of information with which to make important selection decisions.

In order to assess the usefulness of the value metrics used in the 2010 journal title review and identify ways to improve the process for 2011, CDL made an online survey available to UC Collection Development Officers, campus bibliographers, and other librarians who participated in last year’s journal title review. CDL staff members Chan Li and Jacqueline Wilson developed the survey and analyzed the results.  Sixty-two librarians on nine campuses completed the survey. The majority of respondents indicated they were bibliographers.  Participants subject specialties included:  45% social science & humanities; 29% science & technology; 18% life & health sciences; and 13 % other. The survey instrument and full survey results can be found at:  http://www.cdlib.org/services/collections/protected/survey/index.html [password protected]; UC users can request a password here.

The majority of survey participants found the CDL Weighted Value Metrics to be useful, but also wanted to increase the validity of the metrics.  As a result of the survey, CDL will make the following changes to the metrics for 2011:

  • Restructure the weighted value algorithm to encompass additional data points and to better account for missing data elements.  Titles with little data will get low scores so they can be reviewed by the campuses.
  • Explore the feasibility of adding potential ILL costs for titles which may be cancelled
  • Correct usage data anomalies (normalize data & note problems)
  • Add a new major subject category division: Arts and Humanities
  • Seek  additional citation data for social science & arts and humanities titles
  • Bundled titles will be treated as one entity
  • Unsubscribed titles which were  included in packages at no cost will be shown with their publisher list price
  • Eigenfactor use and value will be reconsidered as it is less widely understood and tends to track closely to impact factor

New journal metrics which were of interest to at least 50 percent of the survey participants will be investigated for possible inclusion in this year’s journal title review process.  These metrics include:  # of editors by campus, # of authors by campus, average UC use per article, average UC cost per article, 5 year usage trends, and 3 year usage trends.

Survey participants also identified a number of general changes they would like to see in the next journal review process:

  • Lengthen the time given for campuses to review titles and metrics. Journal package lists up for review will start going out by the end of April in time for campuses to coordinate systemwide decisions with local collection decisions
  • Lists of new and transfer titles for journal packages that are up for review  will go out separately in the early Fall
  • CDL will clarify definitions of “subscribed” & “unsubscribed,” “licensed “and “not licensed,” and  “Historical UC Subscription”
  • CDL will continue to build support for niche titles into the review process
  • CDL will revise voting choices with clearer definitions
  • Usual campus co-shares will be used allocate costs for each journal package; title votes will not be used to re-calculate campus cost shares

CDL will host a number of webinars for campus librarians in May to review the metrics and procedures for this year’s title review process.  More information on the webinars can be found at:  http://www.cdlib.org/services/collections/protected/survey/index.html.

Please contact Chan Li (Chan.Li@ucop.edu) or Jacqueline Wilson (Jacqueline.Wilson@ucop.edu) if you have any questions or would like additional information.