Calculating scholarly journal value through objective metrics
Authored by Jacqueline Wilson and Chan Li
CDL’s Collection Development and Management: Licensed Resources program has developed a value-based strategy which is now used as a major part of the University of California’s journal collection planning process. The strategy involves using objective metrics to calculate the value of scholarly journals and identify titles that make a greater or lesser contribution to the University’s mission of teaching, research, and public service. The value-based process is objective and quantifiable and is based on measures of utility, quality, and cost effectiveness, with a goal of alignment to UC’s user communities and programmatic needs.
A key aspect of this strategy is the use of a Weighted Value Algorithm by Subject Category to assess multiple vectors of value for each journal title under review. This methodology compares each UC e-journal title licensed for systemwide use at the University of California against other UC-licensed titles within the same subject category according to a variety of objective value indicators, in order to arrive at a comparative value for each journal within the UC shared licensing portfolio. The analysis covers over 8,600 journals in 36 UC licensed e-journal packages.
The algorithm takes into account three vectors of value encompassing six data metrics: Utility (usage and citations), Quality (Impact Factor;1 SNIP 2) and Cost Effectiveness (cost per use; cost per SNIP). To establish a baseline for comparison, median values are calculated for each of these metrics within 158 different subject categories. After analyzing a variety of available subject schemes, the Hierarchical Interface to LC Classification (HILCC) 3 developed at Columbia University was selected as the best subject system for this purpose.
The table below demonstrates the differences in median values across all UC licensed titles for two sample subjects under each of the following broad subject categories: Life and Health Sciences, Physical Science and Engineering, Social Sciences, and Arts and Humanities.
A numerical score is assigned to each title for each of these measures depending on whether the value for each metric is above or below the median for that subject. Besides median values, other benchmark values are also established for some metrics to allow for further differentiation. For usage, titles in the lowest quartile of usage in a given subject category receive the lowest possible usage score. The value categories and the CDL algorithm by which they are assigned are shown below.
Finally, each title is assigned an overall value of High, Medium, Low, or Very Low based on its total score that combines the three measurement categories. The Utility category, which includes usage, is given the most weight in determining the overall score.
It is important to note that the value categories assigned using this algorithm are not meant to be absolute indicators of journal quality, but instead are an attempt to apply a set of consistent and rigorous criteria by which to analyze relative value within a specific institutional context. The citation, usage and cost data are all institution-specific measures, as are the median values used by the algorithm. This methodology answers the question “How much value does our institution derive from Journal X compared to other journals that we license in the same discipline?” where ‘value’ is defined as a combined measure of quality, utility, and cost-effectiveness.
In addition to the CDL Weighted Value Algorithm, many other metrics are compiled, calculated, and provided to UC campus librarians by CDL to ensure the richest possible set of information with which to make important journal collection decisions. As the quantity and variety of useful metrics has grown, CDL has found it necessary to develop a database to manage and calculate all this information for the over 8,600 journals that are licensed systemwide. The database was initially populated with a list of UC licensed journals downloaded from CDL’s electronic resource management system; additional data were then imported from numerous sources, including vendor usage statistics, CDL cost information, Impact Factors, UC citation rates, and other sources. The database associates the data for each title and calculates the subject and title-level metrics and score assignments for use in further analysis.
For additional information contact Jacqueline Wilson, Senior Associate for Collection Development and Management, California Digital Library (jacqueline.wilson@ucop.edu) or Chan Li, Senior Data Analyst, California Digital Library (chan.li@ucop.edu)
Foot Notes: