20 Evaluation of Digital Library

Mitesh Pandya


I. Objectives:


The objectives of this module is to impart knowledge on various aspects of evaluation of digital library includes:


•   Basic concept behind evaluation of digital library

•   Need and purpose of digital library evaluation

•   Different approach to digital library evaluation

•   Process and steps involved in digital library evaluation

•   Criteria for evaluation digital library and how to evaluate

•   Different stages of evaluation

•   Models of digital library evaluation



II. Learning Outcomes 


After going through this lesson, the learner would attain knowledge on need, purpose, different approaches, the process and steps involved in evaluation of digital libraries. Users would learn about parameters, criteria and models used for evaluation of digital library and various stages for carrying out the digital library evaluation.



III.   Structure 


1.      Introduction

2.      Need for Digital Library Evaluation

3.      Different Approaches to Digital Library Evaluation

4.      How to evaluate DL: Process and Various Steps

5.      Criteria for Evaluation of Digital Library

6.      What to Evaluate?

7.      Evaluation of the Digital Library during Various Stages

8.      Evaluation of Digital Libraries: Different Perspectives

9.      Models for Evaluation of Digital Library

10.     Summary


1.    Introduction 


The world is changing very fast with increased application of information and telecommunication technology. With the advent of information and communication technology, the concept of the library has gone under complete metamorphosis. Digital libraries extend and integrate approaches adopted in traditional libraries, as well as in distributed information systems, to concede high-end information systems, services, and institutions. Continuing research and innovations in various fields of studies led to an information explosion. In the last couple of years, the library and information centres have changed dramatically in terms of its collection, organization and services. Information and communication technology have changed the concept of the library as never before. Nowadays, libraries and information centers across the world are rapidly moving towards the digital format and collection as digital library has the potentials to reach out to its users without any geographical restrictions. However, it requires sufficient fund to built-up a digital collection using known digital library management system. Development of digital libraries requires immense investment in terms of trained manpower, latest technology and sufficient funds. As such, it is essential to carry out timely evaluation for the digital library.


Evaluation is a general term that includes several aspects of performance measurement and assessment and the activities that include experiments, surveys, time series analysis, online monitoring of user-system, observation of use, and other forms of data collection for the ultimate purpose of evaluation. Evaluation has many intentions ranging from highly focused and well-defined product testing to the highest form of cognitive reflection. The evaluation of the digital library helps in finding its integrity, usability and relevancy. The process allow managers to judge whether a digital library meets the objectives for which was developed. Therefore, evaluation of a digital library is done for the decision maker to take necessary policy decision on the development, continuation and evaluation of digital library. Evaluation should be carried out on a regular interval to meet to keep the digital library up-to-date and relevant.



2.    Need for Digital Library Evaluation 


The first and foremost step in any evaluation process is to establish a strong and a clear cut rationale for evaluation. In simple words, one should begin any evaluation by answering the question: “Why evaluation is necessary?”


Evaluation is carried out for many reasons. Often, evaluation is conducted to comply with some sort of requirement laid down by a funding agency. It may also be done for political reasons, or simply because the people involved in an enterprise may try to find out whether vigorous exercise that is being done is on the right track or not. Various types of professionals organizations carry out evaluation process to help in taking decisions. For example, a doctor inquires about medical histories, conducts examinations, and may perform various medical tests before diagnosis and treatment. Likewise, those who are involved in making decisions about digital libraries, have to perform various tests called evaluations. There are certain advanced areas in digital libraries whose evaluation may require the help of experts. Evaluation of the digital library is a fact-finding process on the basis of collected evidences. It also includes value measuring which are integrated in the management process of digital libraries. It is an assessment of the performances, services, systems, products etc. Evaluation of digital library is often treated as a simple task and many times it is overlooked but, actually it is complex, scientific, necessary for long-term survival and to check the relevancy of a digital library. Because of the complexity and multi dimensionality of the digital library, it becomes quite technical to evaluate it.


3.  Different Approaches to Digital Library Evaluation 


In the evaluation of digital libraries, as in the evaluation of any other system or process, a difficult question arises that clearly affects the results: Where does a digital library under evaluation begin to be evaluated? Where does it end? What are the boundaries? What to include? What to exclude? According to Saracevic (2000) evaluation means an appraisal of the performance or functioning of a system, or part thereof, in relation to some objective. The performance can be evaluated as:


a)      Effectiveness: How well does a system (or any of its parts) perform for which it was designed?

b)      Efficiency: At what cost (costs could be financial or involve time or effort)?

c)      A combination of these two (i.e., cost-effectiveness).


According to Chawdhury & Chowdhury (2002) digital libraries are judged by their effectiveness in terms of cost and use. Marchionini (2000) has pointed out that evaluation is a research process that aims to understand the meaning of a specific phenomenon situated in the context.


The evaluation process for digital library needs to be carried out by the experts in the field, who can evaluate using various approaches. Digital libraries are different than traditional libraries in terms of its collection, execution and support and no evaluation can possibly address all of the aspects altogether. Three important approaches used in evaluation of digital libraries are as follows.


3.1  User and Content Centric Approach: The user centric approach involves assessment of features used by end users which are integrated in the digital library portal. This approach is a bridge between human and system. This approach is used before developing a digital library.


3.2 System Centric Approach: The system centric approach involves the study of software and its performances. It also assesses effectiveness of the system design and technological component. This approach is mostly used at the time of the system design and deployment.


3.3  Economic Approach: Economic approach is basically the study of cost, it measures cost effectiveness of the digital library. This approach will help policy makers to take decision about the worthiness of digital libraries.


4.  How to evaluate DL: Process and Various Steps 


Several complex evaluation models are proposed in scholarly papers, and evaluation specialists often have high qualifications in specific areas. However, the simple evaluation process may have following steps:


•   Identify the activities involved in the digital library;

•   Identify the questions that are to be addressed to inform the decision makers;

•   Identify the evaluation methods, models and tools, which will be used for data collection related to the questions;

•   Bifurcate evaluation in a phased manner so that it becomes effective and efficient; and

•   Write up the finding of the evaluations in accurately so that it can provide meaningful data to make the best possible decisions.


Different form of the evaluation process for traditional library that was established much before the invention of information and communication technology. The evaluation procedure  for digital library is quite different.


The evaluation process for digital library is carried out in a sequential manner starting from planning to evaluation. The evaluation process involves following steps:


•    Surveys and questionnaires


Surveys are used to collect data from a sample on a range of issues. One can use surveys to find out what people think about a particular system or service, including proposed changes, and how services can be improved. A questionnaire-based survey can be used in a variety of formats i.e by post, online, face-to-face and telephone interviews.


•    Observations


A method for data collection in which the area of interest is observed and the relevant facts, actions and behaviors are recorded.


•    Transactional log analysis


Transaction log is a source of user’s information and user’s information behavior. The transaction log system records users interactions for a particular digital library. Traced data in transaction logs are records of interaction which have been used to locate and navigate digital library.


•    Experimentation


Experiments are studies involving intervention by a researcher beyond that required for measurement. The usual intervention is to manipulate some variable in a setting and observe how it effects the subject being studied.


•    Record analysis


Record analysis is the process of systematically applying statistical and/or logical techniques to describe, illustrate, condense and recap, and evaluate collected records.


•    Usage analysis (COUNTER)


Usage analysis is a technique for understanding usage statistics. COUNTER stands for Counting Online Usage of Networked Electronic Resources. Basically, it is used to measure online usage of networked resource.


Fig. 1: Evaluation process of Digital Library


The evaluation process starts with “to-do list” of various activities to be performed at the time of evaluation. Once the activities are planned, methodology is selected based on data to  be collected and analyzed by various analytical systems. The analyzed data is used for taking policy decision, and the decision makers set various criteria on the basis of results for funding the project. Finally, it is decided whether digital library achieves the objectives for which it was developed.


5.  Criteria for Evaluation of Digital Library 


Evaluation is a combination between the selected elements to be evaluated and selected elements of its performance. This leads to selection of stages and levels of evaluation. Digital libraries, like other systems, can be viewed, and evaluated, from a number of slants or levels. Each of these slants can be translated into a criteria for evaluation. An evaluation focuses on decisions about the types of services that the digital library should alter. Some of the questions that might be answered are:


•   Which types of services are offered by digital libraries?

•   Which services the patrons have requested?

•   For which services patrons are willing to pay?

•   What services would require external funding?


Following are some of the important criteria that need to be used for the evaluation of digital library:


5.1  Usability


Usability is one of the important criterion for evaluation of digital library, especially with respect to interface design and operability. This criterion evaluates navigation efficiency, search retrieval efficiency and informative entropy as well.


5.2  Quality of the Collection


One of the important constituent of a digital library is collection/content it holds. The uploaded content should be evaluated on regular interval as there are many innovations that are taking place in a given area of knowledge. The relevancy of the content should be maintained for which a digital library is developed.


5.3  Service Quality


One of the major criticism for digital libraries is lack of a community service, which is one of the important functions for traditional libraries. Community services refer to services that are provided to members of digital library community, such as allowing members to exchange ideas, make announcements, etc.


5.4  Performance of the System


System performance is related to the relevance of retrieval of results and efficiency of the retrieval process. One of the major identified problem regarding system performance is how to increase precision ratio. In general, users care more about precision than recall since they only need enough information to solve their problems. In addition, precision affects the time a user needs to spend, to find relevant information, to solve his/her problems from a digital library.


6.  What to Evaluate? 


According to Chawdhury & Chowdhury (2002), a digital library may be evaluated from a number of perspectives, such as system, access and usability, user interface, information retrieval, content, services, cost as well as overall benefits and impact.


Saracevic (2000) provided a list of elements to be evaluated at the time of evaluation of digital libraries, wherein he has clearly indicated what to include and what to exclude. He suggested that the following elements may be considered for evaluation:


•   Digital collection and digital resources

•   Organizational structure and storage

•   Metadata and access management

•   Physical network and distribution

•   Security, privacy and legal aspects

•   Licenses and cost

•   User interface and retrieval, etc.


7.  Evaluation of the Digital Library during Various Stages


The digital library can be evaluated at the following three stages.


7.1  Evaluation during System Design:


In general, system for digital library is designed after consultation with users. The need to evaluate latest technology that is used for its development is one of the criterion for evaluation. The system needs to be designed on the basis of user’s requirements. At the time of designing system for digital library one has to conduct interviews, observations on user’s need and accordingly the system can be designed.


7.2  Evaluation during System Development 


The second stage of evaluation is carried out during the development of digital library. This includes logical work-flow, user friendly interface and necessary guidelines for the operation of the digital library. Since the development is being done by the technical professional, it needs to be evaluated at the time of development to make it fruitful.


7.3  Evaluation during System Deployment 


Deployment includes system execution, its use and maintenance. Technical professional first release the beta version of the developed system that is tested on the basis of requirements. System deployment means that the software is ready for deployment.


8 Evaluation of Digital Libraries: Different Perspectives 


8.1  User


How well does a digital library support the needs and demands of the user community? This can be very hard to evaluate because of different objectives of the user community. Many complex issues are involved. It should support information needs, tasks and activities of the individual users. The user also expects various facilities such as fast navigation, prompt retrieval, browsing and interaction. Therefore, evaluation needs to be carried to assess users perspectives.


8.2 System 


System is one of the complex aspects of the digital library. One has to evaluate hardware, software and network technology used in developing an infrastructure. During evaluation of system it is also expected to judge about the procedures, techniques, algorithms and operations performed by the digital library.


9.  Models for Evaluation of Digital Library 


To evaluate digital library applications, following two models are popular and are used most often.


9.1  DigiQUAL 


DigiQUAL is a model developed to measure service quality of digital library by the Association of Research Libraries with funding from the National Science Foundation. The DigiQUAL project is a new version of the existing LibQUAL+ protocol which was used to assess the services provided to the user communities of the National Science Digital Library (NSDL) program in US.


9.2  E-metrics 


The e‐metrics is a tool for library professionals to evaluate the use of digital library resources and services which are to be measured for its worthiness. E-metrics is a set of standardized tools and shared understandings about the value of the metrics for making data‐driven, managerial decisions in libraries. E‐metrics are applied to a number of library resources and service domains. It also provides statistics of usage of networked electronic resources collected by external vendors conforming to standards, such as COUNTER (Counting Online Usage of Networked Electronic Resources) and SUSHI (Standardized Usage Statistics Harvesting Initiative).


9.3  PEAK 


PEAK is an evaluation model developed by researchers of the University of Michigan in collaboration with University of Michigan Library. It is developed to measure usage and value of the digital content (electronic journals) subscribed the library. PEAK model provides valuable inputs for taking administrative decision about the digital library.


11.  Summary


Digital libraries are now at the mature stage and well developed in terms of technology, but it is quite necessary to evaluate various strategies used at the time of its development. Service is one of the major parts of the digital library in view of users and therefore, it seems essential to evaluate digital libraries. Thus, it should become an imperative that efforts in the formal evaluation of digital libraries be enlarged and become an integral part of all research  and practice.






  1. Bonn, M. S., MacKie-Mason, J. K., Riveros, J. F., & Lougee, W. P. (1999). A Report on the PEAK Experiment: Context and Design. D-Lib Magazine, 5(6). doi:10.1045/june99- bonn
  2. Buttenfield, B. (1999). Usability Evaluation of Digital Libraries. Science & Technology Libraries, 17(3-4), 39–59. doi:10.1300/J122v17n03_04
  3. Chowdhury, G. G., & Chowdhury, S. (2003). Digital library evaluation. In Introduction to digital libraries. London: Facet Pub.
  4. DigiQUAL. (n.d.). Retrieved April 4, 2014, from http://www.digiqual.org/home
  5. Iris Xie, H. (2006). Evaluation of digital libraries: Criteria and problems from users’ perspectives. Library & Information Science Research, 28(3), 433–452. doi:10.1016/j.lisr.2006.06.002
  6. Lesk, M. (2004). Usability and retrieval evaluation. In Understanding digital libraries. Boston: Elsevier.
  1. Marchionini, G. (2000). Evaluating Digital Libraries: A Longitudinal and Multifaceted View. Library Trends, 49, 304–333.
  2. Nicholas, D., & Huntington, P. (2008). Evaluating the Use and Users of Digital Journal Libraries. In F. Papy (Ed.), Digital Libraries (pp. 211–222). ISTE. Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/9780470611302.ch13/summary
  3. Saracevic, T. (2000). Digital Library Evaluation: Toward an Evolution of Concepts. Library Trends, 49(2), 350–69.
  1. Tsakonas, G., & Papatheodorou, C. (2009). Evaluation of digital libraries: an insight into useful applications and methods. Oxford: Chandos.
  2. Vullo, G. (2010). A Global Approach to Digital Library Evaluation. LIBER Quarterly, 20(2), 169–178.
  3. Zia, L. L. (2001). The NSF National Science, Technology, Engineering, and Mathematics Education Digital Library (NSDL) Program: New Projects and a Progress Report. D-Lib Magazine, 7(11). doi:10.1045/november2001-zia