Remigiusz Sapa The quality of WWW services offered by academic libraries - the usability test[1]
	Library and Information Science Institute, Jagiellonian University
	This article was translated thanks to the grant received from the Open Society Institute

The times, in which it was mainly proper and somewhat fashionable for academic libraries to have a website, have ended without a doubt. These days, it is hard to imagine an academic library without its own web service. These seemingly ordinary statements are concealing a more serious phenomenon. Some five to seven years ago, the very fact of having a website, in a certain way, elevated the library status. Nowadays, users treat it as an expected and required service that every academic library should offer without a question. However, launching a library web service is similar to opening a new library branch. It is impossible to draw satisfaction from the very fact of its launching. If the web service is poorly situated, badly organized and unpleasing to users, the launching can bring the library more loss than gain (e.g. increasing costs with a declining overall user satisfaction).

Therefore, the current problem is not the question of whether an academic library should offer an increasing number of web services, but the issue of how it should do so in order to improve the quality of its services. Unfortunately, due to their relatively poor standard, some academic library web services do not contribute to creating a positive image of libraries. This article seeks to present a commercially established technique, which makes it possible to find and eliminate oversights in planning and developing web services. Furthermore, it seeks to encourage administrators of web services offered by academic libraries to adapt the method to local needs and possibilities of their institutions.

Usability of WWW services

The quality of web services (webpages, websites) is a very complex and multifaceted issue. The activity of service administrators cannot be restricted only to one area. Nonetheless, the contemporary approach towards quality in web services places the primary attention on the user. It is the user's subjective perception, the extent to which his needs are met or, his satisfaction, that mostly determine the standard of the web service. Therefore, the issue of quality has to be approached in such a way that excludes hopeless searching for ideal solutions and, instead, focuses on the relationship with an existing and potential user of a given service.

This article will propose one, but very important, way of looking at the quality of web services, that is, from the viewpoint of usefulness and user-friendliness of interfaces, known as usability. However, by all means, it is not to suggest that other aspects of quality of academic library web services are marginal, nor is it to downgrade other methods and specific techniques of quality analysis. The job of managing the quality of web services must be complex. It cannot restrict itself to solutions presented in this essay.

In a general sense, the concept of usability deals with the relationship between a given product, a web service or some other information system, and its user. Above all, it concentrates on the surface level of the interface and refers to pragmatic and ergonomic aspects.^[2] There are several specific techniques of analyzing the way various products and services are used, such as, different types of interviews, questionnaires and observation. In the field of analyzing the use of computer programs and systems, there is a preference for methods supported by tools, which enable recording the behavior of users. Some of the tools focus on user expectations or, on the context, in which a given product is being used, while others concentrate on identifying obstacles encountered by users. James Hom offers a concise, interesting and practical survey of several most popular methods of usability analysis. ^[3]

It seems that the two most preferred and commonly used methods of analyzing the usability of WWW resources are heuristic evaluation (usability inspection) and conducting usability tests. Both techniques are based on observation and follow a simple rule, as expressed by Jakob Nielsen, the guru of heuristic evaluation, which states "First Rule of Usability? Don't Listen to Users."^[4] Heuristic evaluation and usability tests focus on the actual behavior of users, their way of navigating through a web service and, most of all, their way of accomplishing certain tasks. The only opinions of users that may be taken into account are expressed either during or after using a given service.

Nonetheless, there are fundamental differences between the two methods. In a heuristic evaluation, a small group of experts examines a user interface (WWW service) and assesses how it conforms to the earlier defined usability principles. These so-called heuristics were drawn from several hundred different problems identified in the usability study.^[5] As it is often pointed out, the advantage of this method, in comparison to the empirical usability testing involving actual interface users, is its simplicity, speed and low cost.^[6] However, it also has some serious drawbacks, such as, most significantly, the selection and subjectivity of the experts, and the fossilized canon of principles.^[7] In aiming to offer academic library users WWW services of the highest standard, it is worth applying techniques that, although more complex and time-consuming, are more solid and promising in terms of success. This is especially convincing since academic libraries have the benefit of the potential and resources of parent institutions and they are able to conduct these types of studies, for example, as part of master's or bachelor's theses of library and information science students.

In spite of the popularity of all sorts of surveys in the library community, the observation method maintains its supporters and a tradition. With the implementation and initiation of all the basic functions of computer systems (not just the OPAC), statistical modules and other system-provided programming tools, it became possible to easily monitor user behavior and to collect objective information about the use of the system interface and library resources.^[8] There are similarly great, or maybe even greater, possibilities in terms of observing the actual use of web services, since it is feasible to gather numerous data about the course of specific sessions and users themselves, as well as, to automate the statistical analysis of the collected information. Although this method can provide many valuable guidelines to ensuring the quality of a library and its web services, it is still not sufficient in any way.

Based on many years of experiences and observations of student use of web services in academic libraries, the present author is inclined to argue that student perception of the quality of web services is not determined merely by the extent, to which their primary needs are met; it involves many more factors. Obviously, in this context, the subject cannot be elaborated on but, surely, a combination of minor and major setbacks and problems with service navigation plays a very significant role. Observing the behavior of users as they perform specific tasks in the so-called usability test ^[9] should help in the elimination of these shortcomings. Below is a suggested, basic and general, model for examining the usability of academic library web services by conducting a usability test. Embarking on this type of study requires developing a detailed project that considers local circumstances, especially, the characteristics of users, specific tasks, financial and organizational possibilities of a given library.

Application of the usability test method in examining the usability of WWW services offered by academic libraries

Studies of this sort can be conducted in various stages of developing web services. They can be implemented in the planning stages to help avoid design oversights, as well as, when the service is already launched, to ensure a permanent quality control. There are several basic stages of conducting a usability test:^[10]

• specifying the study objectives,
• designing the test,
• conducting the test,
• analyzing the collected information and drawing conclusions.

When applying usability tests to examine web services of academic libraries, the above stages should be elaborated on appropriately.

Specifying the study objectives. Considering that the strategic goal is to provide a usable web service, it is necessary to describe the objectives in detail in order to make the test more specific. In other words, it is essential to identify the elements of the web service that will be examined. For example, this could include the way of ordering library materials through an on-line catalog offered within the library's web services, ease of compiling bibliographies on a given subject, ease of obtaining full-text documents through a virtual library (if offered), or the way of accessing information about the library, its services, opening hours and information resources. The specific goals of the study should derive not just from the objectives of a particular web service, but also, from the basic role of an academic library to support the advancement of science and the educational process of an institution. This also entails taking into account, for example, the question of accessing different information about the university through the web service of the library.

Designing the test. After the study objectives have been defined, it is possible to proceed to the stage of designing the test. It is essential to identify the participants of the test, prepare detailed scenarios to be completed by them, make technical arrangements and assign the necessary staff.

Test participants. If the study is to be extensive, the test participants should represent the widest possible spectrum of users for whom the service is intended. The participant group cannot solely include students of a given university. It must also consider researchers from various ranks or high school graduates, who are contemplating the choice of universities. In a general sense, two basic groups emerge from the body of users of academic library web services: end-users (students and scholars - key recipients of information services offered by the library) and cooperates (librarians from other institutions searching for bibliographic records, publishers, distributors of scholarly journals, software suppliers, conference organizers, librarians seeking employment, etc.). It cannot be overlooked that test results may vary depending on the fact that test participants can show different levels of familiarity with a given service. The web service should be as friendly to a novice as to an experienced user. Above all, the choice of users should depend on the precise objectives of the study specified in the first stage.

Formulating the tasks. Every test participant will have one or several tasks to complete. Each task should be presented in the form of a so-called behavioral scenario, which defines the starting point of activity, the aim and ways of reaching it.^[11] For example, they can be formulated in the following way:

• "Starting at the home page of the service, enter the library catalog, find and order the item Bazewicz M., Collen A., Podstawy metodologiczne systemów ludzkiej aktywnoœci i informatyki. Warszawa 1995, not using any of the additional guidelines, instructions, or system-provided assistance (help section)" OR
• "Starting at the home page of the service, locate contact information for the person who should to be reached in regards to donating books or periodicals to the library."

It is obvious that tasks need to be presented in a way that would enable the realization of all the specific study objectives considering the selected user categories.

Arranging the technical conditions. The test can be conducted in the library, as well as, in users' homes or any place with a standard personal computer connected to the Internet. It is important to take into account all the different types of Internet connections used to access the service (dial-up, permanent cable connection, Local Area Network…). In Poland, these have a fairly significant influence on the process of accessing WWW resources. While a user who enjoys a direct Internet connection within a LAN may not recognize any problems caused by a service overloaded with graphics or some other special effects, an individual surfer using a slow modem connection will have many negative comments in regards to this aspect. In order to avoid obtaining distorted results, all the possible forms of Internet access used by persons visiting the studied service, should be considered.

Conducting the test. Prior to administering the test, it is important to provide maximum comfort to the participants. The course of the test itself is relatively straightforward. The participants perform the tasks (scenarios) as defined and delivered to them on paper, the staff (observers) watches and records the users' behavior and verbal reactions. Additional information should be obtained by posing questions to participants during and after the test. Also, other supporting methods can be applied, such as, tape-recording responses of the participants encouraged to "think out loud" while executing the experiments, or filming their behavior.

Analyzing the collected information and drawing conclusions. This stage of the study should produce a list of problems to be solved, starting with the most important ones (appearing most commonly, causing the biggest frustrations) and concluding with minor issues in terms of the level of user satisfaction from the library's web service. In order to move through this stage efficiently, it is essential to establish a standard method of documenting the behavior of the participants and to instruct the staff accordingly. This includes basic matters, such as, a standard form of identifying the different webpages that make up the service, as well as, more advanced issues, for example, concerning the range of expressed opinions.

As a final point, it should be emphasized that, if examining the usability of an academic library web service via the usability test is not to serve theoretical generalizations or comparative studies, but the specific goals related to managing the service then, its scale, range and the way it is conducted should be tailored to those specific needs exactly. It seems that, at least in case of several WWW services offered by academic libraries, even very limited and simplified studies involving a small number of users and one specialist would eliminate some blatant mistakes.

Footnotes

[1] The term usability is gaining popularity in the Polish language. It appears in professional literature, as well as, in advertisements of companies offering usability testing services.

[2] Próchnicka, M.: Interfejsy użytkownika do systemów wyszukiwania informacji: kryteria i metody oceny [User interfaces of information retrieval systems: evaluation criteria and methods]. A presentation delivered at the 4th National Forum for Information Science and Technology 'Information - Knowledge - Economy,' Katowice, September 18-20, 2001.

[3] SHom, J.: The Usability Methods Toolbox. Access mode: http://jthom.best.vwh.net/usability; 20.12.2001.

[4] Nielsen, J.: First Rule of Usability? Don't Listen to Users. Jakob Nielsen's Alertbox, August 5. Access mode: http://www.useit.com/alertbox/200010805.html; 20.12.2001

[5] Nielsen, J.: How to Conduct a Heuristic Evaluation. Access mode: http://www.useit.com/papers/heuristic/heuristic_evaluation.html; 12.12.2001.

[6] Levi, M.D., Frederick G. C.: A Heuristic Evaluation of a World Wide Web Prototype. Access mode: http://stats.bls.gov/ore/htm_papers/st960160.htm; 31.12.2001.

[7] The Polish translation of the list of ten basic principles developed by Nielsen appears in: Próchnicka, M. op.cit.; cf. e.g. Buss R., Mühlbach L., Ruschin D.: Results of the Web Services Evaluation. Access mode: http://at.hhi.de/USINACTS/pub/d12.pdf; 31.12.2001.

[8] Sapa, R.: Zachowania informacyjne użytkownika OPAC w Bibliotece Jagiellońskiej. Zagadnienia Informacji Naukowej 1997 no. 2, pp. 70-79.

[9] Nielsen, J.: Usability Engineering. San Francisco.

[10] Hom, J.: op.cit.

[11] Cf. e.g. Buss R., Mühlbach L., Ruschin D.: op.cit., pp. 27-28.

Translated by Marta Sobieszek