HIPNET - Health Information and Publications Network

Part III: Indicators that Measure Usefulness

USEFULNESS INDICATORS
No.
	*Area 1: User Satisfaction*
11	Percentage of those receiving a product or service that read or browsed it
12	Percentage of users who are satisfied with a product or service
13	Percentage of users who rate the format or presentation of a product or service as usable
14	Percentage of users who rate the content of a product or service as useful
15	Number/percentage of users who report knowledge gained from a product or service
16	Number/percentage of users reporting that a product or service changed their views
	*Area 2: Product or Service Quality*
17	Number and quality assessment of reviews of a product in periodicals
18	Number and significance of awards given to a product or service
19	Number of citations of a journal article or other information product
20	Journal impact factor
21	Number/percentage of users who pay for a product or service
22	Number/percentage of information products or services guided by theories of behavior change and communication

Evaluators want to collect both qualitative and quantitative information about the usefulness of an information product to its users. In a way, usefulness is a proxy for perceived value of a product or service. Usefulness indicators help rate the customer’s satisfaction with the product. This type of measurement is important because it can help information specialists design products that respond to the interests and meet the expectations of users. Thus, information specialists can facilitate the use of information, improving the uptake of content into policy, programs, and professional practice.

This part has two sections: user satisfaction and product or service quality. The indicators in Area 1, user satisfaction, measure how useful users deem a specific product to be overall in providing needed information. The indicators in this section can help provide a sense of the intended audience’s preferences as to the presentation of information as well as their perception of its content. Both are important to measure. For example, some users may not need a 200-page report but would appreciate instead a shorter summary of the report’s findings and recommendations. Other audiences may need or want the entire text.

The measures of usefulness in Area 2, product or service quality, relate to users’ perceptions of the quality of products and services in terms of authority, credibility, reputability, and trustworthiness. The indicators in this area include items such as number of times a product is cited in journals or other scientific materials. The assumption is that authors generally cite materials that they perceive to be credible and reputable (except in the less common case when they cite materials they wish to refute). Thus, the more useful, original, and authoritative a product is perceived to be, the more likely that it will be cited and used—other things, such as accessibility, being equal. This type of measurement is valuable for gauging the usefulness of scientific literature because it provides an evidence-based approach to assessing quality. This area also includes a quantitative indicator about payment for products, as a proxy for perceived value.

Area 1: User Satisfaction

Indicator 11:

Percentage of those receiving a product or service that read or browsed it

Definition: To “read” is to receive and comprehend messages in an information product or service. “Browse” is a term often used for online or electronic review of a product or service. It is used to describe how users move from place to place scanning the information contained in an online or electronic resource.

Data Requirements: Self-report from survey.

Data Source(s): User surveys distributed with the product or after a product has been disseminated (bounce-back questionnaires, online surveys, telephone surveys).

Purpose and Issues: This indicator provides critical information about whether a person who has received a product or service has actually read or browsed through it. Even if a publisher has evidence that a user was reached with an information product or service, it is important to verify that the user actually read it or browsed through it before determining the user’s assessment of product or service usefulness.

Indicator 12:

Percentage of users who are satisfied with a product or service

Definition: “Satisfied” refers to a user’s judgment of the adequacy of a product for that user and, therefore, measures perceived quality. Satisfaction reflects the user’s perception of the performance of a product or service. Satisfaction is an overall psychological state that includes cognitive, affective (like/dislike), and behavioral response elements.

Data Requirements: Evaluators can measure attitude/satisfaction using a Likert scale—i.e., by asking users how strongly they agree or disagree with statements (usually, the respondent can choose among “strongly disagree,” “disagree,” “not sure,” “agree,” and “stronglyagree”).

Data Source(s): Feedback forms or user surveys distributed with the product or after a product has been disseminated (bounce-back questionnaires, online surveys, telephone surveys); interviews with users; focus group discussions.

Purpose and Issues: Users’ perceptions are important because they govern users’ decisions about what action to take with the information they have obtained. If the information source passes muster with the user, there is some likelihood the information will influence that user’s actions. But if the user considers the information irrelevant or its presentation impenetrable, it certainly will not be applied. Users’ opinions are often easier to collect than reports of actions taken. Therefore, this indicator often must serve as a rough gauge of the likelihood that the respondent makes use of the information.

To understand users’ applications of, experiences with, and the impact of products, evaluators may need also to conduct in-depth interviews or focus group discussions with selected users.

Example:

When queried on their satisfaction with different aspects of The Pop Reporter (n=512), 94% of respondents strongly agreed/agreed that they were “satisfied with the frequency of the publication,” and 93% strongly agreed/agreed that they were “satisfied with the amount of headlines and links in each issue.” Also, 83% strongly agreed/agreed that “the variety of subject categories in The Pop Reporter meets my needs.”

Indicator 13:

Percentage of users who rate the format or presentation of a product or service as usable

Definition: “Usability” applies to the design features or presentation of a product or service. For Web sites, this indicator can include information on the navigability of the information architecture.

Data Requirements: Evaluators measure attitude/satisfaction, often using a Likert scale to gauge reactions to statements related to writing style and design features, organization of the information, ease of finding information, etc. Using a Likert scale allows respondents to rate statements along a continuum (1 to 5, for example, where “1” is “strongly agree” and “5” is “strongly disagree”).

Data Source(s): Feedback forms or user surveys distributed with the product or after a product has been disseminated; focus group discussions; observation (user testing).

Purpose and Issues: This indicator is intended to provide an overall assessment of how practical readers found a product or service. Format and presentation largely determine practicality. To assess how usable a product or service is, it is helpful to conduct user surveys several months after a product or service has been disseminated, so that users have time to put the product to use.

For online products accessibility and connectivity are important aspects of usability. Consequently, it is important that products delivered via the Internet are specifically designed for those who have low bandwidth (for example by limiting the use of large graphical elements). In addition, it is important to present information in multiple formats (for example, offering text transcripts of audio clips or presenting multiple formats such as Microsoft® Word, PDF, and HTML).

For Web sites, this indicator encompasses whether the organization of a Web site enables a user to find relevant information quickly. Usability testing should be a step in product design. Since the audience’s experience with online resources generally will affect responses, it is important to test with members of the intended audience if at all possible, as well as with staff and colleagues. Usability testing also can be an element of monitoring and evaluation.

What has been seen with Web users may well be true for users of print products as well—that what users say about their perceptions of usability may not coincide with their behavior. Observation of people using a Web site helps overcome this discrepancy. A simple equivalent for print reference materials is to ask audience members to find some specific information in the reference and then to observe if and how they locate it.

Example:

In a user survey conducted in 2002 (n=2854), 83% of the respondents rated the layout and design of Population Reports as good or excellent.

Indicator 14:

Percentage of users who rate the content of a product or services as useful

Definition: “Useful” here reflects the relevance and practical applicability of the content of a product as perceived by the respondent.

Data Requirements: Responses to questions such as “Was the topic(s) covered in the product interesting and useful to you?”

Data Source(s): Feedback forms or user surveys distributed with the product or after a product has been disseminated; interviews; focus group discussions.

Purpose and Issues: This indicator is intended to provide an overall assessment of how useful the content of a product is. It can suggest areas for improvement, which helps to guide product or service development. As with questions concerning usability, asking the question several months after dissemination can help obtain considered responses.

Example:

Readers of The Manager (n=282) ranked the publication as extremely or very useful (77%) or as useful or somewhat useful (22%).

Indicator 15:

Number/Percentage of users who report knowledge gained from a product or service

Definition: This indicator of usefulness gauges whether users feel that they have learned from an information product or service.

Data Requirements: Self-report in survey; anecdotal reports from users.

Data Source(s): Feedback forms or user surveys distributed with the product or service or after a product or service has been disseminated.

Purpose and Issues: Questions can be designed to gauge whether users learned something that they can apply to their work and whether they learned something that they have shared or plan to share with others.

In yes/no format, responses to this question usually are not sufficiently informative. It can be followed up with an open-ended request for the most important point learned or a checklist of topics areas.

Indicator 16:

Number/Percentage of users reporting that a product or service changed their views

Definition:This indicator gauges whether users’ views, attitudes, opinions, or beliefs changed as a result of the information in the product or service. Views may be a favorable or unfavorable state of mind or feeling toward something.

Data Requirements: Self-report in survey; anecdotal reports from users.

Data Source(s): User surveys distributed with the product or service or after a product or service has been disseminated.

Purpose and Issues: Questions about whether users changed their views following the receipt of information can help reveal whether the information was internalized and whether the new knowledge transformed the user in a significant way. Like questions about knowledge gained, questions about views need to cover what views changed and in what direction. People often act in ways that are compatible with their views. Consequently, those who are favorable toward the content presented are likely to adopt desired behaviors in the future (Bertrand and Escudero, 2002).

Example:

The Population Reference Bureau published Abandoning Female Genital Cutting, a report on the prevalence of the practice in Africa and examples of programs designed to end the practice. Readers were asked if and how the publication changed their views. The following are sample responses:

“It made me more determined to include FGM as a subject in our week-long trainings for traditional healers and birth attendants”—Coordinator, Community-based Health Promotion Program, HIV/AIDS Prevention and Control Unit, Dar es Salaam, Tanzania

“I was helpless in ways to stop FGC. Now I know that international organizations are concerned and provide support. I am convening NGOs in northern Ghana to form a network for FGM elimination.”—Director, Community Welfare Foundation, Kumasi, Ghana

“I thought it was impossible to change that culture, but after reading your [report], I have known the methods to employ to end the practice.”—Reproductive Health Trainer, Ministry of Health, Bushenyi, Uganda

Area 2: Product or Service Quality

Indicator 17:

Number and quality assessment of reviews of a product in periodicals

Definition: “Reviewed” refers to a written assessment of a publication that is published in a periodical. “Periodical” may refer to a peer-reviewed research journal, an organizational newsletter, or a book review medium such as Library Journal.

Data Requirements: Information about the number of reviews that a product has received in a periodical and their assessment of the quality of the product.

Data Source(s): Tearsheets, offprints, or reports from periodicals; searches in periodicals.

Purpose and Issues: A positive review not only testifies to the quality of the product but also extends its reach by alerting the review’s readers. A negative review may actually limit reach by discouraging orders.

Reviews are not easy to obtain, because publications usually must be sent to review media before they are published, and in most journals there is competition for the space allotted to reviews. In considering how well the review actually reflects the quality of the product, who wrote and who published the review need consideration.

Example:

In 2002 two MSH publications—Fees for Health Services: Guidelines for Protecting the Poor and Ensuring Equal Access to Health Services: User Fee Systems and the Poor—were each reviewed in Health Economics, 11 (2002): 181–82: “Taken together, these two publications provide a thorough and readable guide to an important area of health policy in developing countries. They are balanced on the advantages and disadvantages of user fees, and eminently practical.”

Indicator 18:

Number and significance of awards given to a product or service

Definition: Awards are defined as formal recognition of the quality of a product by an independent external source that selects the product in a formal or implicit competition.

Data Requirements: Notification that an award has been received, name of award, source of award, feature of product or service that is receiving recognition (e.g., design, content).

Data Source(s): Communication, such as a letter, from the agency or organization making the award.

Purpose and Issues: Sources of awards include the Society for Technical Communication, the National Association of Government Communicators, and the Population Institute’s Global Media Awards. Thomson Gale publishes an international directory entitled Awards, Honors and Prizes.

An award may be made for content, design, dissemination, impact, or other features of an information product. In terms of value, the significance and degree of competitiveness of an award should be considered. For example, it is much more difficult to earn a Pulitzer prize than an honorable mention in a design competition.

Example:

In 2006 the INFO Project won two awards for its Web site http://www.infoforhealth.org. The Web site won the Web Marketing Association’s 2006 WebAward for Outstanding Achievement in Web site Development and the 2006 Spring/Summer World Wide Web Health Award.

Indicator 19:

Number of citations of a journal article or other information product

Definition: This indicator is used to count the numberof times a journal article or other information product (such as a book) is referenced in other information products.

Data Requirements: Data from citation studies or Journal Citation Reports—Science Edition (ISI) and Journal Citation Reports—Social Sciences Edition (ISI).

Data Source(s): Citation studies; Web search engines; citation indexes. Internet search engines such as Google can provide partial information on the number of times a publication is cited online. Citation reports are costly but easy to obtain.

Purpose and Issues: This indicator is a collective measure of the perceived authority and quality of a scientific publication in the research community. The more useful a publication is to those who publish other scientific articles, the more it is cited.

A limitation of indicators based on citation counts is that they do not apply to all types of information products but only to published scientific literature, where influence in the scientific community is a goal and a sign of success. For many information products (e.g., a database, a curriculum), influence in the scientific community is not a goal. Citation counts would not measure whether such a product had achieved its purposes.

Even when influence in the scientific community is a goal, authors in developing countries often face the well-known biases and other limitations that make it difficult for them to make their work known to others in the scientific community. A related limitation is that many relevant journals published in developing countries are not included in some widely used databases such as MEDLINE.

Example:

As of 2004 MSH’s Managing Drug Supply (1997) had been cited 10 times in the following publications: Annals of Tropical Medicine and Parasitology, Health Policy, Health Policy and Planning, Journal of Clinical Epidemiology, Salud Pública de México, and Social Science and Medicine.

Indicator 20:

Journal impact factor

Definition: “The journal impact factor is a measure of the frequency with which the ‘average article’ in a journal has been cited in a particular year. The impact factor will help you evaluate a journal’s relative importance, especially when you compare it to others in the same field (Institute for Science Information, 2000).”

Data Requirements: “The impact factor is calculated by dividing the number of current citations to articles published in the two previous years by the total number of articles published in the two previous years (Institute for Science Information, 2000).”

Data Source(s): Journal Citation Reports

Purpose and Issues: This indicator is an objective measure of the impact of a journal among writers of scientific articles, not that of a particular article in a journal. The acceptance and publication of an article in a journal with a high impact factor, however, is an indicator of the high quality of that article. In 2003, for example, the American Journal of Public Health had an impact factor of 3.363 (López-Abente and Muñoz-Tinoco, 2005), compared with an impact factor in 2004 of 38.570 for the New England Journal of Medicine and 21.713 for The Lancet (Eugene, 2005).

This indicator, calculated by Journal Citation Reports, needs to be used carefully, comparing journals with similar intended audiences. It can be used to compare journals within a field, such as public health. For example in the field of family planning, Perspectives on Sexual and Reproductive Health has an impact factor of 3.417 whereas Studies in Family Planning has an impact factor of 1.311. A large circulation journal that covers a wide variety of topics, such as the New England Journal of Medicine, may have a high impact factor. That does not mean, however, that it reaches reproductive health program decision-makers in developing countries, for example, better than a highly focused publication reaching a small but well-targeted audience.

The impact factor calculation is necessarily based on review of citations in a defined list of major publications. These may not be the publications of the most importance to intended audiences or the most accessible to them.

Indicator 21:

Number/percentage of users who pay for a product or service

Definition: Payment is the exchange of money for a product.

Data Requirements: Number and type of users who pay for products.

Data Source(s): Sales records from the publisher or redistributors; sales receipts, feedback forms or user surveys distributed with the product or after a product has been disseminated.

Purpose and Issues: Many donor-funded projects disseminate information free upon request to users in developing countries or make it available to download from the Internet at no charge. A practical assessment of utility is determining what percentage of users do pay, will or would be willing to pay for a product and/or how much users do or would pay. Willingness to pay indicates that the product is of perceived utility and value. Of course, pricing affects willingness to pay but is not considered here, except to note that it may be appropriate to charge only the cost of production and/or shipping or to set prices to recover the unit cost of the product or to realize a profit so the funds can be used, for example, for reprinting.

Example:

UNC School of Public Health purchases 35 copies each semester of Qualitative Methods for Public Health—written by authors from Family Health International and published by Jossey-Bass—for graduate students in health behavior and health education courses.

Indicator 22:

Number/percentage of informaton products or services guided by theories of behavior change and communication

Definition: Theory helps to guide the creation of effective health messages. Behavior change and communication theory is used to design persuasive messages that will motivate audience members to adopt new behavior.

Data Requirements: Key messages and objectives of information products or services, behavior change or communication theory used.

Data Source(s): Information products and services, project records.

Purpose and Issues: Information products and services are intended not only to inform users about particular health issues, but also to persuade audience members to adopt new behaviors, whether personal behaviors that promote one’s own health or professional behavior that promotes the public health. By using behavior change and communication theory, producers of information products and services can help ensure that they develop persuasive messages. Different types of behavior change and communication theories can inform the development of information products and services. Stage or step theories center on a set of phases that an individual passes through toward behavior change while models of behavioral prediction are centered on cognitive, emotional, and social factors that determine behavior performance or nonperformance (Fishbein et al., 2000; Kincaid, 2000). A tool that shows how writers can apply diffusion of innovations theory to their work can be found in Appendix 5.

Diffusion of innovations is one example of a stage theory (Rogers, 2003). It describes two sets of processes: (1) dissemination of an innovation within a social system (what, where, and to whom) and (2) a decision process where individuals learn about, evaluate, decide, and adopt an innovation (how and why) (Kincaid, 2000). By addressing the attributes of an innovation, producers of information products and services can enhance the likelihood that an individual will try a new behavior. Information products, whether intended for health professionals or consumers, can be designed to incorporate these attributes:

· Relative advantage—demonstrating the benefits of certain practices or approaches over others—particularly of new practices over those of current practices;

· Compatibility—relating emerging policy and program and research practices to current practices;

· Complexity—providing clear steps for application;

· Observability—providing examples and models and facilitating the exchange of experiences among health professionals; and

· Trialability—suggesting easy ways to try these new practices (Rogers, 2000).

Other types of theory focus on a set of cognitive, emotional, and social factors that determine behavioral performance or nonperformance, rather than a set of stages or steps that individuals pass through as they try to change their behavior (Fishbein et al., 2000) . In 1991, scholars agreed upon a list of eight variables or factors that are the best determinants of behavior at a workshop sponsored by the National Institute of Mental Health (Fishbein et al., 2000). At the meeting the group identified a total of eight variables: three direct causes of behavior (intention, skill, environmental constraints) and five indirect causes (attitude, norms, self-standards (image), emotion, and self- efficacy. The latter indirect causes are expected to influence behavioral intentions, which in turn can affect behavior (Fishbein et al., 2000; Kincaid, 2000).

To facilitate behavior change, those developing information products and services can produce products and services that provide instruction, build skills, and promote new behaviors by increasing knowledge, altering attitudes, increasing self-efficacy, and promoting social interaction. Theories that address the eight variables discussed above include cognitive theories (e.g., Theory of Reasoned Action/Theory of Planned Behavior (Ajzen & Fishbein, 1975), Social Cognitive/Learning Theory (Bandura, 1977, 1986)), social process theories, and emotional response theories (Piotrow et al., 1997).

Content analysis is one method that can be used to evaluate the extent to which behavior change and communication theory has guided creation of an information resource. Content analysis is a term that describes a variety of approaches for drawing conclusions from communication resources (text, video, audio)(Bernard, 1995).