HIPNET - Health Information and Publications Network

Part II: Indicators That Measure Reach

This section describes indicators that measure the reach of information products. “Reach” is defined as the breadth and saturation of dissemination, distribution, or referral of the product in both hard copy and/or electronic forms. The purpose of measuring reach is to quantify dissemination so as to inform the planning, promotion, and budgeting of current and future information products and services, and to improve management of product development and printing. Measuring reach also can provide valuable information on the extent to which products get into the hands of intended audiences.

Indicators for reach are grouped into three main areas:

(1)   Primary distribution of the product by the publisher,

(2)   Secondary distribution as a result of user-initiated requests or downloads, and

(3)   Dissemination through referrals, such as Web site links, workshops, or word of mouth.

Primary distribution refers to distribution directly from the publisher to the end user or to a secondary distributor. The end user could be a person or organization on a mailing list or a subscriber to a specific information product—for example, a quarterly magazine. A subscriber may pay for the product or may receive the product at no cost.

Secondary distribution is user-initiated distribution that occurs after the initial mass mailing or listserv announcement. (A listserv is an email mailing or discussion list.) In secondary distribution the user requests or downloads the material.

Primary and secondary distribution relate to the Push-Pull Model for information in the health field. The Push-Pull Model represents the contrast between organizations investing resources in assumed areas of need—push—and populations requesting certain products or services—pull. Indicators under primary distribution refer to the “push” or supply side of the information market. Indicators listed under secondary distribution refer to the “pull,” or the demand side of the information market. There can be a slight overlap in “push” and “pull,” as a user may initially request to be placed on a mailing list, suggesting more of a “pull” than a “push.” After that initial “pull,” however, future unsolicited distribution is considered “push.” In addition, the result of publicity for publications is typically an increase in “pull,” or secondary distribution.

Indicators in the third area of reach—referral—relate to various means through which people can find their way to information resources through Web sites, databases, libraries, word of mouth, and other channels.

Unlike the other indicators under “referral,” the indicator “Number of instances that products are indexed or archived in bibliographic databases” relies on bibliometric research. Bibliometric data, as defined in Journal Citation Reports, are “quantifiable statistical data that provide a systematic, objective way to evaluate products and services and their impact and influence in the global research community.”  One common way of conducting bibliometric research in the field of public health is to use PubMed or similar archives or indices to trace citations. Another area of bibliometric research uses citation analysis to examine linkages among authors through their works. Bibliometric research enables evaluators to determine the impact of a specific study or author on other research publications. In a sense, bibliometric data reflect, in a single measurement, both the reach of a publication in the scholarly or research community and the perceived value and relevance of its content.

Primary distribution refers to distribution directly from the publisher to the end user or to a secondary distributor.

Definition: This indicator refers to the first mass mailing to one or more existing lists (mailed to subscribers or to lists assembled for other purposes (e.g., trainees) or emailing to listservs, newsgroups, or individuals).

Data Requirement: Numbers of copies sent or numbers of recipients on mailing list, subscriber list, listserv subscribers, characteristics of subscribers (e.g., country of origin, type of organization, job type).

Data Source(s): Mailing list, subscriber list, listserv, class registration lists, conference or meeting attendee lists.

Purpose and Issues: This is one of the most basic indicators for measuring dissemination. While organizations continue to disseminate hard copies of products to institutional mailing or subscription lists, many organizations now enhance the scope of dissemination through electronic communication. For example, many organizations also use electronic forums to publicize or announce the availability of new products and to send electronic versions or Web links to potential users. Listservs present a challenge to evaluation because it may not be possible to track the number of subscribers for a given listserv if the organization does not moderate the listserv.

Definition: This indicator refers to distribution that occurs after the initial mass mailing. It is initiated by the publishing organization. For example, an organization may distribute selected products while carrying out technical assistance or e-learning activities, or at workshops, conferences, training sessions, and the annual meetings of professional associations.

Data Requirements: Number of copies, venue, and date distributed.

Data Source(s): Administrative records.

Purpose and Issues: This kind of secondary distribution (dissemination from the supply side) is important to track because it relates directly to the achievement of the goals of technical assistance, including capacity building. This indicator does not measure demand from users, but it does help track distribution that occurs in tandem with other, related events, which could have the effect of increasing the chances that the information will be understood and applied. This could also help build demand for the product.

Secondary distribution is further dissemination of products by the original publisher that may be either solicited or unsolicited.  As an example of solicited requests, a note distributed to colleagues might request them to order products as needed. These orders would be specifically requested by the users.  Similarly, finding a document when browsing the Web or conducting a database search and then downloading it is “pulling” information directly or indirectly from the publisher.

Definition: “Orders” include any user-initiated method of requesting copies of a product, including telephone calls, email, mailing of an order form, online order form, requests to be placed on a mailing list, or asking a colleague from the publishing organization in person.

Data Requirements: Number of phone orders, emails, order forms, interpersonal requests.

Data Source(s): Administrative records; Web statistics; user surveys or questionnaires.

Purpose and Issues: Secondary distribution is an important component of reach because it measures demand and thus indirectly gauges perceived value, appropriateness, and quality. After primary distribution, requests result from learning about a product from the Internet, a library, a colleague, a flier, a listserv announcement, news release, review, index, bibliography, class reading list, etc. Recommendation of a colleague—“word of mouth”—is an effective form of publicity, especially when an organization can provide products to local, national, or international opinion leaders who can help further the reach of the product. This is a relatively easy indicator on which to collect information, at a low cost.

In some instances underreporting is an issue for this indicator, since secondary distribution cannot be tracked in its entirety. For example, it is not possible to count how many copies of the article, which was published in an open-access journal, are circulated among initial readers’ colleagues.

Definition: “File downloads” refers to an Internet user’s transfer of content from a Web site to her or his own electronic storage medium.

Data Requirements: Web server log files.

Data Source(s): Web server log analysis software (e.g., WebTrends®, Google™ Analytics, IBM® Surfaid, HitBox, StatCounter, ClickTracks, and NetGenius).

Purpose and Issues: File downloads captured in a Web server log file provide insight into the types of information products and topics of products that are used the most among those offered on a Web site.  (A log file is a list of actions that have taken place.) Web site usage programs can also be used to get a general sense of which countries or regions and organizations are making most use of a Web site (see Appendix 4). When linked to special events, these statistics also can help pinpoint which promotional events reach online audiences most interested in a topic: Web site activity will increase after a promotional event.

The Web server normally produces log files automatically, so the raw data are readily available. The Web server reliably records every transaction it makes. Furthermore, the data are on the organization’s own servers. It is important to use caution, however, because Web site activity can also change due to unrelated events such as Web site upgrades, broken links, offline servers, and other problems. Also, log files contain information on visits from search engine spiders as well as from human users of the Internet. (A spider is code that automatically searches Web pages and fetches links to those pages.) Although visits by search engine spiders should not be reported as part of the human activity at the Web site, they are important information for search engine optimization of a Web site.

Switching log analysis software, as programs sometimes do for various reasons, limits the ability to assess trends, because different programs collect different information or define common terms in different ways. To address this problem, it may be feasible to use both old and new programs for a time and to compare the results, and/or to analyze historical data with the new program.  However, while log file analysis is easy to conduct, it is often difficult to interpret (see Appendix 4).

Disclaimer: The information provided on this web site is not official U.S. Government information and does not represent the views or positions of the U.S. Agency for International Development or the U.S. Government.