ACKNOWLEDGEMENTS	iii
FOREWORD	viii
PREFACE	ix
PART I: OVERVIEW ...................................................................................................	1
Rationale and Objectives of the Guide............................................................................	1
Intended Audience...........................................................................................................	2
Development of the Guide...............................................................................................	2
Organization of the Guide ..............................................................................................	1
Framework for Monitoring and Evaluating Health Information Products and Services..............................................................................................................................	4
Use of Quantitative and Qualitative Data ………………………………………………..	9
Relative Costs of Data Collection Methods ………………………………………………	10
Summary List of Indicators ………………………………………………………………...	13
PART II: INDICATORS THAT MEASURE REACH.............................................	14
Area 1: Primary Distribution (Push)…………………….…………………………………	15
Area 2: Secondary Distribution (Pull).…………………….…………………………...….	16
Area 3: Referrals..…………………………………………….……………………………...	20
PART III: INDICATORS THAT MEASURE USEFULNESS...............................	24
Area 1: User Satisfaction..................................................................................................	25
Area 2: Product or Service Quality .................................................................................	29
PART IV: INDICATORS THAT MEASURE USE..................................................	35
PART V: INDICATORS THAT MEASURE COLLABORATION AND CAPACITY BUILDING...............................................................................................	41
Area 1: Collaboration.......................................................................................................	41
Area 2: Capacity Building................................................................................................	43
REFERENCES..................................................................................................................	45
APPENDICES..................................................................................................................	47

                                            Acronyms                                                                 

ADRA AED AGI AIDS AIM AOL ASRH BIOSIS CA CCP CD-ROM CHBC COPE CORHA DEC DVD EMBASE FDA FGC FGM FHI FP FS GH   HC HIPNET HIV HTML IEC INFO IR ISI ISP IT IUCD IUD JHU JSI M&E MOH MOU MSH NACO NGO NHS OPRH   OR PAC PAIS PATH PDF PLHA PRB PRH   PVO RH STI TA UNC URC URL US USAID USAID/W WHO

Adventist Development and Relief Agency Academy for Educational Development The Alan Guttmacher Institute Acquired Immune Deficiency Syndrome AIDS Impact Model America On Line Adolescent Sexual and Reproductive Health Biosciences Information Service Cooperating Agency Center for Communication Programs Compact Disc Read-Only memory Community Home-Based Care Client-Oriented Provider Efficient Services Consortium of Reproductive Health Associations (Ethiopia) Development Experience Clearinghouse Digital Video Disc Excerpta Medica Database Food and Drug Administration (USA) Female Genital Cutting Female Genital Mutilation Family Health International Family Planning Field Support United States Agency for International Development Bureau for Global Health Host Country Health Information and Publications Network Human Immunodeficiency Virus Hypertext Markup Language Information, Education, and Communication Information and Knowledge for Optimal Health Project Intermediate Result Institute for Scientific Information Internet Service Provider Information Technology Intrauterine Contraceptive Device Intrauterine Device Johns Hopkins University John Snow, Inc. Monitoring and Evaluation Ministry of Health Memorandum of Understanding Management Sciences for Health National AIDS Control Organization (India) Non-Governmental Organization National Health Service (UK) United States Agency for International Development Office of Population and Reproductive Health Operations Research Postabortion Care Public Affairs Information Service Program for Appropriate Technology in Health Portable Document Format People Living with HIV/AIDS Population Reference Bureau United States Agency for International Development Office of Population and Reproductive Health Private Voluntary Organization Reproductive Health Sexually Transmitted Infection Technical Assistance University of North Carolina University Research Co., LLC Uniform Resource Locator United States United States Agency for International Development United States Agency for International Development/Washington World Health Organization

Foreword

As demand for health resources in developing countries increases, it is imperative that high-quality information products and services flow to health care practitioners and policymakers responsible for improving public health.

What works?  What doesn’t work?  What looks promising?  Can these efforts be replicated elsewhere?  How do you begin?  How do you evaluate?  Where does the emerging evidence lead?  What is sustainable?  Health information products and services must continue to communicate the answers to these and other such questions to audiences throughout the health sector.  A successful health promotion intervention for women in Burkina Faso, for example, might well have a great local impact, but an effective, timely, and accurate account of what makes it work communicated to the right people, in the right way, at the right time will ensure that its benefits spread well beyond Burkina Faso. 

Well-crafted information products and services help assure that project implementation and lessons learned reach others trying to achieve similar results (efficient replication), that wasteful duplication is avoided, and that the next generation of health care providers has a comprehensive knowledge base on which to draw.

While effective face-to-face technical assistance by an experienced consultant often culminates in a successful activity, such consultancies are costly and by their nature narrow in scope.  We must strive to make sure that the results of such efforts are multiplied and expanded through clear and concise information products and services—handbooks, guides, case studies, manuals, distance education courses, CD-ROMs, databases, Web sites, journals, and reports.  As new information technologies come online and more and more developing countries have access to them, the danger of information overload is apparent.   At the same time, once implemented, new technologies can help users access the exact information they are after with a minimum of time and expense.

For these reasons it is crucial that information products and services remain of the highest quality and reach their audiences in the most effective manner.  The Guide to Monitoring and Evaluating Health Information Products and Services provides publishers, knowledge managers, program managers, M&E specialists, and health information communicators with a standardized way to evaluate whether their print or electronic products and services meet the requirements needed to make them effective, used, and adapted by health care practitioners and policymakers in the field. 

The 29 indicators in the Guide measure the reach, usefulness, and use, as well as the collaboration, and capacity building engendered through information products and services.  The innovative “Conceptual Framework for Monitoring and Evaluating Health Information Products and Services” shows how they contribute to the initial, intermediate, and long-term outcomes of health development efforts—something which perhaps we all instinctively recognize but have failed to track in the past.   Such a track record will go a long way to making information products and services an integral part of future global health development efforts.

What makes this guide special is that it brings together knowledge about monitoring and evaluating information products and services from dozens of health organizations—all members of the HIPNET community of practice.  USAID thanks the many health professionals and organizations who contributed to the Guide and hopes this collective effort will help health care providers and policymakers to better access relevant, accurate, and vital information in a user-friendly and efficient manner.  We urge all those in a position to do so to make use of the Guide and to monitor and evaluate their information products and services.  We also invite users of the Guide to provide feedback so that future updates will benefit from their experience. 

Scott Radloff

Director, Office of Population and Reproductive Health

Bureau for Global Health, USAID

Preface

This guide results from a collaborative process involving members of HIPNET (Health Information and Publications Network ) with expertise and interest in monitoring and evaluation (M&E) of information products and services. HIPNET is a working group that facilitates collaboration among organizations that provide information products and services in the field of international health. HIPNET member organizations address the key need for technical information on health, delivered through appropriate technologies that strengthen health care programs around the world.

Over the years members of HIPNET have discussed and exchanged ideas on how best to monitor and evaluate information products and services. Thus HIPNET members recognized the value of a resource that would describe key indicators and discuss methodological issues related to M&E of information products and services. As with other components of health services, it is important that those who produce and disseminate health-related information agree on how best to measure and demonstrate the effect of their work.

The Guide to Monitoring and Evaluating Health Information Products and Services has been produced to meet this need. The set of indicators presented in the Guide reflect the indicators that HIPNET members have used to assess the effect of their products and services. We do not expect that users of this guide will employ all of the indicators presented; rather, users can draw from the “menu” of indicators to apply those most relevant and appropriate to their work. Still, it is important to select indicators from every part of the conceptual framework, including indicators of inputs, processes, outputs, and outcomes (see p. 5). By doing so, evaluators can trace the sequence of results and identify areas of weakness and strength.