Comparative Evaluation of Service Delivery Process & Outcomes Between CommCare Mobile App & Paper-based Household Nutrition Counseling Platforms - Final Report

File Type: PDF | File Size: 2.98 MB

The overall objective of the USAID-funded Zambia Mawa CommCare evaluation is to contribute to the existing body of knowledge on the effectiveness of the CommCare counseling platform as a key resource for sustainable household nutrition behavior change. The Mawa CommCare evaluation includes a comparison of paper-based versus CommCare nutrition counseling job aides to be used during nutrition volunteer household visits to caregivers of children under two (caregivers) and pregnant and lactating women (PLW). The evaluation explores both process (front line worker/service provider-focused) and outcome (program participant/client-focused) indicators associated with the nutrition counseling service.

The research team used a mix of quantitative and qualitative methodologies to conduct the baseline and end line evaluations of the Mawa CommCare nutrition mobile application. Combined samples for baseline and end line studies included household surveys for caregiver/pregnant and lactating women (referred to as “clients”; n=1,181) as well as for nutrition volunteers (n=396); key informant interviews of project staff (nutrition volunteers and base and end line as well as with health promoters at baseline; n=57); focus group discussions among program clients to explore some of the potential causes of baseline findings (n=4); and observations of NVs conducting counseling sessions (n=228), followed by exit interviews with counseling clients (n=228).

Qualitative, observation research results suggest that front line workers (nutrition volunteers, referred to as “NVs”) using the CommCare counseling job aide tended to perform slightly better than their paper-based counterparts against metrics of counseling quality as defined within Mawa’s nutrition counseling methodology (“ASPIRE,” referring to the counseling steps of ask, show, probe, inform, request and examine). That said, both NV groups saw a decline in overall performance at end line and both groups reported regular supplemental use of paper-based versions of the nutrition lessons to effectively counsel their clients. Regardless of primary counseling platform, end line evaluation results demonstrate that over time, NVs have become better at using nutrition counseling lessons and tools as intended, to more flexibly support client counseling and behavior change support needs for adopting essential nutrition and hygiene actions (ENHA) as they are explored during counseling sessions.

In the analysis of infant dietary diversity and meal frequency results at end line, 39% of children aged 6- 23 months (in paper-based and CommCare groups) received a minimum acceptable diet (MAD). Children of clients in camps counseled with the paper-based job aide were statistically more likely to have achieved MAD than their CommCare camp counterparts. Likewise, MAD among Mawa’s clients who participated in this study compares favorably against the MAD prevalence of 9.8% reported for eastern province children 6-23 months in the 2013-2014 Zambia DHS (MAD for children across the country was 11% during this same time period).

Comparative analysis of data suggests that time, rather than counseling platform, was a more likely determinant of knowledge and attitude outcomes among both nutrition counseling clients and NVs.

Respondents participating in Mawa for more than one year—whether as clients or as service providers—tended to have more positive outcomes that support sustained use of ENHA among clients and effective change agency among NVs. Between time points, overall composite scoring of knowledge, attitudes, and practices questions demonstrated that no statistically significant difference was detected between clients counseled with either platform, other than the higher likelihood of clients counseled with the paper-based job aide to engage in ENHA at end line.

Findings also show that neither sex of NVs nor age of clients affect the knowledge, attitudes, or practices of clients. That said, social norms regarding gender roles do influence the various types of household decision-making to which wives and husbands have access, either separately or jointly (e.g., nutrition practices, use of household resources, or child care practices including during a child’s illness).

While the findings of this study do not show that the CommCare counseling platform used in Mawa resulted in overall better outcomes for clients or providers, findings from other contexts may prove differently. There may also be programmatic operational value to be gained through similar, mobile device enabled tools that make the investment worthwhile from the implementers point of view (e.g., the need to make real-time contributions to national health information systems); to that end projects may wish to explore such options, including open source applications.

This study contributes to the evidence of promising practices in intervention design and continuous program improvement for home-based nutrition counseling. Projects that plan to use home-based nutrition counseling should consider the following practices referenced within this study: early initiation of a counseling approach such as ASPIRE and socializing the value of the change agent role played by the counselor whether staff or volunteer; establishing a strong supportive supervision improvement process to support quality counseling service delivery; and using periodic qualitative data collection efforts to better understand trends in incremental client outcomes.

Lastly, future program research teams may wish to further examine this data set to build upon end line findings, including: determining if any differences existed in the socio-demographic characteristics of clients at baseline compared to end line (e.g., age or education level of client, number of children, etc.) and assessing correlations with reported knowledge, attitude or practice levels; analyzing the decision-making domains (e.g., household income for food purchases, crop production, child care during illness, etc.) by clients’ experiences of sole versus couples/joint decision-making; analyzing client meal frequency by breastfeeding status by camp at baseline and end line; and determining if any differences existed in NV knowledge scores (at base and end line) when analyzed by NV mean age or other demographic characteristics.


Publication Date 
September 2017
Resource Type 
Project Reports
Regions/ Countries