2015-2018 Evaluation of the California Black Infant Health Program

The Maternal, Child, and Adolescent Health Division (MCAH) supports the development, implementation and evaluation of the Black Infant Health (BIH) Program.  

The 2015-2018 BIH Program evaluation covered three state fiscal years (July 1, 2015 – June 30, 2018) and focused primarily on implementation of the prenatal group-based model of the BIH Program across 17 sites in 15 local health jurisdictions (LHJs) throughout California. MCAH evaluated the BIH model released in 2015 through a Request for Supplemental Information (RSI).

The evaluation focused on determining if the prenatal group-based model was implemented as intended (see BIH Group Based Model Core Elements) and what impacts on participants' intermediate health outcomes occurred after participation in the Program. Intermediate health outcomes are the short-term health milestones that indicate positive movement toward the long-term goal of healthy birth outcomes¹ for Black people.

The evaluation used a mixed-methods approach of qualitative and quantitative data collection.

• Information about staff and program context came from quarterly and annual reports produced by LHJs, staff surveys, staff group reflections and staff questionnaires.
• Information about how the program was implemented came from data collection forms on group and other service delivery provided by LHJ staff.
• Information about participants came from baseline and follow-up assessments, participant group reflections and post-program satisfaction surveys.
• Comparative population data was included from California's Maternal and Infant Health Assessment (MIHA)² survey and Birth Statistical Master Files³.
• A stakeholder engagement process was undertaken to increase the transparency and validity of the report findings and conclusions.

Navigate the sections below to learn more about the evaluation results and how the evaluation was conducted.

• Results of the BIH Evaluation: Data Briefs
• BIH Program Evaluation Questions and Methods Overview
• Black Infant Health Program Sites and Years of Data Included in the Evaluation
• References
  

Results of the BIH Evaluation: Data Briefs

Intermediate Outcomes Among Prenatal Group Model Participants:  Brief Summary

Read more

• The BIH prenatal group model is achieving its intended intermediate outcomes. Data for a sample of participants that attended an average of 7.2 groups and 5.1 life planning meetings showed significant positive change in 13 of the 18 intermediate health and health-related outcomes examined.
  

• Five indicators related to healthy behaviors were analyzed but did not show change after prenatal group model participation or showed negative change. Further analysis is needed to better understand this result.
  

• Results support BIH as a promising strategy to improve the health of Black birthing people and their families.
  

Services Received and Services Provided During Prenatal Group: Brief Summary

Read more

• Most participants that enrolled in BIH started prenatal group and life planning services attended on average 5.9 out 10 prenatal group sessions (7 are recommended) and 3.3 life planning meetings (4 are recommended) throughout six months of program engagement.
• The Program delivered services mostly as intended. Two trained facilitators delivered most group sessions, with more than 70% of sessions delivered without skipped or modified activities and more than 80% providing the required food during sessions.
• The average size of groups (4.8) was below the recommended (5) and ideal sizes (8-12).

Participant and Staff Perceptions about the Program: Brief Summary

Read more

• Most participants that filled out a satisfaction survey had a positive experience with the Program and most of them would refer others to BIH.
• Both direct service staff and managers appeared to have positive perceptions about their work and work environments, with a few noteworthy exceptions related to available resources and work-related stress.
• Feedback from both participants and staff reflects that their perception is that the Program is achieving its intermediate health outcomes.

Participants Served

Read more

• The BIH Program enrolled a unique subset of the eligible population during the three years examined. These findings could inform program improvements like outreach efforts and structure.
  

• BIH participants appeared more exposed to harmful experiences and faced greater obstacles to health (for example, higher proportion living in neighborhoods with concentrated poverty) than did the eligible population of Black women and birthing people in the same California jurisdictions.

• BIH participants also exhibited key protective factors, including utilization of WIC, low pre-pregnancy alcohol use, and long interpregnancy intervals.

Contextual Conditions that Supported the Implementation of the Prenatal Group Model

Read more

• Sites varied in their abilities to implement the prenatal group model as intended. The contextual conditions sites operated in, as measured by the presence of ideal conditions, also varied. The findings presented in this brief show which contextual conditions were related to more successful program delivery.

• Several ideal contextual conditions were associated with more than one indicator of program implementation success, highlighting the importance of supporting these conditions:

• Local Program Leadership that values BIH and provides supplemental local monetary support;

• Positive staffing practices (e.g., brief vacancies, cultural competency);

• Provision of participant motivators, such as gift cards, door-to-door transportation, and full meals during group meetings;

• Presence of a Community Advisory Board; and

• Staff that believes the Program has value for participants and has positive perceptions of their work environment. Continuous organizational efforts to improve the identified contextual conditions at local sites should lead to improvement in implementation of the BIH prenatal group model.

BIH Program Evaluation Questions and Methods Overview

Below describes the five main overarching evaluation questions that were explored, the sources of data used, samples sizes, and a brief description of the methodological approaches used and their limitations.The results of the evaluation are included in companion data briefs. 

1. How do BIH participants compare to the total eligible population of Black birthing individuals?

   Data Sources

   • BIH information: participation recruitment and baseline assessment
   • Eligible population: California's Maternal and Infant Health Assessment (MIHA)² survey and Birth Statistical Master File (BSMF)³.

   Sample

   N = 3495 first-time enrollees with a baseline assessment

   Methods and Limitations

   • Comparison between BIH enrollees and all eligible Black birthing people in BIH counties using descriptive statistics (means, percentages) with 95% confidence intervals.
   • Limitations: (1) Eligible population may also have been enrolled in BIH during the period of interest (2) MIHA data are collected after birth while BIH data is collected at enrollment (prenatally up to 30 weeks gestation).
     
2. What services did BIH participants receive during the prenatal period? What services did the BIH Program provide for participants?

   Data Sources

   • BIH participant services: referrals, life planning log, short-term goal and goal update, baseline assessment, safety checklist, birth plan
   • BIH group services: group session information, group observation form.

   Sample

   • N=3332 participants who attended at least one prenatal group session or prenatal life planning meeting
   • 386 series that included 3347 prenatal group sessions

   Methods and Limitations

   • Quantitative information on statewide delivery and participation in prenatal group sessions, life planning and other services (e.g., review of home safety checklist, discussion of birth plan) and staff survey data summarized using descriptive statistics (percentages, means, standard deviations).

   • All data reported were collected by local health jurisdictions implementing BIH and submitted to a web-based data system using pre-defined data collection forms. Data were extracted on December 31, 2018 to allow all participants in the sample at least six months to receive prenatal services.   

   • Limitations: (1) We examined data for the project as a whole and did not examine trends over time. (2) Some implementation characteristics that were not measured, most notably the quality of the relationships between staff and participants and among participants, which local partners, and other research, suggest may be key to driving Program impacts.
3. What were participants’ perceptions of the Program? What were staff members’ experiences with and perceptions of the Program?

   Data Sources

   • BIH participant services: satisfaction survey.
   • Staff survey (October-November 2018).

   Sample

   • N=745 participants who filled out a satisfaction survey
   • N=94 BIH Program staff that filled out an online survey.

   Methods and Limitations

   • Qualitative information about participant experiences were analyzed for common themes.
   • Summary quantitative information on staff perceptions of their work environment and the effectiveness of the BIH program.
4. How well are sites implementing the prenatal group-based model and what aspects of implementation are supportive of sites' successful implementation of prenatal group sessions?

   Data Sources

   • Multiple qualitative and quantitative sources including site profiles and annual budgets, site call logs, quarterly and annual reports, staff-surveys and geocoded data on participant and program locations.
   • BIH fidelity measures related to the prenatal group-based model.

   Sample

   Administrative data for 16 BIH sites (July 1, 2017 through December 1, 2018)
   

   Methods and Limitations

   • The level of adherence (or fidelity) to the program policies for the implementation of the prenatal group component of BIH (‘higher,’ ‘intermediate,’ or ‘lower’ fidelity), was examined for four performance indicators and for each site: timely group initiation, minimum group size, group dose and enrollment.
   • The presence or absence of 30 “ideal” contextual conditions that may facilitate implementation was examined for each site.
   • An exploratory subset/superset comparative analysis ^4,5 approach was used to determined which contextual conditions or combinations of conditions were supportive of higher ratings for each of the four fidelity performance indicators of interest. Conditions or combinations of conditions were identified as supportive if they had a coverage score ≥0.75 and a consistency score ≥0.40.
   • These analyses identified conditions (or combinations of conditions) that reliably link to higher implementation fidelity, which may be one of multiple possible pathways to higher fidelity.
   • Limitations: (1) The study included only 18 months of data and the findings may not reflect more recent and overall experiences. (2) The comparability of data across sites was likely affected by variability in both the completeness and quality of the data and in the staffing roles and perspectives of those who provided information. (3) A limited subset of four fidelity measures was selected for these analyses related to the prenatal group model only, and no other BIH components (life planning). (4) The criteria used to differentiate “ideal” from “other than ideal” conditions were developed without input from local staff; however, their feedback was obtained when interpreting the results. (5) Distribution-based cut-offs to define "higher" fidelity were chosen to facilitate the comparative analyses approach used here. This means that the analyses in this chapter focused on relative ('higher,' 'intermediate,' and 'lower') vs. absolute ('high,' 'intermediate,' and 'low') fidelity performance
5. What intermediate health outcomes are participants experiencing after participation in BIH's prenatal group-based model?

   Data Sources

   • BIH participant information: baseline and follow-up assessments; life-planning log.
   • Other participant level data such as number of group sessions attended, number of life planning sessions attended, and average size of groups attended.

   Sample

   N = 1571 first-time enrollees with a baseline assessment and follow-up assessment who participated in at least one prenatal group session

   Methods and Limitations

   • Change in intermediate participants health outcomes after program participation were examined after adjusting for variations by site, participant baseline characteristics, and timing of assessments (using Generalized Linear Mixed Models⁶).  A total of 18 outcomes were analyzed, including social support, empowerment, use of stress management techniques, health knowledge and healthy behaviors, and connection with services.
   • Limitations: (1) this sample was restricted to participants who completed a baseline and follow-up assessment. They more closely resembled the average BIH participant than the average BIH-eligible woman though they attended a higher number of sessions than the average BIH participant. Thus, although we adjusted all analyses to control for baseline characteristics, without a comparison sample we are unable to say how much of the differences observed are a result of preexisting and/or unmeasured differences between participants who attended and those who did not. (2) The evaluation was not designed to test variations in implementation and as such the results must be interpreted with caution.