DEE WILSON CONSULTING

(Originally published February 2023)

​

By Dee Wilson and Toni Sebastian

 

Last month’s commentary discussed the controversy over use of an algorithmic tool (Allegheny County Family Screening Tool (AFST) to screen General Protective Services (GPS) reports in Allegheny County, Pennsylvania. We summarized the findings of an independent study of the use of AFST-1 conducted by researchers from Stanford which found modest initial positive effects on “screening accuracy,” effects that decreased over time. We questioned the logic of evaluating screening decisions based on re-report data that might have been affected by factors other than information contained in GPS reports, e.g., by the motivation and attitudes of mandated reporters.   

 

We were subsequently informed by one of the model’s developers, Rhema Vaithianathan, that AFST was developed to

predict foster care placement, not screening accuracy as defined by the Stanford researchers.  However, it is clear from two evaluations of AFST, the study we discussed and an implementation study by Hornby Zeller Associates, that AFST   was utilized to both predict re-reports and foster care placements. The Stanford researchers then utilized re-report data to assess “screening accuracy” in a way the model’s developers may not have endorsed.  

 

We also learned that investigative caseworkers in Allegheny County were not informed of the AFST score and so could not have been influenced by the score, as we speculated, to offer or provide services to families, though they may have been influenced by factors other than families’ needs or risk of child maltreatment to do so.

 

We concluded that the main effect of AFST was to reduce Allegheny County’s GPS screen out rate from 60% to 47% in 2021 - the average GPS screen out rate in Pennsylvania. It’s plausible that screening in more high-risk reports and fewer low risk reports (as determined by AFST) led to a large increase in GPS families opened for services, though in the absence of an evaluation of services provided to families with open GPS cases this is speculation. It is an odd feature of risk assessment research that studies rarely describe the services to which “high risk” families are linked or the efficacy of those services. This has been true in studies of AFST as well.            

 

Given the goal of the developers to predict foster care placements (which they view as a proxy for serious child maltreatment), it is curious that there has not been a study that we have found of the effects of AFST on entry-into-care rates for GPS reports with the highest algorithmic scores, i.e., 18-20, in Allegheny County. However, we found data indicating that there was a 20% reduction in first entries-into-foster care from 2016-20, the years when AFST was first implemented. The expectation of some critics that use of a predictive tool at intake would be used to justify a large increase in child removal turned out to be greatly exaggerated, to put it mildly. 

 

The most important and potentially useful study of AFST was published in JAMA Pediatrics: “Hospital Injury Encounters of Children Identified by a Predictive Risk Model for Screening Child Maltreatment Referrals,” (Vathianathan, et al, 2020). This study tracked hospital “encounters,” i.e., both emergency room visits and hospitalizations, for 47,305 children with GPS referrals in Allegheny County from April 1, 2010, to May 4, 2016. Rates of any-cause injury and abusive injury were compared for children with an AFST score of 20, i.e., the highest risk 5% of reports, vs. children with AFST scores of 1-10, i.e., the lowest risk 50% of reports.

 

The authors state: “we examined whether a risk model trained to predict future child protection events was sensitive to

several external measures of child harm.” And “our objective was to validate the risk of placement into foster care as a reasonable proxy for child harm …” However, what this study  achieved is something more: the development of a child safety measure more cogent (by far) than rates of re-report and multiple substantiations used by the federal Administration for Children and Families (ACF) and states’ child welfare systems to measure child safety. In addition, the study is a reminder of what is at stake in the debate regarding use of algorithms, i.e., protection of children from serious physical harm, either inflicted or unintentional. Arguably, it is the lack of cogent safety measures tied to concrete physical harm that has made it impossible to ascertain whether child protection systems have – or have not - made incremental progress in achieving their mission. The outcome measure used in this study is not the only measure of child safety that should be utilized by state child welfare systems and ACF, but it is a vital one.                  

 

What the study of hospital encounters found  

​

The authors state: “…among children referred for maltreatment and classified as highest risk (AFST score of 20) the rate of experiencing an any cause injury was 14.5 per 100 compared with children who scored as low risk who had an any cause injury rate of 4.9 per 100. For abuse-associated injury encounters, the rate for high-risk children was 2.0 per 100 and that of low-risk children was 0.2 per 100 … for suicide and self- harm, the high-risk encounter rate was 1.0 per 100 and that of  low-risk children was 0.1 per 100.”  In other words, the  5% of highest risk children had an any-cause injury rate almost three times higher than 50% of the lowest risk children and a rate of abusive injury and self-harm and suicide 10 times the rate of the lowest risk children! AFST is a powerful algorithm when applied to one of the most important safety outcomes in child protection.

 

An algorithm that can identify 5% of children reported to CPS mostly for minor neglect and associated risk factors with rates of any-cause injury or abusive injury 10 times higher than the rates for 50% of the lowest risk children has the potential to direct resources to families that need them the most, and to inform discussion of how to revise definitions of child neglect. The predictive tool can also be used to separate CPS reports into an investigative track vs. an assessment track in child welfare agencies that employ differential response at intake.       

 

High risk vs. low-risk ratios for Black children were not as large as for other groups. For any-cause injury hospital encounters, “the rate for the high-risk group (of Black children) was 16.1 per 100 compared with 8.1 per 100 for low risk.” The abusive injury rate for  the highest risk Black children was 1.9 per 100 vs. 0.5 per 100 for the lowest-risk children. The authors do not speculate why ratios between high-risk children and low-risk children were substantially less for Black children in screening of GPS reports.   

 

Given these important findings, it is surprising that the authors do not discuss the effect of use of AFST on numbers or rates of injury related hospital encounters in Allegheny County. However, a recent randomized control trial of AFST in Colorado found a 32% decline in hospitalizations due to child injury for caseworkers using AFST vs. caseworkers engaged in business as usual. A summary of this study has not yet been published, but a PowerPoint describing the study in a sketchy way is available online. (Grimon & Mills, 2022)

 

If this study, once published, stands up to intense scrutiny, the most important question will not concern the tool or the rules for its use, but rather the services or interventions caseworkers and offices utilized to reduce child injuries. Were more children placed in foster care, or were in-home services provided earlier and to the families who needed them the most? Risk assessment research often gives full attention to a tool and rules for its use while treating services like a bit player, not worthy of attention. However, predictive tools, in and of themselves, achieve nothing. It is the interventions that follow their use which should be given center stage.      

 

Predicting foster placement of young children

​

The Cross Jurisdiction Model replication Project (CJMR) developed by Putnam-Hornstein, Ahn and Vaithianathan, along with scholars from Alaska, Kentucky and Mathematica, utilizes 82 data elements from birth records to predict foster care placement by age 3 for birth cohorts. The authors comment that the study period includes “the year of life when maltreatment reporting, substantiation and foster care placement rates are highest (that is during infancy) and captured the period when about 70 percent of maltreatment fatalities occur (that is before age 3).”  (Putnam-Hornstein, et al, 2022). The predictive tool developed by CJMR was applied to birth cohorts, not CPS reports, a crucial difference.

 

Putnam-Hornstein’s 2011 study of fatal intentional and unintentional injuries among children in California with CPS reports by age 5 found that only one-fifth to one-third of young children who died from injury had been reported to CPS prior to death. To prevent most child injury deaths, prevention programs must reach out to families and offer services before a CPS report is received on the family. For infants and toddlers at highest risk of injury or death reported to CPS, CPS intervention is often too little, too late.

 

Putnam-Hornstein and Vaithianathan have demonstrated that an algorithm developed to predict foster care placement can predict any-cause injuries and abusive injuries. The CJMR algorithm has the potential to be used for this purpose by prevention programs that reach out to parents absent a CPS report or immediately after a CPS report is received, regardless of whether the report is screened in for investigation or screened out. The CJMR algorithm “applied to the Allegheny County cohort … proved more accurate at differentiating risk (of foster placement) than a similar model created using Allegheny County’s own smaller date set.”   

 

The CJMR study found that the cumulative incidence of a first foster care placement by age 3 for the California birth cohorts was 1.8 percent. The model was subsequently tested in Alaska and Kentucky which used much the same methodology.  Children in the birth cohort with the highest decile of risk scores had a placement rate of 0.68% by age 3. The model accurately predicted foster care placement (true positives) for 59.4% of  highest risk children and incorrectly predicted foster care placement (false positives) for 40.6% of children in the highest risk decile. The model was less successful when applied to birth cohorts in Alaska and Kentucky, i.e., the model accurately predicted 34.6% and 32.8% of placements for children in the  highest decile. Curiously, the model was more accurate by far in predicting foster care placement for children in the second highest decile (75.7%, 57.6%, 56.8%) in all three states.

 

Concretely, what this means is that in birth cohorts whose   placement rate to age 3 for the two highest risk deciles of infants and toddlers was less than 1% (1.22% combined) an algorithm that utilized data from birth records accurately predicted foster care placement for one third to three quarters of this population. This is impressive targeting of a population of children at high risk of foster placement, any-cause injury and abusive injury by any reasonable standard.  This model can be accessed on a GitHub website and has been made available to other jurisdictions at no cost.

 

According to the authors, “The model was not designed to identify individual children who would experience foster care placement. Instead, it aimed to use prediction methodologies to understand population wide differences in foster care placement.” Translation: The CJMR is not a case planning tool for working with a specific family; it was designed to inform prevention/early intervention strategies for the highest risk  populations of infants and toddlers in birth cohorts, hopefully prior to CPS reports. Like all risk assessment models, however developed, the CJMR has a high rate of false positives. It would be a misuse to refer to its predictions in dependency actions that  follow involuntary child removal. 

 

How to use an algorithmic tool to prevent any-cause injuries, abusive injuries, and foster care in birth cohorts  

The heated controversy over use of AFST in Allegheny County has had the unfortunate effect of conflating the use of “big data” predictive tools with an expansion of CPS involvement with at- risk children and families, absent allegations of child maltreatment. However, it is possible to utilize the tools to inform expansion of public health services to the highest risk families prior to or separate from a CPS investigation. One does not need a predictive tool to understand the importance of reaching out to parents of young children enrolled in publicly funded substance abuse or mental health treatment, or when there has been a pattern of domestic violent incidents in a family. However, there may be other high risk family profiles that are not obvious, even to experienced caseworkers.   

 

Preventive and early intervention services should include:

• Poverty related services that address basic needs such as food, housing, medical/dental care and transportation.

•  Evidence based skill-oriented services designed to improve emotionally responsive parenting.

•  Emotional and practical support for completion of substance abuse and mental health treatment programs; and DV services.

•  Childcare services and respite care urgently needed to reduce childcare burden in low-income single parent families.

•  Support services for care of disabled and chronically ill children, an element of risk in families often overlooked in CPS assessments.  

• Support for continuing education and job training required to increase family income and spark hope in the future.  

 

Application of a predictive tool at birth and expanding the role of public health nurses is not the only strategy for delivering prevention/early intervention services. Some communities may choose to develop comprehensive family support centers which parents can access as they wish. However, depending on troubled families to seek out services as they feel the need increases the likelihood that early intervention programs will serve more lower-risk motivated parents than high-risk families who want to be left alone. Every prevention program is faced with the challenge that persons who receive services will likely not be the neediest, while families with the greatest need will avoid helpers of any description. Black, American Indian, and Latino families may be reluctant to seek services due to past negative encounters with public agencies and distrust of professionals. 

 

Utilization of a predictive tool applied to all births can enable outreach and offer of services to the highest risk families and children before a family is reported to CPS. 

 

Summary

​

Some key features of the use of AFST and of the CJMR algorithms bear repeating:   

• AFST is a screening tool that identifies the 15%-25% of children at highest risk for foster care and injury related hospital encounters; neither AFST or CJMR are case planning tools utilized by investigators.

• AFST and CJMR can be used with a wide range of services and interventions; they are targeting tools, not an intervention strategy. One jurisdiction may use them well, another poorly.

• The main strength of AFST is its ability to differentiate  5% of high-risk children with a 3-10 times higher rate of  any cause and abusive injury related hospital encounters than the lowest risk 50% of families.

 

 In collaboration with other public and private human service agencies, child protection programs should develop and test practice models for this relatively small group of high-risk children.

 

References

​

Grimon, M. & Mills, C., “The Effect of Algorithmic Tools on Child Welfare Decision Making and Outcomes,” (2022), AEA RCT Registry; https://doi.org/10. 1257/rct 6311-2.0

 

“Allegheny County Predictive Risk Model Tool Implementation: Process Evaluation,” (2018) Hornby Zeller Associates, available online.  

 

Putnam-Hornstein, E., Ahn, A., Vaithianathan, R., Parrish, J, Husa, R., Walton, W., Perry, J., Wulsin, C., Bradley, M. & Varley, B., “What Can We Learn About the Incidence of Foster Care Placement from Birth Records?”  OPRE Report 2022-255, Administration for Children and Families, U.S. Department of Health and Human Services, Washington, DC.  

 

Putnam-Hornstein, E., “Report of Maltreatment as a Risk Factor for Injury Death: A Prospective Birth Cohort,”(2011), Child Maltreatment, 16(3), 163-174.

 

Vaithianathan, V., Putnam-Hornstein, E., Chouldechova, A. Benavides-Prado, D., & Berger, R., “Hospital Injury Encounters of Children Identified by a Predictive Risk Model for Screening Child Maltreatment Referrals: Evidence from the Allegheny Family Screening Tool,”(2020), JAMA Pediatrics, 174(11).  

 

See past Sounding Board commentaries     

​

©Dee Wilson     

  

deewilson13@aol.com

    

​