DEE WILSON CONSULTING

(Originally published December2022)

​

By Dee Wilson and Toni Sebastian

​

(Note: In the interest of full disclosure, the authors had key roles in the development, implementation, and revision of the Washington State risk assessment model during the 1990’s and early 2000’s, and so were not disinterested observers to the events discussed in this commentary. One of the authors participated in several reviews of states’ child protection systems during his employment with a foundation from 2010-15 and had numerous opportunities to discuss use of risk assessment practice models with child welfare staff in several states, including California, Florida, Illinois, Vermont, and Minnesota. One of the authors had a leadership role in the implementation of Washington State’s safety framework based on the Action model, which many child welfare systems around the country utilize.)

 

In the beginning:  The risk assessment “wars”

 

During the 1990s, when many state and large county child welfare systems implemented risk assessment models, there were annual risk assessment conferences that became well known for heated debates among program developers and scholars. These conflicts were obviously motivated to some extent by struggles for funding, influence and (ultimately) preeminence. They also reflected the widespread view that risk assessment had great potential to improve child protection systems and that differences among models (of which tools were a key element) were therefore of great importance. Self-importance led to self-righteousness, and there was no group of developers and scholars engaged in controversies regarding risk assessment who remained unaffected by self-inflation.

 

There have been dozens of studies of risk assessment tools published during the past three decades. Critics had a field day with risk assessment tools and practices from their initial implementation in child welfare systems for reasons that will become clear in this commentary. Frequent attacks on all risk assessment models by scholars, and more recently by parent and social justice advocates, continues to this day. Risk assessment has been blamed for just about everything in child welfare that critics don’t like, including racial disproportionality, unwarranted CPS intrusion into the lives of low-income families, endangering children through misuse of “low risk” classification at intake, and inconsistent assessment practices.           

 

Risk assessment developers revise their claims

​

Initially, risk assessment generated excitement in child protection programs because of widely repeated findings from research that statistical tools did a better job of predicting violence - and recurrence of violence – than clinical judgment. Initially, the expectation of program developers and child welfare managers was that use of risk assessment tools would lead to more accurate prediction of child maltreatment (both occurrence and recurrence) than caseworkers’ unstructured assessments. However, early risk assessment studies found that risk assessment tools had low to modest predictive power in specific cases, and that what risk assessment tools did best was to classify CPS reports or cases according to their risk of rereport or subsequent substantiated investigations. All risk assessment models were found to have high rates of “false positives,” i.e., cases in which recurrence of child maltreatment was predicted but did not occur, as indicated by CPS rereport rates.

 

In accordance with these research findings, arguably the most useful and defensible use of risk assessment tools has been to allocate utilization of limited staffing resources and services in offices with overwhelming workloads and inadequate funding of services by identifying low risk reports and cases which receive a reduced child protection response. When done well, identifying 20-25% of screened in reports as “low risk” has enabled investigators to concentrate the 120 hours per month they have to engage in casework activities on families at highest risk of child maltreatment. When done poorly, sometimes due to misuse of differential response (DR), risk assessment at intake has left children in danger, potentially mortal danger.

 

Once program developers and scholars concluded that the main

goal of risk assessment tools was to separate CPS reports and investigations into low, moderate, and high-risk categories based on risk of rereport, actuarial models, especially Structured Decision Making (SDM), quickly achieved dominance among risk assessment models. Actuarial risk assessment tools had clearly won the risk assessment “wars,” including in Washington, which began using SDM around 2007-08.  

 

The fragmentation of risk assessment   

​

By 2000, program developers (as well as CPS staff) had discovered that risk assessment tools were rarely useful in making placement decisions, which usually depended on a determination of risk of imminent harm, not risk of recurrence per se. Program developers and child welfare agencies began to develop safety tools and/or safety frameworks that were distinct from risk assessments, meaning  a child and family could be categorized as “high risk” but be deemed safe when caseworkers did not find a child to be in immediate danger from child maltreatment. Action, the most widely used safety model in the U.S., utilizes a Safe/Unsafe dichotomy that conflates child safety with the absence of danger, a narrow view of child safety which conflicts with the idea that risk of future maltreatment is a continuum from low risk to high risk, and that risk of imminent harm is at the extreme end of this continuum. By defining child safety as a binary choice, i.e., Safe vs. Unsafe, the Action model could be used to focus CPS concern on Unsafe children and ignore risk that did not rise to the level of danger.     

 

Some states, including Washington, attempted to combine the use of SDM with the Action safety framework, but for both conceptual reasons and workload pressures, this proved difficult to do. By 2018, Washington state’s child protection system had become fully committed to its safety model, while SDM had fallen into disuse or was used in a pro forma way only, as indicated by the DCYF 2022 Annual Progress and Services Report.  

 

In what amounted to a survival of the fittest competition between a safety framework and an actuarial risk assessment model, CPS staff and child welfare managers in Washington chose the safety framework for several reasons: workload pressures, the framing of child safety as either Safe/Unsafe which served the goal of narrowing the grounds for CPS intervention, and diminished interest in providing services to families categorized moderate or high risk after investigation.

 

The gradual loss of interest in the SDM risk assessment tool in Washington was accompanied by a deliberate unlearning of risk assessment fundamentals among some experienced staff, fundamental such as: 

 

• the best predictor of future behavior is past behavior

• risk of child maltreatment is additive, i.e., the more risk factors in a family and more types of risk factors (e.g., child vulnerability factors such as young age and disability) combined with parental impairments due to substance abuse, mental illness, or domestic violence), the higher the risk of severely injurious maltreatment.

 

Today, many inexperienced caseworkers have never acquired these concepts due to inadequate basic training and poor supervision, while some newly hired caseworkers with graduate degrees pride themselves on rejecting risk assessment principles, which they believe conflict with social justice for Black and American Indian families.    

 

The fragmentation of risk assessment did not end with the bifurcation of safety and risk. In the early 2000s, a few scholars presented a cogent argument that risk assessment tools were inadequate for case planning decisions regarding services, and that a needs assessment tool was required for this purpose. Risk assessment tools include factors unrelated to families’ service needs, or which cannot be altered by available services, while ignoring factors which service plans must address such as a child’s trauma history or fear of a parent, or a couple’s marital conflict.

 

Some program developers and agencies began to search for – or develop – needs assessment tools. However, for many offices already struggling with steadily expanding prescriptive frameworks, the idea of adding a needs assessment tool to risk and safety tools and frameworks was a step too far.  Risk assessment in these agencies came to be viewed as ancillary both to safety assessment and services planning, both regarded as essential functions of child protection programs, while risk assessment was consigned to be “documentation only,” little more than a bureaucratic nuisance at case closure. 

 

Caseworkers “weigh in” on risk assessment

​

Risk assessment research and internal audits in Washington and other states and large counties have consistently identified common patterns in how caseworkers use risk assessment: many caseworkers and supervisors assert that they incorporate risk assessment concepts in their decision making, but not through use of tools which are usually completed at case closure.

 

Child welfare managers and program developers have engaged in prolonged discussion regarding how to ensure that tools are employed to inform decisions rather than (mainly) to document decisions after the fact. For the most part, these discussions have been of no avail in offices where caseworkers have had to cope with   overwhelming workload demands. In addition, caseworkers were quick to learn that a risk assessment tool was of no benefit in a large fraction of cases. Experienced caseworkers do not need an assessment tool to determine that families with half a dozen or more CPS reports within a few years are likely to be rereported, or that seriously physically abused or sexually abused children need post- investigation services. Requiring busy caseworkers to complete risk assessment tools is a waste of their time in 30-40% of investigations. This is the kind of prescriptive overkill that has increasingly made U.S. child welfare offices intolerable places to work.  

 

Program developers have been slow to recognize the limitations of their tools or frameworks, in part because they do not have to use them regularly, and also because they often have little regard for  negative feedback from caseworkers which they commonly view as an excuse for refusing to comply with agency policies.  

 

There was more distressing news to come. Middle managers  (including the authors) who reviewed case records following suspected child maltreatment deaths, or other high profile hotly contested cases, found that caseworkers who completed risk assessment tools at case closure often used risk assessment tools to justify their case disposition decisions. A caseworker who had decided to close a case might omit, forget, or even falsify information regarding risk factors inconsistent with their decision. Instead of using risk assessment tools to inform decision making, tools were often used to justify decisions made on other grounds.

 

Some managers learned to read case notes rather than naively accepting as true information on a risk assessment summary. We were occasionally astonished to discover that caseworkers had marked “No risk” or “Low risk” regarding parental substance abuse or mental illness when parents had long histories of drug addiction and mood disorders! Of course, this is not a flaw of risk assessment tools, but it does indicate that child welfare managers are frequently unable to control how child welfare staff utilize them.  

 

In reviewing case records, we also discovered something surprising about our reactions to information contained on risk assessment summaries, i.e., that we paid little or no attention to risk ratings unless they were accompanied by narratives, for example explanations of how substance abuse, mental illness or domestic violence affected parenting. A risk rating without a narrative was not useful to our understanding of case decisions. Craig Schwalbe’s article, “Revisioning risk assessment for human services decision making,” (2004) contains an insightful discussion of the importance of combining risk assessments with case narratives that address causality. This article is among the best scholarly discussions of how to improve utilization of risk assessment in child welfare in recent decades.

 

Risk assessment models do not offer explanatory frameworks with which to develop causal narratives; they are theoretical efforts to use statistical methods to target and manage resources. Causal explanations that connect information regarding multiple risk factors found on assessment tools must be drawn from sources other than these models. It is striking how few interesting hypotheses regarding child maltreatment have come out of more than 30 years of risk assessment research. This is not surprising given the practical goals of these tools, and the widespread interest in finding effective ways to reduce child maltreatment, rather than causal explanations which require more than statistical associations.

 

Thinking Fast and Slow

​

In 2011, Daniel Kahneman, widely regarded as one of the world’s greatest living psychologists, published Thinking Fast and Slow, a summary of several decades of studies in cognitive psychology.  Kahneman is a strong proponent of actuarial risk assessment, even “back of the envelope” tools, yet Thinking Fast and Slow explains clearly why it has been so difficult for child welfare agencies to use risk assessment tools as designed to inform decision making. Kahneman also describes the inherent limitations of risk assessment tools, asserting that they are not designed to predict low base rate phenomena such as child maltreatment deaths, a limitation that cannot be corrected by research.

 

Thinking Fast and Slow became a surprising New York Times best seller because it summarized a large body of research conducted by numerous scholars (including Kahneman and his deceased colleague, Amos Tversky) through a story about the conflict of two types of mental processes, System 1 and System 2. Kahneman acknowledges this is a fiction of sorts, yet one that engages readers’ interest.

 

System 1 is the most compelling character in this story.  System 1 is:

• energetic, always “on” when we’re awake

• quick, intuitive, and headstrong with a powerful emotional component

• able to develop a compelling story from small amounts of information; “What You See Is All There Is” (WYSIATI); “radically indifferent to the quantity and quality of information”

• self-confident regarding its judgments, not given to ambivalence, self-doubt, reflection, or delay

• a way of making decisions under extreme time pressure

• bad at statistics

• likes stories that connect the dots

• is highly vulnerable to a number of heuristic biases, i.e., mental shortcuts, that systematically lead to error, e.g., confirmation bias and halo effect, i.e., the power of first impressions.   

 

System 2 is:

• slow, deliberate

• has limited energetic resources and is therefore lazy

•  is vain regarding its alleged control of the mind, which is mostly an illusion though in specific circumstances System 2 can exercise veto power over System 1 impulses.

 

In the absence of self-reflection, System 1 is usually the decision maker because it is enmeshed with emotions and jumps to conclusions based on a tiny bit of information. Once System 1 has arrived at a decision, confirmation bias kicks in and decision makers look for information that confirms their view and ignore information potentially in conflict with their already formed opinion.

 

In her article, “Common Errors in Reasoning in Child Protection,” (2000) Eileen Munro describes how child protection caseworkers in high profile child maltreatment fatality cases in England were open to new information regarding risk and safety until their mind was made up, at which point they became highly resistant to changing their views and were often unwilling to listen new information or reconsider their assessments.

 

Kahneman acknowledges that the System 1 preference for stories and for emotionally charged information will usually prevail over slower analytical ways of making decisions, especially when System 2 is employing a statistical tool. It is only when decision makers slow down and restrain their impulse to jump to conclusions that any analytical approach to decision making can have influence, much less be given priority.

 

Summary

​

In retrospect, the benefits of risk assessment in child welfare have had little or nothing to do with the tools themselves, or differences among tools or models, because caseworkers under extreme workload pressures have rarely used the tools as anticipated by developers except perhaps at intake, and because of the cognitive proclivities of Kahneman’s System 1.

 

The main benefit of risk assessment has been to ground CPS assessments in a few basic fundamental principles such as using history of child maltreatment as a baseline level of risk and the understanding that level of risk is more influenced by the number of risk factors in a family than by any one specific factor. When  these principles are ignored, children may be left in extreme jeopardy, even as caseworkers assess them to be safe, i.e., not in danger at the moment. This is professional incompetence, not social justice.   

 

Next month’s Sounding Board will discuss use of “big data” algorithms in child welfare, and the contention that expert decision makers in many professions, including child welfare, depend on “holistic pattern recognition” rather than assessment tools.   ©

 

References

 

2022 Annual Progress and Services Report, Washington State Department of Child, Youth and Family Services, September 2022.

 

Doueck, H., English, D., DePanfilis, D. & Moote, G., “Decision Making in Child Protective Services: A Comparison of Selected Risk- Assessment Systems,” (1993), Child Welfare , vol. 72, (5).

 

English, D., Marshall, D., Brumell, S., & Orme, M. (1998), “Risk Assessment in Washington Child Protective Services,” Office of Children’s Research, Washington State Department of Social and Health Services, Olympia, WA.      

 

Kahneman, D., Thinking Fast and Slow (2011), Farrar, Straus & Giroux, New York City.

 

Klein, G., Sources of Power: How People Make Decisions (1999), MIT Press, Cambridge, Mass.   

 

Schwalbe, C., “Revisioning risk assessment for human services decision making,”(2004), Children and Youth Services Review, 26 (6).

 

Shlonsky, A. & Wagner, D., “The next step: integrating actuarial risk assessment and clinical judgment into an evidence-based practice framework in CPS,” (2005), Children and Youth Services Review, vol. 27 (4).

 

Wald, M. & Woolverton, M., “Risk Assessment: The Emperor’s New Clothes?” (1990), Child Welfare, 69.  

 

​

Index of all Sounding Board commentaries 

​

©Dee Wilson     

  

deewilson13@aol.com

    

​