DEE WILSON CONSULTING
How Effective Is Child Protection?
(Originally published April 2013)
Part I
Public opinion regarding child protection in the U.S. and other English speaking countries has been shaped by media coverage of high profile child deaths, and has created an impression of general ineptitude or (worse) dereliction of duty in some egregious failures of child protection. However, drawing conclusions about the performance of child welfare agencies that respond to many thousands of CPS reports annually from a single case or a few extreme possibly atypical cases is unreasonable; and astute advocates, policymakers and scholars will usually look for other better indicators of agency performance.
It is common for policymakers and media critics to fix on trends in child deaths, but the acrimonious conflicts between and among legislators, reporters, advocacy groups and public agency managers over disclosure of information in child death cases or over classification, counting or reporting of child deaths suggest how politically charged child death reviews, data collection and summaries of child death statistics can be. Legislatures sometimes pass laws requiring that public agencies release case information following a child death suspected to have been caused by abuse or neglect; or establish groups of professionals outside the public agency to review child deaths on open or recently open cases and to make annual reports regarding child death trends.
There is a widespread agreement among scholars that child maltreatment deaths are undercounted in the National Child Abuse and Neglect Data System (NCANDS), and since NCANDS data is gathered from state child welfare systems, undercounts in NCANDS means that state agencies are likely undercounting maltreatment related deaths for a variety of reasons. NCANDS rules require that child deaths included in child welfare agencies' counts be substantiated for maltreatment, and substantiation decisions are fraught with difficulty (see below). Some state child welfare agencies depend on coroners' or medical examiners' determinations of causes of death to make substantiation decisions following a child death, and these professional groups rarely classify accidental deaths as neglect related.
There are no widely accepted criteria among state child welfare systems for determining when accidental deaths, for example deaths resulting from house fires or drowning or suffocation, were genuinely accidental or due to parental negligence. In some cases, these judgments are easy to make; but well informed professionals often disagree about the degree of parental culpability in children’s accidental deaths. In practice, arriving at credible counts of maltreatment related deaths on open or recently open cases turns out to be extremely difficult; and as a result confident assertions regarding child death trends in a specific state or nationally, or comparisons of maltreatment death rates in various states are highly questionable. In addition, child maltreatment deaths is a low base rate phenomenon subject to large year- to-year fluctuations both at the state and national level. Policymakers and advocates should be cautious about drawing conclusions about trends from annual or bi-annual changes in numbers or rates of child maltreatment deaths.
There are difficult technical challenges to achieving more accurate counts of child maltreatment deaths, especially in regard to the classification of neglect related deaths; but these challenges could be resolved by bringing together scholars and policymakers to develop rules for classifying and counting child maltreatment deaths and combining the counts of various agencies that make these determinations, e.g, public health, law enforcement, child welfare, and community child death review teams. However, technical challenges become difficult to resolve when public child welfare managers are deeply ambivalent about any improvements in surveillance that will almost certainly increase the numbers of children officially deemed to have died due to abuse or neglect. Improvements in identification and classification of child deaths are likely are to have a high political cost to public agency managers; and in these circumstances it is not unusual to hear both child welfare managers and some scholars question whether more accurate counts of child maltreatment deaths would result in practical benefits to child welfare practice.
For these reasons, neither information regarding high profile child deaths or aggregate multi-year child maltreatment death statistics are a sound way of evaluating U.S. child protection systems at the current time. We strongly support persistent efforts to arrive at more accurate counts of child maltreatment related deaths; because until these counts are more credible there will be no way to know whether CPS systems as currently organized are improving (or not) their ability to reduce child maltreatment deaths. However, we have ceased to believe that technical improvements in surveillance can achieve this goal without a fundamental change in the political milieus in which child protection systems operate. As long as Governors and legislators expect public child welfare agencies to be one hundred percent effective in preventing child deaths on open or recently open CPS cases, the acknowledgement of agency managers that state systems average x number of child maltreatment deaths on open/ recently open cases per year will be politically perilous, and initiatives designed to improve the accuracy of child maltreatment death statistics will often be met with active or passive resistance.
Many child welfare practitioners and managers, and some scholars and advocates, would prefer (we believe) to ignore child deaths and child death statistics and depend on the Child and Family Service Review (CFSR) recurrence measure of child safety to evaluate CPS performance. This measure is as follows: of all children who were victims of a substantiated or indicated allegation during the first 6 months of the reporting period, what percent were not victims of another substantiated or indicated maltreatment allegation within a 6 month period.
This is one of only two measures of child safety required by the federal government’s Administration for Children and Families (the other measure is the rate of children in out-of-home care maltreated annually), and the only measure that most state child welfare systems use to evaluate CPS performance with children living with their birth parents. Most state child welfare agencies have shown improvement on this measure in recent years which some managers and advocates point to as evidence that CPS systems are doing a better job of protecting children from maltreatment.
However, the CFSR recurrence measure is no more credible, and possibly less credible, than counts or rates of child maltreatment related deaths. Substantiation decisions are known to be affected by a variety of factors other than factual evidence pertaining to allegations of child maltreatment. During the many years one of us (Dee Wilson) worked for Washington State's child welfare system as a CPS social worker, supervisor and manager, he participated in numerous discussions regarding whether (or not) to substantiate allegations of maltreatment. Factual evidence of maltreatment was an important factor in these decisions, but so was the question of whether a single incident of abuse or neglect was serious enough to label a parent as abusive or neglectful when to do so might have a long term effect on a parent’s employment opportunities, or negatively affect the caseworker's relationship with the family. The substantiation rate in Washington State declined dramatically in the mid-1990s after DSHS began to utilize administrative law judges to hear parents' appeals of substantiation decisions. When administrative law judges began to reverse caseworkers’ substantiation decisions, often after lengthy quasi-judicial hearings, considerations related to the strength of evidence became increasingly important in caseworkers' thinking about substantiation.
During recent years, the use of the CFSR recurrence measure to evaluate agencies' performance has added a powerful incentive to "game" the measure by limiting the use of substantiation in marginal cases. Declining to substantiate allegations of maltreatment is cost free to practitioners most of the time as there is usually no immediate effect on child safety. Brett Drake has pointed out that the main practical benefit of substantiating allegations of maltreatment in most agencies is to strengthen court actions filed on behalf of abused and neglected children; but caseworkers may view the possibility of filing a dependency action as a remote possibility in many in - home services cases.
In addition to the various reasons and incentives caseworkers have for not substantiating allegations of maltreatment, state child welfare systems have different rules regarding substantiation and different ways of conceptualizing types of maltreatment. In some states, for example Pennsylvania, substantiation is a marker for criminal investigation or prosecution, and, as a result, allegations of neglect are rarely substantiated. In some other states, virtually every substantiated child victim has been neglected. Comparison of recurrence rates in state systems that rarely substantiate neglect allegations with rates of agencies in which almost every child victim has been found to be neglected is not a meaningful indicator of the relative effectiveness of child protection in these states. State child welfare systems vary dramatically in the extent to which they respond to allegations of psychological maltreatment, exposure to domestic violence, educational neglect, prenatal substance abuse, or moral turpitude of parents. Comparing recurrence rates in state systems that have very different definitions of child maltreatment is interesting, but not a way of evaluating the relative effectiveness of these states’ CPS performance.
Until and unless state and county child welfare systems take steps to assure the integrity and increase the consistency of caseworkers’ substantiation decisions along the lines of the initiative undertaken in recent years by the U.S. Air Force (see Smith-Slep and Heyman, 2006), assertions regarding the performance of CPS programs based on their recurrence rates should be taken with a grain of salt.
However, even if there was good reason to have confidence in the CFSR recurrence measure, using this measure as a stand- alone indicator of CPS performance would not be defensible. David Olds, John Eckenrode and Harriet Kitzman have commented that “Investigators have a responsibility to help interested parties to know that there are no single, simple measures to determine intervention impact when examining child maltreatment as an outcome.” Olds, Eckenrode’s and Kitzman’s comment was directed at research of prevention programs, but applies equally to evaluating CPS interventions and outcomes of treatment programs. Recurrence is one dimension of child safety; but severity of harm and chronicity of maltreatment are just as important.
The U.S. child protection system developed in the 1960s and 70s was a response to severely abused young children; and political leaders and advocates are understandably interested in the capacity of CPS programs to protect children from serious inflicted injuries, or serious injuries resulting from parental negligence, following CPS reports. The lack of an NCANDS or CFSR severity measure is difficult to understand without an appreciation of the political damage to public child welfare systems resulting from high profile child deaths and stories of near deaths, or other severe neglect related injuries. However, the lack of a severity measure (that might include counts of maltreatment related deaths on open or recently open cases) leaves public agency managers defenseless when responding to public outrage over a specific case.
There have been several studies published in the past two years ( Berger, et al, 2011; Wood, et al, 2012, Levanthal, et al, 2012) that have found recent increases in hospitalizations of children resulting from abuse; and the researchers who authored these studies have reflected on how the number of substantiated physical abuse victims in NCANDS can be steadily declining while the number and rate of seriously injured abused children has been increasing. One possible answer is that a small fraction of substantiated physical abuse cases involve serious abuse, and fewer still require hospitalization. Substantiation decisions are not a sensitive indicator of serious injuries; ditto for the CFSR recurrence measure. The fact that NCANDS statistics do not reflect an important trend in child safety found by several groups of scholars suggests the inadequacy of current safety measures utilized by public agencies.
Similarly, tracking whether families are substantiated for child maltreatment more than once, or are referred to CPS more than once within a limited time frame, does not reveal the number and percentage of families who are chronically re-referred to CPS, i.e., reported multiple times. The lack of a chronicity measure hides the extent to which CPS programs are, or are not, protecting children from chronic neglect and chronic maltreatment, i.e., combinations of neglect and physical abuse or sexual abuse. A child welfare system in which almost four fifths of substantiated victims have been found to be neglected, but which lacks a chronicity measure, is either poorly informed about neglect or (more likely) does not view prevention of chronic neglect as an important goal of child protection.
In summary, current child safety measures utilized by state child welfare systems and ACF, and agency counts of child maltreatment deaths reflect both the technical difficulties involved in gathering certain types of data, and the political milieu in which data collection and measurement occurs. In addition, the lack of a chronicity measure reflects “the neglect of neglect” and the uncertainty of practitioners and policymakers about how to prevent or reduce neglect in its most extreme form, and contributes to a lack of commitment to finding more effective responses to chronic neglect. The dependence of public child welfare agencies on a single flawed CFSR recurrence measure has left policymakers, managers, practitioners and advocates without a readily available and credible way of answering the question, “how effective is child protection?” and with a fundamental uncertainty about the quality of agencies‘ performance in protecting children.
Read Part II in which we will discuss information, studies and perspectives which can provide a well-informed answer to the question which is the subject of this commentary.
References
Berger, R.P., Fromkin, J.B., Stutz, H., Scribano, P.V., Feldman, K., Tu, L.C. & Fabio, A. (2011), “Abusive Head Trauma During a Time of Increased Unemployment: A multicenter Analysis,” Pediatrics, 128 (4), 637-643.
Drake, B. and Jonson- Reid, M. (2000), “Substantiation and Early Decision Points in Child Welfare,” Child Maltreatment, 5 (3), 227-235.
Levanthal, J.M., Martin, K.D. & Gaither, J.R., “Using US Data to Estimate the incidence of Serious Physical Abuse in Children,” Pediatrics, 129 930, 458-464.
Olds, D., Eckenrode, J. & Kitzman, H. (2005), “Clarifying the impact of the Nurse – Family Partnership on child maltreatment: response to Chaffin (2004), Child Abuse and Neglect, 29, 229-233.
Smith- Slep, A.M. and Heyman, R.E., “Creating and Field- Testing Child Maltreatment Definitions: A Conceptual Reconsideration,” Child Maltreatment, 11 93), 217-236.
Wood, J.N., Medina, S.P., Feudtner, C., Luan, X., Localio, R., Fieldston, E.S., @ Rubin, D.M., “ Local Macroeconomic Trends and Hospital Admissions for Child Abuse, 2000-2009,” Pediatrics, 129 (3), 458-464.