A Proper Review of the Statistics in Lucy Letby’s Trial

It is well known that the statistics used to convict Lucy Letby at her trial were fallacious. The most notorious example being the roster chart which did nothing more than show she was present at certain times when she was present, yet was used to create a sinister connotation.

What has not been done so far as we can tell at least in the public domain, is a proper review of the statistics. To that end, I have recently been taking part in a project to do just that. Collaborating in this includes Peter Elston, who has done extensive work on the statistics of the case already, as well as three other collaborators (all UK-based) who wish to remain anonymous for the time being.

For the exact same reasons as shown in my earlier blog post, I was as certain as one can practically be that a proper statistical analysis of the case would in no way point towards guilt, and indeed that appears to be the case based on our preliminary findings. With that said, all involved are committed to publishing the results regardless of whether they work for or against Lucy. Our ultimate commitment is to the truth. Above all, contrary to what many pitchforkers would have you believe, if I ever found evidence pointing to a serial killer of children, I would promptly make it known to the relevant authorities. I’m not Ravi Jayaram after all!

In the rest of this blog post, I will summarise some key preliminary findings as well as provide some formal housekeeping concerning data protection on this project as required by law.

The Rise in Mortality at CoCH:

Our preliminary findings show that, even with a very prosecution-friendly model, the rise in mortality at CoCH neonatal unit from June 2015 to June 2016 was of a statistical significance one would expect to observe in around one unit per year in the country.

This comes when one accounts, at least in part, for the following factors:

  1. Number of IC patient days in a given period
  2. Uncertainty in the estimate for the mortality rate per IC patient day
  3. Interval length can be varied to produce the most statistically significant “spike” in mortality possible

Moreover, this model operates on the assumption that the hospital continued to perform just as well even in the face of a significant rise in IC days that CoCH was forced to cope with, most likely due to the maternity crisis in North Wales.

If one accepts the common-sense belief that the hospital would have struggled to cope with this increased burden, thus causing the per IC patient-day mortality rate to increase, then the chance of a rise in mortality will be substantially higher than our prosecution-friendly model suggests.

Our model also does not account for other factors. For example, Thirlwall showed that the number of infants with birthweights below <2000g increased during the June 2015 to June 2016 period, which one would expect to increase the chance of a spike in mortality even higher.

How was Lucy Letby Associated with so many Deaths?

This is the current part of the research we are focusing on developing the most, but we have a few preliminary observations.

Firstly, the number of patients in intensive care is substantially higher when Lucy Letby is on shift than when she is not. A potential confounder behind this was her status as the jointly most qualified nurse in her pay bracket. This likely meant that the cheapest way to staff the rota was to use Letby as much as possible for shifts with multiple patients in intensive care, for which she could be used with her QIS qualification, while using the less-qualified Band 5 nurses on shifts with only a single or no patient in intensive care. The fact that Letby was a young nurse and single likely meant she had greater flexibility allowing her to fit such a schedule.

Secondly, heterogeneity is a really important feature. The chance of a death occurring at a given time on the unit is not constant. It varies based on how many patients are present in intensive care at a given time, what their gestational ages and birthweights are, whether or not they are a multiple, whether or not the unit is understaffed at a given time, and any other comorbidities individual patients might have. If one assumes even a small amount of heterogeneity, the chance of an individual nurse being present for a large number of deaths rises dramatically. This is the same phenomenon as I observed in the case of Lucia de Berk, where even a small amount of heterogeneity, combined with correcting other fallacies, meant that the true probability of de Berk being present for the deaths she was at was 1 in 50, a far cry from the 1 in 342 million claimed at her original trial.

Of course, even a 1 in 50 event still implies that a nurse was unlucky to be present at many deaths. But in a country with thousands of nurses, hundreds of thousands in the case of England, you would expect this to happen to a large number of nurses quite regularly. Just as someone is statistically almost guaranteed to win the lottery somewhere, even if their individual chances of doing so are small, it is almost guaranteed that many nurses will be present for many deaths. It does not imply on its own that they were responsible for any wrongdoing.

We would like to develop this work further by using data such as gestational ages and birthweight to account for some of this heterogeneity and obtain a more precise estimate of the chance of a nurse being present for some minimum number of patient deaths.

Lessons for Use of Statistics:

There are several key lessons that should be learned from this story.

One key lesson is that, when the decision around what analysis to do is based on the same data one is analysing, extreme caution must be exercised. Even subconsciously, one can use the data to perform an analysis decision aimed at achieving the very outcome one wants, while ignoring analysis approaches that do not achieve that outcome.

This issue is illustrated repeatedly throughout the case of Letby, including, but by no means limited to, the following:

  1. Defining “suspicious events”, “collapses” or “collapses leading to death” on the basis of whether Lucy was present there in the first place. It is known that at least 10 instances where it was initially thought a baby had been attacked or an otherwise “suspicious” event took place were dropped from the shift chart after Letby was found to be absent from the hospital at the time.
  2. Choosing the specific interval over which to analyse the rise in mortality on the basis of whether a rise in mortality happened to occur.

Another key lesson is that null events are extremely important to keep in mind. The death of 10 patients out of 10 is far more suspicious than the death of 10 patients out of 1000. If an unusual event is merely the result of there being many opportunities for it to happen, and it just occurred eventually at some point, then the fact an event might be rare does not imply that a rare cause must be responsible for it.

Courts seem to be extremely poor at grasping this critical concept. This is evident when the court looked only at collapses when Lucy Letby was present while ignoring other “suspicious events” after it turns out Lucy was never there catastrophically fails to grasp this key principle. It’s rather like evaluating the performance of the Dutch National Football Team by looking at only the matches they have won, while ignoring their defeats like the 2010 World Cup final.

One potential reason courts are so poor at understanding this critical concept is that courts usually handle events that are inherently rare, such as a murder where there’s no dispute a crime has occurred and the only debate is who the perpetrator was. However, determining from observational data whether an exceptionally rare cause even happened at all (such as a serial killer nurse) is a very different task. The more one looks into the case of Lucy Letby, the more one appreciates how ill-equipped courts are for handling these sort of cases.

Data Protection Housekeeping:

The law on data protection in both the UK and EU requires one to fulfil a number of key principles when handling personal data. One of those is transparency, and it is in that spirit that I provide some data protection housekeeping here.

Rights of Data Subjects:

Any data subject is welcome to contact me, including via email, if they want any further information or clarification regarding the project, and we shall endeavour to answer all queries.

Data Subjects ordinarily have a number of rights, the full details of which can be obtained on the UK Information Commissioner’s Office website. See the following link for further details: A guide to individual rights | ICO

These rights include the following:

  1. Right of Access: The right to access and receive a copy of one’s personal data.
  2. Right of Rectification: One has a right for any inaccurate personal data about oneself to be corrected
  3. Right of Restriction: One has a right to restrict how one’s personal data is used in certain circumstances
  4. Right to Object: One has a right to object to the use of one’s personal data, although this right does not apply for processing done under Article 6(1)(e) for reasons of public interest, a provision we rely on (see later for more details)
  5. Right of Erasure: One has a right to have one’s personal data erased in certain circumstances

It is worth noting that there can be restrictions on these rights. In particular, restrictions often apply when exercising the right would substantially impede the academic research we are conducting.

Data subjects also have a right to complain both to the data controller responsible for processing the data, as well as to the appropriate regulatory, in this case the UK Information Commissioner’s Office (ICO).

However, even though such a right to complain exists, we wish to emphasise that we want to work collaboratively with data subjects as much as possible and resolve any queries that they have. If a data subject identifies a flaw in our practices, we would like to work with them and remedy it wherever possible. We also would like the data to be processed in a way that leaves subjects genuinely assured that the use of their personal data in this research project will not result in any adverse effects against them, and we are more than happy to work with them as much as possible to achieve that outcome.

General Public Interest:

There are numerous reasons why this research is so vital, and why we believe it is substantially in the public interest to do:

  1. To help ascertain what really happened at the Countess of Chester Neonatal Unit. In particular, we hope our research can help contribute to ascertaining whether or not Lucy Letby has been the victim of a miscarriage of justice, or if she is indeed guilty of the crimes she has been convicted of.
  2. To provide lessons in the use and misuse of statistics in the legal setting, regardless of whether Lucy Letby has been the victim of a miscarriage of justice.
  3. If it does contribute towards showing Lucy Letby was the victim of a catastrophic miscarriage of justice, there will likely be huge implications for the criminal justice system
  4. Thirlwall has proposed wholesale changes to the NHS, including the abolition of no-blame culture, which will likely make patient care worse. This would be bad enough in any event; it is even worse if it is based on something that never happened. This research can help determine whether that is indeed the case.
  5. Given the public debate around the case of Lucy Letby is likely to continue for the foreseeable future in any event, we hope this research can help make that debate healthier and better informed.

While the practitioners of this research make no secret that we come into it with a very strong prior that Lucy Letby has been the victim of a miscarriage of justice, we do wish to emphasise that our ultimate commitment is to the truth.

Both under academic standards and human standards, we will not hesitate to publish our research if at any time it were to point against our prior belief and suggest Lucy Letby was indeed guilty of the crimes she was convicted of. We believe the public interest behind our research goes far beyond the fate of one person, and our commitment to publishing our findings no matter the result reflects that belief.

Specific Data Used:

The data used in this project will include the nurse’s shift chart, as well as admissions data for 9 out of the 13 months between June 2015 and June 2016. Unfortunately, we don’t have data for the four months where there was no indictment event. The admissions data was obtained by Peter Elston from Lucy’s defence barrister, Mr Mark McDonald and, for the infants admitted to the neonatal unit in those 9 months, contain their gestational ages, birthweights, when they were in the unit and with what category of care (intensive care, high dependency care, special care, normal) on a given day.

How the Data will be Used:

The use of this data is necessary to compare the mortality at CoCH relative to gestational age and birthweight to that of other known environments. The admissions data also allows us to calculate the total IC days over a given time interval, which is also necessary for some of our modelling.

We may require using the data subsequently to develop models around the association of a particular nurse’s presence with mortality, such as discrete time survival models. Should this need arise, we will make sure not to publish anything that could theoretically be reverse-engineered to obtain the original dataset.

It goes without saying this contains very sensitive data, including what GDPR regards as special category data. We therefore make clear at once that nothing will be published that could possibly be used to identify any patient. For example, any graphical displays will be scrambled appropriately to ensure they cannot be reverse-engineered to obtain the original dataset. Nothing will be published pertaining to any individual patient either.

The only reason we use this data is to research the spike in mortality in the Countess, as well as the reasons behind a nurse being present for many of the deaths. In no way will the data be used to make any decision about any individual patient, and, given the summary nature of what we publish, nothing released will have any prospect of doing so either.

Legal Basis for Processing Data:

Formally, the legal bases on which we use this data are article 6(1)(e) (task in the public interest), article 6(1)(f) (legitimate interests) and article 9(2)(j) (archiving, research and statistics) together with section 1 paragraph 4 of Data Protection Act 2018 (DPA). The legitimate interests are the same as those expressed earlier as to why this research is so important, and they form the same reasons as to why we consider this research to be in the general public interest, a key requirement under Article 9(2)(j).

If necessary, at certain points to ensure the completion of the project, we also rely on Schedule 2, paragraph 27 of DPA (the research purposes exemption) and, if necessary, Schedule 2, paragraph 26 of DPA (the academic expression exemption). In particular, with respect to the latter, because the project is done with a view to publishing academic material and we believe that doing so is in the general public interest, this allows us to use the academic expression exemption whenever we reasonably believe a given provision to which the exemption is eligible to be applied would otherwise be incompatible with the academic purpose behind this research.

Time for a new post … on insulin

I would recommend everyone interested in the Lucy Letby case to carefully study the 58 page appeal judgement – a board of three judges refused Lucy’s application to appeal. It is only 58 pages, it is well written and carefully argued… It is just built on heaps of false assumptions. What it says about the insulin evidence is particularly significant.

Three points specifically concern the insulin babies, babies F and L: points 14, 30 and 104 [and another on the evidence of Prof Hindmarsh, which I won’t go into right now.]  Here they are in italics; my underlining.

14. A proposed ground 4 (that the jury were wrongly directed on evidence relating to the persistence of insulin in the bloodstream) was withdrawn following the refusal of leave to appeal by the single judge. 

30. At trial, the integrity of the blood samples and reliability of the biochemical testing was challenged by Mr Myers. However, in her evidence at trial, the applicant [Lucy Letby!] admitted that both babies had been poisoned by insulin, but denied that she was the poisoner. The prosecution relied upon the unlikelihood of there being two poisoners at work on the unit. As the judge expressed it shortly before the jury retired to consider their verdicts: “If you are sure that two of the babies…had Actrapid, manufactured insulin, inserted into the infusion bag that were set up for them 8 months apart in August 2015 and April 2016 respectively, and you are sure that that was done deliberately, you then have to consider whether that may have been a coincidence, two different people independently acting in that way or were they the acts of the one person and, if so, who.”

104. The prosecution made some general points to rebut the allegations of bias and unreliability, including that almost every opinion given by Dr Evans was corroborated by another expert. In addition, it was pointed out that Dr Evans was the person who had identified that two of the babies had been poisoned by insulin (Baby F and Baby L). This was a matter which had eluded the treating medics and went to prove that someone was committing serious offences against babies in the unit; and it was particularly important independent evidence, bolstering Dr Evans’ credibility and reliability. Further, when Dr Evans reached his conclusions, he did so without knowing about other circumstantial evidence relied on by the prosecution in establishing guilt, including the applicant’s Facebook searches, the shift pattern evidence, and the “confession” in the note recovered from the applicant’s home on 3 July 2018

Richard’s comment on point 30: the applicant was told during the trial that it had been proven that two babies had been poisoned. Her reply was not that she admitted this fact, it was more like “well if you say it is proven it must be true. But I didn’t do it”. Notice that for the prosecution, six experts all say that this was true: Evans, Bohin, Hindmarsh, Milan, Wark; the hospital doctors had concurred (Gibbs, in particular). The defence had apparently not even looked for an expert on the insulin matter. They had raised problems about the reliability of the tests but these are ignored because Lucy herself agrees that the babies were poisoned with hospital insulin.

How can a young nurse agree or disagree with a deduction made by half a dozen senior medics that the immunoassay results proved deliberate insulin poisoning? All doctors and nurses have been taught in doctors’ or nurses’ school something about insulin metabolism and know that the ratio of insulin to C-peptide (after someone has been fasting for three hours) should be about 1 to 6. They are not taught anything about the forensic determination of deliberate insulin poisoning, which tells us that an anomalous ratio is a warning sign for something that might have been happened but it should be followed up with completely different tests which exclude a numerous sets of artefacts which can each also cause an anomalous ratio of immunoassays’ numbers. The test result comes back from the lab with a warning note printed in red with exactly this information. It was ignored by hospital doctors. The specialists weren’t called in and never told about it. The babies were nicely recovering from their hypoglycaemia and were rapidly transferred elsewhere.

Note that according to point 14 the defence had submitted and withdrawn a “ground for appeal” concerning the insulin. [I don’t know what it was, someone should find out!]. Note that the insulin is seen by all concerned as proof of presence of a murderer on the unit. The court of appeal sees it as ludicrous to suggest there are two murderers on the unit! They believe that the other evidence (air embolism etc etc) proves Lucy is a murderer. The insulin evidence shows someone tried to murder babies F and L. There can’t be two murderers. Therefore Lucy also tried to murder babies F and L.

The logic of the argument is impeccable. Just the premises of the argument are wrong.

These points do show the utter incompetence of the defence team. 

One of the prosecution experts (Gwen Wark) has even published a paper Marks and Wark (2013) with a concluding list of recommendations which state that the immunoassay only suggests a possibility; in order to prove it other tests must be done. https://pubmed.ncbi.nlm.nih.gov/23751444/ They were not done. No sample was saved so they never can be done. Both babies F and L are alive and well to this day though one has cerebral palsy … linked to problems experienced during birth.

At the trial Dr. Wark confirmed the other experts’ claim that insulin poisoning had been proven. How could she? What had she written in any written testimony provided pre trial to police or CPS? Richard Thomas (Lucy’s solicitor) says that the defence team can’t say whether or not any documents exist and whether or not they saw them. The jury is not shown any such documents.

People say “the jury was given so much more, information…”. That is quite simply not true. The jury in a UK criminal trial hears what is said and it observes the body language. That’s all. It does not receive copies of written testimony of scientific experts. (This is a consequence of “open justice” with a trial by a jury of your peers).

What was the fifth ground in the original application for an appeal? The one which a single judge scrapped and the defence team then scrapped too?

Maybe some journalist should chase after that. I’m afraid the defence team is unlikely to tell us. They are not obliged to, and they can only do it if Lucy instructs them to do so. And nobody is able to talk to her.

How to lie with data

This spreadsheet was shown on TV both yesterday (Friday August 18, the day of the verdicts) and at the start of the trial of Lucy Letby. Apparently, Cheshire Constabulary find this absolutely damning evidence against Lucy. And indeed, many journalists seem to agree.

The 25 events are almost all of the events at which LL was present during the periods investigated. They are suspicious because she was under suspicion when the police started their investigations. Not surprisingly, most nurses are not present at many of these events. And of course, many nurses probably work far fewer hours than LL. Many are often on administrative duties.

The doctors on the ward are of course missing. Doctors were never investigated as suspects but from the start of police investigations apparently always believed to speak gospel truth. During cross-examination, during the trial, some of them have changed various parts of their stories. Of course, unlike Lucy, they do not lie, since they could never (under oath in court, or earlier, when being interviewed as witnesses by police) be saying untruths in order to deceive.

Back to the spreadsheet. When drawing conclusions from any data it is important to know how it was gathered. It is important to know what data is missing, but would be needed draw even the most preliminary and tentative inferences.

There was an NHS investigation into the raised rates of deaths and collapses at Countess of Chester Hospital (CoCH) in summer 2015 and summer 2016. It was published in 2017 by the Royal College of Paediatrics and Child Health (RCPCH). The investigation blamed the consultants for the appalling low standard of care, and the terrible situation regarding hygiene. The RCPCH investigators actually wrote that nurse Lucy Letby could not be associated with the events, but that passage was redacted out of the published report for privacy reasons. We know that already, consultants had presented their fears to hospital management. One of them (successful TV doctor and FaceBook influencer dr Ravi Jayaram) was on TV yesterday proudly telling the world that he had been vindicated. Management was inclined not to believe them, and did not act on them, but they certainly came to the ears of the RCPCH. On publication of the report, four consultants had had enough, and went to the police with their suspicions that LL was a murderer.

Thanks to FOI requests and statistical analysis by independent scientists, we now know that the rate of events (deaths and collapses) is just as much raised when Lucy is not on the ward as it is when she is on the ward. A lot of medical information (as well as the state of the drains at CoCH) points to a seasonal virus epidemic.

The elevated rate went back to normal after the hospital was down-graded (no longer accepting high risk patients), and when the drains were rebuilt, and when the senior consultant retired, all of which happened soon after the police investigation started. Incidentally, the rate of still-births and miscarriages show exactly the same pattern.

Lucy must certainly have been a witch in order to kill babies in the womb and even when she is far from the hospital.

Those familiar with miscarriages of justice involving serial killer nurses will be familiar with this police and prosecution tactic. Is it evil or is it just stupid? (cf. Hanlon’s razor). I think it is quite simply “learnt”. Police and prosecution learn what convinces jurors over the years, and that is why the same “mistakes” are made again and again. They work!

CBS Statistieken, uitkeringsaffaire, uithuisplaatsingen

Deze saga zet zich voort met een nieuwe publicatie van het CBS, https://www.cbs.nl/nl-nl/achtergrond/2023/05/onzekerheidsmarges-onderzoek-toeslagenaffaire-en-jeugdbescherming. Nou ja, het kwam uit anderhalf maand geleden. Ik was met andere dingen bezig …

Hierbij een eerste indruk. Er worden nu betrouwbaarheidsintervallen bepaald en men ziet meteen dat de statistische onzekerheid enorm is. Natuurlijk, worden deze berekeningen gebaseerd op statistische veronderstellingen, en die zijn altijd betwistbaar. Maar op zijn minst kunnen ze geinterpreteerd worden op een pure beschrijvend-data manier als een gevoeligheids analyse. Een brede interval laat zien dat als de data een klein beetje anders was, het antwoord totaal anders zou zijn geweest. We weten zo wie zo dat er allerlei foutbronnen zijn; we weten dat de gegevens in de data bestanden van rijksinstellingen heel ver kunnen afliggen van de ervaringen van de burgers; dat ze afhangen van allerlei definities en afspraken die hun oorsprong hebben in bureaucratische administraties.

Een belangrijke resultaat is het plaatje hieronder, waarbij statistische onzekerheidsmarges toegevoegd zijn aan een plaatje uit de eerste (en omstreden) CBS rapport. Figuur 6.1.1.

Ik heb de “kleine letters” en de “nog kleinere kleine letters” meegenomen, niet om te lezen, maar om te laten zien dat er een hele technisch verhaal bijhoort.

De eerste indruk is dat het lijntje in het midden ongeveer plat is. Dus: de nare ingreep (gedupeerd zijn) in jaar “nul” geen sterke effect heeft. Men ziet over meerdere jaren een lichte toename bij dezelfde 4000 gezinnen van maatregelen van jeugdbescherming wat, zo te zien, beste toevallig had kunnen zijn. De hypothese van “geen impact” kan niet verworpen worden op grond van deze cijfers.

Maar, dat is niet de enige mogelijke uitleg van het plaatje, en die is net zo min te verwerpen. Dat hobbeltje in de grafiek zou ook “echt” kunnen zijn, en bovendien veroorzaakt door de klap wat de belastingdienst in “jaar nul” uitdeelde. Het ziet eruit als een stijging van een half procent per jaar, over meerdere jaren. De meest aannemelijke schatting is dat 20 tot 30 (of zelfs meer) echte dubbele slachtoffers zijn; dubbele slachtoffers in de zin dat gedupeerd zijn door de uitkeringsschandaal werkelijk leidde tot een uithuisplaatsing wat anders niet zou zijn gebeurd.

Het echte effect is gedempt en uitgesmeerd door alle tekortkomingen van het onderzoek. De conclusie moet zijn: het zijn zeker tientallen en mogelijk zelfs honderd.

Overigens, zou ik graag een keer een extra cijfer willen hebben waardoor ik de statistische onzekerheid in het verschil in hoogte van deze twee waardes (blaue en groen) zou kunnen evalueren.

Er zijn ruwweg 4000 gedupeerden en die zijn gepaard één op één met vergelijkbare niet-gedupeerden. We hebben feitelijk te maken met rond de 4000 matched pairs. Het CBS weet van elk lid van elk paar of een jeugdbescherming actie plaatsvond. We hebben feitelijk 4000 waarnemingen van paren, elk waarvan een van de vier waardes kan aannemen (0, 0), (0, 1), (1, 0), (1, 1); noem deze twee gevallen (x, y). Een “1” betekent uit een huisplaatsing (of iets dergelijks), een “0” betekent geen uithuisplaatsing. We zijn geinteresseerd in de gemiddelde van de x‘en minus de gemiddelde van de y‘s. Dat is hetzelfde als de gemiddelde van alle (x – y) waarden; elk ervan is gelijk aan –1, 0, of +1. Ik zou graag het 2×2 tabel willen zien van aantallen van elk van de vier mogelijke gesamenlijke uitkomsten (x, y). Ik zou de standaard afwijking willen uitrekenen van de (x – y) waarden. Dit zou ons inzicht geven in de mate van success van de matching: als het goed is, zouden we een positieve correlatie zien tussen de uitkomsten van de twee groepen. Een correlatie van +1 zou impliceren dat de uitkomst volledig bepaald is door de matching variabelen, dat zou betekenen: gedupeerd zijn maakte werkelijk niks uit. Kom’ns op, CBS!

The bogeyman (Algemene Dagblad, 26 January)

date2023-01-26 09:38:02

Richard Gill. © Rob Voss

Professor Gill helped exonerate Lucia de B., and is now making mincemeat of the CBS report on benefits affair

Top statistician Richard Gill cracks down on the research conducted by Statistics Netherlands (CBS) into custodial placements of children of victims in the benefits affair. ‘CBS should never have come to the conclusion that this group of parents was not hit harder than other parents.’

Carla van der Wal 26-01-23, 06:00 Last update: 08:10

Emeritus professor Richard Gill would prefer to pick edible mushrooms in the woods and spend time with his grandchildren. Nevertheless, the top statistician in the Netherlands, who previously helped to exonerate the unjustly convicted Lucia de B, is now firmly committed to the benefits affair.

CBS should never have started the investigation into the custodial placement of children of victims in the benefits affair, says Gill. “And the conclusion that this group of parents has not been hit harder than other parents, CBS should never have drawn. It left many people thinking: only the tax authorities have failed, but fortunately there is nothing wrong with youth care. So all the fuss about ‘state kidnappings’ was unnecessary.”

After Statistics Netherlands calculated how many children of benefit parents were placed out of home (in the end it turned out to be 2090), it seemed that victims in the affair lost their children more often than similar parents who were not victims. The results were presented on November 1 last year, which Gill now denounces.

Gill is emeritus professor of mathematical statistics at Leiden University and in the past was an advisor to the methodology department of Statistics Netherlands. In the case of Lucia de B. he showed that calculations that would show that De B. had more deaths in her services were incorrect.

CBS abuses

There is a special reason that Gill is now getting stuck in the benefits affair – but more on that later. First about the CBS report. Gill states that Statistics Netherlands is not equipped for this type of research and points out that after two research methods were dropped, only one ‘not ideal, but only option’ remained. He also thinks, among other things, that the more severely affected victims in the benefits affair should be the focus of the investigation. He emphasizes that relatively mildly affected families most likely had to deal with much less drastic consequences. CBS itself also says that it likes to use information about the degree of duping, but that there was none.

CBS also acknowledges some criticisms. “CBS itself has mentioned a number of comments to the report. There seems to be a misunderstanding on one point,” said a spokesperson, who also said that CBS still fully supports the conclusions. CBS will soon be discussing the methodology used with Gill, but in any case CBS sees itself as the right party to carry out the study. “CBS has the task of providing insight into social issues with reliable statistical information and data and has the necessary expertise and techniques. In this case there was a clear social need for statistical insight.”

Gill thinks otherwise and thinks it’s important to raise this. Because he is awakened by injustice. That was also a reason to offer his help when questions arose about the conviction of Lucia de B., who can simply be called Lucia de Berk again since her acquittal. In 2003 she was sentenced to life imprisonment.

Out-of-home placement

With the acquittal in 2010, Gill became not only a top statistician, but also a beacon of hope for people who experienced injustice. And José Booij, a mother of a child placed in care, contacted him many years ago.

Somewhere in Gill’s house in Apeldoorn there is still a box with papers from José. It contains diaries, newspaper clippings and diplomas of hers. She was a little different from other people. A doctor who fell for women, fled the Randstad and settled in Drenthe. There she became pregnant and had a baby. And she had a neighbour with whom she had a disagreement. “That neighbour had made all kinds of reports about José to the local police, said that something terrible would happen to the child.” After six weeks, José’s daughter was removed from home.

State kidnapping

“What happened to José at the time, I also call that a state kidnapping, just as the custodial placements among victims of the benefits affair are now called.” The woman continued to fight to get her child back. But gradually that fight drove her insane. She lost her job, she lost her home. She fled abroad. “Despite a court ruling that the child had to be returned to José, that did not happen. José eventually derailed. I now know that she has left information with more people in the Netherlands to ensure that it is available to her daughter when she is ready. But I can’t find José anymore. I heard she was seen in the south of the Netherlands after escaping from a psychiatric clinic in England.”

Text continues below the photo

Demonstration by victims of the benefits affair. © ANP / ANP

And meanwhile he keeps that box. And Gill thinks of José, when he considers the investigation by the Central Bureau of Statistics into custodial placements of children of victims in the benefits affair. Gill makes mincemeat of it. “The only thing CBS can say is that the results suggest that the differences between the two groups that have been compared are quite small. There should be a lot more caution, and yet in the summary you see bold summaries, such as: ‘Being duped does not increase the likelihood of child protection measures’. I suspect that CBS was put under pressure to conduct this study, or wanted to justify its existence. Perhaps there is an urge to be of service.”

Time for justice

Now is the time to put that right, Gill thinks. Research needs to be done to find out what’s really going on. “I had actually hoped that younger colleagues would have stood up by now, who would take up such matters.” But as long as that doesn’t happen, he’ll do it himself. Maybe it’s in his genes. It was Gill’s mother – he was born in England – who helped crack the enigma code used by the Germans to communicate during World War II. Gill wasn’t surprised when he found out. He already suspected that his excellent mind was inherited not only from his father, but also from his mother.

Love

Yet in the end it was his wife – the love of whom led him settle in the Netherlands – who put him on this track. She pointed Gill to Lucia de Berk’s case and encouraged him to get to work. She may have regretted that. For example, when Gill threatened to burn his Dutch passport during a broadcast of The World Keeps on Turning Round (“De wereld draait door”) if the De Berk case was not reviewed. “She said, ‘You can’t say things like that!'”

In fact, he would like to enjoy his retirement with her now – he has been out of paid work for six years now. Then he would spend his days in the woods looking for edible mushrooms. And spend a lot of time with his grandchildren. But now his calculations also help exonerate other nurses. Last year, Daniela Poggiali was released in Italy after Gill interfered with the case together with an Italian colleague. There are still things waiting for him in England.

And so the benefits affair is here in the Netherlands, which, as far as Gill is concerned, needs more in-depth, thorough research to find out exactly what caused the custodial placements. “That is why I ended up with Pieter Omtzigt and Princess Laurentien, who are also involved in the benefits affair.” Among the people who express themselves diplomatically, he wants to be the bad cop, the man who shakes things up, as he did when he threatened to set his passport on fire. But at the same time, he also hopes that a young statistician will emerge who is prepared to take over the torch.

CBS provided this site with an extensive explanation in response to Gill’s criticism. It recognizes the complexity of this type of research, but sees itself as the appropriate body to carry out that research. An appointment to speak with Gill has already been scheduled. “CBS always tries to explain as clearly and transparently as possible in its reports what has been investigated, how it was done and what the results are.”

Statistics Netherlands also points to nuances in the text of the report, for example after the sentence above a piece of text: ‘Being duped does not increase the chance of child protection measures’. “On an individual level, there may be a relationship between duping and youth protection, which is stated in several places in the report.” Even if ‘on average no evidence is found for a relationship between duping and youth protection’, as Statistics Netherlands notes.

Statistics Netherlands fully supports the research and the conclusions as stated in the report. It is pointed out, however, that there are still opportunities for follow-up research, as has also been indicated by Statistics Netherlands.

De boeman (Algemene Dagblad, 26 januari)

Richard Gill. © Rob Voss

Hoogleraar Gill hielp bij vrijpleiten Lucia de B., en maakt nu gehakt van CBS-rapport toeslagenaffaire

Topstatisticus Richard Gill kraakt het onderzoek dat het Centraal Bureau voor de Statistiek (CBS) uitvoerde naar uithuisplaatsingen van kinderen van gedupeerden in de toeslagenaffaire. ‘De conclusie dat deze groep ouders niet harder is geraakt dan andere ouders, had het CBS nooit mogen trekken.’

Carla van der Wal 26-01-23, 06:00 Laatste update: 08:10

Het liefste zou emeritus hoogleraar Richard Gill eetbare paddenstoelen plukken in het bos, en tijd doorbrengen met zijn kleinkinderen. Toch bijt de topstatisticus van Nederland, die eerder hielp bij het vrijpleiten van de onterecht veroordeelde Lucia de B, zich nu vast in de toeslagenaffaire. 

Het CBS had nooit aan het onderzoek naar de uithuisplaatsing van kinderen van slachtoffers in de toeslagenaffaire moeten beginnen, zegt Gill. ,,En de conclusie dat deze groep ouders niet harder is geraakt dan andere ouders, had het CBS nooit mogen trekken. Die liet velen denken: alleen de belastingdienst heeft gefaald, maar er is gelukkig niets mis met jeugdzorg. Al die ophef over ‘staatsontvoeringen’ was dus onnodig.’’

Nadat het CBS becijferde hoeveel kinderen van toeslagenouders uit huis werden geplaatst (uiteindelijk bleken het er 2090), leek het of gedupeerden in de affaire vaker hun kinderen kwijtraakten dan soortgelijke ouders die geen slachtoffer waren. Op 1 november vorig jaar werden de resultaten gepresenteerd, die Gill nu hekelt.

Gill is emeritus hoogleraar mathematische statistiek aan de universiteit van Leiden en was in het verleden adviseur bij de afdeling methodologie van het CBS. In de zaak van Lucia de B. liet hij zien dat berekeningen die zouden aantonen dat De B. vaker sterfgevallen in haar diensten had, niet klopten.

Misstanden CBS 

Dat Gill zich nu vastbijt in de toeslagenaffaire heeft een bijzondere reden – maar daarover later meer. Eerst nog over het rapport van het CBS. Gill stelt dat het CBS niet is ingericht op dit type onderzoek en wijst erop dat nadat twee onderzoeksmethodes afvielen slechts één ‘niet ideale, maar enige optie’ overbleef. Ook vindt hij onder meer dat zwaarder getroffen gedupeerden in de toeslagenaffaire centraal zouden moeten staan bij het onderzoek. Hij benadrukt dat relatief licht geraakte gezinnen hoogstwaarschijnlijk met veel minder ingrijpende gevolgen te maken hebben gehad. Het CBS zegt overigens zelf ook dat het graag informatie over de mate van gedupeerdheid gebruikt, maar dat die er niet was.

Het CBS erkent ook sommige punten van kritiek. ,,Een aantal heeft het CBS zelf als kanttekening genoemd bij het rapport. Op een enkel punt lijkt sprake van een misverstand’’, aldus een woordvoerder, die ook zegt dat het CBS nog volledig achter de conclusies staat. Over de gebruikte methodologie gaat het CBS binnenkort met Gill in gesprek, maar het CBS ziet zich in elk geval wél als de juiste partij om het onderzoek uit te voeren. ,,Het CBS heeft als taak om met betrouwbare statistische informatie en data inzicht te geven in maatschappelijke vraagstukken en beschikt over de nodige expertise en technieken. In dit geval was een duidelijke maatschappelijke behoefte aan statistisch inzicht.’’

Gill denkt daar anders over en vindt het belangrijk dat aan te kaarten. Want hij ligt wakker van onrecht. Dat was ook reden om zijn hulp aan te bieden toen er vragen rezen over de veroordeling van Lucia de B., die sinds haar vrijspraak gewoon weer Lucia de Berk genoemd kan worden. In 2003 werd ze veroordeeld tot een levenslange gevangenisstraf.

Uithuisplaatsing

Door de vrijspraak in 2010 werd Gill naast een topstatisticus ook een baken van hoop voor mensen die onrecht ervaarden. En nam José Booij, een moeder van een uit huis geplaatst kind, vele jaren geleden contact met hem op.

Ergens in het huis van Gill in Apeldoorn staat nog een doos met papieren van José. Erin zitten dagboeken, krantenknipsels en diploma’s van haar. Ze was een beetje anders dan andere mensen. Een jurist die op vrouwen viel, de Randstad ontvluchtte en neerstreek in Drenthe. Daar werd ze zwanger, kreeg ze een kindje. En had ze een buurvrouw, met wie ze onenigheid had. ,,Die buurvrouw had allerlei meldingen over José gedaan bij de lokale politie, had gezegd dat met het kindje iets vreselijks zou gebeuren.” Na zes weken werd Josés dochtertje uit huis geplaatst.

Staatsontvoering 

,,Wat José indertijd is overkomen, dat noem ik ook een staatsontvoering, net zoals de uithuisplaatsingen onder slachtoffers van de toeslagenaffaire nu worden genoemd.” De vrouw bleef vechten om haar kind terug te krijgen. Maar gaandeweg dreef dat gevecht haar tot waanzin. Ze raakte haar werk kwijt, ze raakte haar huis kwijt. Ze vluchtte naar het buitenland. ,,Ondanks een oordeel van de rechter, dat het kind terug moest naar José, gebeurde dat niet. José is uiteindelijk ontspoord. Inmiddels weet ik dat ze bij meer mensen in Nederland informatie heeft achtergelaten, om te zorgen dat die beschikbaar is voor haar dochter, als die eraan toe is. Maar José heb ik niet meer kunnen vinden. Ik heb gehoord dat ze nog is gezien in het zuiden van Nederland, nadat ze was ontsnapt uit een psychiatrische kliniek in Engeland.”

Tekst gaat verder onder de foto

Gedupeerden in de toeslagenaffaire demonstreren. © ANP / ANP

En ondertussen bewaart hij dus die doos. En denkt Gill aan José, als hij zich buigt over het onderzoek van het Centraal Bureau voor de Statistiek, naar uithuisplaatsingen van kinderen van slachtoffers in de toeslagenaffaire. Gill maakt er gehakt van. ,,Het enige wat het CBS kan zeggen, is dat de uitkomsten suggereren dat de verschillen tussen de twee groepen die zijn vergeleken vrij klein zijn. Er zou veel meer voorzichtigheid moeten zijn, en toch zie je in de samenvatting in vetgedrukte letters stellige samenvattingen, zoals: ‘Gedupeerdheid verhoogt de kans op kinderbeschermingsmaatregelen niet’. Ik vermoed dat het CBS onder druk is gezet om dit onderzoek te doen, of zijn bestaansrecht heeft willen verantwoorden. Wellicht is er sprake van een drang om dienstbaar te zijn.”

Tijd voor rechtvaardigheid

Nu is het tijd om dat recht te zetten, vindt Gill. Er moet onderzoek worden gedaan, om te kijken hoe het echt zit. ,,Ik had eigenlijk gehoopt dat er inmiddels jongere collega’s zouden zijn opgestaan, die dit soort zaken op zouden pakken.” Maar zolang dat niet gebeurt, doet hij het zelf wel. Misschien zit het wel in zijn genen. Het was Gills moeder – hij werd geboren in Engeland – die tijdens de Tweede Wereldoorlog meewerkte aan het kraken van de enigmacode, die door de Duitsers werd gebruikt om te communiceren. Gill verraste het niet, toen hij erachter kwam. Hij had al zo’n vermoeden dat zijn excellente verstand niet alleen een erfenis van zijn vader, maar ook zijn moeder was.

De liefde

Toch was het uiteindelijk zijn vrouw – de liefde zorgde dat hij in Nederland neerstreek – die hem op dit spoor heeft gezet. Zij wees Gill op de zaak van Lucia de Berk en stimuleerde hem ermee aan de slag te gaan. Misschien heeft ze dat wel eens betreurd. Bijvoorbeeld toen Gill tijdens opnames van De wereld draait door dreigde zijn Nederlandse paspoort te verbranden, als de zaak De Berk niet werd herzien. ,,Ze zei: dat kan je toch niet doen?”

Eigenlijk zou hij nu met haar van zijn pensioen willen genieten – hij is inmiddels zes jaar gestopt met zijn betaalde werk. Dan zou hij zijn dagen vullen in het bos, zoekend naar eetbare paddenstoelen. En veel tijd doorbrengen met zijn kleinkinderen. Maar nu helpen zijn berekeningen ook bij het vrijpleiten van andere verpleegkundigen. Vorig jaar werd Daniela Poggiali nog vrijgelaten in Italië, nadat Gill zich samen met een Italiaanse collega met de zaak bemoeide. In Engeland zijn nog zaken die op hem wachten.

En de toeslagenaffaire is er hier in Nederland dus, waar wat Gill betreft diepgravender, gedegen onderzoek naar moet komen, om uit te zoeken wat nu precies de uithuisplaatsingen veroorzaakte. ,,Ik ben daarom terechtgekomen bij Pieter Omtzigt en prinses Laurentien, die zich ook met de toeslagenaffaire bezighouden.” Tussen de mensen die zich diplomatiek uiten, wil hij best de bad cop zijn, de man die de boel opschudt, zoals hij deed toen hij dreigde zijn paspoort in de fik te steken. Maar tegelijkertijd hoopt hij toch vooral ook dat er een jonge statisticus opstaat, die bereid is de fakkel over te nemen.

Het CBS gaf deze site een uitgebreide toelichting, naar aanleiding van de kritiek van Gill. Het erkent de complexiteit van dit soort onderzoek, maar ziet zichzelf wél als aangewezen instantie om dat onderzoek uit te voeren. De afspraak om met Gill te spreken is al ingepland. ,,Het CBS tracht in de rapporten altijd zo duidelijk en transparant mogelijk uit te leggen wat onderzocht is, hoe dat is gedaan en wat de uitkomsten zijn.”

Ook wijst het CBS op nuanceringen in de tekst van het rapport, bijvoorbeeld na de zin boven een stuk tekst: ‘Gedupeerdheid verhoogt de kans op kinderbeschermingsmaatregelingen niet’. ,,Er kan op individueel niveau wél een relatie tussen dupering en jeugdbescherming zijn, dat staat op meerdere plekken in het rapport vermeld.” Ook als er ‘gemiddeld genomen geen bewijs gevonden wordt voor een relatie tussen dupering en jeugdbescherming’, zoals het CBS constateert.

Het CBS staat volledig achter het onderzoek en de conclusies zoals die in het rapport vermeld staan. Wel wordt erop gewezen dat er nog mogelijkheden zijn voor vervolgonderzoek, dat heeft het CBS ook aangegeven.

Nog meer over uitkeringsschandaal en uithuisplaatsingen

Hieronder volgt een poging (20 januari 2023, ‘s ochtends) om het kern van het verhaal op te schrijven in 500 woorden en Jip en Janneke taal. Het lukte niet.

Heeft het CBS de waarheid in pacht?

Velen werden wakker geschud door carabetier Peter Pannekoek’s woorden “1115 staatsontvoeringen”. Maar ze kunnen weer in slaap gesust zijn door het CBS rapport “Jeugdbescherming en de toeslagenaffaire – Kwantitatief onderzoek naar kinderbeschermingsmaatregelen bij kinderen van gedupeerden van de toeslagenaffaire”. Een van de belangrijkste conclusies (samenvatting, eerste bladzijde) luidt

“Gedupeerdheid verhoogt de kans op kinderbeschermingsmaatregelen niet“.

Dat is een krachtige uitspraak. Geen enkel relativering, geen “kleine letters”. Geen melding dat het een uitspraak is die alleen gemaakt kan worden onder een hele reeks veronderstellingen. Helaas, een hele reeks veronderstellingen waarvan velen pertinent onwaar zijn.

Mijn antwoord: misschien geen 1115, maar misschien wel: 115

Nu munt het CBS uit in het doen van beschrijvend statistiek, wat ook hun wettelijke opdracht is. Ze dienen neutraal de feiten te ontsluiten en weer te geven die politiek en bestuur en burgers nodig hebben. Waar het CBS minder expertise in huis heeft, omdat het ook beslist niet tot hun taak behoort, is in het ontwarren van oorzaak en gevolg. Dat noemen we tegenwoordig “Causaliteit” en het is een uiterst actueel, belangrijk, subtiel, en complex onderwerp binnen het wetenschappelijk onderzoek; explosief gegroeid sinds Judea Pearls boek “Causality” uit 2000. Kan je causaliteit concluderen door het waarnemen van correlatie of associatie?

Voorbeeld. Lucia de B maakte vreselijk veel incidenten mee in haar diensten. Veel meer dan men zou hebben verwacht en dat leidde ook tot levenslange gevangenisstraf voor seriële moord. Pas later werd duidelijk dat haar aanwezigheid juist de reden was dat medisch onderzoekers bepaalde gebeurtenissen als incidenten karakteriseerden!

Maar kan geen associatie ook op causaliteit duiden? Jawel! Statistieken kunnen misleiden. Een aansprekend visuele representatie van statistieken des te meer. Mijn oog werd getrokken door Figuur 6.1.2 in het CBS rapport waarin we drie vrolijk gekleurde balkjes zijn, die de percentages 1%, 4% en 4% dienen te representeren. Zie je wel! De percentage uithuisplaatsingen bij de gedupeerden is exact wat je zou hebben verwacht, als al die gezinnen helemaal niet gedupeerd waren geweest!

Ik zou zeggen, dat kan geen toeval zijn. Na studie van het onderzoeksprotocol inclusief de vele door de team hanteerde algoritmes, wordt ook duidelijk dat het geen toeval is. Door de onderzoekskeuzes die het onderzoeksteam zich gedwongen voelde te maken is het verschil in uithuisplaatsingskans tussen “vergelijkbare” wel en niet gedupeerden systematisch verkleind. Het verschil is dus groter dan het lijkt (het lijkt nul te zijn, maar dat is het beslist niet). De juiste conclusie van het onderzoek had moeten zijn, ten eerste, dat er zeker tientallen uithuisplaatsingen “extra” plaatsvonden vanwege de affaire en mogelijk honderd (of zelfs een paar honderd). Een tweede conclusie had moeten zijn dat deze gedurfde pilot studie bewezen heeft dat een totaal ander onderzoeksopzet nodig is oude gestelde vraag te beantwoorden. Mogelijk, iets in de trant van het eerder verworpen onderzoeksvoorstel van Prof. Bart Tromp van de Universiteit Groningen. Overigens, is het nooit nodig om alle dossiers van de hele geschiedenis van alle gedupeerden door te pluizen. Door slim een aselecte steekproef in een verstandig gekozen deelpopulatie te nemen, kan men zich beperken tot het goed uitzoeken van relatief weinig gevallen.

Goede “Data Science” is onmogelijk zonder grote expertise te combineren uit drie gebieden tegelijkertijd: 1) algoritmes en computer mogelijkheden; 2) kansrekening en inferientiele statistiek (dwz het kwantificeren van de onzekerheid in de gevonden resultaten); 3) (last but not least!) vakspecifieke kennis van het beoogde toepassingsgebied; in dit geval psychologie, recht, bestuur.

De verantwoording van mijn claims ben ik momenteel aan het uitschrijven in mijn blog, https://gill1109.com/2023/01/18/de-statistiek-van-slachtoffers-van-uitkeringsschandaal/; het moet nog veel worden uitgebreid met nadere onderbouwing, verwijzingen, enzovoorts.

Ik denk aan een statistische simulatie om mijn punt te illustreren. Die twee getallen “4%” hebben foutbalken nodig van ongeveer +/- 1%. Lastig omdat ik rekening moet houden met de correlatie binnen de paren. We kunnen alleen maar raden hoe groot het is. Dus: meerdere simulaties met verschillende gissingen.

Yet more on the Dutch benefits scandal and child removals

This is a first attempt to summarise my claims in 500 words and simple language. It didn’t succeed.

Does CBS have direct access to the truth?

Many were shaken up by carabetier Peter Pannekoek ‘s words “1115 state kidnappings”. But they may have been lulled back to sleep by the CBS report “Youth protection and the benefits affair – Quantitative research into child protection measures in children of victims of the benefits affair”. One of the main conclusions (summary, first page) reads

“Being a victim of the benefits scandal does not increase the likelihood of child protection measures“.

That’s a powerful statement. No relativization whatsoever, no “small print”. No mention of it being a statement that can only be made under a slew of assumptions. Alas, a slew of assumptions many of which are patently untrue.

My answer: Maybe not 1115, but could well have been 115.

Now CBS excels at doing descriptive statistics, which is also their legal assignment. They should neutrally disclose and represent the facts that politicians, administration and citizens need. Where CBS has less in-house expertise, because it is certainly not part of their task, is in disentangling cause and effect. This is what we call “Causality” today and it is an extremely topical, important, subtle, and complex subject of scientific inquiry; exploded since Judea Pearl’s 2000 book “Causality”. Can you infer causality by observing correlation or association?

Example. Lucia de B experienced an awful lot of incidents in her services. Much more than one would have expected and that also led to life imprisonment for serial murder. Only later did it become clear that her presence was precisely the reason why medical examiners characterized certain events as incidents!

But can *no* association also indicate causality? Yes! Statistics can be misleading. An appealing visual representation of statistics all the more. My eye was drawn to Figure 6.1.2 in the CBS report in which we are three brightly colored bars, which should represent the percentages 1%, 4% and 4%. See! The percentage of custodial placements among the victims is exactly what you would have expected, if all those families had not been victimized at all!

I’d say that can’t be a coincidence. After studying the research protocol, including the many algorithms used by the team, it also becomes clear that this is no coincidence. Due to the research choices that the research team felt compelled to make, the difference in out-of-home placements between “comparable” victims and non-victims has been systematically reduced. So the difference is greater than it appears (it appears to be zero, but it is definitely not). The correct conclusion of the investigation should have been, first, that there were certainly dozens of “extra” custodial placements because of the affair and possibly a hundred (or even a few hundred). A second conclusion should have been that this bold pilot study has proven that a completely different research design is needed to answer an old question. Possibly, something along the lines of Prof. dr. Bart Tromp of the University of Groningen. Incidentally, it is never necessary to go through *all* files of the entire history of all victims. By smartly taking a random sample in a sensibly chosen sub-population, one can limit oneself to properly sorting out relatively few cases.

Good “Data Science” is impossible without combining great expertise from three areas at the same time: 1) algorithms and computing capabilities; 2) probability theory and inferiential statistics (ie quantifying the uncertainty in the results found); 3) (last but not least!) subject-specific knowledge of the intended application area; in this case psychology, law, administration.

I am currently writing out the justification for my claims in my blog, https://gill1109.com/2023/01/18/de-statistiek-van-slachtoffers-van-toeslagsschandaal/; it still needs to be expanded a lot with further substantiation, references, and so on.

I’m thinking of a statistical simulation to illustrate my point. Those two numbers “4%” need error bars of about +/- 1%. Tricky because I must take account of the correlation within the pairs. We can only guess how big it is. So: several simulations with different guesses.

Statistics of victims of the Dutch child-benefits scandal

Commentary on the CBS report

Author: prof.dr. (em.) Richard D. Gill

Mathematical Institute, Leiden University

Monday January 16, 2023

Richard Gill is emeritus professor of mathematical statistics at Leiden University. He is a member of the KNAW and former chairman of the Netherlands Statistical Society (VVS-OR)

=========================================

Mr. Pieter Omtzigt has asked me to give my expert opinion on the CBS report that examines whether the number of child care placements of children by Dutch child protection authorities increased because their families had fallen victim to the child benefit scandal in the Netherlands.

The current note is preliminary and I intend to refine it further. My purpose is to stimulate discussion among relevant professionals of the methodology used by the CBS in this particular case. Feedback, please!

The report gives a clear (and short) account of creative statistical analysis of much complexity. The sophisticated nature of the analysis techniques, the urgency of the question, and the need to communicate the results to a general audience probably led to important “fine print” about the reliability of the results being omitted. The authors seem to me to be too confident in their findings.

Numerous choices had to be made by the CBS team to answer the research questions. Many preferable options are excluded due to data availability and confidentiality. Changing one of the many steps in the analysis through changes in criteria or methodology could lead to wildly different answers. The actual finding of two nearly equal percentages (both close to 4%) in the two groups of families is, in my opinion, “too good to be true”. It’s a fluke. Its striking character may have encouraged the authors to formulate their conclusions much more strongly than they are entitled to.

In this regard, I found it significant that the authors note that the datasets are so large that statistical uncertainty is unimportant. But this is simply not true. After constructing an artificial control group, they have two groups of size (in round numbers) 4000, and 4% of cases in each group, i.e. about 160. According to a rule of thumb calculation (Poisson variation), the statistical variation in those two numbers have a standard deviation of about the square root of 160, so about 12.5. That means that one of those numbers (160) could easily happen to have twice the standard deviation, which is about 25. The conclusion that the benefits scandal did not lead to more children being removed from home than without it would have been the case, certainly cannot be drawn . Taking into account the statistical sampling error, it is quite possible that the control group (those not afflicted by the benefits scandal) would have been 50 less. In that case, the study group experienced 50 more than they would have done, had they not been victims of the benefits scandal.

To make the numbers easier still, suppose there was an error of 40 cases too few in the light blue bar standing for 4%. 40 out of 4000 is 1 out of 100, 1%. Change the light blue bar from height 4% to height 3% and they don’t look the same at all!

But this is already without taking into account possible systematic errors. The statistical techniques used are advanced and model-based. This means that they depend on the validity of many particular assumptions about the form and nature of the relationships between the variables included in the analysis (using “logistic regression”). The methodology uses these assumptions for its convenience and power (more assumptions mean stronger conclusions, but threatens “garbage in, garbage out”). Logistic regression is such a popular tool in so many applied fields because the model is so simple: the results are so easy to interpret, the calculation can often be left to the computer without user intervention. But there’s no reason why the model should be exactly true; one can only hope that it is a useful approximation. Whether it is useful depends on the task for which it is used. The current analysis uses logistic regression for purposes for which it was not designed.

The assumptions of the standard model of logistic regression are certainly not exactly met. It is not clear whether the researchers tested for failure of the assumptions (for example, by looking for interaction effects – violation of additivity). The danger is that the failure of the assumptions can lead to systematic bias in the results, bias that affects the synthetic (“matched”) control group. The central assumption in logistic regression is the additivity of effects of various factors on the log-odds scale (“odds” means probability divided by complementary probability; log means logarithm). This could be true to a first rough approximation, but it is certainly not exactly true. “All models are wrong, but some are useful”.

A good practice is to build models by analyzing a first data set and then evaluating the final chosen model on an independently collected second data set. In this study, not one but numerous models were tested. The researchers seem to have chosen from countless possibilities through subjective assessment of plausibility and effectiveness. This is fine in an exploratory analysis. But the findings of such an exploration must be tested against new data (and there is no new data).

The end result was a procedure to choose “nearest neighbour matches” with respect to a number of observed characteristics of the cases examined. Errors in the logistic regression used to choose matched controls can systematically bias the control group.

Further big questions concern the actual selection of cases and controls at the beginning of the analysis. Not all families affected by the benefits scandal had to pay back a huge amount of subsidy. Mixing the hard-hit and the weak-hit dilutes the effect of the scandal, both in magnitude and accuracy, the latter because maller samples lead to relatively less accurate determination of effect size.

Another problem is that the pre-selection control population (families in general from which a child was removed) also contains victims of the benefit scandal (the study population). That brings the two groups closer together, even more so after the familywise one-on-one matching process, which of course selectively finds matches among the subpopulation most likely to be affected by the benefits scandal.

%%footer%%