It is well known that the statistics used to convict Lucy Letby at her trial were fallacious. The most notorious example being the roster chart which did nothing more than show she was present at certain times when she was present, yet was used to create a sinister connotation.
What has not been done so far as we can tell at least in the public domain is a proper review of the statistics. To that end, I have recently been taking part in a project to do just that. Collaborating in this includes Peter Elston, who has done extensive work on the statistics of the case already, as well as three other collaborators (all UK based) who wish to remain anonymous for the time being.
For the exact same reasons as shown in my earlier blogpost, I was as certain as one can practically be that a proper statistical analysis of the case would in no way point towards guilt, and indeed that appears to be the case based on our preliminary findings. With that said, all involved are committed to publishing the results regardless of whether they work for or against Lucy. Our ultimate commitment is to the truth. Above all, contrary to what many pitchforkers would have you believe, if I ever found evidence pointing to a serial killer of children, I would promptly make it known to the relevant authorities. I’m not Ravi Jayaram after all!
In the rest of this blogpost, I will summarise some key preliminary findings as well as provide some formal housekeeping concerning data protection on this project as required by law.
The Rise in Mortality at CoCH:
Our preliminary findings show that, even with a very prosecution friendly model, the rise in mortality at CoCH neonatal unit from June 2015 to June 2016 was of a statistical significance one would expect to observe in around one unit per year in the country.
This comes when one accounts at least in part for the following factors:
- Number of IC patient days in a given period
- Uncertainty in the estimate for the mortality rate per IC patient day
- Interval length can be varied to produce the most statistically significant “spike” in mortality possible
Moreover, this model operates on the assumption that the hospital continued to perform just as well even in the face of a significant rise in IC days that CoCH was forced to cope with, most likely due to the maternity crisis in North Wales.
If one accepts the common-sense belief that the hospital would have struggled to cope with this increased burden, thus causing the per IC patient day mortality rate to increase, then the chance of a rise in mortality will be substantially higher than our prosecution friendly model suggests.
Our model also does not account for other factors. For example, Thirlwall showed that the number of infants with birthweights below <2000g increased during the June 2015 to June 2016 period, which one would expect to increase the chance of a spike in mortality even higher.
How was Lucy Letby Associated with so many Deaths?
This is the current part of the research we are focusing on developing the most, but we have a few preliminary observations.
Firstly, the number of patient in intensive care is substantially higher when Lucy Letby is on shift than when she is not. A potential confounder behind this was her status as the jointly most qualified nurse in her pay bracket. This likely meant that the cheapest way to staff the rotor was to use Letby as much as possible for shifts with multiple patients in intensive care, for which she could be used with her QIS qualification, while using the lesser qualified band 5 nurses on shifts with only a single or no patient in intensive care. The fact that Letby was a young nurse and single likely meant she had greater flexibility allowing her to fit such a schedule.
Secondly, heterogeneity is a really important feature. The chance of a death occurring at a given time on the unit is not constant. It varies based on how many patients are present in intensive care at a given time, what their gestational ages and birthweights are, whether or not they are a multiple, whether or not the unit is understaffed at a given time, and any other comorbidities individual patients might have. If one assumes, even a small amount of heterogeneity, the chance of an individual nurse being present for a large number of deaths rises dramatically. This is the same phenomenon as I observed in the case of Lucia de Berk, where even a small amount of heterogeneity, combined with correcting other fallacies, meant that the true probability of de Berk being present for the deaths she was at was 1 in 50, a far cry from the 1 in 342 million claimed at her original trial.
Of course, even a 1 in 50 event still implies that a nurse was unlucky to be present at many deaths. But in a country with thousands of nurses, hundreds of thousands in the case of England, you would expect this to happen to a large number of nurses quite regularly. Just as someone is statistically almost guaranteed to win the lottery somewhere, even if their individual chances of doing so are small, it is almost guaranteed that many nurses will be present for a many deaths. It does not imply on its own that they were responsible for any wrongdoing.
We would like to develop this work further by using data such as gestational ages and birthweight to account for some of this heterogeneity and obtain a more precise estimate on the chance of a nurse being present for some minimum number of patient deaths.
Lessons for Use of Statistics:
There are several key lessons that should be learned from this story.
One key lesson is that, when the decision around what analysis to do is based on the same data one is analysing, extreme caution must be exercised. Even subconsciously, one can use the data to perform an analysis decision aimed at achieving the very outcome one wants, while ignoring analysis approaches that do not achieve that outcome.
This issue is illustrated repeatedly throughout the case of Letby including, but by no means limited to, the following:
- Defining “suspicious events”, “collapses” or “collapses leading to death” on the basis of whether Lucy was present there in the first place. It is known that at least 10 instances where it was initially thought a baby had been attacked or an otherwise “suspicious” event took place were dropped from the shift chart after Letby was found to be absent from the hospital at the time.
- Choosing the specific interval over which to analyse the rise in mortality on the basis of it being whether a rise in mortality happened to occur
Another key lesson is that null events are extremely important to keep in mind. The death of 10 patients out of 10 is far more suspicious than the death of 10 patients out of 1000. If an unusual event is merely the result of there being many opportunities for it to happen, and it just occurred eventually at some point, then the fact an event might be rare does not imply that a rare cause must be responsible for it.
Courts seem to be extremely poor at grasping this critical concept. This is evident when the court looked only at collapses when Lucy Letby was present while ignoring other “suspicious events” after it turns out Lucy was never there catastrophically fails to grasp this key principle. It’s rather like evaluating the performance of the Dutch National Football Team by looking at only the matches they have won, while ignoring their defeats like the 2010 World Cup final.
One potential reason courts are so poor at understanding this critical concept is courts usually handle events that are inherently rare, such as a murder where there’s no dispute a crime has occurred and the only debate is who the perpetrator was. However, determining from observational data whether an exceptionally rare cause even happened at all (such as a serial killer nurse) is a very different task. The more one looks into the case of Lucy Letby, the more one appreciates how ill equipped courts are for handling these sort of cases.
Data Protection Housekeeping:
The law on data protection in both the UK and EU requires one fulfils a number of key principles when handling personal data. One of those is transparency, and it is in that spirit that I provide some data protection housekeeping here.
Rights of Data Subjects:
Any data subject is welcome to contact me, including via email, if they want any further information or clarification regarding the project and we shall endeavour to answer all queries.
Data Subjects ordinarily have a number of rights, the full details of which can be obtained on the UK Information Commissioner’s Office website. See the following link for further details: A guide to individual rights | ICO
These rights include the following:
- Right of Access: The right to access and receive a copy of one’s personal data.
- Right of Rectification: One has a right for any inaccurate personal data about oneself to be corrected
- Right of Restriction: One has a right to restrict how one’s personal data is used in certain circumstances
- Right to Object: One has a right to object to the use of one’s personal data, although this right does not apply for processing done under article 6(1)(e) for reasons of public interest, a provision we rely on (see later for more details)
- Right of Erasure: One has a right to have one’s personal data erased in certain circumstances
It is worth noting that there can be restrictions on these rights. In particular, restrictions often apply when deploying the right would substantially impede the academic research we are conducting.
Data subjects also have a right to complain both to the data controller responsible for processing the data, as well as to the appropriate regulatory, in this case the UK Information Commissioner’s Office or ICO.
However, even though such a right to complain exists, we wish to emphasise that we want to work collaboratively with data subjects as much as possible and resolve any queries that they have. If a data subject identifies as flaw in our practices, we would like to work with them and remedy it wherever possible. We also would like the data to be processed in a way that leaves subjects genuinely assured that the use of their personal data in this research project will not result in any adverse effects against them, and we are more than happy to work with them as much as possible to achieve that outcome.
General Public Interest:
There are numerous reasons why this research is so vital, and why we believe it is substantially in the public interest to do:
- To help ascertain what really happened at the Countess of Chester Neonatal Unit. In particular, we hope our research can help contribute to ascertaining whether or not Lucy Letby has been the victim of a miscarriage of justice, or if she is indeed guilty of the crimes she has been convicted of.
- To provide lessons in the use and misuse of statistics in the legal setting, regardless of whether Lucy Letby has been the victim of a miscarriage of justice.
- If it does contribute towards showing Lucy Letby was the victim of a catastrophic miscarriage of justice, there will likely be huge implications for the criminal justice system
- Thirlwall has proposed wholesale changes to the NHS, including the abolition of no blame culture which will likely make patient care worse. This would be bad enough in any event; it is even worse if it is based on something that never happened. This research can help determine whether that is indeed the case.
- Given the public debate around the case of Lucy Letby is likely to continue for the foreseeable future in any event, we hope this research can help make that debate healthier and better informed.
While the practitioners of this research make no secret that we come into it with a very strong prior that Lucy Letby has been the victim of a miscarriage of justice, we do wish to emphasise our ultimate commitment is to the truth.
Both under academic standards, and human standards, we will not hesitate to publish our research if at any time it were to point against our prior belief and suggest Lucy Letby was indeed guilty of the crimes she was convicted of. We believe the public interest behind our research goes far beyond the fate of one person, and our commitment to publishing our findings no matter the result reflects that belief.
Specific Data Used:
The data used in this project will include the nurse’s shift chart, as well as admissions data for 9 out of the 13 months between June 2015 and June 2016. Unfortunately, we don’t have data for the four months where there was no indictment event. The admissions data was obtained by Peter Elston from Lucy’s defence barrister Mr Mark McDonald and, for the infants admitted to the neonatal unit in those 9 months, contains their gestational ages, birthweights, when they were in the unit and with what category of care (intensive care, high dependency care, special care, normal) on a given day.
How the Data will be Used:
The use of this data is necessary to compare the mortality at CoCH relative to gestational age and birthweight to that of other known environments. The admissions data also allows us to calculate the total IC days over a given time interval, which is also necessary for some of our modelling.
We may require using the data subsequently to develop models around the association of a particular nurse’s presence with mortality, such as discrete time survival models. Should this need arise, we will make sure not to publish anything that could theoretically be reversed engineered to obtain the original dataset.
It goes without saying this contains very sensitive data, including what GDPR regards as special category data. We therefore make clear at once that nothing will be published that could possibly be used to identify any patient. For example, any graphical displays will be scrambled appropriately to ensure they cannot be reverse engineered to obtain the original dataset. Nothing will be published pertaining to any individual patient either.
The only reason we use this data is to research the spike in mortality in the Countess, as well as the reasons behind a nurse being present for many of the deaths. In no way will the data be used to make any decision about any individual patient, and, given the summary nature of what we publish, nothing released will have any prospect of doing so either.
Legal Basis for Processing Data:
Formally, the legal bases on which we use this data are article 6(1)(e) (task in the public interest), article 6(1)(f) (legitimate interests) and article 9(2)(j) (archiving, research and statistics) together with section 1 paragraph 4 of Data Protection Act 2018 (DPA). The legitimate interests are the same as those expressed earlier as to why this research is so important, and they form the same reasons as to why we consider this research to be in the general public interest, a key requirement under article 9(2)(j).
If necessary, at certain points to ensure the completion of the project, we also rely on schedule 2, paragraph 27 of DPA (the research purposes exemption) and, if necessary, schedule 2, paragraph 26 of DPA (the academic expression exemption). In particular, with respect to the latter, because project is done with a view to publishing academic material and we believe that doing so is in the general public interest, that allows us to use the academic expression exemption whenever we reasonably believe a given provision, that the exemption is eligible to be applied to, would otherwise be incompatible with the academic purpose behind this research.
