BOAS, Breed, CFR

Relationship between incidence of breathing obstruction and degree of muzzle shortness in pedigree dogs

The little dog in front of van Eyck’s Arnolfini’s is a “Griffon Bruxellois”.
[Arnolfini Portrait. (2022, July 27). In Wikipedia.

This blog post is the result of rapid conversion from a preprint, typeset with LaTeX, posted on as, and submitted to the journal PLoS ONE. I used pandoc to convert LaTeX to Word, then simply copy-pasted the content of the Word document into WordPress. After that, a few mathematical symbols and the numerical contents of the tables needed to be fixed by hand.


There has been much concern about health issues associated with the breeding of short-muzzled pedigree dogs. The Dutch government commissioned a scientific report Fokken met Kortsnuitige Honden (Breeding of short-muzzled dogs), van Hagen (2019), and based on it rather stringent legislation, restricting breeding primarily on the basis of a single simple measurement of brachycephaly, the CFR: cranial-facial ratio. Van Hagen’s work is a literature study and it draws heavily on statistical results obtained in three publications: Njikam (2009), Packer et al. (2015), and Liu et al. (2017). In this paper, I discuss some serious shortcomings of those three studies and in particular, show that Packer et al. have drawn unwarranted conclusions from their study. In fact, new analyses using their data lead to an entirely different conclusion.


The present work was commissioned by “Stichting Ras en Recht” (SRR; Foundation Justice for Pedigree dogs) and focuses on the statistical research results of earlier papers summarized in the literature study Fokken met Kortsnuitige Honden (Breeding of short-muzzled – brachycephalic – dogs) by dr M. van Hagen (2019). That report is the final outcome of a study commissioned by the Netherlands Ministry of Agriculture, Nature, and Food Quality. It was used by the ministry to justify legislation restricting breeding of animals with extreme brachycephaly as measured by a low CFR, cranial-facial ratio.

An important part of van Hagen’s report is based on statistical analyses in three key papers: Njikam et al. (2009), Packer et al. (2015), and Liu et al. (2017). Notice: the paper Packer et al. (2015) reports results from two separate studies, called by the authors Study 1 and Study 2. The data analysed in Packer et al. (2015) study 1 was previously collected and analysed for other purposes in an earlier paper Packer et al. (2013) which does not need to be discussed here.

In this paper, I will focus on these statistical issues. My conclusion is the cited papers have many serious statistical shortcomings, which were not recognised by van Hagen (2019). In fact, a reanalysis of the Study 2 data investigated in Packer et al. (2015) leads to conclusions completely opposite to those drawn by Packer et al., and completely opposite to the conclusions drawn by van Hagen. I come to the conclusion that the Packer et al. study 2 badly needs updating with a much larger replication study.

A very important question is just how generalisable are the results of those papers. There is no word on that issue in van Hagen (2019). I will start by discussing the paper which is most relevant to our question: Packer et al. (2015).

An important preparatory remark should be made concerning the term “BOAS”, brachycephalic obstructive airway syndrome. It is a syndrome, which means: a name for some associated characteristics. “Obstructed airways” means: difficulty in breathing. “Brachycephalic” means: having a (relatively) short muzzle. Having difficulty in breathing is a symptom sometimes caused by having obstructed airways; it is certainly the case that the medical condition is often associated with having a short muzzle. That does not mean that having a short muzzle causes the medical condition. In the past, dog breeders have selected dogs with a view to accentuating certain features, such as a short muzzle: unfortunately, at the same time, they have sometimes selected dogs with other, less favourable characteristics at the same time. The two features of dogs’ anatomies are associated, but one is not the cause of the other. “BOAS” really means: having obstructed airways and a short muzzle.

Packer et al. (2015): an exploratory and flawed paper

Packer et al. (2015) reports findings from two studies. The sample for the first study, “Study 1”, 700 animals, consisted of almost all dogs referred to the Royal Veterinary College Small Animal Referral Hospital (RVC-SAH) in a certain period in 2012. Exclusions were based on a small list of sensible criteria such as the dog being too sick to be moved or too aggressive to be handled. However, this is not the end of the story. In the next stage, those dogs who actually were diagnosed to have BOAS (brachycephalic obstructive airway syndrome) were singled out, together with all dogs whose owners reported respiratory difficulties, except when such difficulties could be explained by respiratory or cardiac disorders. This resulted in a small group of only 70 dogs considered by the researchers to have BOAS, and it involved dogs of 12 breeds only. Finally, all the other dogs of those breeds were added to the 70, ending up with 152 dogs of 13 (!) breeds. (The paper contains many other instances of carelessness).

To continue with the Packer et al. (2015) Study 1 reduced sample of 152 dogs, this sample is a sample of dogs with health problems so serious that they are referred to a specialist veterinary hospital. One might find a relation between BOAS and CFR (craniofacial ratio) in that special population which is not the same as the relation in general. Moreover, the overall risk of BOAS in this special population is by its construction higher than in general. Breeders of pedigree dogs generally exclude already sick dogs from their breeding programmes.

That first study was justly characterised by the authors as exploratory. They had originally used the big sample of 700 dogs for a quite different investigation, Packer et al. (2013). It is exploratory in the sense that they investigated a number of possible risk factors for BOAS besides CFR, and actually used the study to choose CFR as appearing to be the most influential risk factor, when each is taken on its own, according to a certain statistical analysis method, in which already a large number of prior assumptions had been built in. As I will repeat a few more times, the sample is too small to check those assumptions. I do not know if they also tried various simple transformations of the risk factors. Who knows, maybe the logarithm of a different variable would have done better than CFR.

In the second study (“Study 2”), they sampled anew, this time recruiting animals directly mainly from breeders but also from general practice. A critical selection criterium was a CFR smaller than 0.5, that number being the biggest CFR of a dog with BOAS from Study 1. They especially targeted breeders of breeds with low CFR, especially those which had been poorly represented in the first study. Apparently, the Affenpinscher and Griffon Bruxellois are not often so sick that they get referred to the RVC-SAH; of the 700 dogs entering Study 1, there was, for instance, just 1 Affenpinscher and only 2 Griffon Bruxellois. Of course, these are also relatively rare breeds. Anyway, in Study 2, those numbers became 31 and 20. So: the second study population is not so badly biased towards sick animals as the first. Unfortunately, the sample is much, much smaller, and per breed, very small indeed, despite the augmentation of rarer breeds.

Figure 1: Figure 2 from Packer et al. (2015). Predicted probability of brachycephalic dog breeds being affected by brachycephalic obstructive airway syndrome (BOAS) across relevant craniofacial ratio (CFR) and neck girth ranges. The risks across the CFR spectrum are calculated by breed using GLMM equations based on (a) Study 1 referral population data and (b) Study 2 non-referral population data. For each breed, the estimates are only plotted within the CFR ranges observed in the study populations. Dotted lines show breeds represented by <10 individuals. The breed mean neck girth is used for each breed (as stated in Table 2). In (b), the body condition score (BCS) = 5 (ideal bodyweight) and neuter status = neutered

Now it is important to turn to technical comments concerning what perhaps seems to speak most clearly to the non-statistically schooled reader, namely, Figure 2 of Packer et al., which I reproduce here, together with the figure’s original caption.

In the abstract of their paper, they write “we show […] that BOAS risk increases sharply in a non-linear manner”. They do no such thing! They assume that the log odds of BOAS risk , that is: log(p/(1 – p)), depends exactly linearly on CFR and moreover with the same slope for all breeds. The small size of these studies forced them to make such an assumption. It is a conventional “convenience” assumption. Indeed, this is an exploratory analysis, moreover, the authors’ declared aim was to come up with a single risk factor for BOAS. They were forced to extrapolate from breeds which are represented in larger numbers to breeds of which they had seen many less animals. They use the whole sample to estimate just one number, namely the slope of log(p/(1 – p)) as an assumed linear function of CFR. Each small group of animals of each breed then moves that linear function up or down, which corresponds to moving the curves to the right or to the left. Those are not findings of the paper. They are conventional model assumptions imposed by the authors from the start for statistical convenience and statistical necessity and completely in tune with their motivations.

One indeed sees in the graphs that all those beautiful curves are essentially segments of the same curve, shifted horizontally. This has not been shown in the paper to be true. It was assumed by the authors of the paper to be true. Apparently, that assumption worked better for CFR than for the other possible criteria which they considered: that was demonstrated by the exploratory (the author’s own characterisation!) Study 1. When one goes from Study 1 to Study 2, the curves shift a bit: it is definitely a different population now.

There are strange features in the colour codes. Breeds which should be there are missing, and breeds which shouldn’t be there are. The authors have exchanged graphs (a) and (b)! This can be seen by comparing the minimum and maximum predicted risks from their Table 2.

Notice that these curves represent predictions for neutered dogs with breed mean neck girth, breed ideal body condition score (breed ideal body weight). I don’t know whose definition of ideal is being used here. The graphs are not graphs of probabilities for dog breeds, but model predictions for particular classes of dogs of various breeds. They depend strongly on whether or not the model assumptions are correct. The authors did not (and could not) check the model assumptions: the sample sizes are much too small.

By the way, breeders’ dogs are generally not neutered. Still, one-third of the dogs in the sample were neutered, so the “baseline” does represent a lot of animals. Notice that there is no indication whatsoever of statistical uncertainty in those graphics. The authors apparently did not find it necessary to add error bars or confidence bands to their plots. Had they done so, the pictures would have given a very, very different impression.

In their discussion, the authors write “Our results confirm that brachycephaly is a risk factor for BOAS and for the first time quantitatively demonstrate that more extreme brachycephalic conformations are at higher risk of BOAS than more moderate morphologies; BOAS risk increases sharply in a non-linear manner as relative muzzle length shortens”. I disagree strongly with their appraisal. The vaunted non-linearity was just a conventional and convenience (untested) assumption of linearity in the much more sensible log-odds scale. They did not test this assumption and most importantly, they did not test whether it held for each breed considered separately. They could not do that, because both of their studies were much, much too small. Notice that they themselves write, “we found some exceptional individuals that were unaffected by BOAS despite extreme brachycephaly” and it is clear that these exceptions were found in specific breeds. But they do not tell us which.

They also tell us that other predictors are important next to CFR. Once CFR and breed have been taken into account (in the way that they take it into account!), neck girth (NG) becomes very important.

They also write, “if society wanted to eliminate BOAS from the domestic dog population entirely then based on these data a quantitative limit of CFR no less than 0.5 would need to be imposed”. They point out that it is unlikely that society would accept this, and moreover, it would destroy many breeds which do not have problems with BOAS at all! They mention, “several approaches could be used towards breeding towards more moderate, lower-risk morphologies, each of which may have strengths and weaknesses and may be differentially supported by stakeholders involved in this issue”.

This paper definitely does not support imposing a single simple criterion for all dog breeds, much as its authors might have initially hoped that CFR could supply such a criterion.

In a separate section, I will test their model assumptions, and investigate the statistical reliability of their findings.

Liu et al. (2017): an excellent study, but of only three breeds

Now I turn to the other key paper, Liu et al. (2017). In this 8-author paper, the last and senior author, Jane Ladlow, is a very well-known authority in the field. This paper is based on a study involving 604 dogs of only three breeds, and those are the three breeds which are already known to be most severely affected by BOAS: bulldogs, French bulldogs, and pugs. They use a similar statistical methodology to Packer et al., but now they allow each breed to have a different shaped dependence on CFR. Interestingly, the effects of CFR on BOAS risk for pugs, bulldogs and French bulldogs are not statistically significant. Whether or not they are the same across those three breeds becomes, from the statistical point of view, an academic question.

The statistical competence and sophistication of this group of authors can be seen at a glance to be immeasurably higher than that of the group of authors of Packer et al. They do include indications of statistical uncertainty in their graphical illustrations. They state, “in our study with large numbers of dogs of the three breeds, we obtained supportive data on NGR (neck girth ratio: neck girth/chest girth), but only a weak association of BOAS status with CFR in a single breed.” Of course, part of that could be due to the fact that, in their study, CFR did not vary much within each of those three breeds, as they themselves point out. I did not yet re-analyse their data to check this. CFR was certainly highly variable in these three breeds in both of Packer et al.’s studies, see the figures above, and again in Liu et al. as is apparent from my Figure 2 below. But Liu et al. also point out that anyway, “anatomically, the CFR measurement cannot determine the main internal BOAS lesions along the upper airway”.

Another of their concluding remarks is the rather interesting “overall, the conformational and external factors as measured here contribute less than 50% of the variance that is seen in BOAS”. In other words, BOAS is not very well predicted by these shape factors. They conclude, “breeding toward [my emphasis] extreme brachycephalic features should be strictly avoided”. I should hope that nowadays, no recognised breeders deliberately try to make known risk features even more pronounced.

Liu et al. studied only bulldogs, French bulldogs and pugs. The CFRs of these breeds do show within breed statistical variation. The study showed that a different anatomical measure was an excellent predictor of BOAS. Liu et al. moreover explain anatomically and medically why one should not expect CFR to be relevant for the health problems of those races of dogs.

It is absolutely not true that almost all of the animals in that study have BOAS. The study does not investigate BOS. The study was set up in order to investigate the exploratory findings and hypotheses of Packer et al. and it rejects them, as far as the three races they considered were concerned. Packer et al. hoped to find a simple relationship between CFR and BOAS for all brachycephalic dogs but their two studies are both much too small to verify their assumptions. Liu et al. show that for the three races studied, the relationship between measurements of body structure and ill health associated with them, varies between races.

Figure 2: Supplementary material Fig S1 from Liu et al. (2017.) Boxplots show the distribution of the five conformation ratios against BOAS functional grades. The x-axis is BOAS functional grade; the y-axis is the ratios in percentage. CFR, craniofacial ratio; EWR, eye with ratio; SI, skull index; NGR, neck girth ratio; NLR, neck length ratio.

In contradiction to the opinion of van Hagen (2019), there are no “contradictions” between the studies of Packer et al. and Liu et al. The first comes up with some guesses, based on tiny samples from each breed. The second investigates those guesses but discovers that they are wrong for the three races most afflicted with BOAS. Study 1 of Packer et al. is a study of sick animals, but Study 2 is a study of animals from the general population. Liu et al. is a study of animals from the general population. (To complicate matters, Njikam et al., Packer et al. and Liu et al. all use slightly different definitions or categorisations of BOAS.)

Njikam et al. (2009), like the later researchers in the field, fit logistic regression models. They exhibit various associations between illness and risk factors per breed. They do not quantify brachycephaly by CFR but by a similar measure, BRA, the ratio of width to length of the skull. CFR and BRA are approximately non-linear one-to-one functions of one another (this would be exact if skull length equalled skull width plus muzzle length, i.e., assuming a spherical cranium), so a threshold criterium in terms of one can be roughly translated into a threshold criterium in terms of the other. Their samples are again, unfortunately, very small (the title of their paper is very misleading).

Their main interest is in genetic factors associated with BOAS apart from the genetic factors behind CFR, and indeed they find such factors! In other words, this study shows that BOAS is very complex. Its causes are multifactorial. They have no data at all on the breeds of primary interest to SRR: these breeds are not much afflicted by BOAS! It seems that van Hagen again has a reading of Njikam et al. which is not justified by that paper’s content.

Packer et al. (2015) Study. 2, revisited

Fortunately, the data sets used by the publications in PLoS ONE are available as “supplementary material” on the journal’s web pages. First of all, I would like to show a rather simple statistical graphic which shows that the relation between BOAS and CFR in Packer et al.’s Study 2 data does not look at all as the authors hypothesized. First, here are the numbers: a table of numbers of animals with and without BOAS in groups split according to CFR as a percentage, in steps of 5%. The authors recruited animals mainly from breeders, with CFR less than 50%. It seems there were none in their sample with a CFR between 45% and 50%.

BOAS versus CFR group

Table 1: BOAS versus CFR group

This next figure is a simple “pyramid plot” of percentages with and without BOAS per CFR group. I am not taking into account the breed of these dogs, nor of other possible explanatory factors. However, as we will see, the suggestion given by the plot seems to be confirmed by more sophisticated analyses. And that suggestion is: BOAS has a roughly constant incidence of about 20% among dogs with a CFR between 20% and 45%. Below that level, BOAS incidence increases more or less linearly as CFR further decreases.

Be aware that the sample sizes on which these percentages are based are very, very small.

Figure 3: Pyramid plot, data from Packer et al. Study 2

Could it be that the pattern shown in Figure 3 is caused by other important characteristics of the dogs, in particular, breed? In order to investigate this question, I, first of all, fitted a linear logistic regression model with only CFR, and then a smooth logistic regression model with only CFR. In the latter, the effect of CFR on BOAS is allowed to be any smooth function of CFR – not a function of a particular shape. The two fitted curves are seen in Figure 4. The solid line is the smooth, the dashed line is the fitted logistic curve.

Figure 4. BOAS vs CFR, linear logistic regression and smooth logistic regression

This analysis confirms the impression of the pyramid plot. However, the next results which I obtained were dramatic. I added to the smooth model also Breed and Neutered-status, and also investigated some of the other variables which turned up in the papers I have cited. It turned out that “Breed” is not a useful explanatory factor. CFR is hardly significant. Possibly, just one particular breed is important: the Pug. The differences between the others are negligible (once we have taken account of CFR). The variable “neutered” remains somewhat important.

Here (Table 2) is the best model which I found. As far as I can see, the Pug is a rather different animal from all the others. On the logistic scale, even taking account of CFR, Neckgirth and Neuter status, being a Pug increases the log odds ratio for BOAS by 2.5. Below a CFR of 20%, each 5% decrease in CFR increases the log odds ratio for BOAS by 1, so is associated with an increase in incidence by a factor of close to 3. In the appendix can be seen what happens when we allow each breed to have its own effect. We can no longer separate the influence of Breed from CFR and we cannot say anything about any individual breeds, except for one.

 Model 1 
(CFRpct – 20) * (CFRpct < 20)–0.20***(0.05)
Breed == “Pug”:TRUE2.48***(0.71)
Log Likelihood–67.09 
Num. obs.154 
*** p < 0.001; ** p < 0.01; * p < 0.05  
Table 2: A very simple model (GLM, logistic regression)

The pug is in a bad way. But we knew that before. Packer Study 2 data:

 W.out BOASWith BOAS
Not Pug9230
Table 3: The Pug almost always has BOAS. The majority of non-Pugs don’t.

The graphs of Packer et al. in Figure 1 are a fantasy. Reanalysis of their data shows that their model assumptions are wrong. We already knew that BOAS incidence, Breed, and CFR are closely related and naturally they see that again in their data. But the actual possibly Breed-wise relation between CFR and BOAS is completely different from what their fitted model suggests. In fact, the relation between CFR and BOAS seems to be much the same for all breeds, except possibly for the Pug.

Final remarks

The paper Packer et al. (2015) is rightly described by its authors as exploratory. This means: it generates interesting suggestions for further research. The later paper by Liu et al. (2017) is excellent follow-up research. It follows up on the suggestions of Packer et al., but in fact it does not find confirmation of their hypotheses. On the contrary, it gives strong evidence that they were false. Unfortunately, it only studies three breeds, and those breeds are breeds where we already know action should be taken. But already on the basis of a study of just those three breeds, it comes out strongly against taking one single simple criterion, the same for all breeds, as the basis for legislation on breeding.

Further research based on a reanalysis of the data of Packer et al. (2015) shows that the main assumptions of those authors were wrong and that, had they made more reasonable assumptions, completely different conclusions would have been drawn from their study.

The conclusion to be drawn from the works I have discussed is that it is unreasonable to suppose that a single simple criterion, the same for all breeds, can be a sound basis for legislation on breeding. Packer et al. clearly hoped to find support for this but failed: Liu et al. scuppered that dream. Reanalysis of their data with more sophisticated statistical tools shows that they should already have seen that they were betting on the wrong horse.

Below a CFR of 20%, a further decrease in CFR is associated with a higher incidence of BOAS. There is not enough data on every breed to see if this relationship is the same for all breeds. For Pugs, things are much worse. For some breeds, it might not be so bad.

Study 2 of Packer et al. (2015) needs to be replicated, with much larger sample sizes.


van Hagen MAE (2019) Fokken met Kortsnuitige Honden. Criteria ter handhaving van art. 3.4. Besluit Houders van dieren Fokken met Gezelschapsdieren. Departement Dier in Wetenschap en Maatschappij en het Expertisecentrum Genetica Gezelschapsdieren, Universiteit Utrecht.; English translation:

Liu N-C, Troconis EL, Kalmar L, Price DJ, Wright HE, Adams VJ, Sargan DR, Ladlow JF (2017) Conformational risk factors of brachycephalic obstructive airway syndrome (BOAS) in pugs, French bulldogs, and bulldogs. PLoS ONE 12 (8): e0181928.

Njikam IN, Huault M, Pirson V, Detilleux J (2009) The influence of phylogenic origin on the occurrence of brachycephalic airway obstruction syndrome in a large retrospective study. International Journal of Applied Research in Veterinary Medicine 7(3) 138–143.

Packer RMA, Hendricks A, Volk HA, Shihab NK, Burn CC (2013) How Long and Low Can You Go? Effect of Conformation on the Risk of Thoracolumbar Intervertebral Disc Extrusion in Domestic Dogs. PLoS ONE 8 (7): e69650.

Packer RMA, Hendricks A, Tivers MS, Burn CC (2015) Impact of Facial Conformation on Canine Health: Brachycephalic Obstructive Airway Syndrome. PLoS ONE 10 (10): e0137496.

Appendix: what happens when we try to separate“Breed” from “CFR”

 Model 2 
Breed:American Bulldog–43.51(67108864.00)
Breed:Boston Terrier0.35(1.84)
Breed:Cavalier King Charles Spaniel0.82(1.37)
Breed:Dogue de Bordeaux–43.35(67108864.00)
Breed:French Bulldog2.36(1.59)
Breed:Griffon Bruxellois–0.97(1.18)
Breed:Japanese Chin1.70(1.46)
Breed:Lhasa Apso1.75(1.63)
Breed:Mastiff cross1.97(2.60)
Breed:Pug cross–44.79(47453132.81)
Breed:Shih Tzu0.16(1.23)
Breed:Staffordshire Bull Terrier–43.37(47453132.81)
Breed:Staffordshire Bull Terrier Cross2.36(2.07)
Breed:Tibetan Spaniel–44.14(67108864.00)
Breed:Victorian Bulldog–43.16(67108864.00)
EDF: s(CFRpct)1.00(1.00)
Log Likelihood–53.29 
Deviance explained0.48 
GCV score0.03 
Num. obs.154 
Num. smooth terms1 
*** p < 0.001; ** p < 0.01; * p < 0.05  
Table 4: A more complex model (GAM, logistic regression)

The above model (Table 4) allowing each breed to have its own separate “fixed” effect is not a success. That certainly was presumably the motivation to make “Breed” a random, not a fixed, effect in the Packer et al. publication, because treating breed effects as drawn from a normal distribution and assuming the same effect of CFR for all breeds disguises the multicollinearity and lack of information in the data. Many breeds, most of them contributing only one or two animals, enabled the authors’ statistical software to compute an overall estimate of “variability between breeds” but the result is pretty meaningless.

Further inspection shows that many breeds are only represented by 1or 2 animals in the study. Only five are in something a bit like reasonable numbers. These five are the Affenpinscher, Cavalier King Charles Spaniel, Griffon Bruxellois, Japanese Chin and Pug; in numbers 31, 11, 20, 10, 32. I fitted a GLM (logistic regression) trying to explain BOAS in these 105 animals and their breed together with variables CFR, BCR, and so on. Still then, the multicollinearity between all these variables is so strong that the best model did not include CFR at all. In fact: once BCS (Body Condition Score) was included, no other variable could be added without almost everything becoming statistically insignificant. Not surprisingly, it is good to have a good BCS. Being a Pug or a Japanese Chin is disastrous. Cavalier King Charles Spaniel is intermediate. Affenpinscher and Griffon Bruxellois have the least BOAS (and about the same amount, namely an incidence of 10%), even though the mean CFRs of these two species seem somewhat different (0.25, 0.15).

Had the authors presented p-values and error bars the paper would probably never have been published. The study should be repeated with a sample 10 times larger.


This work was partly funded by “Stichting Ras en Recht” (SRR; Foundation Justice for Pedigree dogs). The author accepted the commission by SSR to review statistical aspects of MAE van Hagen’s report “Breeding of short-muzzled dogs” under the condition that he would report his honest professional and scientific opinion on van Hagen’s literature study and its sources.

Repeated measurements with unintended feedback: The Dutch New Herring scandals

Fengnan Gao and Richard D. Gill; 24 July 2022

Note: the present post reproduces the text of our new preprint, adding some juicy pictures. Further editing is planned, much reducing the length of this blog-post version of our story.

Summary: We analyse data from the final two years of a long-running and influential annual Dutch survey of the quality of Dutch New Herring served in large samples of consumer outlets. The data was compiled and analysed by Tilburg University econometrician Ben Vollaard, and his findings were publicized in national and international media. This led to the cessation of the survey amid allegations of bias due to a conflict of interest on the part of the leader of the herring tasting team. The survey organizers responded with accusations of failure of scientific integrity. Vollaard was acquitted of wrongdoing by the Dutch authority, whose inquiry nonetheless concluded that further research was needed. We reconstitute the data and uncover important features which throw new light on Vollaard’s findings, focussing on the issue of correlation versus causality: the sample is definitely not a random sample. Taking into account both newly discovered data features and the sampling mechanism, we conclude that there is no evidence of biased evaluation, despite the econometrician’s renewed insistence on his claim.

Keywords: Data generation mechanism, Predator-prey cycles, Feedback in sampling and measurement, Consumer surveys, Causality versus correlation, Questionable research practices, Unhealthy research stimuli., ©


In surveys intended to help consumers by regularly publishing comparisons of a particular product obtained from different consumer outlets (think of British “fish and chips” bought in a large sample of restaurants and pubs), data is often collected over a number of years and evaluated each year by a panel, which might consist of a few experts, but might also consist of a larger number of ordinary consumers. As time goes by, outlets learn what properties are most valued by the panel, and may modify their product accordingly. Also, consumers learn from the published rankings. Panels are renewed, and new members presumably learn from the past about how they are supposed to weight the different features of a product. Partly due to negative reviews, some outlets go out of business, while new outlets enter the market, and imitate the products of the “winners” of previous years’ surveys. Coming out as “best” boosts sales; coming out as “worst” can be the kiss of death.

For many years, a popular Dutch newspaper (Algemene Dagblad, in the sequel AD) published two immensely influential annual surveys of two particularly popular and typically Dutch seasonal products: the Dutch New Herring (Dutch: Hollandse Nieuwe) in June, and the Dutch “oliebol” (a kind of greasy, currant-studded, deep-fried spherical doughnut) in December. This paper will study the data published by the newspaper on its website of 2016 and 2017—the last two years of the 36 years in which the AD herring test operated. This data included not only a ranking of all participating outlets and their final scores (on a scale of 0 to 10) but also numerical and qualitative evaluations of many features of the product being offered. A position in the top ten was highly coveted. Being in the bottom ten was a disaster.

For a while, rumours had been circulated (possibly by disappointed victims of low scores!) that both tests were biased. The herring test was carried out by a team of three tasters, whose leader Aad Taal was indeed consultant to a wholesale company called Atlantic (based in Scheveningen, in the same region as Rotterdam), and who offered a popular course on herring preparation. As a director at the Dutch ministry of agriculture he had earlier successfully managed to obtain the European Union (EU) legal protection for the official designation “Dutch New Herring”. Products may only be sold under this name in the whole EU only if meticulously prepared in the circumscribed traditional way, as well as satisfying strict rules of food safety. It is nowadays indeed sold in several countries adjacent to the Netherlands. We will later add some crucial further information about what actually makes a Dutch New Herring different from the traditionally prepared herring of other countries.

Enter econometrician Dr Ben Vollaard of Tilburg University. Himself partial to a tasty Dutch New Herring, he learnt in 2017 from his local fishmonger about the complaints then circulating about the AD Herring Test. The AD is based on the city of Rotterdam, close to the main home ports of the Dutch herring fleet in past centuries. Tilburg is somewhat inland. Not surprisingly, consumers in different regions of the country seem to have developed different tastes in Dutch New Herring, and a common complaint was that the AD herring testers had a Rotterdam bias.

Vollaard decided to investigate the matter scientifically. A student helped him to manually download the data published on their website on 144 participating outlets in 2016, and 148 in 2017. An undisclosed number of outlets participated in both years, and initial reports suggested it must be a large number. Later we discovered that the overlap consisted of only 23 outlets. Next, he ran a linear regression analysis, attempting to predict the published final score for each outlet in each year, using as explanatory variables the testing team’s evaluations of the herring according to various criteria such as ripeness and cleaning, together with numerical variables such as weight, price, temperature, and laboratory measurements of fat content and microbiological contamination. Most of the numerical variables were modelled by using dummy variables after discretization into a few categories. A single indicator variable for “distance from Rotterdam’’ (greater than 30 kilometres) was used to test for regional bias.

The analysis satisfyingly showed many highly significant effects, most of which are exactly those that should have been expected. The testing team gave a high final score to fish which had a high fat content, low temperature, well-cleaned, and a little matured (not too little, not too much). More expensive and heavier fish scored better, too. Being more than 30 km from Rotterdam had a just significant negative effect, lowering the final score by about 0.5. Given the supreme importance of getting the highest possible score, 10, a loss of half a point could make a huge difference to a new outlet going all out for a top score and hence position in the “top ten” of the resulting ranking. However, just because outlets in the sample far from Rotterdam performed a little worse on average than those close to Rotterdam, can have many innocent explanations.

But Vollaard went a lot further. After comparing the actual scores to linear regression model predicted scores based on the measured characteristics of the herring, Vollaard concluded:

Everything indicates that herring sales points in Rotterdam and the surrounding area receive a higher score in the AD Herring Test than can be explained from the quality of the herring served.

That is a pretty serious allegation.

Vollaard published this analysis as a scientific paper Vollaard (2017a) on his university personal web page, and the university put out a press release. The research drew a lot of media attention. In the ensuing transition from a more or less academic study (in fact, originally just a student exercise) to a press release put out by a university publicity department, then to journalists’ newspaper articles adorned with headlines composed by desk editors, the conclusion became even more damning.

Presumably stimulated by the publicity that his work had received, Vollaard decided to go further, now following up on further criticism circulating about the AD Herring Test. He rapidly published a second analysis, Vollaard (2017b), on his university personal web page. His focus was now on the question of a conflict of interest concerning a connection between the chief herring tester and the wholesale outlet Atlantic. Presumably by contacting outlets directly, he identified 20 outlets in the sample whose herring, he believed, had been supplied by that company. Certainly, his presumed Atlantic herring outlets tended to have rather good final scores, and a few of them were regularly in the top ten.

We may surmise that Vollaard must have been disappointed and surprised to discover that his dummy variable for being supplied by Atlantic was not statistically significant when he added it to his model. His existing model (the one on the basis of which he argued that the testing team was not evaluating outlets far from Rotterdam using their own measured characteristics) predicted that Atlantic outlets should indeed, according to those characteristics, have come out exactly as well as they did! He had to come up with something different. In his second paper, he insinuated pro-Atlantic bias by comparing the amount of variance explained by what he considered to be “subjective” variables with the amount explained by the “objective” variables, and he showed that the subjective (taste and smell, visual impression) evaluations explained just as much of the variance as the objective evaluations (price, temperature, fat percentage). This change of tune represents a serious inconsistency in thinking: this is cherry-picking in order to support a pre-gone conclusion.

In itself, it does not seem unreasonable to judge a culinary delicacy by taste and smell, and not unreasonable to rely on reports of connoisseurs. However, Vollaard went much further. He hypothesized that “ripeness” and “microbiological state” were both measurements of the same variable; one subjective, the other objective. According to him, they both say how much the fish was “going off”. Since the former variable was extremely important in his model, the latter not much at all, he again accused the herring testers of bias and attributed that bias to conflict of interest. His conclusion was:

A high place in the AD Herring Test is strongly related to purchasing herring from a supplier in which the test panel has a business interest. On a scale of 0 to 10, the final mark for fishmongers with this supplier is on average 3.6 points higher than for fishmongers with another supplier.

He followed that up with the statement:

Almost half of the large difference in average final grade between outlets with and without Atlantic as supplier can be explained by a subjective assessment by the test team of how well the herring has been cleaned (very good/good/moderate/poor) and of the degree of ripening of the herring (light/medium/strong/spoiled).

The implication is that the Atlantic outlets are being given an almost 2 point advantage based on a purely subjective evaluation of ripeness.

More media attention followed, Vollaard appeared on current affairs programs on Dutch national TV, his work was even reported in The Economist,

The AD defended itself and its herring testers by pointing out that the ripeness or maturity of a Dutch new herring, evaluated by taste and smell, reflects ongoing and initially highly desirable chemical processes (protein changing to fat, fat to oil, oil becoming rancid). Degree of microbiological activity, i.e., contamination with harmful bacteria, could be correlated with that, since dangerous bacterial activity will tend to increase with time once it has started, and both processes are speeded up if the herring is not kept cold enough, but it is of a completely different nature: biological, not chemical. It is caused by carelessness in various stages of preparation of the herring, insufficient cooling, and so on. It is obviously not desirable at all. AD also pointed out that one of the Atlantic outlets must have been missed, which actually in the first of the two years had scored very badly. This could be deduced from the numbers of those outlets, and the mean score of the Atlantic-supplied outlets, both reported by Vollaard in his papers.

The newspaper AD complained first to Vollaard and then to his university. With the help of lawyers, a complaint was filed with the Tilburg University committee for scientific integrity. The committee rejected the complaint, but the newspaper took it to the national level. Their lawyers hired the second author of this paper, Richard Gill (RDG), in the hope that he would support their claims. He requested Vollaard’s data-set and also requested that the outlets in the data-set be identified, since one major methodological complaint of his was that Vollaard had not taken account of possible autocorrelation by combining samples from two subsequent years, with presumably a large overlap, but without taking any account of this. Vollaard reluctantly supplied the data but declined to identify the outlets appearing twice or even inform us how many such outlets there were. With the help of AD however, it was possible to find them, and also locate many misclassified outlets. RDG wrote an expert opinion in which he argued that the statistical analysis did not support any allegations of bias or even unreliability of the herring test.

Vollaard had repeatedly stated that he was only investigating correlations, not establishing causality, but at the same time his published statements (quoted in the media), and his spoken statements on national TV, make it clear that he considered that his analysis results were damming evidence against the test. This seemed to RDG to be unprofessional, at the very least. RDG moreover identified much statistical amateurism. Vollaard analysed his data much as any econometrician might do: he had a data-set with a variable of interest and a number of explanatory variables, he ran a linear regression making numerous modelling choices without any motivation and without any model checking. He fit a completely standard linear regression model to two samples of Dutch new herring outlets, without any thought to the data generating mechanism. How were outlets selected to appear in the sample?

According to the AD, there were actually 29 Atlantic outlets in Vollaard’s combined sample. Note, there is some difficulty in determining this number. A given outlet may obtain some fish from Atlantic, some from other suppliers, and may change their suppliers over the course of a year. So the origin of the fish actually tasted by the test team cannot be determined with certainty. We see in Table 1 (according to AD), that Vollaard “caught” only about two thirds of the Atlantic outlets, and misclassified several more.

Atlantic by VollaardNot Atlantic by Vollaard
Atlantic by AD181129
Not Atlantic by AD2261263

Table 1: Atlantic- and not Atlantic-supplied outlets tested over two years as identified by Vollaard and the AD respectively.

At the national level, the LOWI (Landelijk Orgaan Wetenschappelijk Integriteit — the Dutch national organ for investigating complaints of violation of research integrity) re-affirmed the Tilburg University scientific integrity committee’s “not guilty” verdict. Vollaard was not deliberately trying to mislead. “Guilty” verdicts have an enormous impact and imply a finding, beyond a reasonable doubt, of gross research impropriety. This generally leads to termination of university employment contracts and to retraction of publications. They did agree that Vollaard’s analyses were substandard, and they recommended further research. RDG reached out to Vollaard suggesting collaboration, but he declined. After a while, Vollaard’s (still anonymized) data sets and statistical analysis scripts (written in the proprietary Stata language) were also published on his website Vollaard (2020a, 2020b). The data was actually in the form of Stata files; fortunately, it is nowadays possible to read such files in the open source and free R system. The known errors in the classification of Atlantic outlets were not corrected, despite AD’s request. The papers and the files are no longer on Vollaard’s webpages, and he still declines collaboration with us. We have made all documents and data available on our own webpages and on the GitHub page

RDG continued his re-analyses of the data and began the job of converting his expert opinion report (English translation: into a scientific paper. It seemed wise to go back to the original sources and this meant a difficult task of extracting data from the AD’s websites. Each year’s worth of data was moreover coded differently in the underlying HTML documents. At this point he was joined by the first author Fengnan Gao (FG) of the present paper who was able to automate the data scraping and cleaning procedures — a major task. Thus, we were able to replicate the whole data gathering and analysis process and this led to a number of surprises.

Before going into that, we will explain what is so special about Dutch New Herring, and then give a little more information about the variables measured in the AD Herring Test.

Dutch New Herring, ©

Every nation around the North Sea has traditional ways of preparing North Atlantic herring. For centuries, herring has been a staple diet of the masses. It is typically caught when the North Atlantic herring population comes together at its spawning grounds, one of them being in the Skagerak, between Norway and Denmark. Just once a year there is an opportunity for fishers to catch enormous quantities of a particular extremely nutritious fish, at the height of their physical condition, about to engage in an orgy of procreation. The fishers have to preserve their catch during a long journey back to their home base; and if the fish is going to be consumed by poor people throughout a long year, further means of conservation are required. Dutch, Danish, Norwegian, British and German herring fleets (and more) all compete (or competed) for the same fish; but what people in those countries eat varies from country to country. Traditional local methods of bringing ordinary food to the tables of ordinary folk become cultural icons, tourist attractions, gastronomic specialities, and export products.

Traditionally, the Dutch herring fleet brought in the first of the new herring catch in mid-June. The separate barrels in the very first catch are auctioned and a huge price (given to charity) is paid for the very first barrel. Very soon, fishmongers, from big companies with a chain of stores and restaurants, to supermarket chains, to small businesses selling fish in local shops and street markets are offering Dutch New Herring to their customers. It’s a traditional delicacy, and nowadays, thanks to refrigeration, it can be sold the whole year long (the designation “new” should be removed in September). Nowadays, the fish arrives in refrigerated lorries from Denmark, no longer in Dutch fishing boats at Scheveningen harbour.

What makes a Dutch new herring any different from the herring brought to other North Sea and Baltic Sea harbours? The organs of the fish should be removed when they were caught, and the fish kept in lightly salted water. But two internal organs are left, a fish’s equivalent to our pancreas and kidney. The fish’s pancreas contains enzymes which slowly transform some protein into fat and this process is responsible for a special almost creamy taste which is much treasured by Dutch consumers, as well as those in neighbouring countries. See, e.g., the Wikipedia entry for soused herring for more details, According to a story still told to Dutch schoolchildren, this process was discovered in the 14th century by a Dutch fisher named Willem Beukelszoon.

The AD Herring Test

© Marco de Swart (AD),

For many years, the Rotterdam-based newspaper Algemene Dagblad (AD) carried out an annual comparison of the quality of the product offered in a sample of consumer outlets. A small team of expert herring tasters paid surprise visits to the typical small fishmonger’s shops and market stalls where customers can order portions of fish and eat them on the premises (or even just standing in a busy food market). The team evaluated how well the fish has been prepared, preferring especially that the fish have not been cleaned in advance but that they are carefully and properly prepared in front of the client. They judged the taste and checked the temperature at which it is given to the customer: by law it may not be above 7 degrees. A sample was sent to a lab for a number of measurements: weight, fat percentage, signs of microbiological contamination. They are also interested in the price (per gram). An important, though subjective, characteristic is “ripeness”. Expert tasters distinguish Dutch new herring which has not ripened (matured) at all: green. After that comes lightly matured, well matured, too much matured, and eventually rotten.

This information was all written down and evaluated subjectively by each team member, then combined. The team averaged the scores given by its three members (a senior herring expert, a younger colleague, and a journalist) to produce a score from 0 to 10, where 10 is perfection; below 5.5 is a failing grade. However, it was not just a question of averaging. Outlets which sold fish which was definitely rotten, definitely contaminated with harmful bacteria, or definitely too warm got a zero grade. The outlets which took part were then ranked. The ten highest ranking outlets were visited again, and their scores possibly adjusted. The final ranking was published in the newspaper, and put in its entirety on internet. Coming out on top was like getting a Michelin star. The outlets at the bottom of the list might as well have closed down straight away. One sees from the histogram below, Figure [fig:1], that in 2016 and 2017, more than 40% of the outlets got a failing grade; almost 10% were essentially disqualified, by being given a grade of zero. The distribution looks nicely smooth except for the peak at zero, which really means that their wares did not satisfy minimal legal health requirements.

Figure 1: Final test scores, 2016 and 2017.

It is important to understand how outlets were chosen to enter the test. To begin with, the testing team itself automatically revisited last years’ top ten. But further outlets could be nominated by individual newspaper readers, indeed, they could be self-nominated by persons close to the outlets themselves. We are not dealing with a random sample, but with a self-selecting sample, with automatically a high overlap from year to year.

Over the years, there had been more and more acrimonious criticism of the AD Herring Test. As one can imagine, it was mainly the owners of outlets who had bad scores who were unhappy about the test. Many of them, perhaps justly, were proud of their product and had many satisfied customers too. Various accusations were therefore flung around. The most serious one was that the testing team was biased and even had a conflict of interest. The lead taster gave courses on the preparation of Dutch New Herring and led the movement to have the “brand” registered with the EU. There is no doubting his expertise, but he had been hired (in order to give training sessions to their clients) by one particular wholesale business, owned by a successful businessman of Turkish origin, which as one might imagine lead to jealousy and suspicion. Especially since a number of the retail outlets of fish supplied by that particular company often (but certainly not always) appeared year by year in the top ten of the annual AD Herring Test. Other accusations were that the herring tasters favoured businesses in the neighbourhood of Rotterdam (home base of the AD). As herring cognoscenti know, people in various Dutch localities have slightly different tastes in Dutch New Herring. Amsterdammers have a different taste from Rotterdammers.

In the meantime, under the deluge of negative publicity, the AD announced that they would now stop their annual herring test. They did hire a law company who on their behalf brought an accusation of failure of scientific integrity to Tilburg University’s “Commission for Scientific Integrity”. The law firm moreover approached one of us (RDG) for expert advice. He was initially extremely hesitant to be a hired gun in an attack on a fellow academic but as he got to understand the data and the analyses and the subject, he had to agree that the AD had some good points. At the same time, various aggrieved herring sellers were following up with their own civil action against the AD; and the wholesaler whose outlets did so well in the test, also started a civil action against Tilburg University, since its own reputation was damaged by the affair.

Vollaard’s analyses

Here is the main result of Vollaard’s first report.

lm(formula = finalscore ~
                    weight + temp + fat + fresh + micro +
                    ripeness + cleaning + yr2017)
     Min      1Q  Median      3Q     Max
 -4.0611 -0.5993  0.0552  0.8095  3.9866

Residual standard error: 1.282 on 274 degrees of freedom
Multiple R-squared:  0.8268, Adjusted R-squared:  0.816
F-statistic: 76.92 on 17 and 274 DF,  p-value: < 2.2e-16

Estimate Std.Errort value Pr(>|t|) 
weight (grams)

< 7 deg0 (baseline)

7–10 deg –0.6859620.193448–3.5460.000460***

> 10 deg–1.7931390.223113–8.0372.77e–14***

< 10%0 (baseline)


> 14%0.5816020.2500332.3260.020743*
1.8170810.2003359.070< 2e–16***

very good0 (baseline)


bad–0.618397  0.448309  –1.379 0.168897–1.3790.168897



mild0 (baseline)



rotten–4.5987520.503490–9.134< 2e–16***

very good0 (baseline)



Regression model output

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ‘ 1

No surprises here. The testing team prefers fatty and larger herring, properly cooled, mildly matured, freshly prepared in view of customers on-site, and well-cleaned too. We have a delightful amount of statistical significance. There are some curious features of Vollaard’s chosen model: some numerical variables (“temp” and “fat”) have been converted into categorical variables by presumably arbitrary choice of cut-off points, while “weight” is taken as numerical. Presumably, this is because one might expect the effect of temperature not to be monotone. Nowadays, one might attempt fitting low-degree spline curves with few knots. Some categories of categorical variables have been merged, without explanation. One should worry about interactions and about additivity. Certainly one should worry about model fit.

We add to the estimated regression model also R’s standard four diagnostic plots in Fig. 2. Dr Vollaard apparently did not carry out any model checking.

Figure 2a. Model validation: panel 1, residuals versus fitted values
Figure 2b. Model validation: panel 2, normal QQ plot of standardized residuals
Figure 2c. Model validation: panel 3, square root of absolute value of standardized residuals against fitted value
Figure 2d. Model validation: panel 4, standardized residuals against leverage

Model validation beyond Vollaard’s regression analysis

There are some serious statistical issues. There seem to be a couple of serious outliers. The error distribution seems to have a heavier than normal tail. But we also understand that some observations come in pairs — the same outlet evaluated in two subsequent years. The data set has been anonymized too much. Each outlet should at the least have been given a random code so that one can identify the pairs and take account of possible dependence from one year to the next, easy to do by simply estimating the correlation from the residuals, and then doing a generalized linear regression with an estimated covariance matrix of the error terms.

Inspection of the outliers led us to realize that there is a serious issue with the observations which got a final score of zero. Those outlets were essentially disqualified on grounds of gross violation of basic hygiene laws, applied by looking at just a couple of the variables: temperature above 12 degrees (the legal limit is 7), and microbiological activity (dangerous versus low or none). The model should have been split into two parts: a linear regression model for the scores of the not-disqualified outlets; and a logistic regression model, perhaps, for predicting “disqualification” from some of the other characteristics. However, at least it is possible to analyse each of the years separately, and to remove the “disqualified” outlets. That is easy to do. Analysing just the 2017 data, the analysis results look a lot cleaner; the two bad outliers have gone, the estimated standard deviation of the errors is a lot smaller, the normal Q-Q plot looks very nice.

The data-set, now as comma-separated values files and Excel spreadsheets, and with outlets identified, can be found on our already mentioned GitHub repository new- herring.

The real problem

There is another big issue with this data and these analyses which needs to be mentioned, and if possible, addressed. How did the “sample” come to be what it is? A regression model is at best a descriptive account of the correlations in a given data set. Before we should accuse the test team of bias, we should ask how the sample is taken. It is certainly not a random sample from a well-defined population!

Some retail outlets took part in the AD Herring Test year after year. The testing team automatically included last years’ top ten. Individual readers of the newspaper could nominate their own favourite fish shop to be added to the “sample”, and this actually did happen on a big scale. Fish shops which did really badly tended to drop out of future tests and, indeed, some of them stopped doing business altogether:

The “sample” evolves in time by a feedback mechanism.

Everybody could know what the qualities were that the AD testers appreciated, and they learnt from their score and their evaluation each year what they had to do better next year, if they wanted to stay in the running and to join the leaders of the pack. The notion of “how a Dutch New Herring ought to taste”, as well as how it ought to be prepared, was year by year being imprinted by the AD test team on the membership of the sample. New sales outlets joined and competed by adapting themselves to the criteria and the tastes of the test team.

The same newspaper did another annual ranking of outlets of a New Year’s Dutch traditional delicacy, actually, a kind of doughnuts (though without a hole in the middle) called oliebollen. They are indeed somewhat stodgy and oily, roughly spherical, objects, enlivened with currants and sprinkled with icing sugar. The testing panel was able to taste these objects blind. It consisted of about twenty ordinary folk and every year, part of the panel resigned and was replaced with fresh persons. Peter Grünwald of Centrum Wiskunde & Informatica, the national research institute for mathematics and computer science in the Netherlands, developed a simulation model which showed how the panel’s taste in oliebollen would vary over the years, as sales outlets tried to imitate the winners of the previous year, while the notion of what constitutes a good oliebol was not fixed. Taking the underlying quality to be one-dimensional, he demonstrated the well-known predator-prey oscillations (Angerbjorn et al., 1999). Similar lines of thinking have appeared in the study of, e.g., fashion cycles, see e.g. Acerbi et al. (2012), where the authors propose a mechanism for individual actors to imitate other actors’ cultural traits and preferences for these traits such that realistic and cyclic rise-and-fall patterns (see their Figure 4) are observed in simulated settings. A later study, Apriasz et al. (2016), divides a society into two categories of “snobs” and “followers”, where followers copy everyone else and snobs only imitate the trend of their own and go against the followers. As a result, clear recurring cyclic patterns (see their Figures 3 and 4) similar to the predator-prey cycle arise under proper parameter regimes.

The AD was again engaged in a legal dispute with disgruntled owners of low ranked sales outlets, which eventually led to this annual test being abandoned too. In fact, the AD forbade Grünwald to publish his results. We have made some initial simulation studies of a model with higher dimensional latent quality characteristics, which seems to exhibit similar but more complex behaviour.

New analyses, new insights

It turns out that the correlation between the residuals of the same outlet participating in two subsequent years is large, about 0.7. However, their number (23) is fairly small, so this has little effect on Vollaard’s findings. Taking account of it slightly increases the standard errors of estimated coefficients. However, we also knew that according to AD, many outlets were incorrectly classified by Vollaard, and since he did not wish to collaborate with us, we returned to the source of his data: the web pages of AD. This enabled us to play with the various data coding choices made by Vollaard and to try out various natural alternative model specifications. As well as this, we could use the list of outlets certified by AD and Atlantic as having actually supplied the Dutch new herring tested in 2016 and 2017.

First, it is clear from the known behaviour of the test team that a score of zero means something special. There is no reason to expect a linear model to be the right model for all participating outlets. The outlets which were given a zero score were essentially disqualified on objective public health criteria, namely temperature above 12 degrees and definitely dangerous microbiological activity. We decided to re-analyse the data while leaving out all disqualified outlets.

Next, there is the issue of correlation between outlets appearing in two subsequent years. Actually, this turned out to be a much smaller proportion than expected. So correction for autocorrelations hardly makes a difference, but on the other hand, it is easily made superfluous by dropping all outlets appearing for the second year in succession. Now we have two years of data, in the second year only of “newly nominated” outlets.

Going back to the original data published by AD, we discovered that Vollaard had made some adjustments to the published final scores. As was known, the testing team revisited the top ten scoring outlets, and ranked their product again, recording (in one of the two years) scores like 9.1, 9.2, … up to 10, in order to resolve ties. In both years there were scores registered such as  8– or 8+, meant to indicate “nearly an 8” or “a really good 8”, following Dutch traditional school and university test and exam grading. The scores “5″, “6″, “7″, “8”, “9”, “10” have familiar and conventional descriptions “unsatisfactory” or insufficient, “satisfactory” or sufficient, “good”, “very good, “excellent”. Linear regression analysis requires a numerical variable of interest. Vollaard had to convert “9–” (almost worthy of the qualification “very good”) into a number. It seems that he rounded it to 9, but one might just as well have made it 9–𝞮 for some choice of 𝞮, for instance, 𝞮 = 0.01,  0.03, or 0.1.

We compared the results obtained using various conventions for dealing with the “broken” grades, and it turned out that the choice of value of 𝞮 had major impact on the statistical significance of the “just significant” or “almost significant” variables of main interest (supplier; distance). Also, whether one followed standard strategies of model selection based on leaving out insignificant variables has a major impact on the significance of the variables of most interest (distance from Rotterdam; supplier). The size of their effects becomes a little smaller, standard errors remain large. Had Vollaard followed one of several common model selection strategies, he could have found that the effect of “Atlantic” was significant at the 5% level, supporting his prior opinion! As noted by experienced statistical practitioners such as Winship and Western (2016), in linear regression analysis where multicollinearity is present, the regression estimates are highly sensitive to small perturbations in model specification. In our data-set, what should be unimportant changes to which variables are included and which are not included, as well as unimportant changes in the quantification of the variable to be explained, keep changing the statistical significance of the variables which interested Vollaard the most — the results which led to a media circus, societal impact, and reputational damage to several big concerns, as well as to the personal reputation of the chief herring tester Aad Taal.

Having “cleaned” the data by removing the repeat tests, and removing the outlets breaking food safety regulations, and using the AD’s classification, the size of the effects of being an Atlantic-supplied outlet, and of being distant from Rotterdam, are smaller and hardly significant. By varying 𝞮, they change. On leaving out a few of the variables whose statistical significance is smallest, whether the two main variables of interest are significant changes again. The size of the effects remains about the same: Atlantic supplied outlets score a bit higher, outlets distant from Rotterdam score a bit lower, when taking account of all the other variables in the way chosen by the analyst.

By modelling the effects of so many variables by discretization, Vollaard created multicollinearity. The results depend on arbitrarily chosen cut-offs, and other arbitrary choices. For instance, “weight” was kept numerical, but “price” was made categorical. This could have been avoided by assuming additivity and smoothness and using modern statistical methodology, but in fact the data-set is simply too small for this to be meaningful. Trying to incorporate interaction between clearly important variables caused multicollinearity and failure of the standard estimation procedures. Different model selection procedures, and nonparametric approaches, end up with finding quite different models, but do not justify preferring one to another. We can come up with several excellent (and quite simple) predictors of the final score, but we cannot say anything about causality.

Vollaard’s analyses confirmed what we knew in advance (the “taste” of the testers). There is no reason whatsoever to accuse them of favouritism. The advantage of outlets supplied by Atlantic is tiny or non-existent, certainly nothing like the huge amount which Vollaard carelessly insinuated. The distant outlets are typically new entrants to the AD Herring Test. Their clients like the kind of Dutch new herring which they have been used to in their region. Vollaard’s interpretation of his own results obtained from his own data set was unjustified. He said he was only investigating correlations, but he appeared on national TV talk shows to say that his analyses made him believe that the AD Herring Test was severely biased. This caused enormous upset, financial and reputational damage, and led to a lot of money being spent on lawyers.

Everyone makes mistakes and what’s done is done, but we do all have a responsibility to learn from mistakes. The national committee for investigating accusations of violation of scientific integrity (LOWI) did not find Vollaard guilty of gross misdemeanour. They did recommend further statistical analysis. Vollaard declined to participate. No problem. We think that the statistical experiences reported here can provide valuable pedagogical material.


In our opinion, the suggestion that the AD Herring Test was in any way biased cannot be investigated by simple regression models. The “sample” is self-recruiting and much too small. The sales outlets which join the sample are doing so in the hope of getting the equivalent of a Michelin star. They can easily know in advance what are the standards by which they will be evaluated. Vollaard’s purely descriptive and correlational study confirms exactly what everyone (certainly everyone “in the business”) should know. The AD Herring Test, over the years that it operated, helped to raise standards of hygiene and presentation, and encouraged sales outlets to get hold of the best quality Dutch New Herring, and to prepare and serve it optimally. As far as subjective evaluations of taste are concerned, the test was indubitably somewhat biased toward the tastes valued by consumers in the region of Rotterdam and The Hague, and at the main “herring port” Scheveningen. But the “taste” of the herring testers was well known. Their final scores fairly represent their public, written evaluations, as far as can be determined from the available data.

The quality of the statistical analysis performed by Ben Vollaard left a great deal to be desired. To put it bluntly, from the statistical point of view it was highly amateurish. Economists who self-publish statistical reports under the flag of their university on matters of great public interest should have their work peer-reviewed and should rapidly publish their data sets. His results are extremely sensitive to minor variations in model choice and specification, to minor variations in quantifications of verbal scores, and there is not enough data to investigate his assumption of additivity. Any small effects found could as well be attributed to model misspecification as to conscious or unconscious bias on the part of the herring testers. We are reminded of Hanlon’s razor “never attribute to malice that which is adequately explained by stupidity”. In our opinion, in this case, Ben Vollaard was actually a victim of the currently huge pressure on academics to generate media interest by publishing on issues of current public interest. This leads to immature work which does not get sufficient peer review before being fed to the media. The results can cause immense damage.

Statisticians in general should not be afraid to join in societal debates. The total silence concerning this affair from the Dutch statistical society, which even has an econometric chapter, was a shame. Fortunately, the society has recently set up a new section devoted to public outreach.

A huge amount of statistical analyses are performed and published by amateurishly matching formal properties of a data-set (types of variables, the shape of the data file) to standard statistical models with no consideration at all given to model assumptions and to checks of model assumptions. Vollaard’s data-set can provide a valuable teaching resource, and we have published a version with (English language) description of the variables. We have made two versions available: Vollaard’s data-set put together by his student, but now with outlets identified, and the newly constituted data set with Atlantic-supplied outlets according to the AD, which is as well available in our GitHub repository

It would be interesting to add to the data some earlier years’ data, and investigate whether scores of repeatedly evaluated outlets tended to increase over the years. At the very least, it would be good to know which of the year 2016 outlets were repeat participants.

Just before we are about to submit this article, we become aware of Vollaard and van Ours (2021), in which Dr Ben Vollaard made the same accusations with essentially the same false arguments.

More study must be done of the feedback processes involved in consumer research panels. The man behind the herring test: journalist Paul Hovius (r), with herring taster Aad Taal (l), during the AD Herring Test in 2013. © Joost Hoving, ANP

Conflict of interest

The second author was paid by a well-known law firm for a statistical report on Vollaard’s analyses. His report, dated April 5, 2018, appeared in English translation earlier in this blog, He also reveals that the best Dutch New Herring he ever ate was at one of the retail outlets of Simonis in Scheveningen. They got their herring from the wholesaler Atlantic. He had this experience before any involvement in the Dutch New Herring scandals, topic of this paper.


Alberto Acerbi, Stefano Ghirlanda, and Magnus Enquist. The logic of fashion cycles. PloS one, 7(3):e32541, 2012.

Anders Angerbjorn, Magnus Tannerfeldt, and Sam Erlinge. Predator–prey relationships: arctic foxes and lemmings. Journal of  Animal Ecology, 68(1):34–49, 1999.

Rafał Apriasz, Tyll Krueger, Grzegorz Marcjasz, and Katarzyna Sznajd-Weron. The hunt opinion model—an agent based approach to recurring fashion cycles. PloS one, 11(11):e0166323, 2016.

The Economist. Netherlands fishmongers accuse herring-tasters of erring. The Economist, 2017, November 25.

Ben Vollaard. Gaat de AD Haringtest om meer dan de haring? 2017a.

Ben Vollaard. Gaat de AD Haringtest om meer dan de haring? een update. 2017b.

Ben Vollaard. Scores Haringtest. 2020a.

Ben Vollaard. Stata Code Haringtest. 2020b.

Ben Vollaard and Jan C van Ours. Bias in expert product reviews. 2021. Tinbergen Institute Discussion Paper 2021-042/V.

Christopher Winship and Bruce Western. Multicollinearity and model misspecification. Sociological Science, 3(27):627–649, 2016.

Bell’s theorem as a no-go result in classical distributed Monte-Carlo simulation

Abstract and slides of a talk to be given at the IMS conference in London, 27–30 June 2022,

It has long been realized that the mathematical core of Bell’s theorem is essentially a classical probabilistic proof that a certain distributed computing task is impossible: namely, the Monte Carlo simulation of certain iconic quantum correlations. I will present a new and simple proof of the theorem using Fourier methods (time series analysis) which should appeal to probabilists and statisticians. I call it Gull’s theorem since it was sketched in a conference talk many years ago by astrophysicist Steve Gull, but never published. Indeed, there was a gap in the proof.

The connection with the topic of this session [IS18 – Quantum Computing and Statistics – organiser Yazhen Wang, University of Wisconsin-Madison] is the following: though a useful quantum computer is perhaps still a dream, many believe that a useful quantum internet is very close indeed. The first application will be: creating shared secret random cryptographic keys which, due to the laws of physics, cannot possibly be known to any other agent. So-called loophole-free Bell experiments have already been used for this purpose. 

Like other proofs of Bell’s theorem, the proof concerns a thought experiment, and the thought experiment could also in principle be carried out in the lab. This connects to the concept of functional Bell inequalities, whose application in the quantum research lab has not yet been explored. This is again a task for classical statisticians to explore.
R.D. Gill (2022) Gull’s theorem revisited, Entropy 2022, 24(5), 679 (11pp.)

Het interview

Interview in “Stentor”, gepubliceerd vorige Zaterdag

Richard Gill heeft met het weerleggen van statistisch bewijs al twee medische seriemoordenaars vrij gekregen, onder wie Lucia de Berk. © Rob Voss

Deze Apeldoornse wetenschapper redt onschuldige zusters uit de gevangenis: De mens wil helaas niet in toeval geloven

Wetenschapper Richard Gill uit Apeldoorn zorgde er mede voor dat Lucia de Berk werd vrijgesproken. Datzelfde kreeg hij voor elkaar bij een vergelijkbare zaak in Italië en nu gaat hij voor de hattrick in Engeland. Wat drijft hem? 

Anne Boer 28-05-22, 08:00

Pure wetenschappelijke nieuwsgierigheid, dat is wat hem drijft, zegt de internationaal vermaarde wiskundige Richard Gill uit Apeldoorn. Als expert op het terrein van statistieken werkte Gill (70) voor het Openbaar Ministerie en het Internationaal Strafhof. Bijna zes jaar is hij gepensioneerd en staat hij te boek als emeritus professor in de statistiek aan de Universiteit Leiden.

Met zijn kennis over het gebruik van statistieken heeft hij vanuit zijn werkkamer de onschuld kunnen aantonen van twee verpleegkundigen die waren veroordeeld voor seriemoorden: de Nederlandse Lucia de Berk en de Italiaanse Daniela Poggiali. Nu zet hij zich in voor de vrijlating van verpleegkundige Ben Geen uit Engeland.

Klinkklare onzin

Alle drie zouden tijdens hun werk patiënten hebben gedood. Lucia de Berk werd zelfs veroordeeld voor zeven moorden. De bewijslast was vooral gebaseerd op statistieken. Als Lucia werkte, zouden er meer patiënten overlijden dan tijdens de diensten van haar collega’s. Het bleek klinkklare onzin, zoals Gill het fijntjes verwoordt. ,,Kwestie van roddel en achterklap, zoeken naar een zondebok om de reputatie van het ziekenhuis te redden en aannames, terwijl er helemaal geen moord is gepleegd.’’


De mens wil helaas niet in toeval geloven, we willen een oorzaak hebben. Daarom geloven we ook in duivels en goden

Statistisch bewijs speelt een grote rol in onderzoek, ook naar seriemoordenaars in de medische wereld. ,,Maar dan moet je de cijfers wel goed interpreteren’’, vindt Gill. ,,Als er ogenschijnlijk veel mensen overlijden in een ziekenhuis, moet je eerst goed kijken naar de oorzaak. Zijn er misschien meer patiënten dan anders? Zijn ze zieker dan in andere perioden? Is de methode van registreren aangepast? Zijn er wijzigingen in de staf? Als je meteen kijkt welke verpleegkundige aanwezig was, sla je bovendien de belangrijkste vragen over: is er sprake van moord of is het medisch falen of zelfs natuurlijk overlijden?’’

Dat raakt volgens Gill meteen aan een ander pijnlijk punt. ,,Een ziekenhuis is een plek waar mensen doodgaan, maar vaak is de doodsoorzaak niet duidelijk. Dat kan leiden tot clusters van verdachte overlijdens. Je moet wel weten welke doden je telt, anders zoekt de politie bewijs voor beweringen.’’


Volgens Gill moet je altijd in gedachten houden dat er een goede, onschuldige reden kan zijn voor een gebeurtenis. ,,Kijk vooral hoe vaak iemand werkt. Fulltime verpleegkundigen maken meer doden mee dan parttimers. Als iemand fulltime werkt en ook nog gepassioneerd bezig is met haar of zijn vak, is de kans nog groter dat die persoon aanwezig is als iemand overlijdt, dan iemand die een paar dagen per week werkt of strikt de uren werkt die in het rooster staan.”

Wetenschapper Richard Gill. © Rob Voss

Nooit mag je volgens hem een rare samenloop van omstandigheden uitsluiten. ,,Die gebeuren, ook zonder moord. Beroemd is het voorbeeld van een Amerikaans stel dat op één dag in twee verschillende loterijen de hoofdprijs won. Hoe groot is de kans dat zoiets gebeurt? Het gebeurde toch echt. De mens wil helaas niet in toeval geloven, we willen een oorzaak hebben en zoeken een zondebok. Daarom geloven we ook in duivels en goden.’’


Richard Gill is geboren in Engeland. Zijn vader was ook wetenschapper. De liefde brengt hem in 1974 op 23-jarige leeftijd naar Nederland. Hij is zes jaar eerder op vakantie als een blok gevallen voor een dochter van een Nederlandse vriend van zijn vader. Beide vaders werken voor Wavin uit Hardenberg. Na wat omzwervingen belandt Gill begin jaren 80 in Apeldoorn, om er nooit meer weg te gaan. Hij woont in een oud herenhuis, in een zee van weelderig groen. Dit was het ouderlijk huis van zijn vrouw. Om financieel het hoofd boven water te houden, werkt hij extra hard om snel carrière te kunnen maken.

De medische wereld komt al vroeg op zijn pad. Na een studie wiskunde in Cambridge promoveert hij op onderzoek naar de vraag hoelang kankerpatiënten bij bepaalde behandeling overleven. Zijn rekenmethode blijkt een uitkomst en wordt inmiddels ook op andere terreinen toegepast. ,,Het kwam toevallig op mijn bord. Ik had geen onderwerp en mijn promotor haalde dit onderwerp uit zijn la. Het heeft veel impact gehad en de methode wordt nog massaal gebruikt.’’


Zijn vrouw, die historicus is, wijst hem al in een vroeg stadium op de zaak Lucia de Berk, die later veroordeeld zou worden voor zeven moorden in een ziekenhuis. ,,Zij sprak van een heksenjacht en wilde dat ik ernaar keek, zeker toen het ook een heksenproces werd, zoals ze dat noemde. Ze wees me erop dat statistiek als bewijs werd gebruikt en ik er dus wel iets van zou moeten vinden. Ik wilde niet. Er waren al ervaren statistici bij betrokken, ook mensen die ik kende.’’

Lucia de Berk reageert blij na haar vrijspraak © anp

Toen er in 2006 een boek over deze zaak verscheen, ging Gill overstag. ,,Ik werd door een collega op het boek gewezen. Ik wist werkelijk niet wat ik las, was er echt ondersteboven van. Voor mij was zonneklaar dat het vonnis niet deugde en de rechters de cijfers verkeerd hadden geïnterpreteerd.’’ 

De rest is geschiedenis. Gill hielp aantonen dat de cijfers de beschuldiging niet konden onderbouwen en Lucia werd na 6,5 jaar onterechte celstraf in 2010 volledig vrijgesproken.


Als hij in 2014 over een gelijksoortige situatie in Italië leest, besluit hij direct weer in actie te komen. Dit keer wordt een verpleegkundige (Daniela Poggiali) verdacht van zestig moorden. Gill belt zijn collega Julia Mortera van de Roma Tre-universiteit en samen bieden ze hun hulp aan. Met succes, ook deze verpleegkundige is na een eerdere veroordeling tot levenslange gevangenisstraf sinds oktober op vrije voeten.

Statistiek is de wetenschap en de techniek van het verzamelen, bewerken, interpreteren en presenteren van gegevens. Statistische methoden worden gebruikt om grote hoeveelheden gegevens – bijvoorbeeld over het koopgedrag van mensen, de huizenmarkt of het aantal doden in de zorg – om te zetten in bruikbare informatie.

,,De statistiek in deze zaak was totaal amateuristisch, het deugde niet. De aanklagers beweerden dat er meer sterfgevallen waren als Daniela werkte. Tot het moment dat ze werd gearresteerd: toen daalde het plotseling. Wij ontdekten dat het sterftecijfer bij alle personeelsleden hoog was. Daniela was vaak al voor het begin van haar ingeroosterde dienst aanwezig en bleef vaak ook nog helpen nadat haar dienst voorbij was. Daardoor was ze vaker aanwezig als een patiënt stierf. Dat het aantal doden daalde nadat Daniela werd gearresteerd, is simpel te verklaren. Het nieuws over de ‘moordzuster’ was breed uitgemeten in de media. Als gevolg daarvan trok het ziekenhuis minder patiënten. Minder patiënten betekent ook minder sterfgevallen.’’

Lastige kluif

Gill doet nu onderzoek naar de zware beschuldigingen tegen de Engelse verpleegkundige Ben Geen. Dat gebeurt op verzoek van zijn advocaat. Het is vooralsnog een heel lastige kluif, vooral omdat het rechtssysteem in Engeland anders in elkaar zit. Opnieuw is Gill ervan overtuigd dat de verdachte geen moorden heeft gepleegd en dat het recht moet zegevieren.

Uit deze zaken heeft hij belangrijke lessen getrokken die hij wil overbrengen aan iedereen die wereldwijd betrokken is bij de rechtspraak, van advocaten tot rechters en van officieren van justitie tot juryleden. Samen met andere experts schrijft hij een handleiding hoe statistiek in de rechtbank kan worden gebruikt, met name bij strafprocessen tegen zogeheten seriemoordenaars in de gezondheidszorg. Dat gebeurt onder supervisie van het gezaghebbende instituut Royal Statistical Society. Het boek moet later dit jaar verschijnen.


Je kunt aannemen dat een hond vier poten heeft, maar niet dat alles met vier poten een hond is. Als op die manier naar Lucia was gekeken, was ze nooit veroordeeld

Richard Gill

De boodschap die hij heeft, is in hoofdlijnen simpel: gebruik statistische gegevens pas als je je ervan hebt verzekerd dat ze kloppen en gebruik ze goed. ,,Benoem alle factoren. Trek niet te snel conclusies. Vraag onafhankelijke experts om hulp. Onderzoek alle mogelijkheden.’’ Volgens Gill is niet alleen expertise van professionals nodig, maar ook dat rechters en advocaten worden geschoold in een goede interpretatie van statistieken.

Vier poten

Hij geeft een simpel voorbeeld. ,,Je kunt aannemen dat een hond vier poten heeft, maar niet dat alles met vier poten een hond is. Je mag aannemen dat iemand uit Peru Spaans spreekt, maar niet iedereen die Spaans spreekt komt uit Peru. Als op die manier naar Lucia was gekeken, was ze nooit veroordeeld.’’

Kenmerkend vindt Gill dat de verdachten die hij hielp, allemaal opvallende mensen zijn. Ze werkten hard, hadden een duidelijke mening, stootten daardoor waarschijnlijk ook leidinggevenden voor het hoofd en eindigden uiteindelijk als zondebok. ,,Het heeft me echt getroffen hoeveel ze gemeen hebben. Ben Geen wilde legerarts worden en was enorm gedreven in zijn werk. Hij zag zijn werk als meer dan een baan en deed veel extra als het kon. Hij botste ook met managers omdat het ziekenhuis voortdurend tegen grenzen aanliep.’’


Als expert op het terrein van statistieken werkte Gill ook voor het Openbaar Ministerie (moordzaak Tamara Wolvers) en het Internationaal Strafhof (moordaanslag president Libanon). Inmiddels is hij al bijna zes jaar gepensioneerd, maar tijd om zich te vervelen, heeft hij niet. Er ligt nog voor jaren werk op zijn bordje. Puzzels die hij graag helpt oplossen. 

Daarnaast zijn er veel onderwerpen waar hij graag in zou willen duiken, zoals de geruchtmakende Deventer moordzaak, die hem al jaren mateloos intrigeert. ,,Ik houd het nog steeds voor mogelijk dat de veroordeelde Ernest Louwes onschuldig is. Met name de dna-sporen op de blouse van de vermoorde weduwe vind ik interessant. Dna is ook statistisch bewijs en statistiek vertelt ons hoe je met onzekerheden moet omgaan. Er zijn inmiddels nieuwe moleculairbiologische methoden om veel meer uit een spoor te halen.’’


Gill helpt Kamerlid Pieter Omtzigt met het analyseren van data over uithuisplaatsingen als gevolg van het toeslagenschandaal. ,,We maken een tijdlijn om oorzaak en gevolg in beeld te krijgen. Ik heb dus eigenlijk helemaal geen tijd meer om nog meer verpleegkundigen achter de tralies vandaan te halen’’, zegt hij met een glimlach.

Kamerlid Pieter Omtzigt. © ANP

Als er toch weer een zaak van een vermeende moordzuster op zijn pad komt, zal hij waarschijnlijk moeilijk ‘nee’ kunnen zeggen. Hij geniet van het puzzelen en wil voorkomen dat het leven saai wordt. De tekst op de achterkant van zijn trui spreekt wat dat betreft misschien wel boekdelen: ‘Keep calm, en deze opa lost het wel op’. Want ja. Gill, vader van drie kinderen, is opa en zijn vijf kleinkinderen logeren graag bij hem en zijn vrouw in Apeldoorn.


Een van de vele puzzels die hem al jaren bezighoudt en soms zelfs uit zijn slaap haalt, moet hij van zichzelf oplossen: de zaak José Booij, die achttien jaar geleden werd geconfronteerd met de uithuisplaatsing van haar zes weken oude baby Julia-Lynn. 

,,Een onvoorstelbaar en afschuwelijk verhaal. Zij is vermalen door het systeem en daar compleet aan onderdoor gegaan. Ik ben het contact met José verloren, maar nog steeds in het bezit van een doos met persoonlijke spullen van haar, zoals kindertekeningen, diploma’s, dagboeken en krantenknipsels over haar strijd voor haar kind tot de hoogste rechtsorganen in Nederland en Europa aan toe. Wellicht leeft Julia-Lynn nu onder een andere naam en wellicht weet ze haar geboortenaam niet eens. Ik wil dat ze weet wie haar moeder is. Dat die nooit heeft opgegeven. Daar heeft ze recht op. Ik hoop haar ooit te vinden en de spullen van haar moeder te kunnen geven. En weet je, ook deze vrouw is een bijzonder mens, anders dan anderen.’


Published in “The Stentor” last Saturday, auto-translated from the original Dutch by Google Translate. This is the raw version. Corrections still to be made!

Richard Gill has already released two medical serial killers, including Lucia de Berk, by refuting statistical evidence. © Rob Voss

This Apeldoorn scientist saves innocent sisters from prison: Unfortunately, people do not want to believe in coincidence

Scientist Richard Gill from Apeldoorn helped ensure that Lucia de Berk was acquitted. He achieved the same in a similar case in Italy and now he is going for the hat trick in England. What drives him? 

Anne Boer 28-05-22, 08:00

Pure scientific curiosity is what drives him, says the internationally renowned mathematician Richard Gill from Apeldoorn. As an expert in statistics, Gill (70) worked for the Public Prosecution Service and the International Criminal Court. He has been retired for almost six years and is known as emeritus professor of statistics at Leiden University.

With his knowledge of the use of statistics, he was able to prove the innocence of two nurses who were convicted of serial murders from his office: the Dutch Lucia de Berk and the Italian Daniela Poggiali. Now he is campaigning for the release of nurse Ben Geen from England.

Sheer nonsense

All three are said to have killed patients on the job . Lucia de Berk was even convicted of seven murders. The burden of proof was mainly based on statistics. If Lucia worked, more patients would die than during her colleagues ‘ shifts. It turned out to be sheer nonsense, as Gill puts it delicately. “A matter of gossip and backbiting, looking for a scapegoat to save the hospital’s reputation and assumptions when no murder was committed at all. †


Unfortunately, people do not want to believe in coincidence, we want to have a cause. That’s why we also believe in devils and gods

Statistical evidence plays a major role in research, including serial killers in the medical world. ,,But then you have to interpret the figures properly ” , says Gill. “If apparently many people die in a hospital, you first have to look closely at the cause. Are there perhaps more patients than usual ? Are they sicker than in other periods? Has the registration method been adjusted? Are there changes in the staff? If you immediately look at which nurse was present, you also skip the most important questions: is it murder or is it medical failure or even natural death? †

According to Gill, that immediately touches on another sore point. “A hospital is a place where people die, but often the cause of death is not clear. This can lead to clusters of suspicious deaths. You have to know which deaths you count, otherwise the police will look for evidence for claims. †


According to Gill, you should always keep in mind that there can be a good, innocent reason for an event. “Look at how often someone works. Full-time nurses experience more deaths than part-time nurses. If someone works full-time and is passionate about their craft, they’re even more likely to be there when someone dies than someone who works a few days a week or works strictly within the schedule. †

Scientist Richard Gill. © Rob Voss

According to him, you should never rule out a strange combination of circumstances. “They happen, even without murder. A famous example is an American couple who won the top prize in two different lotteries in one day . What are the chances of something like this happening? It really happened. Unfortunately, people do not want to believe in coincidence, we want to have a cause and look for a scapegoat. That is why we also believe in devils and gods. †


Richard Gill was born in England. His father was also a scientist. Love brought him to the Netherlands in 1974 at the age of 23. Six years earlier, on holiday, he had fallen head over heels for a daughter of a Dutch friend of his father’s. Both fathers work for Wavin from Hardenberg. After some wanderings, Gill ends up in Apeldoorn in the early 1980s, never to leave. He lives in an old mansion, in a sea of lush greenery. This was his wife’s childhood home. To keep his head above water financially, he works extra hard to make a quick career .

The medical world crosses his path early on. After studying mathematics at Cambridge, he obtained his doctorate for research into the question of how long cancer patients survive with a particular treatment. His calculation method turned out to be a godsend and is now also being applied in other areas. “It just happened to be on my plate. I didn’t have a topic and my promoter took this topic out of his drawer. It has had a lot of impact and the method is still widely used. †

witch hunt

His wife, who is a historian, points him at an early stage to the case of Lucia de Berk, who would later be convicted of seven murders in a hospital. “She spoke of a witch hunt and wanted me to watch it, especially when it became a witch trial, as she called it. She pointed out to me that statistics were used as evidence and so I should have something to say about it. I did not want to. Experienced statisticians were already involved, including people I knew. †

Lucia de Berk reacts happy after her acquittal © anp

When a book about this case was published in 2006, Gill took the plunge. “I was referred to the book by a colleague. I really didn’t know what I was reading, I was really blown away by it. It was crystal clear to me that the verdict was wrong and that the judges had misinterpreted the figures . †

The rest is history. Gill helped show that the figures failed to substantiate the accusation and Lucia was fully acquitted in 2010 after 6.5 years of wrongful imprisonment.


reads about a similar situation in Italy in 2014 , he immediately decides to jump back into action. This time, a nurse (Daniela Poggiali) is suspected of sixty murders. Gill calls his colleague Julia Mortera from Roma Tre University and together they offer their help. With success, this nurse has also been free since October after a previous sentence to life imprisonment.

Statistics is the science and technology of collecting, processing, interpreting and presenting data. Statistical methods are used to convert large amounts of data – for example about people’s purchasing behaviour, the housing market or the number of deaths in care – into useful information .

,,The statistics in this case were completely amateurish, it was not good. Prosecutors claimed there were more deaths when Daniela worked. Until she was arrested: then it suddenly dropped. We found that the death rate for all staff was high. Daniela was often present before the start of her scheduled shift and often continued to help after her shift was over. As a result, she was more often present when a patient died . It is easy to explain that the number of deaths decreased after Daniela was arrested. The news about the ‘ murder sister ‘ was widely covered in the media. As a result, the hospital attracted fewer patients . Fewer patients also means fewer deaths. †

Difficult bone

Gill is now investigating the serious allegations against English nurse Ben Geen. This is done at the request of his lawyer. It is still a very difficult task, especially because the legal system in England is different. Once again, Gill is convinced that the suspect committed no murders and that justice must prevail.

He has learned important lessons from these cases that he wants to pass on to everyone involved in the justice system worldwide, from lawyers to judges and from prosecutors to jurors. Together with other experts, he is writing a manual on how to use statistics in court, especially in criminal proceedings against so-called serial killers in healthcare. This is done under the supervision of the authoritative Royal Statistical Society. The book is due out later this year.


You can assume that a dog has four legs, but not everything with four legs is a dog. If Lucia had been looked at that way, she would never have been judged

Richard Gill

The message he has is basically simple: do not use statistical data until you have ensured that they are correct and use them well. “Name all the factors. Don’t jump to conclusions too quickly. Ask independent experts for help. Explore all possibilities. ” According to Gill, not only expertise from professionals is needed, but also that judges and lawyers are trained in a good interpretation of statistics.

Four legs

He gives a simple example. “You can assume that a dog has four legs, but not that everything with four legs is a dog. You can assume that someone from Peru speaks Spanish, but not everyone who speaks Spanish is from Peru. If Lucia had been looked at that way, she would never have been convicted. †

Typically, Gill finds that the suspects he helped are all striking people. They worked hard, had strong opinions, probably offended executives as a result, and ended up as scapegoats. “It really struck me how much they have in common. Ben Geen wanted to be an army doctor and was very passionate about his work. He saw his work as more than a job and did a lot extra when he could. He also clashed with managers because the hospital was constantly running into limits. †

criminal court

As an expert in statistics, Gill also worked for the Public Prosecution Service (Tamara Wolvers murder case) and the International Criminal Court (Lebanon Presidential assassination attempt). He has now been retired for almost six years, but he has no time to get bored. There is still work on his plate for years to come. Puzzles he likes to help solve. 

In addition, there are many subjects that he would like to delve into, such as the controversial Deventer murder case, which has intrigued him immensely for years. “I still think it possible that the convicted Ernest Louwes is innocent. I find the DNA traces on the murdered widow’s blouse particularly interesting. DNA is also statistical evidence and statistics tells us how to deal with uncertainties. There are now new molecular biological methods to get much more out of a track. †


Gill helps MP Pieter Omtzigt with analyzing data about custodial placements as a result of the benefits scandal. ,,We make a timeline to get a picture of cause and effect. So I don’t really have time to get any more nurses out of prison , ” he says with a smile.

Member of Parliament Pieter Omtzigt. © ANP

If a case of an alleged murder sister comes his way, he will probably have a hard time saying ‘ no ‘ . He enjoys puzzling and wants to prevent life from getting boring. The text on the back of his sweater might speak volumes in that regard: ‘ Keep calm, and this grandpa will solve it ‘ . Because yes. Gill, a father of three, is a grandfather and his five grandchildren like to stay with him and his wife in Apeldoorn.


He has to solve one of the many puzzles that has occupied him for years and sometimes even wakes him up: the case of Jos é Booij, who was confronted eighteen years ago with the custodial placement of her six-week-old baby Julia-Lynn.

“An unbelievable and horrifying story. She was crushed by the system and completely destroyed by it. I have lost contact with José , but I still have a box with her personal items, such as children’s drawings, diplomas , diaries and newspaper clippings about her fight for her child up to the highest courts in the Netherlands and Europe. . Julia-Lynn may be living under a different name now and may not even know her birth name. I want her to know who her mother is. That he never gave up. She is entitled to that. I hope one day to find her and give her mother’s things. And you know, this woman is also a special person, different from others. †

Sanctuary’s Spin

I have recently been involved in acrimonious discussions in a Google group,, devoted to Bell’s theorem and the interpretation of quantum mechanics. One of the group members, Bryan Sanctuary, insists that two particles leaving a source cannot remain entangled. He claims in preprints in a sequence of recent blog posts that the EPR-B correlations cannot remain valid once the particles are separated. Here is a link to one of the posts, At the same time, he claims that the EPR-B correlations do have a local and realistic interpretation once one introduces a hidden, quantum property which he calls hyper-helicity. He believes that in this way all notions of non-local collapse and weirdness can be banished from quantum mechanics. Another participant, Alexey Nikulov, also thinks that conventional quantum mechanics is wrong, and disbelieves in non-locality and collapse.

A small problem consists in figuring out what actually is Bell’s theorem. A long essay by Sheldon Goldstein (2011) on “Scholarpedia”,, makes it clear that Bell twice quite radically changed his thinking, so in fact, one could say that there are three Bell theorems. The essential mathematics changed from the first to the second, and the physical motivation behind his characterization of a “local hidden variables model” evolved both in the first change and in the second change. The theorem is always the same, that a local hidden variables model cannot reproduce certain quantum mechanics predictions, but its assumptions and interpretation have matured.

Spacetime diagram for EPR–Bell type experiment (Goldstein & Tausk, Scholarpedia)

I have started writing out some notes, maybe they will become a whole paper, to make it clear that “collapse of the wave-function” is an interpretational optional extra, not needed in order to apply quantum mechanics in practice. No interpretation of what is going on behind the scenes is necessary if one’s purpose is to describe nature. An interpretation might be useful if it helps one to understand nature. One must realize that it is possible that understanding nature is something which will forever go beyond our poor facilities. That is not to say that the drive to gain understanding isn’t the true drive behind doing science. Like democracy and justice, understanding is an ideal which we have to continually renew and re-affirm.

That does mean that in introductory expositions it should be clearly labelled as a lie for children.

The usual colourful language involving the non-local collapse of the wave function can be thought just to be a description of a useful computational tool, not a description of physical changes to something existing in physical reality. The only thing assumed to exist are measurement outcomes, and the theory allows us to compute probability distributions of their outcomes, also in complex, composite, sequential, experimental set-ups. One can compute what one needs to know by pretending that the wave function collapse as suggested by the von Neumann-Lüders extension of the Born law is somehow real.  One gets the right answer, as directly as possible. There is, however, no need to think of wave function collapse as being something physical (and necessarily non-local). Such thinking is an optional extra. Some people find it distasteful. Tastes differ. I think it can be usefully thought of as one of those lies for children which need to be seen in a different light as one gains maturity and knowledge.

We are all children! Learning never ends!

The second purpose is to write out the mathematical content of Sanctuary’s claims concerning hyper-helicity. Sanctuary feels that his interpretation of his mathematical formalism will revolutionize our understanding of quantum entanglement. Clearly, he has a long way to go, but I hope my own struggle to understand what he is doing will be helpful to others, if not to him.

José, Kevin, Lucia (JKL)

More than ten years ago I started writing a book on Dutch miscarriages of justice in which I had been involved. I wanted to explore the personality issues in three such cases. In each case, it seemed to me that aspects of the character of the main protagonist led to them being something of a scapegoat of a system under great stress. Some trigger events caused a bad situation to become an utter disaster. Authorities made mistakes and could not admit them, so errors were compounded, and there was no going back, no way to change path any more.

In recent posts, I have told a lot of the story of José Booij. It’s time to start writing about Lucia de Berk and Kevin Sweeney.

Concerning Lucia de Berk there already is an enormous literature. The case started in 2001, seemed to be closed with Lucia in jail for life by 2006 (conviction by the lower court at the first trial in 2003, appeal to higher court failed in 2004, cassation – appeal to the supreme court – failed in 2006) but at that time also a strong movement burst into the public view, calling for a judicial review and a retrial. Lucia was fully exonerated in 2010. The role of statistics in the case is well known though controversial since at the 2004 appeal, she was convicted “on the grounds of incontrovertible medical scientific evidence only”. A “statistical probability calculation” (such as the infamous calculation leading to the spectacular 1 in 342 million) played no part at all in the court’s conclusion, according to her judges.

Yet many things have still not been said in public about the case, except perhaps in literary form. In my future book, I want to say things I have said many times before in ephemeral blog posts, and other removed or hidden web pages.

Concerning Kevin Sweeney, not much has been written at all. He sat out his sentence for the murder of his wife and keeps a low profile.

Waarom maakt een statisticus zich zorgen over José Booij?

In mijn recente blogposts en dutch-family-justice-the-tragic-case-of-jose-booij/ heb ik een glimp gegeven van de beproevingen van José Booij. In dit bericht wil ik uitleggen hoe ik erbij betrokken ben geraakt.

Op 14 april 2010 heeft het gerechtshof in Arnhem Lucia de Berk vrijgesproken van alle aanklachten tegen haar. Ik werkte al sinds het boek van Ton Derksen in 2006 uitkwam aan de zaak, bestudeerde de statistieken, sprak met experts en journalisten, hielp het publiek om van gedachten te veranderen en werkte bovendien nauw samen met Metta de Noo op haar website. Gelukkig is het bewaard gebleven voor het nageslacht,, hoewel de oorspronkelijke URL is overgenomen door een internetcasino dat zichzelf brutaal “Lucia de Berk Casino” noemt. (Daar zit wat poëtische gerechtigheid in: ik zou de Nederlandse Justitie zeker vergelijken met een loterij). Hoe dan ook, in die jaren was ik aan de ontvangende kant van e-mails die naar de webmaster werden gestuurd, en zag ik berichten in de sectie “opmerkingen”. Het is niet verrassend dat we reacties kregen, niet alleen ter ondersteuning van onze strijd voor een nieuw proces voor Lucia, maar ook van verschillende personen die erop wezen hoe zulke soortgelijke verschrikkelijke dingen met hen waren gebeurd. Een van die personen was José Booij, een ander was Kevin Sweeney, Het was moeilijk te achterhalen wat er werkelijk met José was gebeurd; het duurde lang voordat ze zelfs haar naam bekendmaakte. Het is me uiteindelijk gelukt, en eind 2010 ben ik haar gaan opzoeken in haar appartement in Den Haag. Vlak voor de kerst ben ik voor de tweede keer op bezoek geweest: ze wilde namelijk dat ik vele spullen zou meenemen en bewaren “voor haar dochter later” omdat ze, zei ze, op het punt stond haar flat te verlaten. Ik had haast en zei dat tegen haar. Mijn familie wachtte op me elders in Den Haag. Ik laadde snel twee enorme dozen met spullen in mijn gehuurde stationwagen, en ging terug naar haar flat voor de rest. Ze praatte lang over elk boek dat ze voor Julia Lynn uitkoos (erg mooie boeken trouwens), en ik werd ongeduldig, zei iets ongelukkigs, en ze raakte overstuur en gooide me eruit. Ik ging terug na Kerstmis, maar het vogeltje was het nest ontvlucht. Ze was verdwenen.

José geniet van een bezoek aan een voormalige vriend, Patrick

Later bleek dat ze naar verschillende landen was gereisd, in Portugal was beland en overal probeerde mensen te vinden die voor haar zouden vechten om haar baby terug te krijgen. Een half jaar later kreeg ik uit het niets dat rare telefoontje van Delta, een gesloten psychiatrische afdeling die voornamelijk bewoond werd door junkies. Delta maakt tegenwoordig deel uit van een organisatie genaamd “Antes”. Het kliniek in kwestie was niet ver van een dorp genaamd, vreemd genoeg, Portugaal; ten zuiden van Rotterdam en dicht bij de “Oude Maas”,

De psychiaters aldaar hadden haar verhaal niet geloofd dat haar baby was gestolen door de Nederlandse staat en dat ze een brief aan de Koningin had geschreven en de Koningin had beloofd haar te helpen. Typisch psychotisch gebrabbel, denk ik (maar ik ben geen psychiater). Afgezien van haar vreemde terminologie was het helemaal waar. Ik mocht haar bezoeken en praten met de arts die de leiding had over haar behandeling. Ik heb een klein netwerk van vrienden opgezet die voor haar mede konden zorgen in Den Haag en omgeving. Ik werd getroffen door de enorme overeenkomsten tussen de zeer uiteenlopende zaken van José Booij, Kevin Sweeney en Lucia de Berk (“JKL”). In elk geval was dit een persoon die een traumatische jeugd had gehad, die moeilijkheden had overwonnen en dankzij hard werken en natuurlijke gaven echt iets van hun leven maakte. Maar ze vielen allemaal op in een menigte, en ze gedroegen zich niet helemaal zoals ‘normale’ mensen. Ze hadden ook pech – er gebeurden nare dingen als ze in de buurt waren en ze werden gezien als de oorzaak. Lucia had uiteindelijk ook wat geluk, en dat leidde tot haar vrijstelling. Maar het heeft lang geduurd en heel veel inspanning gekost van vele mensen.

Ik ben erin geslaagd dat Delta aan José toestemming gaf om te vertrekken, nadat we een woning hadden geregeld, genoeg geld om te overleven en nieuwe afspraken met advocaten en met nieuwe psychiaters. Na een tijdje ging José dus in een nieuw appartement in Den Haag wonen en leefde van haar arbeidsongeschiktheidsuitkering. Ik heb haar zover gekregen om het Psychotrauma Diagnose Centrum te bezoeken, toen gevestigd op de Universiteitscampus van Utrecht, en nu onderdeel van een landelijk netwerk, zie Dit resulteerde in een zeer duidelijk, gedetailleerd en coherent rapport waarin ernstige PTSD werd geïdentificeerd en een geschikte behandeling werd aanbevolen. Een heel andere behandeling dan wat ze bij Delta had gekregen. Ondertussen ontving ik de post van José terwijl ik een jaar verbleef op het Netherlands Institute for Advanced Studies in Wassenaar (NIAS), niet ver van Den Haag. Mijn project bij NIAS was om een ​​boek te schrijven rondom JKL. José wilde niet dat haar echte adres bekend was bij de autoriteiten: daarom had ik Delta mijn adres bij NIAS gegeven en had ik José een eigen reserve bankrekening geleend. José wist dat verschillende instanties en personen nog steeds geld van haar wilden hebben, en weigerde botweg haar eigen bankrekening te openen, omdat ze beweerde dat haar geld dan onmiddellijk in beslag zou worden genomen door “de overheid”.

Helaas betekende dit dat ik de brieven te zien kreeg die ze van de Nederlandse belastingdienst kreeg, evenals verzoeken om haar oude studiebeurs voor haar studie Medicijnen aan de Rijksuniversiteit Groningen terug te betalen. Elke organisatie ging ervan uit dat ze nog steeds (d.w.z., ook na 2004) haar oude salaris bleef ontvangen. Ik eiste dat we dit zouden regelen. Ik kon haar niet helpen bij het ontduiken van belasting. Ze werd opnieuw woedend en vluchtte rond Kerstmis 2011 het land uit.

Ik was nu een slecht mens, net als alle andere personen van wie ze eerder had gehoopt dat ze haar zouden helpen haar baby terug te krijgen. Voor mij was het bovendien heel gênant dat ik geld van haar op een bankrekening van mij had staan. Af en toe waren er tekenen dat ze nog leefde (maar geen goede tekenen), namelijk problemen veroorzaakte, en op verschillende verre plaatsen: Lissabon, opnieuw; later, kort, in Freiburg in Duitsland, dicht bij Basel. Later nog in Engeland, opgesloten in Medway Maritime Hospital in Kent. Elke keer slaagden we erin om wat van haar geld bij haar te krijgen. Ik had trouwens meteen tegen de arbeidsongeschiktheidsdienst laten weten dat ze was verhuisd en dat haar locatie nu niet bekend was. Haar maandelijkse toelage kwam niet meer naar mij toe, onmiddellijk. Op dezelfde manier vertelde ik alle verschillende autoriteiten die haar aanmaningen stuurden om oude rekeningen te betalen dat ze niet meer op dat adres was. Hun brieven kwamen nu ook niet meer bij mij binnen.

Zoals iedereen eigenlijk, was José een vat vol tegenstrijdigheden. Haar ervaringen hadden die tegenstrijdigheden uitvergroot. Aan de ene kant was ze intelligent, levenslustig, gezellig; zeer geïnteresseerd in het leven, politiek, sociale ontwikkelingen, de natuur, in het bijzonder planten, vogels. Ze was goed georganiseerd, professioneel, praktisch. Aan de andere kant was ze (en werd ze steeds meer) monomaan. Ze zou haar moeilijke jeugd overwinnen. Haar jeugdtraumas zouden niet haar leven bepalen. Ze had een enorme behoefte om zelf een kind te hebben terwijl ze zichzelf als lesbisch beschouwde en bovendien alleen haar kind wilde opbrengen (of het laatste zo verstandig was, weet ik niet). Maar: ze had immers ook zichzelf helemaal alleen groot gebracht? Ze zocht relaties op dating sites, maar eigenlijk wilde ze geen relatie, eigenlijk wilde ze alleen een oudewetse zaaddonor vinden. Ze heeft dus menig man diep gekwetst door hun het idee te geven dat het om hun ging, terwijl het eigenlijk alleen ging om het biologische resultaat van seksuele gemeenschap. Wat kan haar hebben bezield om te denken dat ze zou overleven als alleenstaande moeder op het Drentsche platteland? Het meest achterlijk gebied van nederland, met een rijke geschiedenis van volledig negeren van centraal gezag. Het wilde Westen, maar dan in het Oosten.

Nadat haar kind was weggehaald stortte haar wereld in. Dapper vocht ze terug, maar steeds hadden de autoriteiten de troefkaarten in handen. “Het belang van het kind”. Alsof de autoriteiten het uitlokten, gooide ze steeds haar eigen ruiten in. Liep boos weg of ging schreeuwen in plaats van gedwee mee te spelen. Keer op keer, kreeg ze steun van een oude of een nieuwe vriend, van een advocaat of van haar oude therapeut Beata Bakker. Beata kwam op het geniale idee dat José en de biologische vader samen zouden doen alsof ze samen het kind wilden opvoeden. Dan moest Jeugdzorg zich toch anders gaan opstellen. José kon echter het toneel niet verdragen. Eigenlijk herkende ze ook haar eigen kind niet meer – die was inmiddels zo veranderd. De foto’s van haar, Peter de Koningh, en een inmiddels een jaar of zo oude Julia Lynn zien er ook vreselijk onecht uit. José kon het toneel niet opvoeren.

Elke keer deed haar huidige “ridder op een wit paard” iets fouts, waardoor ze razend werd, en hem (meestal, maar niet altijd, was het inderdaad een man) verwierp.

Ze is naar de Hoge Raad gegaan. Ze is naar de ombudsman gegaan. Ze is naar de European Court of Justice gegaan. Elke keer werd haar klacht op formele gronden onontvankelijk gevonden. Ze ging naar de Koningin. De Koningin (dus: eigenlijk een lid van het Kabinet van de Koningin) instrueerde Hare Minister van Justitie om de zaak uit te zoeken. De minister probeerde contact met José op te nemen, maar ze was er niet meer. Dat hield dan gauw op. Natuurlijk, ze had ook geschreven dat als haar brief gelezen werd, ze er niet meer zou zijn.

Dit is allemaal heel klassiek. Ik kan ook de psychologische en psychiatrische literatuur over PTSD lezen. José had nodig dat mensen begrepen dat fouten waren gemaakt, en ze had nodig dat die fouten erkend werden. In plaats daarvan werd ze, Kafkaësk, van het ene loket naar het andere gestuurd, waar steeds de ene ambtenaar naar het andere constateerde dat ze bij het verkeerde loket was. De hele tijd werd tijd kostbare tijd verloren, en zorgde ze zelf, door haar eigen snelle psychische aftakeling, dat het niet meer goed kon komen.

Ik vind dat het onrecht die haar gedaan is door de Nederlandse staat erkend moet worden. (Dat was uiteindelijk ook het enige dat ze wilde). Ze is niet de enige slachtoffer van Jeugdzorg en Kinderbescherming. Alle slachtoffers verdienen een monument.

Verder moet haar dochter mogen weten dat José enorm van haar kind hield, fantastisch goed voor haar zorgde, en als een tijgerin gevochten heeft om haar te beschermen. Ik heb laatst een verslag gelezen van Kinderbescherming zelf, over hoe Julia Lynn zich gedroeg in de eerste maanden nadat ze bij pleegouders was geplaatst. Het ging niet goed. Het is schrijnend.

Tenslotte heb ik nog kindertekeningen van José, schoolrapporten, zelfs haar diploma (medisch arts) bij de Rijksuniversiteit Groningen; gegevens over haar biologische ouders; oude familie foto’s. Haar dagboeken en notitieboeken. Alle juridische uitspraken van talloze rechtbanken. Rapporten van psychiatrische evaluaties. Deze zou ik graag willen zien toekomen aan haar dochter. Daar heeft Julia Lynn Booij ook recht op.

Why is a statistician concerned about José Booij?

In my recent blog posts and I’ve given a glimpse of the trials and tribulations of José Booij. In this post, I want to explain how I got involved.

On April 14, 2010, the appeal court in Arnhem pronounced Lucia de Berk innocent of all charges against her. I had been working since Ton Derksen’s book came out in 2006 on the case, studying the statistics, talking to experts and journalists, helping the public to change their mind, and moreover working closely with Metta de Noo on her website. Fortunately, it has been preserved for posterity,, though the original URL has been taken over by an internet casino which brazenly calls itself “Lucia de Berk Casino”. (There is some poetic justice in that: I would certainly compare Dutch Justice to a lottery). Anyway, during those years I was at the receiving end of emails sent to the web-master as well as noticing posts placed on the “comments” section. Not surprisingly, we got reactions not just in straight support of our struggle for a re-trial for Lucia, but also reactions from various individuals who pointed out how such very similar terrible things had happened to them. One of those individuals was José Booij, another was Kevin Sweeney, It was hard to find out what had actually happened to José; indeed it took a long time for her to divulge her name. At last, I succeeded and went to see her in her apartment in the Hague towards the end of 2010. Just before Christmas, I made a second visit: she wanted me to take a lot of stuff away and keep it “for her daughter later” because she was, she said, about to be forced to leave her flat. I was in a hurry and told her so. My family was waiting for me elsewhere in the Hague. I quickly loaded up two huge boxes of stuff and put them in my hired station car, then went back to her flat for the remainder. She was taking a long time talking about every single book she was choosing for Julia Lynn (very nice books by the way), and I got impatient, said something unfortunate, and she flew off the handle and chucked me out. I went back after Christmas but the bird had fled the nest. She’d disappeared.

José enjoying a visit to a former friend, Patrick

Later it transpired that she had travelled to several countries, ending up in Portugal, trying everywhere to find people who would fight for her to get her baby back. Half a year later, out of the blue, I got that weird phone call from Delta, a closed psychiatric ward inhabited mainly by sectioned junkies. Delta is nowadays part of an organisation called “Antes”. The specific hospital in question was not far from a village absurdly called Portugaal, south of Rotterdam and close to the “Oude Maas”,

The psychiatrists there had not believed her story that her baby had been stolen by the Dutch state and that she had written a letter to the queen and the queen had promised to help her. Aside from her odd terminology, it was completely true. I got to visit her and talk to the doctor who was in charge of her treatment. I set up a small network of friends who could care for her in the Hague and neighbourhood. I was struck by the enormous similarities between the very disparate cases of José Booij, Kevin Sweeney, and Lucia de Berk (“JKL”). In each case, this was a person who had had a traumatic childhood, had overcome those difficulties, and thanks to hard work and natural gifts were really making something of their lives. But they all stood out in a crowd, not quite behaving as “normal” people do. They had bad luck – bad things happened when they were around and they were seen to be the cause. Lucia eventually had some good luck and that led to her exoneration. But it took a long long time and a lot of work.

We managed to get José allowed to leave, having arranged a place to live, enough money to survive, and appointments with lawyers and psychiatrists. So after a while, José was living in a new apartment in the Hague, living off her disability allowance. I got her to visit the Psychotrauma Diagnosis Centre, then located on Utrecht University campus, and now part of a national network, see This resulted in a very clear, detailed, and coherent report identifying severe PTSD and recommending suitable treatment. Very different treatment from what she had been getting at Delta. Meantime, I was the recipient of José’s post while staying a year at the Netherlands Institute for Advanced Studies in Wassenaar (NIAS), not far from The Hague. My project at NIAS was to write a book around JKL. José did not want her actual address known to the authorities: that’s why I had given Delta my address at NIAS, and lent José a spare bank account of my own. José knew that various agencies and persons still wanted to get money from her and flatly refused to open her own bank account because she claimed that her money would then be immediately confiscated by “the government”. Her disability allowance therefore now went to an otherwise sleeping account of mine, and she took out the money she needed from an ATM using the bank card of the account.

Unfortunately, this meant that I got to see the letters she got from the Netherlands tax authorities as well as demands to repay her old government university study grant. Each different organisation assumed that she had continued to be getting her old salary since 2004. I demanded that we got this sorted out. I could not assist her in evading tax. She became furious yet again, and fled the country, around Christmas 2011.

I was now an evil person like all the other persons who she had earlier hoped would help her get back her baby. For me, it was moreover very embarrassing that I had money of hers in a bank account of mine. Occasionally, there were signs that she was alive (but not well), causing trouble and in various distant places: Lisbon, again; later, briefly, in Freiburg in Germany close to Basel. Later still in England, locked up at Medway Maritime Hospital in Kent. Each time we managed to get some of her money to her. By the way, I had immediately told the disability allowance agency that she had moved and that her location was now unknown. Her monthly allowance stopped coming to me, immediately. Similarly, I told all the various authorities who were sending her demands for money that she was no longer at that address. The letters slowly stopped coming.

Like everyone else, José was a barrel full of contradictions. Her experiences had magnified those contradictions. On the one hand, she was intelligent, lively, sociable; very interested in life, politics, social developments, nature, especially plants, birds. She was well organized, professional, practical. On the other hand, she was (and became more and more) a monomaniac. She would overcome her difficult childhood. Her childhood trauma would not define her life. She had a great need to have a child of her own while considering herself a lesbian and wanting to raise her child alone (whether the latter was the wisest thing to do, I don’t know). Hadn’t she also raised herself all by herself? She looked for relationships on dating sites, but she really didn’t want a relationship, really just wanted to find an old-fashioned sperm donor. So she deeply hurt many a man by giving them the idea that it was about them, when in fact it was only about the biological result of sexual intercourse. What could have possessed her to think that she would survive as a single mother in the countryside of Drenthe? The most backward area of ​​the Netherlands, with a rich history of completely ignoring central authority. The wild West, but in the East.

After her child was taken away, her world collapsed. She fought back bravely, but the authorities always held the trump cards. “The best interests of the child”. As if the authorities were provoking it, she kept smashing her own windows in. Walked away angry or yelled instead of meekly playing along. Time and again, she received support from an old or a new friend, from a lawyer or from her old therapist Beata Bakker. Beata had the genial idea that José and the biological father could act together as if they wanted to raise the child together. Then Youth Care would have to adopt a different attitude. José, however, could not bear the play-acting. In fact, she no longer recognized her own child – she had changed so much by now. The photos of her, Peter de Koningh, and Julia Lynn, who is now a year or so old, also look terribly fake. Jose couldn’t play her part of contrite and loving mother and wife.

Each time, her current “knight on a white horse” did something wrong, making her furious, and rejecting him (usually, but not always, it was indeed a man).

She went to the Supreme Court. She went to the ombudsman. She went to the European Court of Justice. Each time, her complaint was found to be inadmissible on formal grounds. She went to the Queen. The Queen (so: actually a member of the Queen’s Cabinet) instructed Her Minister of Justice to investigate the matter. The minister tried to contact José, but she was no longer there. That soon stopped. Of course, she had also written that when her letter was read, she would no longer be on this earth!

This is all very classic. I can also read the psychological and psychiatric literature on PTSD. José needed people to understand that mistakes had been made, and she needed those mistakes to be acknowledged. Instead, Kafkaesque, she was sent from one official to another, and one official after another determined that she was at the wrong counter. All the while, precious time was wasted, and she herself, through her own rapid psychological decline, made certain that things wouldn’t get any better.

I think the injustice done to her should be recognized by the Dutch state. (That was all she wanted, after all). She is not the only victim of Youth Care and Child Protection. All victims deserve a monument.

Furthermore, her daughter should know that José loved her child very much, took great care of her, and fought like a tigress to protect her. I recently read a report from Child Protection Services itself, about how Julia Lynn behaved in the first months after being placed with foster parents. It didn’t go well. It’s harrowing.

Finally, I still have children’s drawings by José, school reports, even her diploma (medical doctor) at the University of Groningen; data on her biological parents; old family photos. Her diaries and notebooks. All the legal rulings of numerous courts. Reports of psychiatric evaluations. I would like to see this delivered to her daughter. Julia Lynn Booij is also entitled to that.