Why the models got the Western Cape so wrong
The first official prediction of how many ICU beds would be used in the Western Cape by Covid-19 patients, made back in May, overestimated by between 12 times and 16.5 times the critical care beds needed.
Updated projections in June were better, but still out by a factor of two: modellers estimating that 780 ICU beds would be needed in July, when only 320 were needed at the peak.
A later model, estimating short-term death projections for the Western Cape at 4,750 by July 13, may also not have been accurate — though this is a little unclear because it did say the best-case scenario would be 2,510 deaths, which isn’t far off the actual figure of 2,385.
All of this illustrates how, at almost every step of SA’s Covid-19 battle, the official models have been of little help.
The SA Epi Model Consortium, a group of modellers from various universities led by the National Institute for Communicable Diseases, publicly released its long-term model on May 19, with the disclaimer that there was much “uncertainty” in the projections.
While the official model of deaths countrywide from Covid-19 by July 1 was largely correct, the modelling of hospital bed use in the Western Cape was way off.
The model predicted that in the best-case scenario, by the middle of July there would be 2,000 people in ICU in the Western Cape, and 5,300 in the worst case at the peak.
In reality, the number of patients currently needing high care in the Western Cape is between 270 and 280 today. At the peak, 320 ICU beds were used.
Also, while the model predicted the Western Cape’s peak would happen between mid-July and mid-August, National Institute of Communicable Diseases epidemiologist Dr Harry Moultrie says: “Hospital admissions for Covid-19 in the Western Cape appear to have peaked on June 22.”
Of course, the much lower numbers of ICU hospital patients in the Western Cape is a good thing. And, to be fair, the short-term model in June was much more accurate in predicting the number of beds the province would need than the May model.
But still, the minimum number of ordinary hospital beds the June model predicted was wrong. It said that by July 13 there would be between 4,380 and 6,820 patients needing beds in the Western Cape. Yet the most beds used at one time were 1,900 -less than half the projections.
Cost of error
The point is, what was the cost of getting these models so wrong?
Pandemic Data & Analytics (Panda), a group of economists, statisticians and actuaries, have publicly criticised modellers by name, making them highly unpopular. But Panda members argue that getting it wrong has serious economic implications for the country.
Says Panda founder Nick Hudson: “The modellers pretend that their models have no bearing on government decisions. We think it is obvious that their outlandish death forecasts have a bearing on policy-making …The models provide a fig leaf of rationality and are hence part of the causal structure of lockdown.”
Had there been no forecasts, he says, Panda would be taking the government to court, arguing to end the state of disaster on the basis that it is irrational.
Of course, it’s hard to say now whether the predictions for nationwide deaths by year-end will be proven correct.
The models for countrywide deaths, including Sacema’s first March estimates, predicted anything from 87,900 to 351,000 deaths. The Actuarial Society of SA model in April estimated between 88,000 and 43,400 deaths. And the official Epi Model on May 19 predicting between 40,000 and 48,000 deaths for the whole year.
As of today, SA has 5,173 deaths — about a tenth of the lowest estimates for Covid-19 fatalities this year.
Moultrie asks: “Did we overestimate Covid-19 deaths in the Western Cape? The brief answer is we don’t know yet.”
While the model’s fatality predictions for SA as a whole appear to be roughly accurate, Moultrie admitted that Covid-19 deaths have “unexpectedly” plateaued in the Western Cape over the past three weeks.
He says this “plateau is not in keeping with the modelling consortium projections”.
There are many reasons why the official model may have overestimated deaths in the Western Cape. This includes assuming that everyone is susceptible to the disease, better treatment in ICUs that has lowered deaths, undercounting of Covid-19 deaths that take place at home, and the impact of social distancing
Clearly, many of the assumptions were wrong.
The very first models, called susceptible-exposed-infectious-recovered (SEIR), assumed everyone who is exposed to Sars-CoV-19 will become infected. Globally, this has not been borne out by reality.
For one thing, it is now thought by some scientists that some people’s immune systems fight off the disease without producing antibodies. This innate immunity is very hard to measure, so it’s difficult to know what percent of people have this response to Covid-19.
Moultrie explains: “Panda has promoted this [innate immunity] thesis for a few months, [but] the evidence remains scanty.”
He says another reason the consortium may have estimated deaths incorrectly is that tests were limited in the Western Cape in late May to people who were very sick or in hospital. So people died at home without their relatives knowing they had Covid-19.
Moultrie says it is “likely that Covid-19 deaths are underreported”.
The SA Medical Research Council estimates that there were 3,694 “excess deaths” in the Western Cape between May 6 and July 7, whereas the confirmed number of Covid-19 deaths by July 7 was 2,192.
However, the missing 1,500 deaths are not all Covid-19 related, says Moultrie.
[This] is likely a result of a combination of unreported Covid-19 deaths and increased deaths from other causes arising from delays or difficulties in accessing health care.”
Also, a few of the extra deaths may date from before May, but were not reported immediately as home affairs offices were closed during the lockdown.
Moultrie says people’s behaviour also probably played a role in lowering the number of infections and death rates.
“Communication campaigns in the Western Cape, combined with increasing deaths, could have resulted in better adherence to precautions including masks, hand washing and social distancing, and those most at risk for severe Covid-19 disease taking additional precautions to isolate themselves”.
Moultrie says it’s hard to model this. “Since individual behaviour changes are unpredictable and difficult to model, these are not included in our models.”
Another factor which may have skewed the projections is that people are getting better treatment in ICU, compared to when the outbreak started, while there is more knowledge on what drugs to use. For example, high pressure oxygen, administered through the nose, is now given to the very ill lying on their stomachs, instead of using ventilators.
Social distancing, better hospital treatment and cross immunity could all play a role in lowering deaths. “We simply don’t know yet,” says Moultrie.
Panda, which predicted the Western Cape deaths to date far more accurately, feels there is no excuse for getting it so wrong. They wants modellers to retract their models.
Moultrie disagrees, and says Panda’s vilification is misplaced.
“Instead the focus should be on trying to understand why the Western Cape has a lower than predicted death rate, including being lower than global case fatality rates, and on urgently planning for the peak that is expected in other provinces,” he says.
Would you like to comment on this article or view other readers' comments?
Register (it’s quick and free) or sign in now.
Please read our Comment Policy before commenting.