A Vox.com piece about decision-making caught my attention this morning.
The story was compelling. A 12-year-old boy had intractable seizures from a leaking vascular malformation in the brain. A first neurosurgeon would not operate and recommended radiation therapy instead. The patient’s mother sought another opinion from a Mayo Clinic neurosurgeon who was adamant that an operation should be undertaken. The second surgeon surgeon was undeniably right. The patient is now a bright, fully-functional researcher at the University of California San Francisco.
So far, so good? Not so, according to Vox. That there should be a smart mom making a smart decision, and a smart doctor carrying out a successful surgery is apparently a problem.
Why? Because the more cautious surgeon had a different opinion and, had the mom compliantly accepted his recommendation, the child could have been worse off. Variability in judgment, as always, is the enemy.
Vox quotes a recent Annals of Surgery study that “supports” the concern. When surgeons are given clinical vignettes, they vary widely in how they estimate the risks and benefits of an operation. Consequently, the decision to operate also varies widely—at least, on vignette paper.
This variation seemed to come down to surgeons’ perceptions of risks and benefits, the researchers wrote: “Surgeons were less likely to operate as their perceptions of operative risk increased and their perceptions of nonoperative benefit increased.”
And those risk perceptions were very predictive of whether or not a surgeon would recommend an operation: “Surgeons were more likely to operate as their perceptions of operative benefit increased and their perceptions of nonoperative risk increased.”
The remedy? Risk calculators which take “high-quality data from millions of patients” who have had similar operations to come up with estimates on the risks of surgery.
Dr. Ashish Jha, a Harvard professor of health policy concurs: “It’s clear we need to develop more resources like this to be additional input beyond personal experience for surgical decision-making.” Accordingly, had the first surgeon, truly known the actual risk of the operation, he may have acted differently, and saved the mother and her child the delay of care and any additional expense.
Imagine the conversation:
Mom: my child is not doing well. Will you operate?
Surgeon: Too risky, I don’t want to.
Health Policy Expert: Hey, check out this cool risk calculator!
Surgeon: What was I thinking? Of course, I’ll open the skull!
What strange minds doctors seem to have! As it turns out, this peculiar understanding of how clinical decisions are made has been around for decades. Quoted in the Annals‘ article is a famous JAMA paper from 1990 in which author David Eddy, a Harvard physician, mathematician, and healthcare analyst, describes clinical decision-making according to a two-step process of inputs and outputs:
In other words, the doctor’s mind is a rather simple computer. Again, from the Annals article:
According to normative decision theory, treatment decisions under uncertainty should be based on an evaluation for each available treatment option of: (i) the probabilities of possible outcomes; and (ii) the relative attractiveness or unattractiveness (ie, the utilities) of these outcomes.
Of course the computational theory of mind is not unique to the medical field. But it has been criticized widely and, in my opinion, quite effectively, notably by John Searle and by the late Hilary Putnam. As Robert Epstein put it just a couple of days ago: “Your brain does not process information, retrieve knowledge, or store memories. In short, your brain is not a computer.” But the memo, it seems, hasn’t reach the ivory towers of healthcare analysis.
I have previously pointed to the work of psychologist Gary Klein (see here and here) who studies decision-making in real life settings. Klein shows that decision-making is clearly unlike what the computer model would lead one to believe. In fact, excellent decisions are made, at times, on the basis of inexpressible knowledge which, to an outside objective observer, could seem like purely subjective idiosyncrasies.
Health policy experts, however, remain wedded to the computer model of the surgical mind and view the objective risk of an operation and its quantified benefit as essential “inputs” for decision-making. Predictive analyses are then carried out with implacable logic: Add the probability of benefit, subtract the probability of harm…Uncertainty is almost conquered.
Unfortunately, healthcare analysts seem to forget that a 5% mortality risk for resecting an aneurysm is only an average risk. That number may be called into question when applied to a particular surgeon. What if the doctor at hand had a drinking problem? Or what if she or he “sensed” they had a personal technical limitation and preferred not to operate in a given case? Should we use the average risk number to convince them otherwise?
Of course, the analysts always concede that calculation is not the end-all-be-all. “Yes,” they say, “use your judgment along with the risk calculation.” The risk calculator is simply offered as a tool to help “lift the bottom up,” so to speak, and should not interfere with sound judgment.
But here’s the rub about lifting the bottom up. How does a doctor at the bottom know that his judgment is bad? By definition, bad surgeons misjudge risk or benefit unknowingly. If they knew they were misjudging, they would judge otherwise. And even if bad surgeons were to agree to employ the calculator, could we still trust them to use judgment along with calculation—as if that concept had any real meaning, anyway.
Faced with the grim reality of unequal surgical talent, health policy will inevitably lean on doctors to use more and more calculation and less and less judgment. That’s the natural history of these managerial devices. After all, how else can we expect to have a “standard of care?” At that point, top surgeons, too, will necessarily end up making decisions rotely, and meld with the mass of undifferentiated surgical care providers.
The irony is that the boy in the story who was saved by a heroic surgeon is now a PhD student in epidemiology and seeks to develop “decision support systems…[which] can supersede the personal biases and subjectivity of physicians.” Shouldn’t he know that getting rid of the “personal biases and subjectivity of surgeons” cuts both ways? If variability is to be overcome, so will excellence be defeated.
[Related post: Make decisions, not calculations]
[Related post: Statistics and the rise of medical fortunetellers]
It seems to me that biological variability is also a casualty in the vilification of subjectivity. In his book Do No Harm, neurosurgeon Henry Marsh describes operative procedures that he knew went wrong right at the time and others where there were severe post-operative complications with no clear problem with the surgery.
Surgery is not like making Toyotas. The only way that you can say that the problem here is the “personal biases and subjectivity of surgeons” if you lack an appreciation of biological complexity.
I couldn’t agree more, thank you.