Risky decisions

Pay-for-performance, shared decisions, and the science of risk

Share with your friends










Submit

I had a sort of epiphany about risk and medical decision-making last Thursday evening, even though I was particularly tired and in a state of mental fog.

I had been thinking about two recent articles dealing with risk prediction, but I also was preoccupied with risky career decisions I have made, and how these are affecting me and my family (I am pursuing a quixotic version of the “triple threat:” independent physician, apprentice schoolman, blogger).  So risk was very much on my mind.

One of the articles I had read was a recent viewpoint in JAMA by Allan Sniderman, Ralph d’Agostino, and Michael Pencina titled “The role of the physician in the era of predictive analytics.”  The other was a response to that article written by Bill Gardner at the Incidental Economist blog.

In their paper, Sniderman et al. discuss an important difficulty in the medical science of risk prediction.  The difficulty is apparent when one contrasts the notion of population risk with the notion of individual risk.

Population risk is a simple frequency of events and can be estimated empirically.  For example, 10% of people with a blood cholesterol level of x will die of a heart attack in 10 years.

Individual risk, on the other hand, is a problematic idea.  One is tempted to tell Mrs. Jones, whose blood cholesterol level is x, that she has a 10% risk of dying of a heart attack in the next 10 years.  But what does that mean?  As Sniderman et al. note:

Contrary to what is thought, this [10%] risk level is not that person’s personal risk because probability is not meaningful in an individual context (emphasis mine).

This is not a new idea.  Probability theorist Richard von Mises said as much in 1957:

We can say nothing about the probability of death of an individual even if we know his condition of life and health in detail. The phrase ‘probability of death’, when it refers to a single person, has no meaning at all for us.

Personal lives are not repeatable observations.  According to von Mises, no repeatable Mrs. Jones, no “individual risk” for Mrs. Jones.

And if the notion of individual risk “has no meaning at all,” then the whole enterprise of guideline medicine (including its pay-for-performance component) is also highly problematic, since it looks to group risk as proxy for individual risk.  Furthermore, individual risk is what we are supposed to discuss with patients when we obtain informed consent or engage in shared decision-making, the new examplar of patient-centered care.

Sniderman et al. recognize the insecure character of individual risk, but they are not so categorical as von Mises in denouncing it.  To salvage the concept—and with it, the clinical guidelines enterprise—they meet von Mises only halfway: individual risk is not meaningful, but that’s only because our knowledge is limited.

Thus they boldly propose that:

Hypothetically, if all past, present, and future predictors and processes that contribute to future events were known and quantifiable, algorithms could be constructed that produce perfect risk estimates for individuals—that is, they would predict with perfect accuracy whether an event would occur or not in every individual.

Although they don’t elaborate further on this hypothetical, Sniderman et al. seem to hark back to old school determinism (Laplace’s demon comes to mind), a position that is rather difficult to defend nowadays.  At the very least, this position is controversial enough that to slip it into an argument as casually as they do seems intellectually careless.

But the JAMA authors could have disagreed with von Mises without explicitly embracing nineteenth century determinism.  They could have said that a probability does not have to always represent a proportion of events from a uniform sequence of observation.   It can also represent a degree of belief, according to a “subjectivist” or Bayesian school of thought on probability theory.¹

That is precisely the position taken by Bill Gardner in his response to that JAMA article.  In the blog post, Gardner mostly agrees with Sniderman et al.’s defense of guideline medicine, but rebukes them for “badly going off the rails” in rejecting the notion of individual risk.  On the contrary, Gardner argues,

…it’s perfectly rational to use probabilities to express a consistent set of beliefs about the likelihoods of individual future events. Otherwise, saying “this patient will probably survive the operation” would be a logical absurdity.

Now, I have long taken the same position as von Mises regarding the notion of individual risk, i.e., that it is meaningless under any circumstances.  But I must admit that I have also struggled to reconcile my position with what I seem to be doing on a day-to-day basis when I talk to patients and advise them on the best course of action.

After all, I’m as likely as anyone else to say something like “well, Susan, the risk of x is not so high compared to doing y, and therefore I recommend…”  Am I engaging in a sort of mental reservation when I do this, speaking in generality about group risk, but knowing full well that Susan must be thinking instead about her “personal risk?” Am I being hypocritical?  Or am I a subjectivist in spite of myself?

Gardner’s statement must have hit a nerve, because I felt compelled to Tweet him a retort:

But retreating into a linguistic argument was not the most compelling point I could make, and I knew it.

So how did I resolve the mental tension I was experiencing about this tricky subject last Thursday?

The details are fuzzy but I remember thinking that the solution would not come from getting into the nitty-gritty details of the dispute between frequentistist and Bayesian theorists, as if the distinction between the two camps were clear cut (it’s not).  Anyone who dares to read about probability theory will soon enough realize how murky these abstract concepts can be.²

What eventually hit me, though, is the following realization:  doctors should not engage in risk prediction at all!

That was my ah ha! moment.  A seemingly outlandish one, for sure, so let me restate it more carefully:

When a doctor is engaged in the act of healing—which, after all, is what the medical profession is about—he or she should not first enter into a calculation of benefit, risk, cost, and the like before deciding on the proper course of action.

The key term is calculation.  In other words, in the course of helping patients, doctors should not explicitly add the pros and cons of the available options or engage in formal risk-benefit estimation, even if this estimation in only semi-quantitative (that is, not strictly-speaking arithmetic).

Of course,  I have a lot of explaining to do!  But I’m not coming to this conclusion without arguments to back me up.  In fact, I suspect this proposal of mine is not altogether original, but we’re already well past the 1000-word mark for this blog post, so I’ll leave the supportive evidence for my claim until next time.

For now, though, I’d be very interested in getting readers’ reaction to this idea.  What do you think?  Is a physician’s good counsel fundamentally a matter of doing a risk-benefit calculation for the patient?  Remember, your answers could put the future of pay-for-performance at risk!

This post is part of an on-going series on medical decision-making

Notes:

1. The fact the Laplace himself was also the originator of Bayesian theory would suggest that this school of thought is bound to a deterministic worldview.  Nevertheless, I don’t imagine that all those who invoke Bayesian theory necessarily espouse that worldview.

2. Hacking, Ian.  The Emergence of Probability: A Philosophical Study of Early Ideas about Probability, Induction and Statistical Inference.

Additional references of interest:

Accad, M. Statistics and the rise of medical fortunetellers.  Tex Heart Inst J. (2009) 36(6):508-509 Free access

Accad, M and Fred HL.  Risk-factor medicine: an industry out of control?  Cardiology. (2010) 117:64-67 Paywall

Goodman, S.  Toward evidence-based medical statistics 1.: The P value fallacy. Ann Intern Med (1999) 130:995-1004 Paywall

Goodman, S.  Toward evidence-based medical statistics 2.: The Bayes factor. Ann Intern Med (1999) 130:1005-13 Paywall

If you enjoy what you read, don’t forget to share the content with friends and colleagues so they too might become Alert and Oriented! Also, sign-up at the upper right-hand corner of your browser (or at the bottom of the page on mobile devices) to receive a free monthly digest of all my posts. Thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *

6 thoughts on “Risky decisions

  1. Your commentary got me to thinking again about this general topic.So far, I don’t think my views have changed much since I wrote about it in 2013 with a posting called “Individual risk assessment-Is that a concept that resists meaning?”
    http://mdredux.blogspot.com/2013/12/individual-risk-assessment-is-that.html
    But every time I try and think it all through at the end I find myself with more that a little amount of uncertainty. I am anxious to hear your further thoughts.

    James

  2. With this article you have begun to answer the question I was going to pose on your previous post: namely when you are counseling a patient how do you make decisions and what gives you confidence in those decisions.

    I look forward to the next post but at the moment am skeptical. I wish I could elaborate more on why but my one year old is demanding my attention.

    • Thank you for your interest, Bob.

      Yours is a two-part question and both parts are important. I will certainly begin to address them a little more specifically in my next post but in a way, this website is an ongoing exploration for those answers.

      One thing I would say, though, is that we (physicians and patients) are all conditioned by our current medical culture, good and bad. On a day-to-day basis, it is very hard and maybe inadvisable to take some of the opinions expressed here and “put them in practice.” We have to be mindful that we operate under given circumstances and with certain expectations (legal or otherwise) from patients and from society, that cannot be ignored.

      Michel

  3. I completely agree that trying to calculate an individual’s risk is not only impossible, but pointless, and that doctors should not be doing this or counseling their patients as such. However, the information gained from the risk in a population is extremely valuable in trying to counsel patients about their individual risk factors, and what they can be doing about them.

    As a soon to be famous physician once said “people ain’t widgets”. Unfortunately this is the dawning of the age of Obamacare, and bureaucrats are on the ascendance. I for one will go on treating my patients as best as I can and for their individual benefit, social justice be damned.

    • Thank you, Thomas.

      I certainly agree with you that there is value in epidemiological studies and these need not be tossed out. The value, however, is not so much in how precisely they quantify population risk. First of all, the studies themselves have limitations and the risk estimates can vary widely. Secondly, if we think that individual risk calculation is pointless, then quantification of population risk is not so valuable, at least for the clinician (it may have its use in public health).

      Social justice is important, but I don’t think it is carried out by applying population metrics to individual care. Rather, the other way around, by treating each person with the individual respect they deserve, regardless of their social circumstances–and I think that’s what you are saying too.

      Michel

  4. Statistics are for populations, not individuals. The doctor – patient relationship may be the most complex in all of free enterprise. One cannot be blind to statistics in decision making but neither can they be slaves to it. Data can have unintended consequences. Poor surgery risk patients will be denied care “for numbers” if third party payers rule. The surgeon who takes on high risk patients will be run out of practice and the lives otherwise saved are now lost. In Emergency Medicine, I can assure you, patients are never widgets and intangibles rule the day beyond any traditional economic model.