The devolution of evidence-based medicine

Share with your friends










Submit

Anish Koka recently wrote a great piece entitled “In Defense of Small Data” that was published on The Health Care Blog.

While many doctors remain enamored with the promise of Big Data or hold their breath in anticipation of the next mega clinical trial, Koka skillfully puts the vagaries of medical progress in their right perspective.  More often than not, Koka notes, big changes come from astute observations by little guys with small data sets.

In times past, alert clinicians would make advances using their powers of observation, their five senses (as well as the common one) and, most importantly, their clinical judgment.  They would produce a case series describing their experience so others could try to replicate the findings and judge for themselves.

Today, this is no longer the case.  We live in the era of “evidence-based medicine,” or EBM, which began about fifty years ago.  Reflecting on the scientific standards that the medical field has progressively imposed on itself over the last few decades, I can make out that demands for better scientific methodology have ratcheted up four levels:

Beginning in the late 1960s, and then throughout the 1970s, some began to call attention to the need for better statistical science in research publications. Chief among those emphasizing this problem was Alvan Feinstein.  Another important figure was Stanton Glantz, and there were others as well.  See, for example, this editorial in Circulation in 1980.

Then, in the late 1980s and early 1990s the movement that coined the term EBM emerged, notably from McMaster University in Hamilton, Ontario.  Names associated with that movement include David Sackett, Gordon Guyatt, Salim Yusuf, and others.  Beyond the proper use of statistics, they demanded that clinical studies be designed properly, and they anointed the randomized double-blind clinical trial as the supreme purveyor of medical evidence.  Others, like Ian Chalmers and the folks at the Cochrane Collaboration developed methods of “meta-analysis” whereby different trials and studies on a particular question could be analyzed in aggregate.

The mid-to-late 1990s were the heydays of EBM.  I remember the excitement in 1992 when the Evidence-Based Medicine Working Group published its teaching series in JAMA on how to properly interpret the medical literature.  I was a senior in medical school at that time, and that series of papers quickly became the go-to reference for self-respecting doctors who wanted to demonstrate that they weren’t taking the conclusions of medical publications at face value.  We would be able to judge for ourselves the validity of any new claim!  And we began to systematically doubt old claims that were not backed by sufficient evidence.  (The autobiography of EBM was recently sketched in JAMA here).

By the late 1990s, however, it became clear that EBM was having two major effects, and the promotion of clinical judgment was not one of them.  On the one hand, EBM gave rise to “guideline medicine” and cookbook recommendations that would soon provide insurers and government agencies a method to gauge “quality of care” on a large scale and to tie performance to that quality.  On the other hand, the methodology also played into the hands of large pharmaceutical companies who, through the implementation of large clinical trials, could now identify small effects that physicians would feel compelled to apply to large populations of patients.

The former effect was not so problematic for the promoters of EBM who, by and large, were made up of “systems experts” and outcomes researchers for whom the “rational” provision of care by way of clinical guidelines and “appropriate use criteria” seemed like a great boon.

But the latter effect, i.e., the benefits the pharmaceutical and medical device industry reaped through the use of the mega-trial, was a tough pill to swallow.  Many in the EBM movement were not particularly thrilled to see their pet methodology be at the service of major corporations, but they were hard-pressed to be able to criticize them on that basis.

An opportunity to correct that trend came when the Vioxx scandal brought to light the fact that too cozy a relationship between scientists and corporations could lead to tampering of the precious evidence.  It would not be long before the next level of demand on clinical science would be made: in addition to being proficient in the use of statistics and in the designs of clinical studies, scientists would now also have to disclose any potential conflicts of interest, i.e., any financial relationship with a sponsoring corporation.  Soon the sun would shine on the greed of doctors and scientists and, it was hoped, the truth would finally emerge victorious.

But even that has failed to satisfy the skeptics.  After all, conflict of interest is undetectable and unavoidable.  And the emphasis on conflicts of interest seems to have poisoned the atmosphere.  Witness any debate between scientists on Twitter, and you will experience a new version of Godwin’s law:

[see related posts on conflicts of interest in medicine and medical science]

The latest remedy concocted to deal with the vexing problem of scientific truth is data sharing, which I mentioned in a recent post.  Soon, scientists may need to make all data forms, correspondence, and even their inner reflections, available to third parties on demand, so that, you know, we can finally be sure that the results are “true.”

I predict that statistics, optimal study designs, conflict of interest disclosures, and data sharing will never satisfy our yearnings for clinical truth.  In recent days, I have been particularly struck by the epistemiological crisis in which we seem to find ourselves, despite the fact that we have access to huge data sets of information.  There is no longer any medical fact or truth that can be considered reliable anymore.  Skepticism is rampant and everything is in doubt, which is strange since there has never been more attention paid to being meticulous about science and methods.

It’s like a paradox: the more we insist on scientific reliability, the less certain our knowledge seems to become.  As I mentioned before, I believe this paradox arises in part because we are bent on applying to medicine quantitative methods that are more suited for the study of falling stones and quarks than for the proper understanding of human beings. Randomized trials provide us with general rules, but the business of medicine is to particularize.  When the time comes to apply our knowledge at the bedside, the evolution of EBM seems like a devolution against sound clinical judgment.

Also, quantitative methods have taken prominence because medicine has ceased to be a private affair between doctors and patients, but has turned instead into a public health endeavor.  Yet patients still expect us to treat them as individuals, and most doctors hope to practice on a human scale in the context of small communities.  The focus on big data and big methods is a major distraction, and it has yet to convince me that we are better doctors as a result of it.

In his magnificent essay, Koka notes

I realize it has become dangerous to use one’s clinical experience to inform one’s views….Forcing all progress through the funnel of a million strong clinical trial is a bar too high, a bar that protects us from medicine itself.  No one benefits from that.

I agree with my colleague Koka.  It’s time to return to the small data set.  It’s time to reclaim our ability to think, judge and reason for ourselves.  It’s time to put an end to the EBM madness.

11 Comments

  1. Dr. Accad, an excellent analysis of EBM.
    It is absurd to think that massive documentation of processes or outcomes can create cost control through mining big data population statistics for rewarding quality or its lack. For example, there is no quality magic that could have an effect on population and individual health compared to what happened over more than the last 100 years. [MMWR Weekly July 30, 1999;48(29):621-629. ] During that time public health departments were formed, public water supply and sewage disposal systems were cleaned up, vaccinations became common, and wondrous technologies including antibiotics became available.
    Today, population health is mostly a function of socioeconomic and cultural factors. [The Health of Nations: Why inequality is harmful to your health The New Press NYC, NY;2002:58-60. Health Aff:2002;21(2)113-118. Health Aff;2002:21(2):60-76. JAMA 2006;295(11):1304-1307.
    Socio-economic factors are beyond clinic control. The role of physicians is to care for patients, not populations of corporate or government clientele. Doctors are not public health officers and ought not to be population insurance underwriters gatekeeping “resource use” profitable to 3rd party corporations or government agencies.
    Thanks, Bob

    1. Thank you for these comments, Bob. I completely agree that a public health role has been somewhat foisted on physicians, making them serve two masters.

  2. Mega kudos to you and to Dr. Koka. I cannot remember when I have read regarding medicine and its practice anything so high on the “important insight scale” than these two commentaries.

  3. Thank you, Dr. Accad for such a balanced and intellectually honest approach to medical research! I find that with the “small guys” doing case studies and the like, they may also have more passion in regards to promoting a particular original treatment, or in the case of nutrition–a certain plan, and the unavoidable self-interest is less about the money with a lot of those “small guys” and more about the therapy and patient benefit. There are of course exceptions. Thank you again!

    1. Thank you, Brian. I suspect you’re correct. Doing “powerful” science invites all kinds of temptations.

  4. Great article. Totally agree with you. Especially for young doctors that have grown under the scope of the EBM.
    Every day I claim the need to go back to the clinician point of view. Taking into consideration an equal combination between EBM, physiology, clinical experience and common sense. It is time for the individuals. We should put strength on each patient rather than the specific diseases.

    Thanks

  5. Thank you Dr. Accad, as a young physician, I found my self sometimes lost in between guidelines and beside treatments in my Country. Lastly I Have the feeling that sometimes the things that I’m reading as the God derived Truth in Medicine are misleading conclusions from big pharma and “big”physicians…. Thanks again.

  6. The fallacy of EBP in education. Can this be applied to the medical model?

    Metaevaluation.
    Educational evaluation cannot be “done done.” It’s an impossible dream. If Ten is full-and-accurate determination of the value of an educational program, we sometimes get to Three, usually not past Two. The RFP calls for Michelangelo, and we are finger-painting. We differ among ourselves as to the meaning of the words, “to evaluate,” and we advise folks to do a lot of different things in the name of “evaluation.”

    It means to determine the quality of something. Everybody evaluates all the time: “You there are wearing your best shoes.” “That melon at lunch tasted so good.” The best of our evaluations allow people to falsely presume that a complete evaluation has been done. We should not be satisfied that quality of teaching is known by student ratings, or by student test scores, or by peer reviews, or by teacher of the year awards. Teaching is a situationally responsive act, a role a hundred times more complicated than the best checklist or set of standards.

    Even the best of our evaluations allow people to falsely presume that a complete evaluation has been done. We should not be satisfied that quality of teaching is known by student ratings, or by student test scores, or by peer reviews, or by teacher of the year awards. Teaching is a situationally responsive act, a role a hundred times more complicated than the best checklist or set of standards. Its meaning is constructed by the folks-involved every bit as much as the meaning of mathematics is constructed by children. The value is embedded in the situation, only in small part accessible to evaluators, supervisors, or the teachers themselves. Every child is shaped in part by teachers, for good or not, and most of the good they do, and most of the ill they do, is God’s truth alone.

    Todd Denny M.S.W.
    WESTERN NEW MEXICO UNIVERSITY
    School Of Social Work

Leave a Comment

Your email address will not be published.