A Critical Review of a Critical Review of e-Prescribing ... Or Is It CPOE?

In PLoS medicine, the following article was recently published by researchers at the University of New South Wales in Australia:

Westbrook JI, Reckmann M, Li L, Runciman WB, Burke R, et al. (2012) Effects of Two Commercial Electronic Prescribing Systems on Prescribing Error Rates in Hospital In-Patients: A Before and After Study. PLoS Med 9(1): e1001164. doi:10.1371/journal.pmed.1001164

The section I find most interesting is this:

We conducted a before and after study involving medication chart audit of 3,291 admissions (1,923 at baseline and 1,368 post e-prescribing system) at two Australian teaching hospitals. In Hospital A, the Cerner Millennium e-prescribing system was implemented on one ward, and three wards, which did not receive the e-prescribing system, acted as controls. In Hospital B, the iSoft MedChart system was implemented on two wards and we compared before and after error rates. Procedural (e.g., unclear and incomplete prescribing orders) and clinical (e.g., wrong dose, wrong drug) errors were identified. Prescribing error rates per admission and per 100 patient days; rates of serious errors (5-point severity scale, those ≥3 were categorised as serious) by hospital and study period; and rates and categories of postintervention “system-related” errors (where system functionality or design contributed to the error) were calculated.

Here is my major issue:

Unless I am misreading, this research took place in hospitals (i.e., "wards" in hospitals) and does not seem to focus (if even refer to) discharge prescriptions.

I think it would be reasonable to say that what are referred to as "e-Prescribing" systems are systems used at discharge, or in outpatient clinic/offices to communicate with a pharmacy selling commercially and not involved in inpatient care.

From the U.S. Centers for Medicare and Medicaid Services (CMS), for example:

E-Prescribing - a prescriber's ability to electronically send an accurate, error-free and understandable prescription [theoretically, that is - ed.] directly to a pharmacy from the point-of-care

I therefore think the terminology used in the article as to the type of system studied is not well chosen. I believe it could mislead readers not experienced with the various 'species' of health IT.

This study appears to be of an inpatient Computerized Practitioner Order Entry (CPOE) system, not e-Prescribing.

Terminology matters. For example, in the U.S. the HHS term "certification" is misleading purchasers about the quality, safety and efficacy of health IT. HIT certification as it exists today (granted via ONC-Authorized Testing and Certification Bodies) is merely a features-and-functionality "certification of presence." It is not like an Underwriter Labs (UL) safety certification of an electrical appliance that the appliance will not electrocute you.

(This is not to mention the irony that one major aspect of Medical Informatics research is to remove ambiguity from medical terminology, e.g., via the decades-old Unified Medical Language System project or UMLS. However, as I've often written, the HIT domain lacks the rigor of medical science itself.)

I note that if this were a grant proposal for studying e-Prescribing, I would return it with a low ranking and a reviewer comment that the study proposed is actually of CPOE.

That said, looking at the nature of this study:

The conclusion of this paper was as follows. I am omitting some of the actual numbers such as confidence intervals for clarity; see the full article available freely at above link for that data:

Use of an e-prescribing system was associated with a statistically significant reduction in error rates in all three intervention wards. The use of the system resulted in a decline in errors at Hospital A from 6.25 per admission to 2.12 and at Hospital B from 3.62 to 1.46. This decrease was driven by a large reduction in unclear, illegal, and incomplete orders. The Hospital A control wards experienced no significant change. There was limited change in clinical error rates, but serious errors decreased by 44% across the intervention wards compared to the control wards.

Both hospitals experienced system-related errors (0.73 and 0.51 per admission), which accounted for 35% of postsystem errors in the intervention wards; each system was associated with different types of system-related errors.

I note that "system related errors" were defined as errors "where system functionality or design contributed to the error." In other words, these were unintended adverse events as a result of the technology itself.

The authors conclude:

Implementation of these commercial e-prescribing systems resulted in statistically significant reductions in prescribing error rates. Reductions in clinical errors were limited in the absence of substantial decision support, but a statistically significant decline in serious errors was observed.

The authors do acknowledge some limitations of their (CPOE) study:

Limitations included a lack of control wards at Hospital B and an inability to randomize wards to the intervention.

Thus, this was mainly a pre-post observational study, certainly not a randomized controlled clinical trial.

Not apparently accounted for, either, were potential confounding variables related to the CPOE implementation process (as in this comment thread).

In that thread I wrote to a commenter [a heckler, actually, apparently an employee of HIT company Meditech] with a stated absolute faith in pre-post studies that:

... A common scenario in HIT implementation is to first do a process improvement analysis to improve processes prior to IT implementation, on the simple calculus that "bad processes will only run faster under automation." There are many other changes that occur pre- and during implementation, such as training, raising the awareness of medical errors, hiring of new support staff, etc.

There can easily be scenarios (I've seen them) where poorly done HIT's distracting effects on clinicians is moderated to some extent by process and other improvements. Such factors need to be analyzed quite carefully, datasets and endpoints developed, and data carefully collected; the study design and preparation needs to occur before the study even begins. Larger sample sizes will not eliminate the possible confounding effects of these factors and many more not listed here.

The belief that simple A/B pre-post test that look at error rate comparisons are adequate is seductive, but it is wrong.

Stated simply, in pre-post trials the results may be affected by changes that occur other than the intervention. HIT implementation does not involve just putting computers on desks, as I point out above.

In other words, the study was essentially anecdotal.

The lack of RCT's in health IT are, in general, one violation of traditional medical research methodologies for studying medical devices. That issue is not limited to this article, of course.

Next, on ethics:

CPOE has already been demonstrated in situ to create all sorts of new potential complications, such in at Koppel et al.'s "Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors", JAMA. 2005;293(10):1197-1203. doi: 10.1001/jama.293.10.1197 that concluded:

In this study, we found that a leading CPOE system often facilitated medication error risks, with many reported to occur frequently. As CPOE systems are implemented, clinicians and hospitals must attend to errors that these systems cause in addition to errors that they prevent.

CPOE technology, at best, should be considered experimental in 2012.

In regards to e-Prescribing proper, there's this: Errors Occur in 12% of Electronic Drug Prescriptions, Matching Handwritten and this: Upgrading e-prescribing system can bump up error risk to consider; in other words, the literature is conflicting, confirming the technology remains experimental.

This current study confirmed some (CPOE) errors that would not have occurred with paper did occur with cybernetics, amounting to "35% of postsystem errors in the intervention wards."

In other words, patient Jones was now subjected to a cybernetic error that would not have occurred with paper, in the hopes that patients Smith and Silverstein would be spared errors that might have occurred without cybernetic aid.

Even though the authors observe that "human research ethics approval was received from both hospitals and the University of Sydney", since patient Jones did not provide informed consent to the experimentation with what really are experimental medical devices as I've written often on this blog [see note 1], I'm not certain the full set of ethical issues have been well-addressed. It's not limited to this occasion, however. This phenomenon represents a pervasive, continual world-wide oversight with regard to clinical IT.

Furthermore, and finally: of considerable concern is another common limitation of all health IT studies, which I believe is often willful.

What really should be studied before justifications are given to spend tens of millions of dollars/Euros/whatever on CPOE or other clinical IT is this:

The impact of possible non-cybernetic interventions (e.g., additional humans and processes) to improve "medication ordering" (either CPOE, or ePrescribing) that might be FAR LESS EXPENSIVE, and that might have far less IT-caused unintended adverse consequences, than cybernetic "solutions."

Instead, pre-post studies are used to justify expenditures of millions (locally) and tens or hundreds of billions (nationally), with results sometimes like this affecting an entire country.

There is something very wrong with this, both scientifically and ethically.

-- SS

Note:

[1] If these devices are not experimental, why are so many studying them to see if they actually work, to see if they pose unknown dangers, and to try to understand the conflicting results in the literature? More at this query link: http://hcrenewal.blogspot.com/search/label/Healthcare%20IT%20experiment

Addendum Feb. 10, 2012:

An anonymous commenter points out an interesting issue. They wrote:

The study was flawed due to its failure to consider delays in care and medication administration as an error caused by these experimental devices.

Delays are widespread with CPOE devices. One emergency room resorted to paper file cards and vacuum tubes to communicate urgency with the pharmacy. Delays were for hours.

I agree that lack of consideration of a temporal component, i.e., delays due to technology issues, is potentially significant.

I, for example, remember a more than five-minute delay in getting sublingual nitroglycerin to a relative with apparent chest pain due to IT-related causes. The problem turned out to be gastrointestinal, not cardiac; however, in another patient, the hospital might not be so lucky.

Addendum Feb. 12, 2012:

A key issue in technology evaluation studies is to separate the effects of the technology intervention from other, potentially confounding variables which always exist in a complex sociotechnical system, especially in a domain such as medicine. This seems uncommonly done in HIT evaluation studies. Not doing so will likely inflate the apparent contribution of the technology.

A "control ward" where the same education and training, process re-engineering, procedural improvements, etc. were performed as compared to the "intervention ward" (but without actual IT use) would probably be better suited to pre-post studies such as this.

A "comparison ward" where human interventions were implemented, as opposed to cybernetic, would be a mechanism to determine how efficacious and cost-effective the IT was compared to less expensive non-cybernetic alternatives.

-- SS

BUY IT NOW!

Just Say "No" to the Term "Anecdotes"; and HIT as a Medical Metadevice

A New Year's thought: there needs to be a push in healthcare for dropping of the word "anecdote" to describe case reports of health IT-related errors.

This word even appears in the late 2011 IOM report on HIT safety (PDF), e.g., the preface:

... We found that specific types of health IT can improve patient safety under the right conditions, but those conditions cannot be replicated easily and require continual effort to achieve. We tried to balance the findings in the literature with anecdotes from the field but came to the realization that the information needed for an objective analysis and assessment of the safety of health IT and its use was not available.

The "A" word needs to be dropped from the healthcare IT lexicon, since such reports from reliable sources are in fact incident reports purposed for risk management activities.

Incident reports do not need peer review for consideration for that purpose.

Of note, I do not believe the incident reports filed in hospitals when something awry occurs are labelled "anecdotes", either.

See the Aug. 2011 post "From a Senior Clinician Down Under: Anecdotes and Medicine, We are Actually Talking About Two Different Things" for more on this topic.

And on another vein, the issue of HIT being a medical device:

As the good State Rep. Marino of my home state of Pennsylvania and others oddly proffer - that 'certification' of health IT, having nothing to do with safety or usability, relieves HIT from being a device [1] - and as the IOM itself debates exactly what to call HIT and under what guidelines to regulate it, another term/category for HIT devices is needed.

In the spirit of the naming of the UMLS Metathesaurus, and in consideration of HIT's informational governance/orchestration of other medical devices and personnel (including the 'carbon units' known as clinicians and patients) -- I suggest the term "metadevice" for HIT.

Healthcare metadevices need their own specific regulation, apart from traditional medical devices.

-- SS

Note:

[1] As in line 21- 24 on page 6 of the "Safeguarding Access For Every Medicare Patient Act" Bill (PDF) that I wrote about here. The Bill states: "CLARIFICATION OF AUTHORITY. Certified EHR’s shall not be considered a device for purposes of the Federal Food, Drug, and Cosmetic Act."

(This proposal, of course, raises the question of whether Rep. Marino believes non-certified HIT shall be considered a medical device, a topic for another time.)

BUY IT NOW!

From a Senior Clinician Down Under: Anecdotes and Medicine, We are Actually Talking About Two Different Things

A poster who wishes to remain anonymous, a Senior Clinician in the state of Victoria, Australia, added this comment to my March 2011 post on 'anecdotes.' (That post was entitled "Australian ED EHR Study: An End to the Line "Your Evidence Is Anecdotal, Thus Worthless?".)

He makes a critical point I think has gotten lost in the HIT domain (emphases mine):

Anonymous August 15, 2011 9:26:00 PM EDT said...

Anecdote and Medicine.

We are actually talking about two different things here.

1. Anecdotal reporting of a new and potentially exciting finding in Medicine is NEVER a reason to widely implement a new treatment or procedure. It represents the lowest category of evidence in any systematic review In any orthodox system of medicine in the developed world a new intervention would not be ratified or re-imbursed without EXTENSIVE study in Randomised trials - ie the mandatory three phase trial arrangement for new drugs.

2. Anecdotal reporting of side effects/failure for an implemented treatment is a crucial part of any risk management strategy within a healthcare setting. Individual incident reporting of harmful events AND near misses is crucial to help organisations (and regulatory agencies) understand where risks to patients and staff are to be found. A root cause analysis can then be undertaken and corrective measures introduced and their subsequent impact assessed. The process of 'closing the audit loop' is required or no reduction in risk can be verified. This is separate from the regular audit cycle which each Department should apply to aspects of its work, usually reviewing a particular intervention in rotation.

In the above debate the pro e-Health lobby find themselves mis-interpreting both definitions.

They support the positive anecdotes for the adoption of Electronic Health Records WITHOUT proper randomised evidence being available and they decry the anecdotes of negative experiences of implemented systems that in reality represent episodes of incident reporting.

It appears the over-exuberant proponents of e-Health in general and for everything, need to attend revision courses in research methodology and risk management... [a fascinating observation - ed.]

nb Irony of irony;

Victoria now has State-wide implementation of risk management software which for all healthcare staff is the obligatory reporting mechanism for all incidents and near-misses.

The system is so user unfriendly and time demanding virtually none of the busy hospital doctors I have spoken to access the system, even though 'training' has been undertaken. The system has been widely condemned at our Medical Staff Committee.

Victoria has recently congratulated itself for a fall in annual number of critical incidents occurring in public hospitals!

I will leave it to you to work out how such a positive risk management statistic could be generated in a healthcare system working near capacity with increasing year on year demand for its services...

Per these observations:

A critical distinction that seems to have become lost, even among the Medical Informatics academic elite (see, for instance, my Sept. 2010 post "The Dangers of Critical Thinking in A Politicized, Irrational Culture"), is the distinction between research observations on the one hand, and risk management-relevant incident reports on the other.

It seems a form of erroneous thinking or logical fallacy.

The lost distinction between research methods and risk management methods, that require very essential, very different consideration of "anecdotes", and the conflation of the two types of "anecdotes", are brilliant observations.

Finally, the loss of consideration of the distinctions between the two different types of "anecdotal reporting" is part of what I have termed the lack of the rigor of medicine itself in HIT.

-- SS

BUY IT NOW!

Those Who Dismiss Healthcare (and Healthcare IT) Adverse Events Reports as Mere "Anecdotes" Have Lost - Supreme Court-Style

At my Sept. 2010 post "The Dangers of Critical Thinking in A Politicized, Irrational Culture" I wrote:

... It's the EMR "anecdotalists" (as opposed to the "Markopolists") who say that "anecdotes" of HIT-related injury are meaningless. They deem reports of safety issues and HIT-related misadventures and risk as simply "anecdotal", and that "anecdotes don't make evidence" (or "anecdotes don't make data").

For "anecdotes" of patient harm due to medical devices even from the most reliable of sources to be counted as "evidence" of device risk, apparently, the stories need to be blessed with Statistical Holy Water. The Holy Water must also be of a brand approved by the academic pundits.

For me, this is no longer merely a professional debate. My elderly relative became one of those "anecdotes" in May last year.

I address the casual, Dogbert-style, waving-of-the-hand "Bah!" dismissal of health IT harm "anecdotes" at numerous other posts as well, such as "EHR Problems? No, They're Merely Anecdotal" and "Health IT: On Anecdotalism and Totalitarianism".

Bah! Your Health IT adverse events reports are anecdotes, and anecdotes don't make data!

In those posts I also mention how Australian informatics professor Dr. Jon Patrick had essentially hit the flaws of this argument out of the Southern hemisphere with a short editorial in the journal "Applied Clinical Informatics" entitled "The Validity of Personal Experiences in Evaluating HIT." That essay is free at the link and is worth reading.

Interestingly and thankfully, the "anecdotes are meaningless" crowd have now lost, and lost big - Supreme Court style. In fact, the U.S. Supreme Court has shown far more common sense than many esteemed academics and industry pundits.

As noted in this post at Derek Lowe's pharmaceutical industry "In the Pipeline" blog, the company that made "Zicam", a zinc-based over-the-counter cold remedy, tried to defend shareholder suits that the company withheld case reports of Zicam causing permanent loss of smell via arguing that such reports "did not reach a level of statistical significance", i.e., were "anecdotal." The case went to the U.S. Supreme Court.

The Supreme Court would have none of that argument:

"Matrixx’s [Zicam's manufacturer - ed.] premise that statistical significance is the only reliable indication of causation is flawed. Both medical experts and the Food and Drug Administration rely on evidence other than statistically significant data to establish an inference of causation. It thus stands to reason that reasonable investors would act on such evidence.

The full court decision is at this link: http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf (PDF file), but a passage I consider key to this issue is as follows:

... We conclude that the materiality of adverse event reports cannot be reduced to a bright-line rule ... Because adverse reports can take many forms, assessing their materiality is a fact-specific inquiry, requiring consideration of their source, content, and context.

This is common sense incarnate. It applies not just to drugs, but to medical devices, to health IT, and to other domains as well.

In essence, it is saying that adverse events reports, especially repeated ones, from trustworthy sources are not to be lightly dismissed, but should serve at the very least as red flags that there may be a systemic problem requiring further investigation.

One wonders how and if public healthcare IT vendors will begin disclosing "anecdotal" reports of their products causing patient harm to their own stockholders.

One also wonders if the academic anecdotalists (up to the level of the chair of the Office of the National Coordinator for Health Information Technology at HHS) will cease their unfettered dismissal of health IT AE reports as mere "anecdotes" and therefore let's roll out this 100% beneficent technology nationwide ASAP:

"Nothing [ONC has] found would give them any pause that a policy of introducing EMR's could impede patient safety." - David Blumenthal

That sounds a bit like the refrain of the makers of Zicam.

One might also wonder if the anecdotalists merely lack common sense, or are using this form of epistemological dementia to obscure conflict of interest.

On a final note, my favorite comment at the aforementioned "In the Pipeline" blog story is this by anonymous commenter "Still Scared of Dinosaurs":

One of the most important ideas real statisticians must get into their heads is "Thou shalt not worship the 0.05 threshold". The whole concept of "statistical significance" for AEs is idiotic and the fact that Matrixx based any part of their defense on it indicates that their stupidity did not end when they named the company.

Perhaps this Dilbert cartoon is apropos to the Supreme Court decision:


-SS

Addendum:

I thought it appropriate to share these thoughts with the leadership of the Joint Commission, the organization that accredits healthcare organizations in the United States:

From: Scot Silverstein
Sent: Sunday, March 27, 2011 10:41 AM
To: MGiuntoli, Anita; Chassin, Mark; Schyve, Paul; Legaspi, Shirley
Cc: Ross Koppel; 'David Kreda'
Subject: Re: MATRIXX INITIATIVES, INC., ET AL. v. SIRACUSANO ET AL.

Not a complaint this time [about health IT failure - ed.], but an observation.

The JC has noted health IT risks in the Sentinel Events Alert "Safely implementing health information and converging technologies" of 2008.

The company that made "Zicam", a zinc-based over-the-counter cold remedy, tried to defend shareholder suits that the company withheld case reports of Zicam causing permanent loss of smell via arguing that such reports "did not reach a level of statistical significance", i.e., were "anecdotal." The case went to the U.S. Supreme Court.

The Supreme Court would have none of that argument:

"Matrixx’s [Zicam's manufacturer] premise that statistical significance is the only reliable indication of causation is flawed. Both medical experts and the Food and Drug Administration rely on evidence other than statistically significant data to establish an inference of causation. It thus stands to reason that reasonable investors would act on such evidence.

The full court decision is at this link: http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf (PDF file), but a passage I consider key to this issue is as follows:

... Because adverse reports can take many forms, assessing their materiality is a fact-specific inquiry, requiring consideration of their source, content, and context.

This is common sense incarnate. It applies not just to drugs, but to medical devices, to health IT, and to other domains as well.

I believe JC should start to pay serious attention to "anecdotal reports" of health IT-caused patient injury, and consider reliable reporting of these events as an Accreditation standard.

As I noted in my July 2009 JAMA letter to the editor "Health Care Information Technology, Hospital Responsibilities, and Joint Commission Standards" in response to Koppel and Kreda's JAMA article on HIT industry practices, "hold harmless" and "gag" clauses must go, and be replaced with proactive reporting of healthcare IT-related "events."

Scot Silverstein

-- SS

BUY IT NOW!

Australian ED EHR Study: An End to the Line "Your Evidence Is Anecdotal, Thus Worthless?"

At my Sept. 2010 post "The Dangers of Critical Thinking in A Politicized, Irrational Culture" I lamented that while my early mentor in biomedicine Victor P. Satinsky MD taught the wise credo "critical thinking always, or your patient's dead" in the 1970's, our culture had become so perverse that this credo had been largely supplanted with:

"Critical thinking anytime, and your career's dead."

That post was in reaction to continued heckling on a professional mailing list, the American Medical Informatics Association (AMIA) Clinical Information Systems Working Group (cis-wg). On this list, Oregon Health Sciences University professor William Hersh expounded on how the evidence of health IT dangers was largely "anecdotal" therefore to be discounted, and how I, specifically, "didn't know the literature on health IT."



In that post I put the lie to the latter figment. On the former prevarication Dr. Jon Patrick, author of the recent thorough dissection of problems with the ED EHR system being rolled out in public hospitals in New South Wales, Australia (see my Mar. 5, 2011 post "On an EMR Forensic Evaluation by Professor Jon Patrick from Down Under: More Thoughts"), hit the ball out of the park:

Prof. Patrick to AMIA cis-wg:



I think such defences are particularly unuseful especially with respect to the dismissal of personal stories and experiences as "anecdotes", hence committing them to the realm of folklore. I offer these notions as a counterpoint.



Discounting Anecdotes:



1. Is a perfidious and specious act.



2. It denies early warning signs of problems.



3. It denies a voice and disempowers the working clinical community who have to operationalise decisions made by others.



4. It denies a route to process improvement within an institution - which is most important for EBM and incremental review of local processes.



5. It defends software manufacturers from fault rectification - cuts off even a need to deliberate on it. Critics of the value of anecdotes are squarely on the side of the faulty and deficient manufacturer.



6. A rule of project management is that projects consist of 3 components, cost, quality and time and if their needs to be a compromise it has to be on quality. Anecdotes are early warning signs of such a compromise.

Prof. Patrick had to once again put the lie to this refrain at a comment on the HISTalk blog yesterday, where the anonymous proprietor had written in a review of Patrick's EHR study:

... On the other hand, I wouldn’t say it’s [Prof. Patrick's Cerner FirstNet study] necessarily unbiased, it focuses on implementation of a single department application that didn’t go well for a variety of reasons (despite many successful FirstNet implementations elsewhere), it uses the unchallenged anecdotal comments of unhappy users who make it clear they liked their previous EDIS better, and it nitpicks (I wasn’t moved to find a pitchfork when I learned that the primary keys in the Millennium database aren’t named consistently).



[Considering the complexity and changeability of healthcare and the corresponding software lifecycle, I duly note that that latter attitude about 'failing to find pitchforks' regarding breaches of sound software engineering practices seems to be a symptom of the larger health IT disease that Prof. Patrick writes about - ed.]

Dr. Patrick then knocked the ball out of the Southern Hemisphere with a comment of his own about anecdotalism and a link to an expansion of the aforementioned ideas he'd shared on "discounting anecdotes":

Prof. Patrick to HISTalk owner:



Your Comment “it uses the unchallenged anecdotal comments of unhappy users ” is not only unfair but unreasonably inaccurate. The comments made by the users are the Directors of 7 EDs and so they have a right to carry authority by virtue of the experience but also the number of 6 out 7 presenting a view of Firstnet as unfit for purpose has numeric validity, which they justify with about 20+ pages of their comments – see Part 2 Appendix 2.



I would also point readers to my editorial about the role of personal experiences being the most useful information to understand the nature of socio-technical failures. http://aci.schattauer.de/en/contents/archive/issue/1124/manuscript/15463/show.html

The essay at that link, "The Validity of Personal Experiences in Evaluating HIT", is an editorial in Johns Hopkins informaticist Chris Lehmann's brilliant new journal "Applied Clinical Informatics."



The editorial is available free, and is a must-read for anyone in a decision-making or managerial role in mission critical domains, including our elected representatives.



In the editorial Dr. Patrick concludes. similarly to his earlier AMIA mailing list opinion:

... the denial of recounted personal experiences in discussion and analysis of HIT is biased and specious and has the effect of:



1. Denying early warning signs of problems.



2. Denying a voice for the working clinical community who have to operationalise decisions made by others and thus disempowers them.



3. Denying process improvement within an institution – which is most important for Evidence Based Medicine and incremental review of local processes.



4. Discourages staff from engaging in any form of process improvement hence worsening the sense of disenchantment.



Every legitimate personal experience of a HIT deserves to be considered on its merits lest we wish to retreat from process and product improvement. Mechanisms of censorship both implicit due to contrived processes of disinformation and disempowerment or explicit due to contractual specifications will lead to more waste, lost productivity, contempt for the providers, and distress among frontline staff rather than increased productivity and improved patient health and safety as we all desire.

In my view, the drivers or motivators for the "anecdotalist" accusation are these, singly or in combination:

• Too much "education" to see the nose on one's face, as in, to think zebras and unicorns instead of horses when hearing hoofbeats outside one's midwest U.S. abode (eggheads);
  
  
• Too little common sense (fools), as in Scott Adams' example: "IGNORING ALL ANECDOTAL EVIDENCE - Example: I always get hives immediately after eating strawberries. But without a scientifically controlled experiment, it’s not reliable data. So I continue to eat strawberries every day, since I can’t tell if they cause hives";
  
  
• Too much concern for the possible interruption of flow of money or power in one's direction (gonifs).
  
  

In conclusion, the anecdotalist refrain of "your evidence is anecdotal" [therefore of little or no value] when used repetitively against competent observers is the refrain of eggheads, fools and gonifs.



In healthcare, the end result is "your patient's dead."



My "anecdotal relative" injured in a mid-2010 HIT mishap is sadly an example.



[June 2011 addendum: my relative, after much suffering, has now died of complications of the "anecdotal HIT mishap" - ed.]



As for myself, I am a Markopolist (see my Sept. 2010 post "Health IT: On Anecdotalism and Totalitarianism").



-- SS

BUY IT NOW!

The Dangers of Critical Thinking in A Politicized, Irrational Culture

My early mentor in biomedicine Victor P. Satinsky MD lived by the credo "critical thinking always, or your patient's dead."

Unfortunately, the motto of today's degraded culture in biomedicine (and other domains) might well be "critical thinking, and your career is dead."

At "Health IT: On Anecdotalism and Totalitarianism" I posted these thoughts:

At the article Blumenthal on EMRs: Debate "raging" over competition vs. standards, ONC czar David Blumenthal is cited as saying several interesting things:

... EMRs make him a better physician, he said, recounting personal anecdotes of discovering patients' allergies through automated EMR alerts and using stored image date to more quickly get a diagnosis for a patient without subjecting them to more radiation and toxic radiation agents ...

It's the EMR "anecdotalists" (as opposed to the "Markopolists") who say that "anecdotes" of HIT-related injury are meaningless. They deem reports of safety issues and HIT-related misadventures and risk as simply "anecdotal", and that "anecdotes don't make evidence" (or "anecdotes don't make data").

Yet anecdotal reports of EMR "saves" are used by a czar to justify tens of billions of dollars of expenditures?

To the anecdotalists, I say: you can't have it both ways.

I also posted nearly the same complete Healthcare Renewal post to several mailing lists of the American Medical Informatics Association including the Clinical Information Systems working group (CIS-WG). CIS-WG is a mailing list read by something over 1000 healthcare informatics professionals at last time I had access to the statistics a few years ago.

I received some supportive replies from colleagues, including collaborators on the AHIMA (not AMIA) book we co-authored in 2009 entitled "H.I.T. or Miss: Lessons Learned from Health Information Technology Implementations" - itself not exactly a popular exercise among the strictly positivist informatics leadership class.

Now, I thought my posting on the double standard regarding "anecdotes" highly straightforward. From a high ranking academic leader of a major national informatics program, Bill Hersh at OHSU, however, the following reply was posted:

Scot,

For someone who is a faculty in informatics, I am surprised at how unfamiliar you are with the literature. There is solid evidence, much more than anecdotes, on the efficacy of health IT. Even Dr. Blumenthal himself has posted on that. (I think you are taking this quote out of context.

I am then served a platter of literature I must be "unfamiliar with" such as:

Goldzweig, C., Towfigh, A., et al. (2009). Costs and benefits of health information technology: new trends from the literature. Health Affairs, 28: w282-w293.

[Note - I had commented on and linked to this very article at this Aug. 29, 2010 post - ed.]

Garg, A., Adhikari, N., et al. (2005). Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Journal of the American Medical Association, 293: 1223-1238.

Amarasingham, R., Plantinga, L., et al. (2009). Clinical information technologies and inpatient outcomes: a multiple hospital study. Archives of Internal Medicine, 169: 108-114.

Longhurst, C., Parast, L., et al. (2010). Decrease in hospital-wide mortality rate after implementation of a commercially sold computerized physician order entry system. Pediatrics, 126: 14-21.

Now, aside from the serious breach of academic etiquette of attacking the competence of your colleagues in a public forum, I seem to be hearing that it's OK to purvey positive anecdotes about health IT (usually based on weak retrospective observational studies alone, not randomized clinical trials), but not anecdotes of HIT malfunctions or of HIT-related adverse outcomes, since there's 'solid evidence' of the efficacy of health IT.

In plain English, an ad hominem fallacy is followed by an appeal to authority of sorts ("the literature") to justify public Pollyanna attitudes towards HIT by high ranking officials. And since the literature is so glowing, negative anecdotes must be of low worth.

[Jan. 2011 addendum - perhaps the literature's not so glowing - ed.]

Actually, the response in its entirety was a non sequitur to my post.

Others in cis-wg took affront. One of my book co-authors responded that:

I didn't read Scot's comment as saying that there is no data in support of EHRs .... despite a body of evidence, Dr. Blumenthal made a statement only about personal experience in what Scot quoted.

At the same time, ONC has asked for EHR users to share their positive experiences, but has not (as far as I have seen) asked for their failures. Quite frankly, the failures would be more instructive and would constitute a very valuable repository. ONC has also not shared the studies on the dangers and failures of EHR implementations with nearly the same passion as the successes. My point is that there is data and there is anecdote for both sides and ONC has not presented a balanced picture so that we can adequately address the real risks.

Another CIS-WG reader shared valuable observations:

Regarding Goldzweig:

Regarding studies conducted by the HIT leaders (e.g. Partners Vandy, Regenstrief, IHC, etc...): "Many of the new studies report modest or even no benefits of the new applications or changed functionalities."

Regarding studies of commercial HIT systems: "These study results were similar to those reported by the health IT leaders—most studies demonstrated modest benefits, some demonstrated no benefits, and a few demonstrated marked benefits."

Regarding Adhikari:

The CDSS improved practitioner performance in 62 (64%) of the 97 studies assessing this outcome,

52 trials assessed 1 or more patient outcomes, of which 7 trials (13%) reported improvements.

And so on.

In other words, the literature's mixed.

Finally, the knock-the-ball-out-of-the-park response came from a Medical Informatics researcher Down Under:

I think such defences are particularly unuseful especially with respect to the dismissal of personal stories and experiences as "anecdotes", hence committing them to the realm of folklore. I offer these notions as a counterpoint.

Discounting Anecdotes:

1. Is a perfidious and specious act.

2. It denies early warning signs of problems.

3. It denies a voice and disempowers the working clinical community who have to operationalise decisions made by others.

4. It denies a route to process improvement within an institution - which is most important for EBM and incremental review of local processes.

5. It defends software manufacturers from fault rectification - cuts off even a need to deliberate on it. Critics of the value of anecdotes are squarely on the side of the faulty and deficient manufacturer.

6. A rule of project management is that projects consist of 3 components, cost, quality and time and if their needs to be a compromise it has to be on quality. Anecdotes are early warning signs of such a compromise.

I, of course, added that ignoring "anecdotes" of HIT problems was even more cavalier if one recognized the context of the stories, that is, that they arise in an environment hostile to diffusion through contractual arrangements, poorly recognized reporting resources, fear, etc. Understood in context, they should be receiving more research attention than otherwise, and certainly not ignored.

However, the comment about my purported lack of knowledge of the literature was sent out to 1000+ people by a nationally-recognized informatics leader, people who may or may not read the followup in detail.

This is unfortunate and perhaps reflects the ethos of our day.

-- SS

Addendum:

Also see the Aug. 2011 post "From a Senior Clinician Down Under: Anecdotes and Medicine, We are Actually Talking About Two Different Things" for a truly stunning takedown of the "anecdotes" canard, which amounts to conflating risk management with scientific discovery.

BUY IT NOW!