At my Nov. 2011 post "Lifespan (Rhode Island): Yet another health IT glitch affecting thousands - that, of course, caused no patient harm that they know of - yet" I wrote:
Add another case to the health IT glitch file, under the "do we feel lucky today?" patient risk category.
From the Pittsburgh Post-Gazette (I am quoted):
First, as my aforementioned Nov. 2011 post and its contained links point out, these events are not as "rare" as they should be. (The asteroid colliding with Earth that caused the extinction of the dinosaurs - now that's a "rare" event.)
Second, as multiple posts on this blog have pointed out, the claims that "no patients were harmed" is both misleading and irrelevant:
Such claims of 'massive EHR outage benevolence' are misleading, in that medical errors due to electronic outages might not appear for days or weeks after the outage, depending on what information was corrupted/lost/misindentified/or otherwise mishandled after it is 'backloaded' once the system is up. All it takes is one med lost to cause misery and death. (I can speak about that from unfortunate personal experience.
Claims of 'massive EHR outage benevolence' are also irrelevant in that, even if there was no catastrophe directly coincident with the outage, their was greatly elevated risk. Sooner or later, such outages will maim and kill.
Faced 'some challenges?' In other words, care was compromised by the outage and the 'challenges' were to avoid medical error (and, of course, to make sure billing was unaffected):
Compromised -
I cannot, for example, think of a single instance where thousands of paper records went unavailable simultaneously (unless, that is, someone lost the key to the Medical Records department), were made available to identity thieves en masse, or where thousands of medical orders were scrambled or truncated in a relatively short period of time as in Rhode Island.
These amplified risks could wipe out any advantages of EHR's over paper in a microsecond.
A partial list of facilities apparently affected in this latest episode of EHR mayhem, from this list:
That accounts for several thousand active patients, I am sure.
(12/28 Addendum: Bed counts of PA hospitals are here. Searching on "University of Pittsburgh Medical Center", it can be seen that thousands of beds were indeed involved.)
This seems at best an insensitive and perhaps even inhumane bit of P.R. More "cumbersome" for the clinicians? What about the poor patients? How would Ms. Zellner feel, I wonder, if it were her mother, child or significant other on the Operating Room table or having an acute MI when the EHR/CPOE systems went down?
It appears P.R. is not very high on the list for receiving information when a crisis arises. I may have known of the outage before they did.
(That is, it's not our fault, it's the fault of the database vendor. Hospitals, I regret to inform you - you are responsible for unapproved medical devices used in your facilities, no matter what the source.)
And there's that word "glitch" again, accompanied by the equally banal "bug."
Me again in the Post-Gazette:
Somehow, when a clinician makes a mistake, the terms "bug" and "glitch" are never used. In fact, when clinicians fail to meet accepted professional standards of healthcare practice, it is called "malpractice."
I think we can all agree that a major near-full-day outage of an enterprise EHR affecting multiple hospitals and thousands of patients does not meet accepted professionals standard of life-critical computing practice. Yet, all this merits is the word "glitch." It seems to me that if patients are harmed by, in reality, what is (on its face) IT malpractice during such events, not only the clinicians affected should be held liable.
The usual refrain. Let me repeat my definition of "compromised:"
Compromised -
A simple question - if extended EHR outages like this never seem to "compromise" care, then why not eliminate health IT entirely and spend the hundreds of millions saved on patient care?
I leave it to the readers to ascertain the computer expertise levels and reasonableness of what Ms. Zellner thinks a "crash" is.
They "don't know what triggered the 'problem'" - is a proper translation that they have no idea what went wrong?
In fact, regarding another Cerner EHR system which was extensively studied (see "A Study of an Enterprise Information System" at this link), Dr. Jon Patrick came to the conclusion that one of the sources of catastrophic failures is poor software engineering that has made the behavior of the studied system "non-deterministic." Further, software upgrades are not protected from incremental changes made by maintenance and customization staff, and may introduce even more instability.
A software upgrade without clearly understanding "what triggered the problem" is simply asking for more trouble. (My bet, however, is that they attempt it anyway.)
What's to say?
How about this:
Repeating UMPC's statement from the article that appeared after I gave my quotes to the reporters: "Doctors and nurses continued to have access to patients' electronic records through backup systems, [the UPMC spokesperson] said. They also had to resort to using old-fashioned paper records for documentation and orders."
My stated fears of disruption and increased risk due to compromised care seem well-grounded.
Wait ... I thought I'd heard these events were "rare." Two in the same city within six months?
---------------------------
Truth be told:
The primary rule in computing is:
Clearly the latter was the case here.
The following questions arise:
Further, the soft-selling of these incidents must end. The use of terms such as "bug" and "glitch" must also end. What occurred here, echoing my newspaper quote, was a disruptive, potentially catastrophic major malfunction of a life-critical enterprise medical device.
System-wide EHR crashes are not merely ‘glitches’ or ‘bugs.’ They need to be considered, as in medicine itself, as 'never events.' From AHRQ:
Further, re: "patient care was never compromised." How do they know that? In fact, this is 'spin' and word games on its face. By definition, if CPOE and chart updating was unavailable, patient care was compromised, where "compromised" means "increased levels of risk for error were created, requiring workarounds."
Further, as mentioned earlier, harms might not show up for some time. Lost orders, corrupted data, errors of omission or commission transcribing backup paper records into the computer ("backloading"), etc. can take their toll later. Post-outage vigilance is essential, putting even more stress on clinicians that increases likelihood of further error and that they certainly do not need. Clinicians are stressed enough already.
Finally:
IT personnel have not only deliberately inserted themselves into clinical affairs (e.g, via the HITECH Act of 2009), they have also done so with a stunning arrogance and unproven braggadocio about their systems "revolutionizing" medicine (whatever that means).
Indeed, they need to accept the medical responsibility and obligations this territorial intrusion entails.
On its face, this massive outage was the result of issues that did not meet accepted professional standards of IT practice for life-critical environments. Res ipsa loquitur.
Something was not vetted properly, there was a lack of redundancy, the IT personnel were NOT in control of their systems.
Just as when physicians don't provide care that meets accepted professional standards of healthcare, this incident and others like it are, by definition, a result of IT malpractice.
If patients are harmed, IT personnel and their management (often non-IT C-level officers) involved in this system need to be held accountable.
If they can't take the clinical heat (as clinicians do daily since the time they enter medical or nursing school), then they need to get out of the clinical kitchen.
-- SS
Note: see this take on these matters at the HIStalk blog:
Now we know who the unnamed "mystery database vendor" is...
-- SS
Dec. 29, 2011 Addendum:
Was UPMC acting as a "proving ground" for some Oracle-Cerner-UPMC experimental health IT technology that resulted in the crash? The claim of being an IT "proving ground" has been made in the past:
As I wrote at "Proving Ground for IT Tests On Children: Pioneers in Health IT, or Pioneers in Ignoring the Past?":
"A hospital and patients are not a learning lab for HIT vendors. The appropriate "proving ground" for new medical technology is the controlled clinical trial where participants (in this case, patients and healthcare professionals alike) have freedom of choice whether or not to participate, and a chance to give (or deny) consent after being fully informed of potential risk."This is a fundamental human rights issue.
-- SS
There's been yet another health IT "glitch" that, of course, caused no patients to be harmed. See other "glitches" here, here, here and at other posts which can be found by searching this blog on the banal term 'glitch'.
Add another case to the health IT glitch file, under the "do we feel lucky today?" patient risk category.
From the Pittsburgh Post-Gazette (I am quoted):
Computer outage at UPMC called 'rare' Systemwide disruption potentially dangerous, expert warns Saturday, December 24, 2011 By Jonathan D. Silver, Pittsburgh Post-Gazette
UPMC's electronic medical records system for inpatients went offline for more than 14 hours at nearly all its hospitals in the region, marking what the health system called a "rare" outage, but one that it claims did not harm patients.
First, as my aforementioned Nov. 2011 post and its contained links point out, these events are not as "rare" as they should be. (The asteroid colliding with Earth that caused the extinction of the dinosaurs - now that's a "rare" event.)
Second, as multiple posts on this blog have pointed out, the claims that "no patients were harmed" is both misleading and irrelevant:
Such claims of 'massive EHR outage benevolence' are misleading, in that medical errors due to electronic outages might not appear for days or weeks after the outage, depending on what information was corrupted/lost/misindentified/or otherwise mishandled after it is 'backloaded' once the system is up. All it takes is one med lost to cause misery and death. (I can speak about that from unfortunate personal experience.
Claims of 'massive EHR outage benevolence' are also irrelevant in that, even if there was no catastrophe directly coincident with the outage, their was greatly elevated risk. Sooner or later, such outages will maim and kill.
The outage affected a system designed by Cerner Corp., a global electronic records company, and customized by UPMC that doctors and nurses rely on for communication about patient records, medical orders and prescriptions.
It was unavailable from about 8:45 p.m. Thursday to 11 a.m. Friday at almost all of UPMC's hospitals except for Children's and UPMC Hamot in Erie, spokeswoman Wendy Zellner said.
"This is rare. This kind of widespread, extensive downtime would be rare," Ms. Zellner said.
Doctors and nurses continued to have access to patients' electronic records through backup systems, she said. They also had to resort to using old-fashioned paper records for documentation and orders.
"These things happen. They have really well spelled-out procedures for what to do when something goes down," Ms. Zellner said.
She acknowledged that doctors and nurses faced some challenges.
Faced 'some challenges?' In other words, care was compromised by the outage and the 'challenges' were to avoid medical error (and, of course, to make sure billing was unaffected):
Compromised -
a. To expose or make liable to danger, suspicion, or disrepute
b. To reduce in quality, value, or degree; weaken or lower.
Thousands of patients were affected, again reinforcing my point about how IT can and does greatly amplify the risks of paper -- as in my Rhode Island post -- such as errors and confidentiality breaches.I cannot, for example, think of a single instance where thousands of paper records went unavailable simultaneously (unless, that is, someone lost the key to the Medical Records department), were made available to identity thieves en masse, or where thousands of medical orders were scrambled or truncated in a relatively short period of time as in Rhode Island.
These amplified risks could wipe out any advantages of EHR's over paper in a microsecond.
A partial list of facilities apparently affected in this latest episode of EHR mayhem, from this list:
That accounts for several thousand active patients, I am sure.
(12/28 Addendum: Bed counts of PA hospitals are here. Searching on "University of Pittsburgh Medical Center", it can be seen that thousands of beds were indeed involved.)
"Whenever people aren't working in their native system and workflow I have to believe that is more cumbersome for the clinicians, but these folks are well-trained in what to do when these things happen."
This seems at best an insensitive and perhaps even inhumane bit of P.R. More "cumbersome" for the clinicians? What about the poor patients? How would Ms. Zellner feel, I wonder, if it were her mother, child or significant other on the Operating Room table or having an acute MI when the EHR/CPOE systems went down?
Ms. Zellner said UPMC's public relations staff was unaware of the outage until contacted by a reporter.
It appears P.R. is not very high on the list for receiving information when a crisis arises. I may have known of the outage before they did.
The outage was caused by a "bug" or glitch in software designed by a vendor affiliated with Cerner, Ms. Zellner said. She refused to identify the company.
"We're not trying to point fingers at different vendors. It's a database bug, that's all I can tell you."
(That is, it's not our fault, it's the fault of the database vendor. Hospitals, I regret to inform you - you are responsible for unapproved medical devices used in your facilities, no matter what the source.)
And there's that word "glitch" again, accompanied by the equally banal "bug."
It's just a "bug." Cute little critter!
Me again in the Post-Gazette:
Scot M. Silverstein, a doctor and assistant professor Healthcare Informatics at Drexel University in Philadelphia, disagreed with the use of the terms "bug" and "glitch."
"What occurred here was a disruptive, potentially dangerous major malfunction of a life-critical enterprise medical device," he said.
Somehow, when a clinician makes a mistake, the terms "bug" and "glitch" are never used. In fact, when clinicians fail to meet accepted professional standards of healthcare practice, it is called "malpractice."
I think we can all agree that a major near-full-day outage of an enterprise EHR affecting multiple hospitals and thousands of patients does not meet accepted professionals standard of life-critical computing practice. Yet, all this merits is the word "glitch." It seems to me that if patients are harmed by, in reality, what is (on its face) IT malpractice during such events, not only the clinicians affected should be held liable.
Ms. Zellner said the problem was not a "crash" of the system because there were alternate methods used to cope that prevented patient care from being compromised.
The usual refrain. Let me repeat my definition of "compromised:"
Compromised -
a. To expose or make liable to danger, suspicion, or disrepute
b. To reduce in quality, value, or degree; weaken or lower.
A simple question - if extended EHR outages like this never seem to "compromise" care, then why not eliminate health IT entirely and spend the hundreds of millions saved on patient care?
"This is not a crash of Cerner either," Ms. Zellner said. "I think a crash is, 'Oh my God, the sky is falling,' nobody can get anything."
I leave it to the readers to ascertain the computer expertise levels and reasonableness of what Ms. Zellner thinks a "crash" is.
Technicians from UPMC, Cerner and the third company [the 'mystery' database company? - ed.] worked together on-site to identify and fix the problem. Ms. Zellner said she did not know why it took 14 hours to fix and the underlying cause was still unclear.
"They know what the problem is and I believe it's been fixed, but we really don't know what triggered it," Ms. Zellner said. "I think the next step would be some actual software upgrades."
They "don't know what triggered the 'problem'" - is a proper translation that they have no idea what went wrong?
In fact, regarding another Cerner EHR system which was extensively studied (see "A Study of an Enterprise Information System" at this link), Dr. Jon Patrick came to the conclusion that one of the sources of catastrophic failures is poor software engineering that has made the behavior of the studied system "non-deterministic." Further, software upgrades are not protected from incremental changes made by maintenance and customization staff, and may introduce even more instability.
A software upgrade without clearly understanding "what triggered the problem" is simply asking for more trouble. (My bet, however, is that they attempt it anyway.)
A Cerner representative could not be reached for comment.
What's to say?
How about this:
Dr. Silverstein said based on what he was told about the computer outage, it means that hospital medical staff would have been unable to update patient charts and probably would not have been able to issue any orders through the system during the time it was off line.
He also questioned how up-to-date the hospital's redundant records were.
Repeating UMPC's statement from the article that appeared after I gave my quotes to the reporters: "Doctors and nurses continued to have access to patients' electronic records through backup systems, [the UPMC spokesperson] said. They also had to resort to using old-fashioned paper records for documentation and orders."
My stated fears of disruption and increased risk due to compromised care seem well-grounded.
In May, Allegheny General Hospital had to shut its electronic medical records computer system down because of problems with the vendor's hardware.
The hospital used backup procedures to continue care for patients, including using paper orders and record-keeping.
Wait ... I thought I'd heard these events were "rare." Two in the same city within six months?
---------------------------
Truth be told:
The primary rule in computing is:
Either you are in control of your information systems, or they are in control of you.
Clearly the latter was the case here.
The following questions arise:
- Was the software containing the "bug" properly vetted before being used on live patients? This is not just the vendor's obligation.
- If it was not vetted properly, why not?
- Was it an "upgrade" or patch? (If so, the same vetting rules apply.)
Further, the soft-selling of these incidents must end. The use of terms such as "bug" and "glitch" must also end. What occurred here, echoing my newspaper quote, was a disruptive, potentially catastrophic major malfunction of a life-critical enterprise medical device.
System-wide EHR crashes are not merely ‘glitches’ or ‘bugs.’ They need to be considered, as in medicine itself, as 'never events.' From AHRQ:
The term "Never Event" was first introduced in 2001 by Ken Kizer, MD, former CEO of the National Quality Forum (NQF), in reference to particularly shocking medical errors (such as wrong-site surgery) that should never occur. Over time, the list has been expanded to signify adverse events that are unambiguous (clearly identifiable and measurable), serious (resulting in death or significant disability), and usually preventable.
Further, re: "patient care was never compromised." How do they know that? In fact, this is 'spin' and word games on its face. By definition, if CPOE and chart updating was unavailable, patient care was compromised, where "compromised" means "increased levels of risk for error were created, requiring workarounds."
Further, as mentioned earlier, harms might not show up for some time. Lost orders, corrupted data, errors of omission or commission transcribing backup paper records into the computer ("backloading"), etc. can take their toll later. Post-outage vigilance is essential, putting even more stress on clinicians that increases likelihood of further error and that they certainly do not need. Clinicians are stressed enough already.
Finally:
IT personnel have not only deliberately inserted themselves into clinical affairs (e.g, via the HITECH Act of 2009), they have also done so with a stunning arrogance and unproven braggadocio about their systems "revolutionizing" medicine (whatever that means).
Indeed, they need to accept the medical responsibility and obligations this territorial intrusion entails.
On its face, this massive outage was the result of issues that did not meet accepted professional standards of IT practice for life-critical environments. Res ipsa loquitur.
Something was not vetted properly, there was a lack of redundancy, the IT personnel were NOT in control of their systems.
Just as when physicians don't provide care that meets accepted professional standards of healthcare, this incident and others like it are, by definition, a result of IT malpractice.
If patients are harmed, IT personnel and their management (often non-IT C-level officers) involved in this system need to be held accountable.
If they can't take the clinical heat (as clinicians do daily since the time they enter medical or nursing school), then they need to get out of the clinical kitchen.
-- SS
Note: see this take on these matters at the HIStalk blog:
UPMC’s Cerner systems go down for 14 hours at most campuses last Thursday and Friday, forcing them to go back to paper. The PR person blamed “a database bug,” which makes the above Oracle press release from this past summer a particularly fun read. Cerner and UPMC have an atypical vendor-customer relationship since they’ve invested big money together in innovation projects and UPMC runs a Cerner implementation business overseas.
Now we know who the unnamed "mystery database vendor" is...
-- SS
Dec. 29, 2011 Addendum:
Was UPMC acting as a "proving ground" for some Oracle-Cerner-UPMC experimental health IT technology that resulted in the crash? The claim of being an IT "proving ground" has been made in the past:
Pittsburgh Tribune
May 2, 2006UPMC partners with technology provider
The University of Pittsburgh Medical Center is taking another step in a quest to commercialize new medical technology.
UPMC on Monday signed a three-year deal with health care information technology provider Cerner Corp. to develop and market medicine-related technological advances. Both parties will contribute $10 million in cash, services and intellectual property to the effort.
The deal is a smaller version of an April 2005 deal between UPMC and information technology behemoth IBM.
As is the case in the IBM deal, UPMC will serve as a built-in proving ground for jointly developed technologies and products, with Cerner marketing the products and UPMC awarded a share of profits.
As I wrote at "Proving Ground for IT Tests On Children: Pioneers in Health IT, or Pioneers in Ignoring the Past?":
"A hospital and patients are not a learning lab for HIT vendors. The appropriate "proving ground" for new medical technology is the controlled clinical trial where participants (in this case, patients and healthcare professionals alike) have freedom of choice whether or not to participate, and a chance to give (or deny) consent after being fully informed of potential risk."This is a fundamental human rights issue.
-- SS
