Rewiring the VA: Decision to use outside contractors to replace key pieces of vaunted VistA IT system draws criticism from experts, original architects

The question is whether the Veterans Health Information Systems and Technology Architecture, or VistA—the clinical information system that powers the VA health system—will wither or bloom in the months and years ahead. It’s an issue that has implications not only for millions of veterans but also millions of other potential users of open-source and proprietary versions of VistA, both in the private and public sectors in the U.S. and abroad.

The VA runs a vast, national healthcare enterprise. VA officials expect to treat 5.8 million patients in the current fiscal year, up 1.6% over 2008, including more than 333,000 veterans from the war in Iraq and some 40,000 from the war in Afghanistan, according to the VA’s fiscal 2009 budget request to Congress.

Though highly praised, the IT program at the VA also has come under fire.

Just last week, VA officials agreed to pay up to $20 million to settle lawsuits for damages following a 2006 data breach in which portions of the records of 26.5 million veterans were put at risk when a laptop computer was stolen during a home burglary of a VA employee. The laptop was turned in to the FBI, whose forensic analysts said no records were exposed.

Earlier in January, the Associated Press reported that a software glitch within VistA intermittently caused some data errors in patients’ records. According to the VA, there were nine incidents in which a doctor’s orders to stop the administration of intravenous drugs—most commonly the blood thinner heparin—failed to display in the system. The VA says it caught the errors with no harm occurring to patients. The problem was traced to a recent software update introduced last October, but several VA programmers interviewed for this story wondered whether the glitch was a symptom of a larger problem in how IT is being handled at the VA.

In 2007, however, the VistA system in Northern California suffered a far more serious problem, an eight-hour outage that J. Ben Davoren, a physician who is director of clinical informatics at the 132-bed San Francisco VA Medical Center, in written testimony before Congress, called “the most significant technological threat to patient safety VA has ever had.” Davoren linked the outage and other IT problems to a reorganization and centralization of IT management at the VA in the Office of Information and Technology.

Last month, retired four-star Army Gen. Eric Shinseki was confirmed as the new VA secretary in the Obama administration. On Dec. 7, 2008, in announcing Shinseki as his choice to head the department, then President-elect Barack Obama said, “We need to build a 21st century VA,” and that included “fully funding VA healthcare.”

But what does it mean to build a 21st century healthcare information technology system at the VA when its largely home-grown clinical IT system, VistA, remains light years ahead of all but the most elite IT programs in the most-wired hospitals and healthcare systems in the U.S.? Does that mean it’s possible the VA could return to the decentralized, collaborative and iterative software development process that was key to the creation and improvement of VistA?

The former VA software development process, according to VistA historians, was a kissing cousin to the open-source model of software evolution that yielded the Linux computer operating system as well as the Apache family of software, which powers much of the Internet. At its core, the VA process relied on the iterative development of software between hospital-based programmers and clinicians who focused on a single clinical problem and—to borrow from IT marketing jargon—actually provided “solutions.” This close collaboration is a process that has nearly vanished within the VA over the past decade, according to current and former VA programmers.

Since work on the VistA software was paid for with taxpayer dollars, much of its code is in the public domain. Copies can be obtained without charge under the federal Freedom of Information Act. As such, a cottage industry began forming in the early part of this decade around the VistA system. Development of VistA outside the VA got a big boost in 2003 when the Pacific Telehealth & Technology Hui, a Honolulu-based partnership between the VA and the Defense Department, sponsored the creation of an open-source version of VistA and later turned it over to a not-for-profit organization, WorldVistA. Several for-profit companies in addition to WorldVistA now offer versions of the VistA system for use in other government and private-sector healthcare organizations, both in the U.S. and abroad.

Installations of versions of VistA are complete or under way in non-VA hospitals, clinics and nursing homes in Arizona, Colorado, Hawaii, Texas, Idaho, Oklahoma, West Virginia, New York and American Samoa. A recent, headline-grabbing VistA contract announced last summer involved Perot Systems Corp., Plano, Texas, which is installing the open-source WorldVistA EHR at two hospitals and a clinic in Amman, Jordan.

The VA way of doing things was cited recently as a model for meeting one of the premier challenges of healthcare IT in the future: designing advanced, clinically “smart” computerized decision support tools, according to an academic IT expert.

William Stead, chief information officer for 833-bed Vanderbilt University Medical Center in Nashville, served as co-editor of a recently released IT report by an committee under the National Research Council of the National Academies. In conducting research for the report, committee members visited the VA hospital in Washington, D.C.—one of eight healthcare organizations the committee selected for site visits because they were well-known for their IT excellence.

“The way the (VA) system was put together, that iterative work that went on in several of the hospitals in parallel, where it was working to solve the problems of the individual hospitals, what our report says, that was actually the right model to do this stuff,” Stead says. “The idea that we can create a monolithic system and distribute it and have the system enable rapid, iterative improvements in the process, those ideas are absolutely counter to one another.”

And yet, in 2006, Congress ordered the VA to gain greater control and efficiency over the IT programs of its three disparate departments—healthcare, benefits and burials—by placing them under one chief information officer, a mandate, interestingly enough, that came in the wake of an IT outsourcing fiasco that had nothing to do with VistA. The $247 million write-off in 2004 of the work by defense contractor BearingPoint, McLean, Va., on the VA’s failed Core Financial and Logistics System, or CoreFLS, made national headlines and sparked congressional investigations.

But a more recent outsourcing effort, and more disconcerting according to some VistA community members, was the decision, reached in 2006, to replace the VistA laboratory information systems module with proprietary software purchased from a commercial vendor. Cerner Corp., Kansas City, Mo., announced in November 2007 it had won the lab contract. Cerner would not provide a company official to be interviewed for this story, referring queries to the VA.

The VA also would not provide an official for an interview, but Josephine Schuda, a VA public relations officer, responded to written questions in an e-mail.

The VA contract with Cerner was for nearly $2.7 million for the first year, with additional one-year contract extensions for up to eight years available as options priced on a more open-ended “indefinite delivery, indefinite quantity basis,” Schuda says. The contract calls for development, testing and national implementation of Cerner’s lab system throughout the VA. The VA performed a comparison of cost estimates between outsourcing and doing an upgrade to the VistA system’s existing lab module, but a request to release the comparison was referred to VA legal counsel, according to the e-mail. The results of the comparison were not made available by deadline.

Work on the lab integration project is still under way, the spokeswoman says, with the lab team performing “integration and user acceptance testing in preparation for alpha-testing in the field,” according to Schuda. “The software is not installed in a production environment yet; national deployment is set to start in 2010.”

According to the VA’s fiscal 2009 budget, the VA decided to replace the VistA lab information systems module because it “was created more than 20 years ago and is inefficient, limits revenue collection, does not meet current regulatory requirements, potentially jeopardizes patient safety, and is unable to support planned quality improvements to patient care.”

Author Phillip Longman, however, is aghast at the VA’s decision. A research director at the New America Foundation, Longman was so inspired by what he found in reporting for a magazine article about the VA’s health system, he went on to write a book about it, Best Care Anywhere: Why VA Health Care is Better Than Yours. Contracting out the lab system is merely a symptom of a larger ailment at the VA, he says.

A disappearing culture

“Front-line workers are no longer involved in going to the next generation of VistA,” Longman says. “I was speaking in the spring” at the 142-bed Durham (N.C.) VA Medical Center, he says. “It’s an extremely impressive hospital; they have robots running around with medications, there are robots dispensing them, they have computerized medication administration, but if you go down in the basement where all of the IT stuff is, you have about 10 people in these windowless rooms and all they are doing is maintaining the day-to-day functions of the system. Nobody is doing any programming. The culture where a doctor might meet someone in the hall and say, ‘Let’s put our heads together and get something done’ is gone.” Instead, he says, there is the push to install proprietary software.

Longman says that the VA has been fighting a decades-long public relations battle with its horrendous, 1970s-era image.

“In the 1970s, we didn’t have the concept of open-source, but that’s basically what they were doing” at the VA, Longman says. “That’s how the VA reinvented itself. It’s one of the most remarkable stories in the annals of management and institutional reform. And now, every trend in the VA is toward recentralization. The leaders of the VA have lost sight of the reasons for the VA’s rebirth. It’s a strong organization and it’s still ticking, but there is a real danger if this outsourcing goes on long enough, we will lose the VA culture that has so many accomplishments behind it.”

Former VA programmer Brian Lord, chief executive officer and owner of Sequence Managers Software, Durham, N.C., a vendor of VistA-based clinical IT systems, says the decision to contract out the lab software shows hostility toward the very notion of the government-created IT system becoming and remaining a public resource. “You’ve got these people who don’t want (VistA) to exist,” says Lord, who accuses the government of trying to take away a low-cost alternative to multimillion-dollar systems.

“That’s my frustration,” Lord says. “For the past 10 years, there have been steps taken to disable that and they keep spending millions and millions of dollars to pay these huge companies.”

Physician Scott Shreeve co-founded Medsphere Systems Corp., Carlsbad, Calif., one of a number of vendors selling open-source VistA to healthcare organizations outside the VA. Shreeve has since left the company but remains a frequent commentator on healthcare IT. He wrote in reaction to the VA/Cerner announcement a provocative column titled “Diabetic VistA—the First Amputation,” published on his health IT blog, Crossover Healthcare.

In that 2007 column, Shreeve praised the high quality of the Cerner lab software, conceding that the VA’s own lab module was in serious need of an upgrade, and yet he wondered, “How could the VA allow a critical, integral part of VistA to languish for more than a decade?”

“I fear this is the first amputation in a long and steady surgical removal of VistA from the VA,” Shreeve wrote. “Piece by piece, subsystem by subsystem, the VA appears to be looking to take a best-of-breed approach. All the beautiful and inherent advantages of a single, integrated software solution get thrown out the window as a patchwork of best-of-breed solutions gets thrown into the mix.”

Fourteen months after writing the column, Shreeve says, “I still feel sad when I think about it. To not invest in a system for 10 or 15 years, and then decry its lack of functionality, and then spend so much money on its replacement, that’s what I’m concerned about. I just don’t think they can justify it. Why not just reinvest in your own system and make it better?”

Former VA programmer Cam-eron Schlehuber began work in 1978 on the VA’s computer system, including its lab module, and kept at it until he retired from the VA in 2006. Schlehuber, who has been active in developing an open-source version of VistA through the WorldVistA community, says he can trace the explosion in spending on contracted government services to a specific document, the decades-old Office of Management and Budget Circular A-76.

“It said anything that can be done by the private sector, let’s start moving towards contracting out,” Schlehuber says. “Now, with this decade, it’s just gone wild. We’re contracting out the Army, we’re contracting out torture, and the contracting companies now are even writing their own scope of work.”

VistA historians still debate this point, but according to many, work on what was to become the VistA system began back in late 1977. Its FileMan database manager was fully deployed by the early 1980s and onto it were added many “modules,” software applications for lab, pharmacy, admission, discharge and transfer, billing, bar code-based medication administration (a VA innovation), radiology and other programs to address specific clinical and business needs. Computerized physician order entry, or CPOE, still a rarity in U.S. hospitals outside the VA, was in widespread use at the VA in the 1990s. Today, VistA has more than 100 modules. Many of them were developed, tried and perfected at local VA hospitals.

Historically, this hospital-based, distributed computing process has long had its detractors within the VA and in Congress. Back in the late 1970s, initial work on the distributed model began under the auspices of the Computer Assisted System Staff, or CASS, within the then-Department of Medicine and Surgery, the latter being the equivalent of the Veterans Health Administration today. But the VA also had a centralized department of computer technology, the Office of Data Management and Telecommunications, similar to today’s Office of Information and Technology. Over a period of several years, the hospital-based CASS programmers fought a bureaucratic trench war with what they called “the enemy” at ODMT over control of clinical systems development, according to a history of VA programming written by one of its pioneers, George Timson.

The infighting led to employee firings and reassignments, Timson recalls. Hospital computers were suddenly crated up and hauled away. There was even a computer-room fire that VistA community members to this day view with suspicion. During the battle, hospital-based clinicians and programmers felt compelled to write and distribute software via a clandestine effort its participants would come to call the “Underground Railroad.”

Tom Munnecke, a pioneer VA programmer in Loma Linda, Calif., had printed up a few hundred Underground Railroad business cards bearing a cartoon steam locomotive and distributed them to members of the cabal, cards that remain badges of honor to veteran VA programmers. Because the distributed development model was faster, cheaper and produced better results, it finally won out. By 1982, its work products were officially recognized by the VA brass and named the Decentralized Hospital Computer Program, which was renamed VistA in the 1990s, according to Timson’s history.

But in 2001, the VA began an initiative called HealtheVet to modernize its medical information system and supplant VistA. According to a Government Accountability Office report released last summer, the VA plans to replace the 104 VistA modules and programs with 67 new applications, including proprietary systems. From fiscal 2001 through 2007, the VA reported spending almost $600 million on just eight projects related to the replacement of VistA by HealtheVet. The time frame for completing the projects and the HealtheVet system as a whole was 2012, but the projected completion date has now been delayed until 2018. In April 2008, the VA released an $11 billion cost estimate for completion of HealtheVet.

Munnecke, who now lives in San Diego, worked at the VA from 1978 to 1986 and is credited with being one of the visionaries who came up with the multi-layered, “onion” architecture of the VA’s IT system.

“Personally, I would just sit there and ruminate on it,” Munnecke says. “I was just obsessed with it and then it would just come out in a burst of code. I used to dream in code. In the morning, I knew I’d been up and saw four or five pages of code on my desk, and it would work.”

Munnecke says inserting a proprietary module as important as the lab system into VistA “is a horrible approach.”

“It stops the propagation of an open system,” he says. “It’s only as open as its most closed link. So, by closing off the lab data, they’ve basically shut down the entire value of the whole approach. We should be going the other way and have open source and everybody contributing to higher values in the change.”

The matter of cost

The high cost of healthcare IT systems is the most-cited barrier to adoption, but money is a particularly acute problem for safety net providers. Some 60% of 1,300 community health centers are looking to implement healthcare IT systems, according to Johanna Barraza-Cannon, director of the Division of Health Information Technology Policy within the Health Resources and Services Administration, an arm of HHS. The IT tab for those providers could run upwards of $300 million or more, depending on the cost of the software, Barraza-Cannon says.

The HRSA is funding pilot projects in West Virginia and Arizona in which VistA versions are being deployed by safety net providers.

Theresa Cullen, a physician who is chief information officer at the Indian Health Service, strove mightily to be diplomatic in discussing the VA contract with Cerner, but it is clearly a concern. “Well, the politically appropriate response as a CIO of a smaller federal agency, I don’t know that whatever decision they make will be able to be extended to the Indian Health Service,” Cullen says. The agency, which provides healthcare for 1.8 million people at about 400 IHS and tribal healthcare locations, uses an electronic health record system based on VistA called the Resource and Patient Management System.

Cullen says the IHS would probably stick with the VA’s older lab module that it currently uses in RPMS because it would not be able to afford the license fees for the Cerner lab system once it gets folded into the VA software.

Lower price is a big selling point for commercial vendors of VistA-related systems, particularly open-source versions, because they reduce or eliminate proprietary software and its attendant software license costs.

Not everyone sees the VA’s decision to contract out the lab module to a proprietary software vendor as the death knell for VistA.

First, the installation and interfacing with the rest of VistA has to be made to work, which is far from a foregone conclusion, given the integral nature of labs in VistA system, according to Frederick Marshall, a former VA programmer and a past-president of WorldVista.

“When it crashes and burns, we’ll have a new lab package ready,” Marshall predicts.

According to VistA pioneer Gordon Moreshead, one of the original authors of the VA’s lab software program, the VA could put together a team of its own employees and contract out for additional programming help and upgrade VistA labs for a lot less money than it will spend contracting with Cerner or any other vendor of off-the-shelf lab software.

Moreshead currently is president of Informatix Laboratories Corp., a Salt Lake City-based company that develops patient billing and accounting systems. He started working with the VA in 1970 while completing a master’s degree in bio-engineering at the University of Florida, Gainesville. He moved to Salt Lake City in 1973 and was a member of the earliest of planning committees that led to the development of the VA’s clinical computing system.

“I was involved in a lot of internal technology at the hospital in Salt Lake and started to solve problems for physicians, like getting lab results out where they worked,” Moreshead says. “We were delivering lab results to the nursing units in about 1980,” Moreshead recalls. “Around the mid-1980s, I tried to do a cost analysis on what had been spent by the time we had the (lab) package completed. On the development side, it was less that $5 million and on the distribution and training side, it was less than $5 million. So, if you triple it today, you’re not going to spend the money that you’ll spend on Cerner.”

With the VA lab module, Moreshead says, “The mistake the VA made was in the late 1980s to early 1990s, they said, ‘Oh, it’s done.’ If you want to modernize it and keep it viable, you can’t do that. You have to keep investing in that, because those domains of lab and radiology aren’t stable. They’re changing.”

AFGE Events

Event Calendar is for Members Only. Please Log In to see our calendar of events.


SUBSCRIBE Latest news & info