Debugging the Healthcare.gov Hearings
At Thursday’s House Energy and Commerce Committee hearing on healthcare.gov, Cheryl Campbell of CGI Federal and Andrew Slavitt of Optum/QSSI provided few useful details about the site’s architecture or problems. Nor would they take much responsibility for the failure of the system. But there was one line of questioning they had no trouble answering. When Bill Cassidy, R-La., asked how much they’d been paid in their contracts, Campbell immediately gave a detailed rundown: CGI Federal, lead contractor on healthcare.gov, has received $112 million for this year, $196 million for this phase of the contract, and a Total Contract Value (TCV) of $293 million. Slavitt of QSSI, which built healthcare.gov’s data services hub, likewise gave his answer—$85 million—without hesitation.
Campbell and Slavitt, a senior vice president and an executive vice president, respectively, are not techies; they are not project managers. Their expertise is in extracting as much money as possible from the government in procurement contracts, and judging by the numbers, they are very good at it.
But if you place these kinds of managers on the critical communication chain of a software project, you immediately endanger its success. Project quality is sacrificed for the sake of appearances—meeting the letter of the contract with indifference toward the actual practical outcome. Even if you put “the best and the brightest” (to borrow the administration’s own phrase) on a project, the mere presence of such managers can make it impossible to do good work, because the lines of communication will be broken.
In terms of tech jobs I’d never want to have, I’d estimate that programming for a CGI-like contractor would be better than being a World of Warcraft gold farmer in China, but worse than working for Pax Dickinson. Campbell and Slavitt made many gaffes that would ensure that any programmer with common sense would never work under them. Let me debug their testimony a bit.
When asked about QSSI’s testimony at a Sept. 10 hearing that everything was fine and dandy, Slavitt replied, “We mentioned the data services hub would be ready. It indeed was ready.” Alas, minutes earlier, Slavitt had said, “We absolutely take accountability for those first days when our tool was part of the issue in terms of the volume. Today the data services hub and the EIDM tool”—the registration and access management tool—“are performing well.” In other words, it was “ready,” but it wasn’t ready—because Slavitt is defining “readiness” not in terms of the product actually working, but in terms of meeting contract demands. They handed something off and ticked all the boxes, so what’s the problem? This attitude goes some ways toward explaining the last few weeks.
There was a lot of talk Thursday about end-to-end testing. Slavitt and Campbell both claimed they weren’t responsible for it. As Bill Johnson, R-Ohio, pointed out, the Pre-Operational Readiness Review in their contracts “required end-to-end testing results.” Campbell declared, “We have not been the systems integrator and we have never been the systems integrator.” Campbell saying it doesn’t make it so—CGI clearly was in charge of integration.
Campbell’s stonewalling on this point became embarrassing. When CGI Federal said it owned the front-end, Campbell explained, it meant that the company owned the front of a building but not the front door—which, by the way, was broken, which was all QSSI’s fault. Slavitt unhelpfully added, “I think the front door is a bit of a term of art. We supply a tool.”
Let me explain how end-to-end testing works in integrating large systems owned by multiple vendors. Each vendor works out detailed specifications for how the systems should interact. These are made as clear as possible so that when something goes wrong—and it always does—you can point to the spec and say, “You weren’t supposed to do that and that’s why our component appeared to misbehave.” In order to meet the specs, each vendor simulates end-to-end testing by building a prototype of the larger system.
In the case of healthcare.gov, QSSI should have had test scaffolding that could simulate the functionality of what CGI Federal was building, and vice versa. Each vendor needed to know the broad definition of what the other was building, and it was their responsibility to make sure they knew it. Campbell and Slavitt’s refusal to acknowledge this basic fact is both frightening and mortifying, and accounts for their inability to give any clear answers as to exactly which portions of the system are failing. They don’t seem to understand the difference between acceptable and unacceptable bugs, and worse, they don’t seem to know that there is a difference.
Campbell said, “We were quite optimistic that our portion of the system would work when the system went live.” That is either a lie or a revelation of ghastly incompetence, because no competent programmer or manager would ever display a shred of optimism until full end-to-end testing had been done. To say that “our portion of the system would work” is akin to saying that you know a computer will work before you’ve hooked a monitor up to it, just because it turns on. There’s no half-credit in these cases.
Everyone at the hearing, contractors and representatives alike, seemed ignorant of what is demanded by this sort of testing, with the exception of Jerry McNerney, D-Calif., a mathematics Ph.D. and ex-programmer. He asked about such specifications for integrations. Campbell said, “There were use-cases and things of that sort,” evidently unfamiliar with what specifications are (hint: not use-cases). Slavitt said, “We believe we received appropriate specifications.”
This is dismal not least because this is exactly where a good manager would know how to save face: by saying the specifications were bad in spite of your best efforts. Now that QSSI’s “front door” is apparently fixed, Campbell admits that there are ongoing “system performance issues” within CGI’s piece of the site, the Federally Facilitated Marketplace (FFM), which will operate in states that are not building their own insurance exchanges. A competent buck-passer would say that the performance guidelines in the specifications had proved inaccurate, or that she had warned the government that she could not meet the guidelines. As it is, she more or less admits CGI’s incompetence in her testimony—perhaps without meaning to.
Slavitt’s worst facepalm, meanwhile, was when he defended the security of the data hub by saying, “Our systems don’t hold data. They just transport data through it.” Mike Rogers, R-Mich., immediately jumped on this: “You don’t have to hold it to protect it.”
Back in March, Henry Chao of the Centers for Medicare and Medicaid Services said, “We are under 200 days from open enrollment, and I’m pretty nervous,” adding, “Let’s just make sure it’s not a third-world experience.” That sort of pessimism is a lot more realistic, and should have been a warning sign to anyone listening to him. Chao is one of the people Campbell was reporting to, and she shamelessly pointed the finger at him multiple times in the hearing as someone to whom she “alerted” test problems. As Chao is the only person I can find who expressed public worries about healthcare.gov pre-rollout, he should be given a promotion and asked who ignored his concerns.
Worst of all, when asked if the Spanish-language site developed by CGI Federal would be “functional” were it to be released, Campbell answered unequivocally, “It would be.” Of all the fatuous statements made in these dispiriting hearings, this was the worst. The only acceptable answer would have been, “The Spanish-language site would reflect some of the same chronic problems that currently exist in the system which we are currently working to fix.” But Campbell and her ilk don’t think in terms of systems; her job, and the job of her counterparts, is to think in terms of money and politics. Her agenda is not to make healthcare.gov work, but to protect CGI Federal.
When I worked at Microsoft, there was the dreaded “BillG review,” where a team’s top people would meet with Gates and he would “ask questions,” which in reality meant rapid-fire interrogation and belittlement. Campbell and Slavitt never would have survived a BillG review. By sending people so ignorant of specifications, process, and technology to represent their work at a government hearing, CGI and QSSI displayed contempt for the crucially important job they were tasked with. Campbell, Slavitt, and those like them should be immediately removed from the healthcare.gov project. They are bugs.
David Auerbach is a writer and software engineer based in New York. His website is http://davidauerba.ch.