‘All models are wrong, some are useful’

Image via FlexSim Healthcare.
Image via FlexSim Healthcare.

Image via FlexSim Healthcare.

There are good models and there are bad models.

For example, how many times have you purchased what amounted to be a garbage sack because it looked so much like a beautiful dress on the airbrushed model in the picture online? If it’s half as many times as I have, then you know that models are not the real deal.

Whether it’s an architectural model of the new building going up downtown, the model train you’ve got set up in your parents’ basement, or the mannequins and airbrushed photos in our glossy magazines, models are sometimes fairly inaccurate.

In contrast, last week medical professionals traveled from five states (Mass., Conn., R.I., N.Y., Maine) to a healthcare simulation workshop at Northeastern to learn how to build models of healthcare systems that actually are representative and useful to help them improve critical problems.

When it comes to a computational model, “just because it’s not real, doesn’t mean it can’t be used in real and useful ways,” said industrial and mechanical engineering professor James Benneyan, director of the Healthcare Systems Engineering Institute, at a recent two-day workshop for Boston area healthcare systems personnel to teach them how to use computer simulation to improve critical problems. He quoted a popular adage among his lot, uttered some years ago by statistician George E. P. Box: “All models are wrong, some are useful.”

The workshop was part of Benneyan’s grant from the Centers for Medicare and Medicaid to create a national network of centers that partner with local healthcare systems to help improve critical problems with systems engineering methods. While I was there I met a few people from the Center for Clinical Excellence at Brigham and Women’s Hospital, an in-house consultancy group that aims to help the hospital establish more effective quality, safety, and operations measures. They told me that they often use simulation among their clinical staff to look at practices and identify more efficient approaches. But for them, simulation has always been of the “table-top” sort.

In healthcare systems engineering, practitioners can draw from a number of tools in their arsenal. Most of the problems they encounter, Benneyan likes to say, are pretty basic and can be solved with simple approaches. But when you get into the complex, weedy problems—like how to better manage of ICUs and emergency departments, schedule primary care appointments, or redesign screening policies for common diseases—basic approaches no longer cut the mustard.

I’d never heard of table-top simulation before the workshop, but it’s exactly what it sounds like. You simulate the real world scenario on the table: Think of Stannis Baratheon’s painted table in Game of Thrones or the XO’s war room map in Battlestar Galactica. You can play out what did happen and then make changes to see how it might have happened if you’d done something differently.

This is all well and good until the number of variables in your scenario—and their unreliability—starts rising. In the healthcare world, these are often quite significant, as it deals with perhaps the least reliable entity on the planet: humans.

“How do we design good processes given there’s so much out there we can’t control?” Benneyan asked the workshop participants. Well, you stop playing around with the table-top-models and pull out the big guns: computer models.

Lou Keller, an expert in computer simulation who has been involved in simulation of healthcare systems his entire career spanning military and civilian healthcare systems, walked the participants through the FlexSim Software which he helped develop, as an example tool for these more complex situations. The number-one rule, he said, is to always model what is supposed to happen first,” he said. “Once you know what’s supposed to happen then you can go looking for what it is that’s keeping it from happening.”

Healthcare costs our nation $3 trillion each year. Benneyan says that one third of that could be eliminated by dealing with the waste and inefficiencies built into the system. The most common method for doing so is simulation. “These are really critical problems we are trying to help our healthcare colleagues with, so I’m thrilled with the interest and engagement,” said Benneyan. “For me, this also was a lot of fun, as it’s the first time I’ve taught with someone who was a bit of a mentor when I was starting my career.”

The mission of the Healthcare Systems Engineering Institute at Northeastern is to have broad national impact on healthcare improvement and redesign through research, education, and applications of systems engineering methods.