Tracking foodborne illness with Yelp

An estimated 48 million Americans (about 1 in 6) are stricken with food poisoning each year. So says the Centers for Disease Control and Prevention (CDC), but it really is, as my italicizing implies, just an estimate. It’s difficult to say how many people get sick from meals annually because only a fraction of these cases are ever reported. Unless victims of foodborne bugs are hospitalized (about 128,000) or die (about 3,000) the majority of them never take the time to share their suffering with doctors or health officials. (And who can blame them. Why throw up in a doctor’s office waiting room when you can throw up in the comfort of your own home?) Instead they complain to the more accessible outlets – to their families, to their roommates, and, most importantly, to the Internet. Online review sites offer users the chance not just to praise or pan an establishment’s food, but also to warn others away from the diner whose dubious lasagna led to several days of gastrointestinal calamity. Having noticed that people are more eager to share tales of food gone wrong on Yelp than with their primary care physicians, the New York City Department of Health and Mental Hygiene (DOHMH) created a pilot project aimed at using this online data to find previously unreported outbreaks of food poisoning from New York restaurants.

Working with Yelp (for data) and Columbia University (for computer wizardry), DOHMH spent the period of July 1, 2012 to March 31, 2013 analyzing restaurant reviews for tell tale signs of food poisoning outbreaks. Each week Columbia’s computer program plucked any reviews that could be indicative of foodborne illness – basically those containing keywords such as “sick” or “vomit,” and with incubation periods of 10 hours or more (the time it takes for the majority of foodborne microbial maladies to show themselves).

Over the 9-month period, about 294,000 reviews were screened, with 893 being flagged by the software as possible contenders. Unfortunately, computers tend to miss certain nuances, so a human epidemiologist had to then read through the flagged reviews and separate the potentially useful (“I spent the next day vomiting”) from the totally irrelevant (“my entrée looked like someone’s cat had vomited on it” or “I’m so sick of these hipster restaurants and their stupid bacon”). Those are my hypothetical examples, btw, not actual reviews. For the purposes of this project, food poisoning episodes where only one person became ill were excluded, which eventually pared the reviews down to just 129 worthy of further investigation. DOHMH sent this elite group emails inviting them to discuss their experiences in more detail via phone interview, an offer that only 27 of them accepted (not ideal, but a 21% response rate is better than nothing).

From these interviews, three unreported foodborne outbreaks were identified, and the restaurants involved investigated. The full report, published in the CDC’s Morbidity and Mortality Weekly Report, doesn’t out the restaurants by name. It does, however, list the likely contaminated food items in each case (including, egad, something called “macaroni and cheese spring rolls”) and the findings of the restaurant inspections (your usual battery of offenses – poor sanitation, lackluster refrigeration, and the occasional rodent).

It’s a clever use of social media to benefit public health. Of about 468 reviews initially designated (by human, not just computer, analysis) as likely food poisoning cases, only 3% had also been reported to health officials. So the online review approach could suss out otherwise overlooked outbreaks and force the restaurants involved to clean up their act. On the downside, it’s all pretty resource intensive. Nine months and 893 reviews just to find out that you should maybe avoid the macaroni and cheese spring rolls. It’s a bit anticlimactic. Additionally, since the food poisoning sufferers weren’t seen by doctors and the restaurant inspections didn’t turn up any specific microbial evidence, we don’t know the pathogens responsible for these illnesses. And while the authors don’t mention it, I can’t help but wonder what percentage of “I got sick” type reviews are spiteful responses to non food-related gripes. Hopefully, any such individuals declined the phone interviews.

DOHMH will keep working to fine-tune its project, with tweaks to catch more cases (e.g., including other review websites) and increase response rates. The authors also suggest (though haven’t incorporated into their project) what I think could be the best approach in terms of effort to reward ratio – having Yelp and its ilk provide reviewers with links to local health departments’ reporting systems while they’re posting. In theory, this could just be a part of the general review interface, a message that comes up before you press the submit button. Though I’m envisioning a more targeted, keyword-driven approach. Perhaps a pop-up saying, “Your review indicates that you may be complaining about food poisoning. If you would like to complain some more, please visit yadayadayada.gov.” (Disclaimer: not a real website…yet.)