Lies, Damned Lies and Dairy Safety: How Poorly Applied Statistics Could Lead to the Worse Public Policy

Volume 137, No. 41 - Friday, April 12, 2013

In my last article, I mentioned some of the problems involved with recent reports on foodborne illness. In particular, there is a report that was published In the CDC’s Journal of Emerging Infectious Diseases, Volume 17, Number One, from January 2011i, quoted widely, even in this wonderful journal. Now published on the CDC website as official estimates, they have ignited a debate from which major policy decisions will be made.

The data does not support their conclusions: what data they use compares apples to oranges, and they make no attempt to separate the signals from the noise, basing their calculations on the assumption of a normal distributionii, something that rarely exists in complex messy systems like the food chain, and they make no attempt to verify. These predictions are based on nothing more than “Stuff.”

So I downloaded the actual data, the same source they used, and applied the skills I have been lucky enough to learn, and the results point to very different conclusions than those posted on the CDC website, with some promising potential solutions, if we keep in mind that they must be tested first.

I was looking for pragmatic ways to have real impact on lowering the incidence of foodborne illness in dairy products. To my surprise I uncovered some commonly shared illusions surrounding the relative safety of dairy products, both pasteurized and raw.iii

1998-2010 Confirmed or Suspected Dairy Related Food Outbreaks

Tables of data can be misleading; they don’t provide enough context. It would be easy to assume from this that there is a huge difference in risk between pasteurized and raw milk, one of the illusions commonly shared.

This would miss the point, because what you are looking at is Non-Commercial occurrences from private homes, farmhouses, church events, and picnics. The “homemade” in fact is homemade ice cream. The available data is not clear if raw milk outbreaks came from farmers drinking their own milk, or from those who buy raw milk locally from the farmer. More research would need to be done.

The numbers for commercial products either direct from the processor, or served in restaurants, schools, prisons, etc., that is, approved facilities, tell a different story, don’t they?

The raw milk controversy may be somewhat of a red herring. Rather than trying to distance itself from raw milk products, the industry would be better served trying to ensure the industry as a whole is not compared to non-commercial products.

It is unfair to include commercially produced with non-commercial product in the same analysis, and then draw conclusions affecting both. It is reasonable to assume that many more dairy products are consumed commercially than non.

The single death reported from confirmed commercial product over the last 12 years occurred in a pasteurized cheese bought in a supermarket in Oregon in 2006, and was caused by listeria. Even among non-commercial deaths, most came from “bath-tub” producers of fresh Mexican style cheese, both raw and pasteurized.

I am not trying to minimize the tragedy of anyone dying, nor the inherent risks in raw milk, but trying to understand how to deal with what really happened and is likely to happen again. To prevent it, we need to find the real causes and stop blaming the easy targets.

(Keep in mind that the “numbers” don’t count, just the trends.)

Commercial Incidents
The largest number of outbreaks took place in Restaurants, and the largest number of illnesses per outbreak in Schools and Camps. This makes sense since the number of people served the same food in camp or school is greater, and kids immune systems may not yet be fully developed.

Where the Commercial IncidentsTook Place

Two-thirds of the 904 illnesses in the schools took place in only five incidents from 2001 to 2005. Three of those were caused by salmonella, with one from pasteurized 2 percent milk, and two others from foods made with cheese, which, based on experience, raises the question of food handling, rather than the innate integrity coming from the factory. In fact, with the exception of the two incidents from grocery stores, most of the other incidents involve post process food handling.

A Pareto chart is a useful tool that helps us discern the relative few that really matter.iv In the chart above we can see that organized by most outbreaks to least. This tool helps us figure out where to begin to look to improve the right processes, the ones that will make the most difference. Focusing on Restaurants and Schools would have a huge impact, based on the data we have, and assuming that the rest of the story follows the same trends.

Looking at it from the point of view of illnesses caused, the first place to find a solution would be in schools, the second restaurants. Schools would be easier as the number of illnesses per outbreak tend to be greater. If solutions can be found, these two would eliminate over half the illnesses reported for Dairy Products, if the trends hold and a root cause and solution should be found. (TESTING REQUIRED)

Again, the arrow points to schools and restaurants as the first point of entry to analyze and find root causes and improve the system.

This table includes both commercial and non-commercial sources in the more reliable data. While the most outbreaks were caused by Campylobacter, the most illnesses were caused by Salmonella, by a large margin.

Putting Salmonella aside for a moment, one part of the story in the data becomes clear when you consider the link between Campylobacter and Raw Milk: only a tiny number of outbreaks from this germ are linked to anything else.
This is good news. If the root cause can be found and processing improved, a huge number of cases could be minimized or eliminated.

With most illnesses coming from Enteric Salmonella a close study would have to be done where the incidents happen to find out how the contamination takes place. It will most certainly involve food handling, as most of the incidents involve secondary processing on-site of pasteurized dairy products.v

When we get to the fatal we run into yet another story: a very modern story. How is it that this weak little germ, present in moist environments since forever, has become such a relative threat? It certainly underscores the need for continual improvement in how we deal with Listeria. Should facilities try treating their drains with the same phage that is used in vacuum packed meats to kill any residual listeria for instance?

For my final thoughts, check in again in a month, I will propose what the data shows to be powerful potential solutions, that could really impact the problem, rather than kneejerk reactions that will waste a lot of valuable resources, and make things worse. DS

i http://wwwnc.cdc.gov/eid/article/17/1/pdfs/p1-1101.pdf

iian orderly, even distribution of results around the average.

iii I downloaded all reported food outbreaks from 1998 to 2010 from http://wwwn.cdc.gov/foodborneoutbreaks/Default.aspx. I removed all with unconfirmed causes, or where the food involved was unclear, (which whacked off about 60% of the data.) I kept any that had “sus-pected” causes, as having been involved in a few incidents myself, and know how hard it is to fix the exact cause, and what suspected means: a pretty good idea. I removed all that came from bacteria that are not associated with Dairy Foods, and that can only contaminate after the product leaves the processing plant, like staph. The quality of the data would make it difficult or impossible to determine if signal or noise. But some reasonable assumptions can be made, based on an understanding of statistical thinking, and the intensity of some things, as you will see are clearly signals, based on knowledge of food production, handling and distribution.

iv The Pareto Principle teaches us that less than 20% of occurrences are generally responsible for 80% of the outcomes.

v pasteurization eliminates salmonella in the milk before it has a chance to multiply.

Dan Strongin is managing partner and owner of Edible Solutions, a consulting company focused on helping companies making great food make a profit. He encourages your comments regarding this column. Comments can be made anonymously to columnists@cheesereporter.com. For previous Strongin columns, visit: www.cheesereporter.com.

Dan Strongin is managing partner and owner of Edible Solutions, a consulting company focused on helping companies making great food make a profit. He will be writing a monthly column in Cheese Reporter. Strongin can be reached via phone at (510) 224-0493, or via e-mail at dan@danstrongin.com. You can visit and blog with Dan at www.managenaturally.com.

Other Strongin Articles written for Cheese Reporter

Is Dairy Safe Is The Wrong Question, Part 1 Not All Data Is Information Start From Where You Are Learning About Your Customer The Vision Thing Customer Service? NOT! Collaboration: The Road To A Better Future Resolution Water In Memoriam: Ignazio Vella 1928-2011 Of Cheese, Seals, And Deming In Their Own Words: Lettie Kilmoyer In Their Own Words: Fritz Maytag In Their Own Words: Paula Lambert Show Me the Money: Cost Accounting Cost Accounting Chokes, Part 2: Inventory Cost Accounting Is Choking Your Business, Part 1 It Ain’t Over ‘til It’s Over Raw Reason A Story For The Holiday Season, Part II A Story For The Holiday Season Truth In Labeling This Too Shall Pass or "What were we thinking?" Marketing Language That Resonates When Will We Ever Learn? Cheese Competitions In The Context Of Marketing Economy Even The Best Laid Plans Go Astray Root Causes: Communication Partners Diamond Cutting: It's What You Don't Know That Can Hurt You Integrity and Ethics Pricing:  The Perceived Value Designing the Effective Sell Sheet Common Sense It All Begins in The Mouth Of Cars... The Gathering Storm As Our Industry Evolves, So Should Our Terminology:

Other Cheese Reporter Guest Columnists

Visit John Umhoefer Visit Neville McNaughton