🎵 He’s Making a PlasticList, He’s Checking it Twice… 🎵

(Thank you to the many people who subscribed because of the executive order tracking document. I have just updated it with EOs from the last couple days and updated forecasts.)

1. What is a PlasticList?

A team put together by Nat Friedman, the former CEO of Github, inspired by concerns around endocrine-disrupting chemicals, just spent six months testing hundreds of food products for plastic-related compounds: phthalates, phthalate substitutes, and bisphenols. They released their report and an accompanying website just before the new year, and I was immediately impressed. I was particularly interested in this endeavor for a few reasons:

• I’d done a ton of similar analytical chemistry over the last few years, although I’d been testing for heavy metals rather than phthalates, using mostly ICP-MS rather than GC-MS or LC-MS, and testing primarily water (and tea) samples, not solid food products.

• Their research was extremely public-facing, with a really well-made webpage, and they were explaining their methodology and data limitations far better than the vast majority of the academic literature on the subject (more on that later), despite orienting themselves as a public service initiative rather than an academic study.

• I had a gut response of deep annoyance and frustration at the people commenting and discussing the report on social media. This is no fault of the PlasticList team, of course.

2. How the Sausage is Made

Now, in your life, you’ve probably seen tons of articles of the format: “Scientists found toxins in your food.” Generally speaking, there’s a pipeline for how this news filters down to the public.

First, a research lab studying toxin X has run out of good ideas, so they buy a few dozen consumer products and test each of them for toxin X. Then, they publish a paper called something like “A multi-year heavy metal analysis of 72 dark chocolate and cocoa products in the USA.” The conclusions are usually something like, “We found toxin X in most of these products, and in a few of them there was a lot of it.” No surprises there. You’ll always find a few nanograms of whatever you’re looking for (heavy metals, forever chemicals, pesticides, microplastics, etc) in basically any food item / water sample / consumer product. What’s more interesting than whether it’s there is how much of it is there, obviously. The dose does indeed make the poison. However, journalists don’t really have any clue whether 1 microgram of toxin X per hamburger is horrendously high or blissfully low. So they write an article called something like “Scientists Keep Finding Heavy Metals in Dark Chocolate. Should You Worry?” Betteridge’s Law of Headlines obviously applies here.

Now, I won’t pretend to have any deep insights into the unique properties of phthalates, bisphenols, and other endocrine disruptors, which PlasticList has diligently tested for. You can decide for yourself whether you should or shouldn’t be worried about them. In general, the literature on compounds like these is somewhat mixed, with a handful of studies (in mice) showing that these things are not too worrisome at the concentrations people might be intaking by using plastic stuff for our entire lives, and another handful of studies (in mice) linking these compounds to mild increases in the rates of a wide variety of diseases. Unlike, say, methylmercury, there’s not an explicit observable disease that any of these compounds is directly implicated in (yet). These compounds also attract typically hyperbolic discourse online about how they’re causing infertility, lowering male testosterone, causing the downfall of western civilization, and the other usual things.

In an ideal world, there would be very scientific and rigorously tested limits for food items for each of these compounds, and if certain food products exceeded these limits, they would be taken off the shelves until the issue was addressed.

However, we run into some challenges:

• Most compounds like these don’t explicitly have established limits in food (i.e., from the FDA). The established limits that do exist are often based on very uncertain science (in mice).

• These also should probably vary depending on the food item. Most people aren’t ingesting half a kilogram of coffee grounds per day, whereas people might be eating that amount of, say, rice. Thus, if your coffee grounds have the same concentration of toxin X as your rice on a per mass basis, the rice should be a lot more concerning to you!

• Some of these compounds are themselves poorly characterized and have different variants with varying toxicity that for regulatory or convenience reasons are lumped together.

• Some things are bad for infants or children but not really that bad for adults.

So, it’s challenging to actually determine what a safe amount of a given toxin in a given food product might be. And besides, who’s going to be responsible for testing this stuff? If it’s the manufacturer, there’s a massive regulatory burden in testing a statistically significant fraction of your food products for every single toxic substance under the sun. Even doing that annually probably exceeds the entire budget of… well… anyone. Thus, the burden usually falls to you, the taxpayer. The FDA holds the threat of randomly sampling and testing certain food products, which they generally only do for “high risk” products for the particular contaminants that they suspect might be getting into them. The USDA does something similar for meat and eggs. The CDC might get involved in testing stuff if it’s related to public health considerations (bacterial or viral contamination, typically). And generally this type of testing is done on imported or domestically produced raw ingredients, less so on the processed foods that are assembled from these ingredients. And with chemicals like phthalates, one might be more concerned that these are being added during production, processing, packaging, or preparation (I promise the alliteration was unintentional)!

That last bit right there is a good explanation for why one might be motivated to conduct one’s own large-scale testing of these sorts of compounds in consumer products, like PlasticList did. And it’s also one reason why this work is far more interesting than your typical academic slop haphazardly testing stuff for evil molecules.

3. It’s Hard to Know Things

In their report, they have a big disclaimer:

“An important disclaimer: we have refrained from drawing high-confidence conclusions from these results, and we think that you should, too. Consider this a snapshot of our raw test results, suitable as a starting point and inspiration for further work, but not solid enough on its own to draw conclusions or make policy recommendations or even necessarily to alter your personal purchasing decisions. These results represent point-in-time results of a small number of product samples and may not be representative of actual product contents. These tests, like all tests, have inherent uncertainties, and different testing methodologies are likely to yield different results. And the existence of a chemical in a food doesn't necessarily imply a safety issue. We'd be thrilled to see serious efforts to replicate our results and we are open to any corrections you may have.”

This is admirable, and… unfortunately did very little to stop the internet from reading this report and drawing conclusions ranging from “oh no there are phthalates in everything,” to “OH NO THE MICROPLASTICS ARE GONNA KILL US ALL!” [no microplastics were tested in this report, but that didn’t stop anyone]

Since the FDA doesn’t set limits on these compounds, PlasticList had to use EPA and EU-EFSA exposure limits (which are quite divergent for some compounds) to back-calculate a sort of daily limit per serving. These values, as you can imagine, have multiple compounding sources of uncertainty:

• First, there’s the uncertainty in the testing that goes into the animal model to determine toxicity.

• Second, there’s the uncertainty in taking that exposure limit and deciding what threshold is relevant for a “maximum dose” for regulatory purposes.

• Third, there’s the uncertainty in converting from that animal model to a human model, including through introducing an “uncertainty factor” which tries to be conservative and assume that certain compounds might be much worse for humans than for mice.

• Fourth, there’s the uncertainty in PlasticList’s assumptions in converting from an exposure limit for humans (which is a “reference dose” or “tolerable daily intake”) into some sort of daily limit for food products, which is not precisely the same idea.

PlasticList then displays, for each of the tested food products, the amount of each of the 18 tested compounds on multiple bases: ng/g, ng/serving, and % of EPA and EU-EFSA limits for both toddlers and adults. Since they’re not primarily academics, they didn’t include error bars or confidence intervals for these values (as almost any journal would immediately demand of them). Ultimately, I don’t think that matters much, because most scientists would just turn around and report the instrumental uncertainty, which would be quite small (maybe a couple % for each data point). Well…

4. It’s Really Hard to Know Things (remember this part every time anyone tells you with any certainty the amount of some toxic thing in some other thing for the rest of your life, please!)

Let’s say we want to know the true amount of dibutyl phthalate (DBP) in the average Wendy’s Baconator. Okay, so I buy a Wendy’s Baconator, bring it to the lab, take a little piece of it, and test it in my mass spectrometer for dibutyl phthalate. I get a reading of 100 ng/g.

First, we have to consider the error in the measurement itself. My mass spectrometer calculates a standard error value (due to boring stuff like signal-to-noise ratio and the shape of the peak in the spectrum, etc), along with my value of 100 ng/g, and it’s probably quite small, let’s say 2%.

But actually, these mass spectrometer calculations require a calibration curve. It’s possible my calibration samples are old, or I didn’t weigh them out perfectly (you can never do it perfectly), or one of my calibrating samples got some anomalous reading in the instrument which throws off the calibration curve calculation by a small amount. This isn’t quite accounted for in the instrument’s own error calculation, so let’s add a tad more uncertainty, say, another 2%.

There are also artifacts in mass spectrometry. It’s possible there’s some other compound we haven’t accounted for that overlaps with our peak or interferes with the measurement. This usually isn’t a problem, but sometimes it is. It’s also possible that a bit of stuff got stuck in the mass spectrometer sampler or in the tubes or in the machine itself. Let’s say 5% of the time there’s an artifact that throws off our calculation by an average of 20%. That’s another 1% of error.

Oh wait, there’s variation between tests too. The MS instrument sometimes just gives you a different value on the same sample tested twice. The PlasticList team found this to be a whopping 20% on average. Why is this not accounted for in the instrument’s own standard error value? Why does no one really talk about this ever (kudos to the PlasticList team for admitting it)? Who knows!? To convert from RPD to % error, I’m going to divide by two (assuming that the “true” value is the average): 10%.

Uh-oh, there’s also some uncertainty in sampling within the Baconator. Let’s say that a certain part of the burger has most of the dibutyl phthalate. Well, the PlasticList team was diligent and homogenized their burgers into a pulp before testing. However, there’s always some sneaky error here. When you homogenize your burger, are you causing it to lose or gain water weight due to the exposed surface area of little burger particles? Are some of the compounds going to sink to the bottom of your Baconator goo for whatever reason so that when you sample from the top of the goo, you’re getting a non-representative sample? Are some of the compounds adsorbing to the walls of the blender? Who knows! I’d estimate at least another 10% error here, given my experiences in testing for heavy metals and seeing weird stuff happen to concentrations due to sample prep.

And there’s also variation between different samples of the same thing! If I get two different Wendy’s Baconators, they might have different values of DBP. PlasticList found that if they tested two samples, they differed by on average 59%! They also found that samples tested in triplicate had lower divergence from each other (33%), but I assume this means divergence from the average of the three? I’m not entirely sure how they calculated this, or if they mean that for whatever reason, when they sampled three times instead of two, there happened to be lower divergence between each pair within those subsets. Regardless, I’m going to take the 59% and divide by two to estimate an error of ~30%. This is by far the largest contributor to uncertainty in these calculations. Since hopefully these sources of error aren’t especially correlated, this will matter the most.

Okay, so now I go around the city and test burgers from different restaurants and make a list of the burgers that have the most and least dibutyl phthalate. I find that at one location, the burger has only 10 ng/g of DBP, whereas at another location, the burger has 200 ng/g. Is this difference real? Hard to say. If I tested 100 burgers, I’m bound to find some outliers, just from random variability, that are 2-3 standard deviations away from their true probability, even if all the burger joints have the same “true” average level of DBP. This is the essential problem with this kind of methodology.

Moreover, if I’m now testing each burger for a dozen different things, I’m basically guaranteed to find a few “surprising” outliers, even if all the burger joints again have the same “true” average levels. This becomes magnified if I start to do this testing regularly over time!

And so, even if a group like PlasticList does everything flawlessly and thoughtfully, without motivated reasoning, their table showing 18 compounds tested over hundreds of products will invariably be mostly noise. A few of the outliers (and in trace detection of these sorts of compounds, outliers means more of the toxin, since an outlier in the other direction would be a negative amount, which is not possible) seem significant.

For instance, they found that Korean War rations from the 1950s had somewhat concerning levels of several contaminants. But this isn’t particularly surprising. I personally wouldn’t put 70-year-old military hard-tack in my body, regardless of how little plastic-related compounds are in it.

However, I don’t think that someone should make any other decisions based on this data. They seemed to think that water purification tablets and some boba tea might have high amounts of BPA, but BPA is (1) probably not bad for you at all at these levels and (2) there’s so much sample-to-sample variability that it doesn’t matter anyway. The vast majority of everything that was tested fell substantially under any regulatory limits, even the very cautious EU limits.

5. So… We’re Gonna be Okay, Then?

As far as risk factors in your life, I would say phthalates and bisphenols should probably rank fairly low. I’d guesstimate that if you’re living in a country like the United States, the food you consume probably accounts for ~10% of the variance in your health outcomes (there are other things: genetics, exercise, trauma, nutrition, stress, viruses, car accidents, etc). Within that, toxins in that food are probably only ~10% of that total variance. So, maybe 1% of variance in your health outcomes could be explained by toxins in your food, much of which is fairly priced in after you stop being a child. And a decent portion of that is heavy metals. If given the choice between completely eliminating plastic-related compounds from your food and driving 5% less, I recommend you do the latter. Car accidents are the leading cause of death for young adults.

Will this stop scientists around the world from publishing hundreds of papers a year doing similar analyses to the one PlasticList performed, just much, much worse? No, it will not. Nor will it stop major news publications from continuing to write very badly articulated press reports about that science. But perhaps you can decide to stop worrying about it.