CrossFit: We Got the Science?

12/15/10

In 11/22/2010, CrossFit.com posted an interesting video about a planned study. In it, Greg Glassman claimed

This will dwarf anything ever conceived of in Framingham or Harvard Nurses
The discussion that followed naturally had people asking about this. The questioning is understandable, given the status and results of the Framingham Heart Study (FHS), the Nurses' Health Study, and the studies derived from these studies.

Tony Budding, a trainer for CrossFit, claimed that

In terms of Framingham, the study was highly flawed.
When people, including myself, defended the Framingham Heart Study (FHS), the conversation turned really wierd.

Greg Glassman's father, Jeffrey Glassman, the Chief Scientist for CrossFit, chimed in.

When Coach set his sights on dwarfing the Framingham and Harvard studies he was aiming too low. If one tries to read official reports on the results and conclusions of those studies, he is going to be disappointed. Why is explained well by Michael R. Eades, MD, famous and popular especially in the CrossFit community for his book, Protein Power, written with his wife, Mary Dan Eades, MD. Read about the "Framingham follies" on his blog. http://www.proteinpower.com/drmike/cardiovascular-disease/framingham-follies/ . For some key information about an unfortunate consequence of the Harvard Nurses's Health Study, see his blog on "Observational studies". http://www.proteinpower.com/drmike/bogus-studies/observational-studies/ . These studies are important and statistically significant, but a disaster for conventional medical health wisdom.

What Coach is trying to do may not dwarf Framingham and Harvard Nurses on the calendar scale, but it could on the number of participants and perhaps even on the number of man years. He could blow away these studies because while they were epidemiology observational studies, his plan is for an experimental study, in itself a rare event in health science. He'll be like a climatologist who can actually vary the weather.

First, the claim
...an experimental study, in itself a rare event in health science.
is false. There are clinical trials, for example, and there are literally thousands of active ones in the United States alone. See ClinicalTrials.gov for more information. Note that the definition of "health science" is pretty broad as well, and therefore the number of experimental studies is much larger than just the number of clinical trials.

In any case, I responded

Let's see... properly reviewed medical studies peer reviewed and published over many decades, or blog posts by people with merchandise to sell their point of view, who typically pick apart said studies but offer nothing in return. I'd much rather be disappointed by limitations of the former than be amused by the latter.
Jeffrey Glassman responded
Being "peer reviewed and published over a many decades" appears not to have coaxed any useful results or conclusions from the principal investigators. The best I could find from the PIs was a list of published papers. But what would you expect from purely observational studies?

Discovery is far from the whole of science. Predicting and validating is everything.

And if the fact wasn't clear to you at the outset that these were observational studies, Eades points that out for you. He also points out that these studies had nothing by way of conclusions beyond de-correlating what were a couple of conventional cause and effect relationships.

To which I responded, but they chose not to publish:
Jeffrey Glassman #59, thank you for your reply.

You wrote:

"Being "peer reviewed and published over a many decades" appears not to have coaxed any useful results or conclusions ..."

Many people disagree with you, of course including experts in the field, which is understandable given the embarrassment of riches that the Framingham Heart Study (FHS) and the Harvard Nurses Study (HNS) produced and are producing.

You wrote:

"...from the principal investigators. The best I could find from the PIs was a list of published papers."

Please correct me if I am misunderstanding you on this point, but it seems an extremely odd way of arguing a point to purposefully limit yourself to the original study, especially considering FHS is ongoing by design.

There have been literally thousands of scientific journal articles (http://www.framinghamheartstudy.org/biblio/index.html) and popular pieces published based on FHS alone.

Let's just focus on some "small" things FHS showed us:

What do you predict the CrossFit study will show us?

You wrote:

"Discovery is far from the whole of science. Predicting and validating is everything."

OK, let's run with that, because FHS results have a lot to say about those things. They coined the term "risk factor", after all. The datasets alone have spawned numerous research projects.

Understandably the predictions aren't 100% right all the time (not that anyone claimed that, however), and have at times overestimated, underestimated, not performed as well as other methods, or need tweaking to fit a specific population. That applies to basically all prediction studies, however. In any case, a very quick search verified that FHS has been/are being used in prediction:

Is the Framingham risk function valid for northern European populations? A comparison of methods for estimating absolute coronary risk in high risk men.
Haq IU, Ramsay LE, Yeo WW, Jackson PR, Wallis EJ.
"Taking PROCAM as the external standard, the Framingham function separates high and low CHD risk groups and is acceptably accurate for northern European populations, at least in men."

Validity of an adaptation of the Framingham cardiovascular risk function: the VERIFICA Study.
Marrugat J, Subirana I, Comín E, Cabezas C, Vila J, Elosua R, Nam BH, Ramos R, Sala J, Solanas P, Cordón F, Gené-Badia J, D'Agostino RB; VERIFICA Investigators.
"The Framingham function adapted to local population characteristics accurately and reliably predicted the 5-year CHD risk for patients aged 35-74 years, in contrast with the original function, which consistently overestimated the actual risk."

Predictive value for the Chinese population of the Framingham CHD risk assessment tool compared with the Chinese Multi-Provincial Cohort Study.
Liu J, Hong Y, D'Agostino RB Sr, Wu Z, Wang W, Sun J, Wilson PW, Kannel WB, Zhao D.
"The original Framingham functions overestimated the risk of CHD for CMCS participants. Recalibration of the Framingham functions improved the estimates and demonstrated that the Framingham model is useful in the Chinese population."

Prediction of first coronary events with the Framingham score: A systematic review
Klaus Eichler MD, MPH, Milo A. Puhan MD, PhD, Johann Steurer MD, MME and Lucas M. Bachmann MD, PhD
"The Framingham score is well calibrated to predict first coronary events in populations from the United States, Australia, and New Zealand. Overestimation of absolute risk in European cohorts requires recalibration procedures."

Assessment of claims of improved prediction beyond the Framingham risk score.
Tzoulaki I, Liberopoulos G, Ioannidis JP.
"The majority of examined studies claimed that they found factors that could offer additional predictive value beyond what the FRS could achieve; however, most had flaws in their design, analyses, and reporting that cast some doubt on the reliability of the claims for improved prediction."

Validating the Framingham Hypertension Risk Score: results from the Whitehall II study.
Kivimäki M, Batty GD, Singh-Manoux A, Ferrie JE, Tabak AG, Jokela M, Marmot MG, Smith GD, Shipley MJ.
"These data suggest that the Framingham hypertension risk score provides a valid tool with which to estimate near-term risk of developing hypertension. "

Predicting the 30-year risk of cardiovascular disease: the framingham heart study.
Pencina MJ, D'Agostino RB Sr, Larson MG, Massaro JM, Vasan RS.
"Standard risk factors remain strong predictors of hard CVD over extended follow-up. Thirty-year risk prediction functions offer additional risk burden information that complements that of 10-year functions."

Evaluation of the Framingham risk score in the European Prospective Investigation of Cancer-Norfolk cohort: does adding glycated hemoglobin improve the prediction of coronary heart disease events?
Simmons RK, Sharp S, Boekholdt SM, Sargeant LA, Khaw KT, Wareham NJ, Griffin SJ.
"The Framingham risk score predicts CHD in this cohort."

Metabolic syndrome and Framingham risk score for prediction of cardiovascular events in Caribbean Indian patients with blood glucose abnormalities.
Jaquet A, Deloumeaux J, Dumoulin M, Bangou J, Donnet JP, Foucan L.
"In these Caribbean Indian patients with blood glucose abnormalities, unlike the FRS, MS failed to identify subgroups at high cardiovascular risk in the short term (8.5 years)."

A risk score for predicting near-term incidence of hypertension: the Framingham Heart Study.
Parikh NI, Pencina MJ, Wang TJ, Benjamin EJ, Lanier KJ, Levy D, D'Agostino RB Sr, Kannel WB, Vasan RS.
"The hypertension risk prediction score can be used to estimate an individual's absolute risk for hypertension on short-term follow-up, and it represents a simple, office-based tool that may facilitate management of high-risk individuals with prehypertension."

Metabolic syndrome vs Framingham Risk Score for prediction of coronary heart disease, stroke, and type 2 diabetes mellitus.
Wannamethee SG, Shaper AG, Lennon L, Morris RW.
"Although MetS does not predict CHD as well as the FRS, ..."

Enhanced risk assessment in asymptomatic individuals with exercise testing and Framingham risk scores.
Mora S, Redberg RF, Sharrett AR, Blumenthal RS.
"After FRS adjustment, low HRR and METs individually were highly significant predictors of CVD death, ..."

Probability of Stroke: A Risk Profile From the Framingham Study
Philip A. Wolf, MD; Ralph B. D'Agostino, PhD; Albert J. Belanger, MA; and William B. Kannel, MD
"We are now ready to obtain the predicted probability that a man with a selected set of risk factors will develop a stroke within t years. This is obtained by..."

Prediction of coronary heart disease mortality in Busselton, Western Australia: an evaluation of the Framingham, national health epidemiologic follow up study, and WHO ERICA risk scores.
Knuiman MW, Vu HT.
"The use of Framingham and other similar risk scores will not be misleading in white Australian populations."

A comparison of the Framingham risk index, coronary artery calcification, and culprit plaque morphology in sudden cardiac death.
Taylor AJ, Burke AP, O'Malley PG, Farb A, Malcom GT, Smialek J, Virmani R.
"The prediction of sudden cardiac death using the Framingham risk index and the measurement of coronary calcification are distinct methods of assessing risk for sudden cardiac death. Excessive reliance on either method alone will produce errors in risk classification, particularly for patients at risk of plaque erosion, but their combination may be complementary. "

Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation.
D'Agostino RB Sr, Grundy S, Sullivan LM, Wilson P; CHD Risk Prediction Group.
"The sex-specific Framingham CHD prediction functions perform well among whites and blacks in different settings and can be applied to other ethnic groups after recalibration for differing prevalences of risk factors and underlying rates of CHD events."

The applicability of the Framingham coronary heart disease prediction function to black and minority ethnic groups in the UK
T P Quirke, P S Gill, J W Mant, and T F Allan
"The effectiveness of the Framingham prediction function should ideally be compared to incidence of CHD within a prospective cohort study."

A Coronary Heart Disease Risk Score Based on Patient Reported Information
Arch G. Mainous, III, PhD, Richelle J. Koopman, MD, MS, Vanessa A Diaz, MD, MS, Charles J. Everett, PhD, Peter W.F. Wilson, MD, and Barbara C. Tilley, PhD
"Moreover, the Personal HEART score has predictive utility similar to the FRS and the European SCORE."

Is pulse pressure useful in predicting risk for coronary heart Disease? The Framingham heart study.
Franklin SS, Khan SA, Wong ND, Larson MG, Levy D.
"In the middle-aged and elderly, CHD risk increased with lower DBP at any level of SBP>/=120 mm Hg, suggesting that higher PP was an important component of risk. Neither SBP nor DBP was superior to PP in predicting CHD risk."

Cardiac risk stratification: role of the coronary calcium score.
Sharma RK, Sharma RK, Voelker DJ, Singh VN, Pahuja D, Nash T, Reddy HK.
"The Framingham Risk Score predicts CHD events only moderately well where family history is not included as a risk factor."

Influence of pharmaceutical care on health outcomes in patients with Type 2 diabetes mellitus.
Al Mazroui NR, Kamal MM, Ghabash NM, Yacout TA, Kole PL, McElnay JC.
"British National Formulary and Framingham scoring methods were used to estimate changes in 10-year coronary heart disease risk scores in all patients."

Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort study.
Schnabel RB, Sullivan LM, Levy D, Pencina MJ, Massaro JM, D'Agostino RB Sr, Newton-Cheh C, Yamamoto JF, Magnani JW, Tadros TM, Kannel WB, Wang TJ, Ellinor PT, Wolf PA, Vasan RS, Benjamin EJ.
"From clinical factors readily accessible in primary care, our risk score could help to identify risk of atrial fibrillation for individuals in the community, assess technologies or markers for improvement of risk prediction, and target high-risk individuals for preventive measures."

Importance of a patient's personal health history on assessments of future risk of coronary heart disease.
Mainous AG 3rd, Everett CJ, Player MS, King DE, Diaz VA.
"Accounting for an individual's history improves risk assessments based on current measures."

Application of cardiovascular disease risk prediction models and the relevance of novel biomarkers to risk stratification in Asian Indians
S Kanjilal, VS Rao, M Mukherjee, BK Natesha, KS Renuka, K Sibi, SS Iyengar, and Vijay V Kakkar
"To this effect, the risk assessment defined by the Framingham Study researchers was a great leap forward (Wilson et al 1998). This lead to the development of multiple predictive CVD risk score calculators, such as ..."

Risk prediction of coronary heart disease based on retinal vascular caliber (from the Atherosclerosis Risk In Communities [ARIC] Study).
McGeechan K, Liew G, Macaskill P, Irwig L, Klein R, Sharrett AR, Klein BE, Wang JJ, Chambless LE, Wong TY.
"..., the incremental predictive ability over that of the Framingham model was modest and unlikely to translate meaningfully into clinical practice."

Incidence of coronary artery disease in siblings of patients with premature coronary artery disease: 10 years of follow-up.
Vaidya D, Yanek LR, Moy TF, Pearson TA, Becker LC, Becker DM.
"In women, the observed CAD event rate was 7.1% (p <0.001 vs men), modestly but not significantly greater than the 6.3% predicted by the FRE (p = 0.34)."

Coronary artery calcium score combined with Framingham score for risk prediction in asymptomatic individuals.
Greenland P, LaBree L, Azen SP, Doherty TM, Detrano RC.
"These data support the hypothesis that high CACS can modify predicted risk obtained from FRS alone, ..."

A simple score for predicting coronary artery disease in patients with chest pain.
Wu EB, Hodson F, Chambers JB.
"The chest pain score had additive predictive value with Framingham risk analysis and Duke's score."

Application of the updated Framingham risk score to Japanese men.
Suka M, Sugimori H, Yoshida K.
"..., the updated Framingham risk score could provide a reasonable rank ordering of CHD risk and could identify Japanese men (and possible other individuals) at high risk for CHD with considerable accuracy."

You wrote:

"But what would you expect from purely observational studies?

There is obvious importance in long-term observational (longitudinal) studies. Repeated observation at the individual level and the order of events are pretty important stuff. They can't randomize who has CHD after all, or things like blood pressure, etc. How would you go about improving the science of predicting heart disease?

You wrote:

"Peer review proceeds in secret to vague, unpublished standards or worse -- dogma."

I'd be interested in your ideas to improve peer review, as scientists already know the potential issues that can occur with it. Is publishing your climate paper on your website or publishing it in you son's fitness journal a cure? I ask these questions in all seriousness. I will note ahead of time that any improvements to the peer review system that people do come up with are, paradoxically, a result of them sort of peer reviewing the peer review system itself.

But back to the real topic (FHS is not the real topic): the CrossFit study. As the Chief Scientist for CrossFit, you can perhaps tell us what CrossFit is promising to do, but actually hasn't done yet. I think an issue is that many people don't know how to translate "Dog we got the science. And we got it by the balls" or see how talk about DOD, NSA, and chaos theory is relevant.

What will the CrossFit study do to correct what you perceive as flaws that were supposedly present in FHS or HNS? What is the experimental design?

Thank you for your time,

Shortly after this, I received

I will post updates here on CrossFit's science. I don't expect much honestly, judging by their misunderstanding of FHS, and other obvious issues.

If you are interested in reading more about the Framingham Heart Study, please check out the book Change of Heart: Unraveling the Mysteries of Cardiovascular Disease. The book provides a great overview of the history, science, and people involved in the FHS.

Please anonymously VOTE on the content you have just read:

Like:
Dislike: