Although the prevalence and incidence of type 2 diabetes mellitus (DM) continue to increase at an alarming rate,¹ many at-risk patients do not develop DM.² Accurate prediction of DM onset may help target interventions to prevent DM and facilitate the efficient allocation of healthcare resources. A few DM prediction models have been developed,^3-5 but none are widely used in clinical practice. Investigators from the Framingham Offspring Study (FOS)⁶ recently proposed a new simpler algorithm for estimating 8-year DM risk. The algorithm uses a point score derived from a small set of  variables easily obtainable at a clinic visit. The authors’ methods followed those of the original Framingham Heart Study, which created a point score for predicting coronary heart disease (CHD).⁷ The now widespread acceptance and use of the FOS CHD point score coupled with the simplicity of the FOS DM risk score may lead to rapid adoption of the FOS DM risk score in clinical practice. However, as the FOS investigators recommend, the DM prediction models and the corresponding point score must first be validated in other population samples. We sought to replicate the FOS models using electronic medical records from a large integrated health plan and to compare the DM risk estimated by the point score with the observed DM incidence.

METHODS
The study sample was drawn from adult members of Kaiser Permanente Northwest (KPNW), a closed group practice health maintenance organization located in Portland, Oregon. Kaiser Permanente Northwest maintains comprehensive medical utilization data that includes an electronic medical record of all patient encounters, laboratory results that are analyzed by a single regional laboratory using standardized methods, and dispensings from pharmacies located in all clinics.

For this study, we selected 20,644 persons aged 26 to 82 years (same age range as the FOS sample) who had no prior evidence of DM and who had all the data elements necessary to estimate the models described herein recorded during the baseline period (July 1, 1999, to June 30, 2000). Patients were followed up from July 1, 2000, through June 30, 2007, for new onset of DM, defined as 1 inpatient diagnosis or 2 or more outpatient diagnoses (International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] codes 250.xx), a dispensing of an antihyperglycemic medication, or a fasting plasma glucose level exceeding 125 mg/dL (to convert glucose level to millimoles per liter, multiply by 0.0555). To replicate the FOS methods, we excluded patients who died or left the health plan before the end of the follow-up period without first developing DM.

In addition to age and sex, the baseline predictor variables used in the FOS simple clinical model included parental history of DM, body mass index (BMI), blood pressure (or receipt of antihypertensive therapy), and high-density lipoprotein cholesterol (HDL-C), triglyceride, and fasting glucose levels. We used an ICD-9-CM code of family history of DM (code v18.0) recorded anytime during observation as a proxy for parental history. All other data were extracted from values recorded in the medical record in the baseline year, using the latest value when multiple values were available. Following the FOS, we categorized metabolic syndrome traits according to revised National Cholesterol Education Program Adult Treatment Panel III criteria.⁸ Patients having a blood pressure of 130/85 mm Hg or higher or receiving treatment for hypertension were considered to have elevated blood pressure; those with a fasting glucose level of 100 to 125 mg/dL were considered to have fasting hyperglycemia; a fasting triglyceride level of 150 mg/dL or greater was considered hypertriglyceridemia (to convert triglyceride level to millimoles per liter, multiply by 0.0113); and an HDL-C level of less than 40 mg/dL in men or less than 50 mg/dL in women was considered low (to convert HDL-C level to millimoles per liter, multiply by 0.0259).

Our primary focus was to examine the generalizability of the FOS models by reestimating 3 logistic regression models reported by the FOS. The first was a personal model based on information known to an individual without laboratory data. The second was a simple clinical model that used categorical values of the variables already listed. The third used continuous values for age, BM I, systolic blood pressure, and HDL-C, triglyceride, and fasting glucose levels. The discriminatory capability of the models was evaluated using C statistic, also known as the area under the receiver operating characteristic (ROC) curve. All models were estimated using commercially available statistical software (SAS version 8.2; SAS Institute, Cary, North Carolina). We provide the FOS results for comparison (with permission) but do not possess the source data from which the results were derived. Therefore, we cannot statistically compare the 2 samples or resulting models.

In addition to reestimating the FOS models, we examined how well the FOS equations fit our population by applying the FOS function to the KPNW data using the Hosmer-Lemeshow χ2 statistic as the measure of calibration. We then recalibrated the models using methods suggested by D’Agostino et al.⁹

We also compared the 8-year risk of DM estimated by the point score developed by the FOS investigators from the simple clinical model with the actual DM incidence observed in our data at each point level. We first made this comparison in the total KPNW sample. However, we observed that the overall risk of incident DM in the total KPNW sample (16.5%) was substantially greater than the 5.1% reported in the FOS data. To determine whether the accuracy of the point score was affected by this difference, we created a subsample of the KPNW sample by randomly selecting 926 of 3406 patients who developed DM, such that the proportion developing DM was identical to that in the FOS sample. We then again compared the 8-year estimated risk of DM with the observed DM incidence in this subsample.

RESULTS
The baseline characteristics of 20,644 KPNW subjects are given in Table 1, along with the characteristics of 3140 persons in the FOS sample. The KPNW sample seemed to be somewhat older (mean age, 57.4 vs 54.0 years) and more obese (mean BMI [calculated as weight in kilograms divided by height in meters squared], 30.3 vs 27.1). Greater proportions of the KPNW sample were hypertensive (67.8% vs 44.2%) and had elevated triglyceride levels (41.8% vs 31.8%).

The proportions in each sample with impaired fasting glucose level (100-125 mg/dL) were almost identical (26.8% vs 27.0%). The proportion developing DM was substantially greater in the KPNW sample compared with that in the FOS sample (16.5% vs 5.1%).

Reestimated Multivariate Models
The FOS and KPNW personal models for DM prediction using only categories of age, sex, family history of DM, and BMI are given in Table 2. Age younger than 50 years and BM I less than 25.0 were the referent categories. Although ages 50 to 64 years and 65 years or older significantly increased the risk of DM in both samples, the odds ratios were much greater for these variables in the FOS sample. Male sex produced a similar odds ratio in both samples but was not statistically significant in the FOS sample. In both samples, BM I of 25.0 to 29.9 approximately doubled the risk of DM, while BM I of 30.0 or higher increased the risk by about 6 times. The FOS model seemed to yield somewhat better discrimination than the KPNW model (area under the ROC curve, 0.724 vs 0.676).

Table 3 gives the results of the simple clinical models predicting DM development that added categories for hypetension, dyslipidemia, and impaired fasting glucose level to the personal model. In the FOS and KPNW samples, adjustment for these additional variables changed the direction of age as a predictor for DM. Male sex was not significant in either sample. Other variables, especially BMI, triglyceride level of 150 mg/dL or higher, and fasting glucose level of 100 to 125 mg/dL, performed remarkably similarly in the 2 samples. In addition, the discriminatory ability of the model in the 2 samples was similar (area under the ROC curve, 0.852 vs 0.824).

Results of the simple clinical models using continuously distributed variables are given in Table 4. Again, the odds ratios in the 2 samples were almost identical. Note that the FOS model included waist circumference (not statistically significant), which was not available in the KPNW sample. The area under the ROC curve of 0.881 in the FOS sample was again somewhat higher but similar to that of 0.840 the KPNW sample.

Calibration Analysis
The calibration analysis of the simple clinical model using categorical variables is given in Table 5. Applying the FOS function directly to the full KPNW sample produced a high χ2 statistic, indicating poor calibration. Recalibrating the model substantially lowered the χ2 statistic, but the value was still highly significant. Although still lower, the χ2 statistic for the reestimated equation was also statistically significant. In all 3 applications, calibration was better among the KPNW subsample.

FOS Point Score and DM Incidence
The FOS algorithm to estimate DM risk assigns 10 points for a fasting glucose level of 100 to 125 mg/dL, 5 points for an HDL-C level of less than 40 mg/dL in men or less than 50 mg/dL in women, 2 points for a BMI of 25.0 to 29.9, 5 points for a BM I of 30.0 or higher, 3 points for parental history of DM, 3 points for a triglyceride level of 150 mg/dL or higher, and 2 points for blood pressure of 130/85 mm Hg or higher or receiving treatment. The 8-year risk of type 2 DM as projected by the sum of the points as developed by the FOS is plotted in the Figure, ranging from less than 3% among patients who score 10 or fewer points to greater than 35% among patients who score 25 or more points. Actual DM incidence for the total KPNW sample at each point level is plotted according to the points assigned from baseline values, ranging from 5% (<10 points) to almost 60% (>25 points). Actual DM incidence at each point level among the subset of the KPNW sample randomly selected to match the FOS proportion developing DM is also plotted, closely following the estimated risk of the FOS point score.

DISCUSSION
In this observational study of the electronic medical records of 20,644 health maintenance organization members, we connfirmed the DM prediction equations developed by the FOS. In the simple clinical models, the odds ratios of the predictor variables were remarkably similar to those reported by the FOS, whether using categorical or continuous measures. The overall discrimination as measured by the area under the ROC curve was also similar. The validation of the FOS equations supports the conclusion that complex models are not needed to determine risk of type 2 DM. A set of clinical variables that is easy to collect (namely, BMI, the presence of hypertension, family history of DM, and HDL-C, triglyceride, and fasting glucose levels) produced an area under the ROC curve of 0.84.

However, we did not confirm the FOS point score predicting 8-year DM risk. Although we found that more points equated to greater risk, the accuracy of the point score risk  estimate seemed to be contingent on the overall incidence of DM in the population being similar to the 5.1% found in the FOS data. In our study sample, which had a much higher DM incidence of 16.5%, the point score dramatically underestimated individual risk. When we randomly selected a subset of patients who developed DM at the same 5.1% rate found in the FOS data, the expected risk predicted by the point score and the actual proportion developing DM at each point total were almost identical. This finding has important ramifications for the applicability of the simple point scores to higher risk populations. Applying the FOS point score to an individual patient could lead to severe overestimation or underestimation of the patient’s risk if the underlying mean incident  rate is substantially different from that in the FOS sample. However, because actual DM incidence increased with each point at a rate almost identical to that suggested by the FOS score, the number of points accurately represents increasing DM risk. Therefore, if managed care plans with data resources similar to those of KPNW plotted their actual DM incidence at each point level, the point score could be accurately adapted to their specific populations.

Race/ethnicity mix may explain some of the difference in the proportions of the FOS and KPNW samples that developed DM. The FOS sample was 99% white compared with approximately 90% of the KPNW sample. Selection bias likely explains much more of the difference. Although the FOS sample was population based, it was composed of volunteers, who may have been healthier than nonvolunteers. Because the KPNW sample was observational, we were limited to analyzing approximately 20% of KPNW patients for whom all the necessary clinical data were available. Among the total KPNW population aged 26 to 82 years (those with and without all the clinical measures), the 7-year incidence of DM was 6.5%. That 16.5% of our study sample experienced DM during the same 7 years suggests that these patients had the necessary measures because their clinicians may have been concerned about their DM risk. However, these are precisely the type of patients to which a clinician would likely apply the FOS point score. Although the FOS point score might accurately assess DM risk on a populationwide basis, it seems to substantially underestimate that risk among clinical attendees at all point levels.

In addition to the KPNW sample being composed of healthcare seekers, there were some specific differences between the KPNW and FOS samples that likely affected the underlying incidence of DM. In particular, the mean BM I of the KPNW sample was 30.3 compared with 27.1 in the FOS sample. Because obesity is one of the strongest risk factors for DM, it is not surprising that DM incidence was much greater in the KPNW sample. Nonetheless, if the predictive power of BM I is continuous, the FOS risk score will underestimate the effect of very high BMI because patients with a BMI of 30.0 receive the same 5 points as patients with a BM I of 35.0. This is an important consideration given the continuing trend of rising obesity prevalence.^10-12 In addition, the KPNW sample had a much higher proportion of subjects with hypertension and high triglyceride levels, both well-known risk factors for DM incidence.¹³

To our knowledge, this is the first study to examine the applicability of the FOS prediction equations of incident DM. However, it is not the first to find Framingham equations to be inaccurate. Previous investigations of the Framingham CHD risk equations have shown that underestimation of risk occurs in patients with DM.^14,15 This is due to an underlying risk of CHD in DM that is greater than that in the general population on which the Framingham equations were based. When such systematic underestimation occurs, a process known as recalibration can be used, which replaces the Framingham mean predictor values and mean incident rate with the mean values and mean incident rate from the new cohort. Recalibration has been shown to correct overestimation or underestimation in ethnic groups with different underlying CHD risk but does not produce a better model fit than reestimating the equations with each site’s data.⁹ We similarly found that recalibration greatly improved the fit of the FOS function but not more so than reestimating risk in our data using identical variables.

The FOS is not the first study to attempt DM risk prediction. Most of those efforts have focused on predicting prevalent but undiagnosed DM.^3,16-18 A simple tool that relied on questionnaire-based data was developed in Finland, but that risk score predicted DM medication initiation, not incident DM per se.¹⁹ Another longitudinal risk score using 4 variables (age, sex, and triglyceride and fasting glucose levels) has been proposed, but the area under the ROC curve for that score was 0.71, considerably lower than that in the FOS model. The San Antonio Heart Study investigators developed an incident DM prediction model similar to the FOS model in the variables used and in performance but did not propose a risk score.⁴ Given the simplicity of the FOS risk score and the widespread use of a similar score for predicting CHD risk, the FOS tool might be rapidly adopted into clinical practice. Although the FOS equations seem valid for predicting DM incidence, the point score did not accurately  predict incident DM in a population at much greater risk. Therefore, the accuracy of a point score requires that the underlying incidence of the individuals to whom it is applied be similar to the population from which it was derived. Because the clinician is unlikely to know whether this is indeed true, using the point score to estimate DM risk of individual patients must be done cautiously. Health plans that can recalculate the risk associated with each point level can accurately adapt the point score to predict DM risk in their populations.