Development and Validation of Prediction Scores for Early Mortality at Transition to Dialysis

Objective: To develop and validate a risk prediction model that would help individualize treatment and improve the shared decision‐making process between clinicians and patients. Patients and Methods: We developed a risk prediction tool for mortality during the first year of dialysis based on pre–end‐stage renal disease characteristics in a cohort of 35,878 US veterans with incident end‐stage renal disease who transitioned to dialysis treatment between October 1, 2007, and March 31, 2014 and then externally validated this tool among 4284 patients in the Kaiser Permanente Southern California (KPSC) health care system who transitioned to dialysis treatment between January 1, 2007, and September 30, 2015. Results: To ensure model goodness of fit, 2 separate models were selected for patients whose last estimated glomerular filtration rate (eGFR) before dialysis initiation was less than 15 mL/min per 1.73 m2 or 15 mL/min per 1.73 m2 or higher. Model discrimination in the internal validation cohort of veterans resulted in C statistics of 0.71 (95% CI, 0.70‐0.72) and 0.66 (95% CI, 0.65‐0.67) among patients with eGFR lower than 15 mL/min per 1.73 m2 and 15 mL/min per 1.73 m2 or higher, respectively. In the KPSC external validation cohort, the developed risk score exhibited C statistics of 0.77 (95% CI, 0.74‐0.79) in men and 0.74 (95% CI, 0.71‐0.76) in women with eGFR lower than 15 mL/min per 1.73 m2 and 0.71 (95% CI, 0.67‐0.74) in men and 0.67 (95% CI, 0.62‐0.72) in women with eGFR of 15 mL/min per 1.73 m2 or higher. Conclusion: A new risk prediction tool for mortality during the first year after transition to dialysis (available at www.DialysisScore.com) was developed in the large national Veterans Affairs cohort and validated with good performance in the racially, ethnically, and gender diverse KPSC cohort. This risk prediction tool will help identify high‐risk populations and guide management strategies at the transition to dialysis.

T he number of incident cases of end-stage renal disease (ESRD) in the United States has risen over time and exceeded 120,000 in 2014. 1 There is substantial heterogeneity in the incident ESRD population in which some patients may die early after dialysis initiation while others may experience greater longevity with dialysis. Overall, mortality is exceptionally high during the first year of dialysis. 1 Plans for the initiation of maintenance dialysis therapy, including whether to initiate dialysis, integrate conservative care, or organize end-of-life support, are affected by various factors such as comorbid conditions, personal beliefs, and cultural perspectives. 2,3 A risk prediction model to predict early mortality could help individualize treatment and support the shared decision-making process among clinicians, patients, and patients' family members. For example, if a patient has high anticipated risk of mortality within 1 year of dialysis, conservative management with infrequent dialysis therapy could be a potential option. [4][5][6] Additionally, it may provide little or no benefit but rather harm in such a case to provide aggressive treatments for cancer or certain chronic conditions such as hypertension, atrial fibrillation, or hyperphosphatemia.

Study Population and Data Source
The Transition of Care in Chronic Kidney Disease study included 2 historical cohorts with incident ESRD: (1) (Figure 1, A). We further restricted patients in the KPSC cohort to those without missing data on the variables selected in the final model ( Figure 1, B  Table 3, available online at http://www.mayoclinicproceedings.org). We defined pre-ESRD comorbidity status by the ICD-9-CM diagnosis and surgical codes recorded in both CMS and administrative databases (Supplemental Table 4 All data on body mass index and laboratory tests were obtained before dialysis initiation. We calculated eGFR using the Chronic Kidney Disease Epidemiology Collaboration formula. 21 The last measurement of each variable before ESRD transition was used in this study.

Statistical Analyses
Differences between groups were compared by standardized differences because of the large sample size of this study. 22,23 Differences in patient characteristics between 2 groups were compared by standardized difference, of which 0.8, 0.5, and 0.2 in absolute value were considered large, medium, and small differences, respectively.
Prediction models for mortality during the first year of dialysis were developed on the basis of survival data up to 14 months after transition to dialysis using Cox proportional hazards models. Follow-up started at dialysis initiation and continued until death or the date of final follow-up assessment (September 2, 2014). Information on date of censoring events was obtained from the records of the CMS, USRDS, and respective administrative database. As candidate predictors, we a priori selected clinically relevant variables less likely to be intentionally modified (Table 1) and excluded medications and easily modifiable variables (ie, hemoglobin, potassium, calcium, phosphorus, intact parathyroid hormone, bicarbonate, ferritin, and lipids). The proportions of missing data were approximately 3% for body mass index, serum sodium, and serum urea nitrogen and approximately 10% for white blood cell, serum albumin, and serum alkaline phosphatase. Missing data in the VA cohort were imputed by their mean values.
The VA cohort was randomly divided into a two-thirds development set (n ¼ 24,030) and a one-third internal validation set (n ¼ 11,848). Final models with reduced number of predictors were obtained in the development data set using backward selection based on Akaike information criterion. To address potential model overfitting (optimism), we estimated a linear shrinkage factor (g) using 100 bootstrap samples [24][25][26] and adjusted the risk score by the shrinkage factor. Overfitting, particularly in small data sets, results in regression coefficients being overestimated (overfitted) for prediction. Thus, a shrinkage factor (0 g 1) is estimated and applied to the risk score to shrink the regression coefficients towards zero so that predictions will more likely show better calibration on new patients (ie, validation data). Furthermore, our study has 2 additional protections against overfitting, ie, internal and external validation cohorts.
Model calibration was assessed by calibration plots and a group-based goodness-of-fit test for survival model. 27 In both the internal and external validation data sets, model predictive discrimination was assessed using the index of concordance, or C statistic. 25 The final prediction models based on the shrunken prognostic score (PS) can be used to estimate the predicted probabilities of all-cause death at a given time t (month). That is, the shrunken PS* that will be used to predict the outcomes of new/future patients will be PS* ¼ g Â b, where b is the collection of estimated coefficients in the final prediction model. The predicted survival at time t for a new patient can be obtained as S 0 (t) exp(PS*) , where S 0 (t) is the baseline survival estimate from the final model. Kaplan-Meier survival estimates were compared with predicted mortality at months 3, 6, 9, and 12 across 5 risk score categories based on cutoff points at 5th, 35th, 65th, and 95th percentiles in respective eGFR groups of the internal validation cohort. However, predicted survival rates and their 95% CIs were not provided for the external validation cohort due to data confidentiality because it required merging of the VA and KPSC data at the individual level.
Analyses were performed in SAS version 9.3 PROC PHREG (SAS Institute) and R version 3.12 (R Project for Statistical Computing) using libraries RMS and SURVIVAL.

Baseline Characteristics of the VA Cohort
The 35,878 patients in the VA cohort had a mean (SD) age of 69.4AE11.1 years, 35,215 (98%) were male, 24,344 (68%) were white, 2474 (7%) were Hispanic, and 17,182 (48%) had diabetes as the cause of their ESRD ( Table 1). The last eGFR before dialysis initiation was 12.4 mL/min per 1.73 m 2 (interquartile range [IQR], 8.6-18.6 mL/min per 1.73 m 2 ), and the prevalence of low and high eGFR (ie, <15 or !15 mL/min per 1.73 m 2 ) were 63% (n¼22,605) and 37% (n¼13,273), respectively. When compared with patients with low eGFR, those with high eGFR were older, more likely to be white, and more likely to have a history of hyperlipidemia, cardiovascular disease, and chronic pulmonary disease (absolute standardized difference, >0.2). There were no meaningful differences between the development and internal validation data sets (Supplemental Table 5, available online at http://www.mayoclinicproceedings.org). The time from the last eGFR measurement to dialysis initiation was 24 days (IQR, 3-109 days). The prevalence of hemodialysis and peritoneal dialysis as the initial dialysis modality was 94% (11,162) and 5% (579), Values are presented as mean AE SD, median (IQR), or No. (percentage). Percentages may not total 100 because of rounding. c SI conversion factors: To convert white blood cell values to Â10 9 /L, multiply by 0.001; to convert sodium values to mmol/L, multiply by 1.0; to convert albumin values to g/dL, multiply by 10; to convert urea nitrogen values to mmol/L, multiply by 0.357; to convert alkaline phosphatase values to mkat/L, multiply by 0.0167. d Differences in patient characteristics between 2 groups were compared by standardized difference, of which 0.8, 0.5, and 0.2 in absolute value were considered large, medium, and small differences, respectively. respectively, and dialysis modality was uncertain among 107 patients (1%). There were a total of 9813 deaths (4525 and 5288 deaths among patients with low and high eGFR, respectively) during the median follow-up of 14 months (IQR, 9-14 months). Six-month and 1-year mortality rates were 10% and 18% among patients with low eGFR, respectively, and 25% and 37% among patients with high eGFR, respectively.

Development of the Prediction Score
A single prediction score resulted in poor model fit (P<.001 for goodness-of-fit test). We then evaluated model fit by stratifying patients according to age, race, diabetes as the cause of ESRD, initial dialysis modality, and time from the last eGFR measurement to dialysis initiation, and only stratification based on the last eGFR resulted in significant improvement (P>.1 for goodness-of-fit test) (Supplemental Table 6, available online at http://www.mayoclinicproceedings.org). This led to the development of 2 separate prediction scores for patients stratified by low and high eGFR. The final early mortality prediction model coefficients are presented in Table 2. The performances of our reduced/simplified models as quantified by C statistics were practically the same as the full models (data not shown).
Supplemental Figure 1, A and B (available online at http://www.mayoclinicproceedings. org) shows the predicted probability of 1-year mortality after transitioning to dialysis as a function of risk score (ie, risk plots). Calibration plots comparing observed vs predicted 1-year survival graphically exhibited consistent results (Supplemental Figure 2, A and B, available online at http://www.mayoclinicproceedings.org). We also compared Kaplan-Meier estimates and predicted survival at months 3, 6, 9, and 12 after dialysis initiation across 5 categories with cutoff points at 5th, 35th, 65th, and 95th percentiles of risk score in respective eGFR groups (Figure 2, A and B). Predicted survival was slightly lower than actual in the lowest-risk group but was otherwise consistent (well calibrated) with the observed survival.

Internal Validation Within the VA Cohort
The performance of the prediction score was tested in the internal validation data set of 11,848 veterans. The model discrimination was acceptable among patients with low eGFR, with a C statistic of 0.71 (95% CI, 0.70-0.72). The prediction model showed a C statistic of 0.66 (95% CI, 0.65-0.67) among those with high eGFR. Subgroup analyses including initial dialysis modality (ie, hemodialysis vs peritoneal dialysis) showed consistent C statistics except for patients aged 65 years or older, in whom C statistics were 0.66 (95% CI, 0.65-0.68) and 0.63 (95% CI, 0.61-0.64) for those with low and high eGFR, respectively ( Table 3). Risk plots are shown in Supplemental Figure 1, C and D.
The group-based goodness-of-fit tests showed no significant difference between observed and predicted 1-year mortality after transitioning to dialysis in both the low and high eGFR groups (overall P>.1) (Supplemental Table 6). Calibration plots exhibited consistent results (Supplemental Figure 2, C and D). Predicted vs observed Kaplan-Meier estimates were also consistent, as were those in the development cohort (Figure 2, C and D).  Table 7, available online at http://www.mayoclinicproceedings.org). The last eGFR before dialysis initiation in the KPSC cohort was 10.5 mL/min per 1.73 m 2 (IQR, 7.7-14.1 mL/min per 1.73 m 2 ), and the prevalence of low and high eGFR (<15 or !15 mL/min per 1.73 m 2 ) was 79% (3363) and 21% (921), respectively. Time from the last eGFR measurement to dialysis initiation was 11 days (IQR, 5-25 days), and the prevalence of hemodialysis and peritoneal dialysis as the initial dialysis modality was 87% (3727) and 13% (557), respectively. Supplemental Table 8 summarizes data on predictors in the final model stratifying KPSC patients by sex and last eGFR before dialysis initiation. There were a total of 824 deaths (552 and 272 deaths among patients with low and high eGFR, respectively) during the median follow-up time of 14 months (IQR, 14-14 months). Six-month and 1-year mortality were 9% and 15% among patients with low eGFR, respectively, and 20% and 29% among patients with high eGFR, respectively.

External Validation
Risk plots for the KPSC cohort show the same pattern of relationship between mortality and risk score (Supplemental Figure 1, E and F). However, observed survival was typically higher than model-predicted survival, especially in the high-risk groups (Supplemental Figure 2, E and F). Nevertheless, the Kaplan-Meier estimates (Figure 2, E Model parameter estimate before application of shrinkage factor (ie, 0.970 and 0.965 for low and high eGFR groups, respectively). Estimated 1-year survival probabilities can be obtained as: 1 À S 0 exp(PS*) , where PS* ¼ g Â LP, g is the shrinkage factor, LP is the linear predictor using the given parameter estimates, and S 0 (t). S 0 (t) is the baseline survival estimate from the final model. S 0 at months 3, 6, 9, and 12 were 0.9722, 0.9446, 0.9234, and 0.9011 for patients with last eGFR <15 mL/min/1.73 m 2 , respectively, and 0.9313, 0.8790, 0.8369, and 0.7994 for patients with last eGFR !15 mL/min/1.73 m 2 , respectively. c Last available value during 1 year before dialysis initiation. eGFR <15 mL/min per 1.73 m 2 Internal development data set in the VA cohort  5% to <35% 35% to <65% 65% to <95% >95% Points and bars represent means and 95% CIs of predicted survival for the risk score categories. Cutoff points were selected at 5th, 35th, 65th, and 95th percentiles in respective eGFR groups of the internal validation cohort and were consistent across cohorts. Predicted survival rates and their 95% CIs were not provided for the external validation cohort due to data confidentiality because it requires merging of the VA and KPSC data at the individual level. and F) show the same consistent ordering of survival and risk score percentile (ie, high risk score, high mortality), and the prediction score showed similar or even better discrimination in the KPSC cohort than in the VA cohort; C statistics were 0.77 (95% CI, 0.74-0.79) and 0.74 (95% CI, 0.71-0.76) among men and women with low eGFR, respectively, and 0.71 (95% CI, 0.67-0.74) and 0.67 (95% CI, 0.62-0.72) among men and women with high eGFR, respectively.

DISCUSSION
In this study, we developed a prediction tool for mortality during the first year of dialysis based on pre-ESRD data in a large contemporary national cohort of US veterans. Although the VA cohort has inherently unique characteristics compared with other populations, these new risk scores appeared generalizable given that these were externally validated in the racially, ethnically, and gender diverse KPSC cohort with reasonable discrimination in most studies groups. In order to facilitate practical implementation, we have created an online risk score calculator at www.DialysisScore.com that provides the predicted risk of mortality at months 3, 6, 9, and 12 after dialysis initiation. The discriminatory performance of our risk score may be improved by adding other factors such as socioeconomic status, 28 medication adherence, 29 and timing of referral to nephrologists. 30,31 In particular, physical and cognitive function may play an important role in predicting outcomes among elderly patients, in whom our risk score showed C statistics of less than 0.7. Indeed, geriatric syndromes such as cognitive impairment and frailty are prevalent in the dialysis population, 32 and the decline in cognitive and physical function has been shown to accelerate after dialysis initiation, especially among elderly patients. [33][34][35] These variables are difficult to capture because trained personnel and careful consideration are necessary for their evaluation, but comprehensive risk models accounting for additional measures specific to the elderly population may help predict mortality more precisely.
Several previous prediction models focused on the elderly population 7,10-12 given the differences in clinical consequences (eg, high mortality and hospitalization rates) and patient Initial dialysis modality was uncertain among 65 and 42 patients with last eGFR <15 and >¼15 ml/min/1.73m 2 , respectively, and they were excluded in the subgroup analyses based on initial dialysis modality due to small sample sizes.
characteristics (eg, more severe comorbid conditions and dialysis initiation with higher eGFR) among older patients. 36,37 However, we obtained satisfactory model fit with stratification by last eGFR before dialysis initiation but not with stratification by age (data not shown), suggesting that the association between risk factors and outcomes is dependent on comorbid conditions rather than age. Higher eGFR may reflect unmeasured clinical conditions at the time of dialysis initiation, such as severely impaired cardiac function requiring more strict fluid management than usual and overestimation of actual kidney function due to low muscle mass. Indeed, the increased mortality risk associated with higher eGFR in our study was consistent with that in previous studies, 7 and these heterogeneous conditions may explain why model fit was improved by stratification according to last eGFR and why we observed lower C statistics among patients with high eGFR. The patterns of mortality during the first year of dialysis have been reported to differ by age and treatment modality. 1 For patients undergoing hemodialysis, mortality is especially high during the first several months, and this early mortality is more remarkable among older patients. In contrast, patients undergoing peritoneal dialysis have relatively constant mortality over the course of the year. Nevertheless, a previous study reported similar outcomes between dialysis modalities when accounting for patient characteristics, 38 and our risk scores had consistent C statistics across subgroups of age and treatment modality (Table 3). Predicted survival was also concordant with observed survival across risk groups and time points except for month 6 in the highest risk score group (ie, !95th percentile of risk score) among patients with a last eGFR of 15 mL/min per 1.73 m 2 or higher ( Figure 2). This group was the oldest population, and their overestimated survival may be explained by the early mortality phenomena among the elderly.
Our study results should be interpreted with several potential limitations in mind. First, the model performance may depend on the accuracy of data on comorbid conditions. We used specific ICD-9-CM codes in the CMS and administrative databases but were not able to confirm their accuracy. However, our prediction model was externally validated in the KPSC cohort where clinical practice is different from that of the VA, suggesting little impact of this potential bias. Furthermore, by using diagnostic codes to identify the presence of comorbid conditions, our score lends itself to automated implementation in electronic medical records. Second, although prediction performances (C statistics) were similar for the development and internal validation for the VA cohort as well as the external KPSC cohort, overall calibration was better for the interval validation cohort (Supplemental Figure 2) as expected because of the survival differences in the KPSC and VA cohorts. Future model refinements may include a mixture of patient cohorts more representative of the general dialysis population. Third, timing to use this risk score would also affect the model performance because values in selected variables change over time. Although C statistics were consistent between veterans with different time periods between the last eGFR measurement and dialysis initiation (ie, <6 vs !6 months) (Table 3), our model showed greater discrimination in the lower eGFR groups and the KPSC cohort, in whom the lag period was shorter than that of their counterparts. Fourth, our cohort consisted mostly of whites and blacks. Although the KPSC cohort included 11% Asians, predicted mortality should be interpreted with caution in races other than whites or blacks before being validated in a specific race. Lastly, we did not account for vascular access. However, several studies have found that the mortality risk associated with vascular access may largely be accounted for by patient characteristics. 39,40 CONCLUSION A new prediction tool for mortality during the first year of dialysis was developed and externally validated among incident dialysis patients in the United States. This tool is available at our website www.DialysisScore.com, which allows clinicians, patients, and patient family members to obtain the predicted mortality at months 3, 6, 9, and 12 after dialysis initiation. Our new tool could help improve nephrology care and may facilitate shared decision making for individualized preparation for dialysis initiation in late stages of chronic kidney disease.

ACKNOWLEDGMENTS
Drs Streja, Kovesdy, and Kalantar-Zadeh are employees of the US Department of Veterans Affairs. The interpretation and reporting of these data are the responsibility of the authors and should in no way be considered as official policy or interpretation of the US Department of Veterans Affairs or the US government. Funders/ sponsors had no role in analysis and interpretation of the data, preparation, review, or approval of the submitted manuscript, and decision to submit the manuscript for publication.

SUPPLEMENTAL ONLINE MATERIAL
Supplemental material can be found online at http://www.mayoclinicproceedings.org. Supplemental material attached to journal articles has not been edited, and the authors take responsibility for the accuracy of all data.