Background

The validity of quality of care assessments relies upon data quality, yet little is known about the relative completeness and validity of data sources for evaluating the quality of care.

Objectives

We evaluated concordance between ambulatory medical record and patient survey data. Levels of concordance, variations by type of item, sources of disagreement between data sources, and implications for quality of care assessment efforts are discussed.

Design and subjects

This was an observational study that included 1270 patients sampled from 39 West Coast medical organizations with at least 1 of the following: diabetes, ischemic heart disease, asthma or chronic obstructive pulmonary disease, or low back pain.

Measures

Items from both data sources were grouped into 4 conceptual domains: diagnosis, clinical services delivered, counseling and referral, and medication use. We present total agreement, kappa, sensitivity, and specificity at the item and domain-levels and for all items combined.

Results

We found good concordance between survey and medical records overall, but there was substantial variation within and across domains. The worst concordance was in the counseling and referrals domain, the best in the medication use domain. Patients were able to report with good sensitivity on memorable items.

Conclusions

Quality ratings are likely to vary in differing directions, depending on the data source used. The most appropriate data source for analyses of components of and overall quality of care must be considered in light of study objectives and resources. We recommend data collection from multiple sources to most accurately portray the patient and provider experience of medical care.

Few studies of the concordance between patient self-report and medical record data have examined how concordance varies with patient characteristics, and results of such studies have been mixed. Given discrepancies in the quality of care received across patient cohorts, it is important to understand the degree to which concordance metrics are robust across patient characteristics. We hypothesized that concordance between ambulatory medical record and patient survey data varies by patient demographic characteristics, especially education, income, and race/ethnicity. We present the results of bivariate and multivariate analyses including data from 1,270 patients with at least one of: diabetes, ischemic heart disease, asthma or COPD, or low back pain sampled from 39 West Coast medical organizations. We present total agreement, kappa, and survey sensitivity and specificity, stratified by patient demographic and health status characteristics. We conducted logistic regressions to test the impact of patient demographic characteristics, domain of medical care, and health status on these three measures of concordance. Survey sensitivity varied significantly by race/ethnicity in bivariate analyses, but this effect was erased in multivariate analyses. Our findings do not support the hypothesis that patient education, income, or race/ethnicity have an independent effect on concordance when controlling for other factors. However, concordance varied significantly by patient health status. The medical record and patient self-report do not measure quality comparably across patient cohorts. We recommend continued efforts to improve survey data collection across different patient populations and to improve the quality of clinical data. © Springer Science+Business Media, LLC 2006.

Objective

To illustrate, using empirical data, methodological challenges associated with patient responses to longitudinal surveys regarding the quality of process of care and health status, including overall response rate, differential response rate, and stability of responses with time.

Data sources/study setting

Primary patient self-report data were collected from 30,308 patients in 1996 and 13,438 patients in 1998 as part of a two-year longitudinal study of quality of care and health status of patients receiving care delivered by 63 physician organizations (physician groups) across three West Coast states.

Study design

We analyzed longitudinal, observational data collected by Pacific Business Group on Health (PBGH) from patients aged 18-70 using a four-page survey in 1996 and a similar survey in 1998 to assess health status, satisfaction, use of services, and self-reported process of care. A subset of patients with self-reported chronic disease in the 1996 study received an enriched survey in 1998 to more fully detail processes of care for patients with chronic disease.

Data collection/extraction methods

We measured response rate overall and separately for patients with chronic disease. Logistic regression was used to assess the impact of 1996 predictors on response to the follow-up 1998 survey. We compared process of care scores without and with nonresponse weights. Additionally, we measured stability of patient responses over time using percent agreement and kappa statistics, and examined rates of gender inconsistencies reported across the 1996 and 1998 surveys.

Principal findings

In 1998, response rates were 54 percent overall and 63 percent for patients with chronic disease. Patient demographics, health status, use of services, and satisfaction with care in 1996 were all significant predictors of response in 1998, highlighting the importance of analytic strategies (i.e., application of nonresponse weights) to minimize bias in estimates of care and outcomes associated with longitudinal quality of care and health outcome analyses. Process of care scores weighted for nonresponse differed from unweighted scores (p<.001). Stability of responses across time was moderate, but varied by survey item from fair to excellent.

Conclusions

Longitudinal analyses involving the collection of data from the same patients at two points in time provide opportunities for analysis of relationships between process and outcomes of care that cannot occur with cross-sectional data. We present empirical results documenting the scope of the problems and discuss options for responding to these challenges. With increasing emphasis in the United States on quality reporting and use of financial incentives for quality in the health care market, it is important to identify and address methodological challenges that potentially threaten the validity of quality-of-care assessments.