Measuring collaboration and transdisciplinary integration in team science

As the science of team science evolves, the development of measures that assess important processes related to working in transdisciplinary teams is critical. Therefore, the purpose of this paper is to present the psychometric properties of scales measuring collaborative processes and transdisciplinary integration. hundred-sixteen researchers and research staff participating in the Transdisciplinary Tobacco Use Research Centers (TTURC) Initiative completed the TTURC researcher survey. Conﬁrmatory-factor analyses were used to verify the hypothesized factor structures. Descriptive data pertinent to these scales and their associations with other constructs were included to further examine the properties of the scales. Overall, the hypothesized-factor structures, with some minor modiﬁcations, were validated. A total of four scales were developed, three to assess collaborative processes (satisfaction with the collaboration, impact of collaboration, trust and respect) and one to assess transdisciplinary integration. All scales were found to have adequate internal consistency (i.e., Cronbach (cid:1) ’s were all (cid:1) 0.70); were correlated with intermediate markers of collaborations (e.g., the collaboration and transdisciplinary-integration scales were positively associated with the perception of a center’s making good progress in creating new methods, new science and models, and new interventions); and showed some ability to detect group differences. Conclusions: This paper provides valid tools that can be utilized to examine the underlying processes of team science—an important step toward advancing the science of team science. Internal consistency. The SPSS reliability subroutine was used to compute internal consistency (Cronbach’s coefﬁcient (cid:1) ) for the collaboration and transdisciplinary-integration scales. Using the lower-bound criteria for internal consistency, a Cronbach’s (cid:1) of at least 0.70 was considered adequate. 28


Methods:
Two hundred-sixteen researchers and research staff participating in the Transdisciplinary Tobacco Use Research Centers (TTURC) Initiative completed the TTURC researcher survey. Confirmatory-factor analyses were used to verify the hypothesized factor structures. Descriptive data pertinent to these scales and their associations with other constructs were included to further examine the properties of the scales.

Results:
Overall, the hypothesized-factor structures, with some minor modifications, were validated. A total of four scales were developed, three to assess collaborative processes (satisfaction with the collaboration, impact of collaboration, trust and respect) and one to assess transdisciplinary integration. All scales were found to have adequate internal consistency (i.e., Cronbach ␣'s were all Ͼ0.70); were correlated with intermediate markers of collaborations (e.g., the collaboration and transdisciplinary-integration scales were positively associated with the perception of a center's making good progress in creating new methods, new science and models, and new interventions); and showed some ability to detect group differences.
Conclusions: This paper provides valid tools that can be utilized to examine the underlying processes of team science-an important step toward advancing the science of team science.
(Am J Prev Med 2008;35(2S):S151-S160) © 2008 American Journal of Preventive Medicine Background S everal studies [1][2][3][4] have documented that, since the mid-1950s, the natural, behavioral, and social sciences have made a pronounced shift from individually oriented research toward team-based scientific initiatives. This trend toward greater teamwork in science is paralleled by a growing emphasis on crossdisciplinary approaches to research and training. [5][6][7] Substantial investments by government agencies and private foundations in cross-disciplinary centers and teams have triggered a lively debate about the relative merits of individual-versus-team-based models of research and the emergence of a new area of program evaluation research, namely, the science of team science. 8 -11 Evaluations of team science initiatives aim to identify, measure, and understand the processes and outcomes of large-scale research collaborations. Given the substantial amount of federal and private resources that have been allocated to establish and maintain team science initiatives, it is essential that concerted efforts be made to evaluate both their near-, mid-, and longerterm collaborative processes and outcomes. [12][13][14] The science-of-team-science field is at a relative early stage in its development and can benefit from the development of psychometrically valid and reliable measures of collaborative processes, especially those involving cross-disciplinary synergy and integration. As these initial collaborative processes may be integrally linked to the achievement of subsequent and farreaching benefits to science and society, it is important to develop reliable and valid measures of these con-structs early-on as a basis for evaluating their influence on the cumulative contributions of a team initiative over a longer period.
Findings are presented here from an early-stage evaluation of the National Cancer Institute's (NCI) Transdisciplinary Tobacco Use Research Centers (TTURC) initiative. 15 The overall goals of the study were (1) to create and validate new methods and metrics for assessing crossdisciplinary collaboration and transdisciplinary integration within the context of the TTURC initiative, and (2) to develop and preliminarily assess a conceptual logic model linking the sequential phases, processes, and outcomes associated with large team science initiatives more generally. The TTURC program 15 is one of four large-scale, cross-disciplinary initiatives organized and funded since 1999 by the Division of Cancer Control and Population Sciences within NCI. a Currently, the total NIH investment into those four initiatives (TTURC, the Centers of Excellence in Cancer Communications Research, the Centers for Population Health and Health Disparities, and the Transdisciplinary Research on Energetics and Cancer centers) that address both basic and applied research in cancer control is approximately $286 million. 15-18,b Conceptual Foundations of the TTURC Initiative Evaluation Study The TTURC initiative is rooted in Rosenfield's conceptualization of transdisciplinary scientific collaboration. 19,20 Rosenfield describes a continuum of collaborative research ranging from unidisciplinary and multidisciplinary to interdisciplinary and transdisciplinary approaches. According to Rosenfield, transdisciplinary collaborations (compared to multidisciplinary and interdisciplinary forms of cross-disciplinary research) lead to the development of shared conceptual frameworks that not only integrate but also transcend the individual disciplinary perspectives represented by various members of the research team. These transdisciplinary conceptual frameworks, integrating the concepts and methods drawn from multiple disciplines and analytic levels, have the greatest potential to generate truly novel scientific and societal advances-reflected, for example, in a more comprehensive understanding of nicotine-addiction processes, the development of more-powerful smoking prevention strategies, and a substantial reduction of tobaccorelated disease and mortality in the population. 21,22 As a basis for understanding the conceptual and empirical links among cross-disciplinary collaboration, transdisciplinary integration, and the more distal scientific achievements and health outcomes generated by the TTURC initiative, Trochim and colleagues 23 developed a comprehensive logic model to evaluate large initiatives (ELI). TTURC investigators, funders, and other stakeholders (staff and scientific consultants) first completed a web-based concept-mapping exercise for the purpose of deriving key constructs associated with effective transdisciplinary-team initiatives and understanding the temporal relationships among the different constructs. They later developed a researcher survey that was designed to assess key components of the ELI logic model. The logic model ( Figure 1   health impacts. The collaboration cluster subsumes the dimensions of training, collaboration, and transdisciplinary integration. These constructs serve as proximal, or early-stage, markers of team effectiveness during the initial phase of the TTURC initiative. To the extent that the TTURCs are effective over the course of their initial and later phases, the levels of intellectual collaboration and transdisciplinary integration will be higher at the outset, thereby prompting changes in investigators' methods and models. Those methodologic and conceptual changes, in turn, enable translations of transdisciplinary knowledge into new health promotion interventions, policy innovations, and improved health outcomes. This hypothesized sequence of changes is ultimately expected to facilitate greater recognition of the value of transdisciplinary science and the broad-based adoption or institutionalization of transdisciplinary approaches to tobacco-use research. 23 Operationalizing the constructs included in the ELI logic model is an important starting point for evaluating the potential benefits of transdisciplinary research and is the focus of this paper. The findings reported below focus on two major components of the ELI logic model, namely, the collaboration and transdisciplinary-integration constructs. Although the effectiveness of collaborative teams has been studied extensively in nonscientific venues, the measures employed in those contexts often do not generalize readily to scientific settings. 24 -26 Therefore, some major purposes of this paper are to examine the factorial validity and internal consistency of three collaboration scales and one transdisciplinary-integration scale that were developed in the context of the TTURC initiative as well as to evaluate their associations with other constructs included in the ELI logic model ( Figure 1).

Participants
Participants consisted of all TTURC investigators (principal investigators, co-investigators, project directors, research associates, and scientists); research staff; and trainees who were identified by each center's principal investigator as eligible respondents for the researcher survey. As part of the TTURC evaluation, each principal investigator completed a center survey, which provided a quick profile of the center and the number of staff who would be eligible to complete the researcher survey. Among the seven TTURCs, there were 234 eligible respondents (Nϭ234); 216 completed the researcher survey, for an overall 92% response rate.

Data-Collection Protocol
The data were collected in the context of a program evaluation during the third year of the initiative. The TTURC principal investigators were primarily responsible for identifying someone who would serve as the point of contact for distributing the survey and reminding eligible respondents to complete it. The researchers and research staff were asked to complete the survey and mail it back in a self-addressed pre-paid envelope to the data processing center. To increase compliance, the data processing center compiled on a weekly basis the total number of Researcher Surveys received by each Center. The contact person received an anonymized update on their center's response rates, as well as response rates of the other centers (anonymized as well). The contact person was asked to send reminders to their colleagues and research staff to ensure an adequate response rate to the survey. At all times, the contact person or anyone involved were never aware of who responded to the Researcher Survey. Although the PIs, researchers, and research staff were encouraged to fill out the survey, their participation was completely voluntary.

TTURC Researcher Survey Development
The TTURC Researcher Survey is a 12-page instrument that included indices and scales that represented all the dimensions assessed by the ELI logic model ( Figure 1). Concept mapping served as the basis for the ELI logic model, and also served to provide much of the initial content for developing the researcher survey. Additionally, because the conceptmapping process consisted of clustering statements into dimensions, the statements within these clusters formed the initial theoretical operationalization of the dimensions. The researcher-survey development was led by a methodology team (WTM, LCM, and SM co-authors) and was developed in collaboration with TTURC funders, TTURC researchers, and input from a consulting committee. The researcher survey went through several expert reviews and revisions, and received final approval from a consulting committee for administration to the TTURCs. Of particular interest to this paper are the sections that focused on collaboration and transdisciplinary research (see Appendixes A and B for a description of the items).

Collaboration
The researcher survey included 23 items that assessed collaboration. All items used a 5-point, Likert-type response format. Fifteen items used the stem Please evaluate the collaboration within your center with the following response anchors: inadequate, poor, satisfactory, good, and excellent. The other eight items started with Please rate your views about collaboration with respect to your center-related research with the response anchors strongly disagree, somewhat disagree, not sure, somewhat agree, and strongly agree. It was determined a priori that the factor structure of the collaboration scale would have three correlated factors. One factor was designed to assess satisfaction with collaboration using eight items: acceptance of ideas, communication, researchers' strengths, organization, resolution of conflict, working styles, outside involvement, and discipline involvement. A second factor, designed to assess the impact of collaboration, used 6 items: meeting productivity, products productivity, overall productivity, research productivity, quality of research, and time burden. A final factor, designed to assess trust and respect in the collaborative context, used four items: being comfortable in showing limits, trusting colleagues, being open to criticism, and respect). Five of the initial collaboration items were excluded from the analyses as they did not measure the above constructs.

Transdisciplinary Integration
The researcher survey had 15 items that measured attitudes about transdisciplinary research. Respondents were asked to indicate their attitudes about transdisciplinary research and to provide interpretations based on their understanding or perception of transdisciplinary research. All items used a 5-point, Likert-type format with the response options strongly agree, somewhat disagree, not sure, somewhat agree, and strongly disagree. It was determined a priori that the items likely measured one factor that assessed transdisciplinary integration.

ELI Intermediate Markers of Progress Toward Collaboration and Transdisciplinary Integration
Although the researcher survey included a number of indexes that corresponded to the ELI logic model (Figure 1), only four of the indexes (methods, science and models, improved interventions, and publications) were used here. These were seen as intermediate markers of progress within the centers. It should be noted that for these constructs, index measures were created. Overall, these indexes measured how much progress had been achieved by the TTURCs in these areas. The methods index was computed by averaging 7 items (e.g., development or refinement of methods for gathering data); 17 items were averaged for the sciences-and-models index (e.g., understanding multiple determinants of the stages of nicotine addiction); 12 items were averaged to measure improved interventions (e.g., progress in pharmacologic interventions); and, finally, the publications index was the sum of submitted and published articles and abstracts.

Data Analysis
Factor structure. All negatively worded items were reversecoded for the analyses. Confirmatory-factor analyses, using the LISREL 8.8 software, served to validate the a priori-factor structure of the collaboration and transdisciplinary-integration scales. Parameter estimates were obtained using the maximumlikelihood method of estimation. As there are no agreed-upon standards for determining model fit, the criteria established by Hu and Bentler 27 for evaluating fit were followed. The chisquare goodness-of-fit test served to determine the overall fit of the factor structure, with a p-value Ͻ0.15 indicating that the residuals were no longer significant-hence, a good fit. Given that the chi-square is highly affected by sample size and the Finally, the corrected Akaike's information criteria (CAIC) served to compare the fit of different models while accounting for the number of parameters estimated in the model; a lower CAIC was indicative of a better fit. Standardized factor loadings ranged from Ϫ1.00 to 1.00, and a value of Ͻ0.30 was used to assess items that loaded poorly on the hypothesized factor.
Relationship with ELI outcomes. It was hypothesized that the collaboration and the transdisciplinary-integration scales would be significantly correlated with select intermediate markers on the ELI logic model (methods, science and models, improved interventions, and publications). To assess these bivariate relationships, the potential clustering effect of the center was accounted for by first regressing each scale on the center (coded as a set of dummy variables) and then computing a Pearson product moment correlation between the resulting residuals.
Group differences. Finally, one-way ANOVAs were computed for each scale to examine if differences existed on these scales by respondent's role and by center, using the general linear model procedure in SAS to take into account the nested structure of the data. Posthoc analyses were conducted (as appropriate) using the least-significant-differences method. Although some differences were expected, these analyses were mainly exploratory. All analyses used a p-value Ͻ0.05 to determine significance.

Demographic Information
Of the valid responses (nϭ202), 50% of the respondents (nϭ101) indicated that they had been with their center for Ն2 years, and 66.3% reported having worked Ͻ40 hours per week on TTURC-related efforts. The largest percentage of respondents (nϭ100) characterized their research role in the Center as investigator (49.3%), while others indicated their role as professional staff (25.1%); student (16.3%); and other (9.4%).
Respondents were asked to report their primary, secondary, and tertiary disciplinary affiliations. The most commonly reported disciplinary affiliations were psychology (nϭ88); public health (nϭ50); and behavioral medicine (nϭ44). Respondents also reported considerable collaboration with new disciplines in association with their TTURCrelated efforts. While 76.9% (nϭ166) of the respondents had collaborated with at least one new discipline over the past year, 62.5% (nϭ135) reported collaborating with two or more new disciplines. The most-frequently mentioned new disciplines with which researchers reported collaborating included genetics (27.3%); public health (26.9%); communications (24.5%); epidemiology (22.7%); and biostatistics (20.8%), reflecting a broad spectrum of disciplines from the biological sciences to population health.

Factorial Validity of the Collaboration Scales
The confirmatory-factor analysis results for the collaboration scales are summarized in Table 1. The results showed that the a priori three-factor structure did not fit the data very well (the RMSEA, SRMR, and residuals were high). The results suggested that Item 14, time burden (collaboration has posed a significant time burden in your research), did not load on the factor that assessed the impact of collaboration. Of the 18 items, this was the only item that was negatively worded. Given that the factor loading was extremely low (0.01), the solution was run without this item (Model 2). As shown in Table 1, the fit of Model 2 significantly improved compared to Model 1, but the solution remained inadequate (the RMSEA, SRMR, and residuals were high). Examination of the modification indexes revealed a weakness in the factor structure, suggesting the addition of two correlated-error terms to Model 2. A correlation between Items 7, outside involvement, and Item 8, discipline involvement, was added, as well as a correlation between Item 12, research productivity, and Item 13, quality of research.
It should be noted that adding these correlations suggests that the solution does not account for all of the correlations that exist among these four items. To address this issue, Model 3 added these two extra correlations (Table 1), which resulted in an adequate fit as well as a significant improvement in the fit of the model. The final three-factor solution is presented in Figure 2. The factor loadings (standardized paths) ranged from 0.42 on Item 15, showing limits, to a high of 0.88 on Item 11, overall productivity. Correlations among the factors were moderately high (the correlation between impact of collaboration and trust and respect was 0.65) to high (the correlation between satisfaction with collaboration and impact of collaboration was 0.90, and between satisfaction with collaboration and trust and respect was 0.81). Cronbach's ␣ for each scale was adequate: 0.91 for satisfaction with collaboration, 0.87 for impact of collabora-tion, and 0.75 for trust and respect. Item and subscale means were high; on the 1-to 5-point Likert scale, the means were (in general) closer to the 4-point-indicative of overall satisfaction with the collaborative process. Overall item means and scale means were high, indicating satisfaction in these areas.

Factorial Validity of the Transdisciplinary-Integration Scale
The confirmatory-factor-analysis results of the transdisciplinary-integration scale are summarized in Table 2. The results showed that the hypothesized one-factor structure for the transdisciplinary items did not fit very well (inadequate RMSEA, SRMR, and standardized residuals). Examination of the modification indexes suggested that the correlation between two items (Item 6, changes my research ideas, and Item 7, improved my research) was not well-explained by the solution. Given that the content of these two items was related, a correlated-error term was added to the model (Model 2). Adding this correlated-error term significantly improved the fit of the model, but the solution remained inadequate (high RMSEA, SRMR, and standardized residuals). Re-examination of the modification indexes revealed that the correlations among all the negatively worded items (Items 2, 3, and 4) remained high.  Note: Model 1: Hypothesized three-factor structure; Model 2: hypothesized three-factor solution minus the item that assesses "time burden"; Model 3: Model 2 plus two correlated-error terms (one between Items 7 and 8, and a second between Items 12 and 13) CAIC, corrected Aikaike's information criterion; CFI, comparative fit index; NNFI, non-normed fit index; RMSEA, root mean square residuals; SRMR, standardized root mean square residuals To remedy this, a new model was fitted that included extra correlated-errors terms among all negatively worded items (Model 3), and resulted in an adequate fit and significant improvement in the fit of the model. As shown in the final solution (Figure 3), the factor loadings (standardized paths) for the negatively worded items (Item 2, knowledge interference; Item 3, less productive; and Item 4, fewer publications) were borderline adequate (Ͼ0.30) to inadequate (Ͻ0.30), indicating that although the overall fit of the model was improved by the addition of a correlated-error term among these items, these items remained poor indicators of transdisciplinary integration. Table 3 summarizes the associations for the collaboration and transdisciplinary-integration scales with select intermediate ELI outcomes. The results showed that the three scales for collaboration and the transdisciplinary-integration were significantly correlated with the following ELI outcomes: methods, science and models, and interventions. Table 4 presents collaboration and transdisciplinaryintegration scales by respondent's role and by center. The analyses revealed significant betweengroup differences by respondent's role for the trust-and-respect collaboration scale only (Fϭ3.47 [dfϭ3, 183], pϽ0.05) and revealed no significant differences for the other collaboration scales and the transdisciplinary-integration scale by respondent's role. Posthoc comparisons revealed that on the trustand-respect factor, investigators' scores were significantly higher than those of "other" research staff (pϽ0.05), and students' scores were significantly higher than the scores of both the professional support staff scores (pϽ0.05) and the "other" research staff (pϽ0.05). Finally, the results comparing differences by center revealed significant between-center differences for all the collaboration factors: satisfaction with collaboration ( Posthoc results are available upon request and are not reported here, as the anonymity of the data precludes any meaningful interpretation; however, the results are presented to demonstrate the power of these scales to detect differences among centers.

Discussion
The purpose of this paper was to examine the psychometric properties of scales that measure collaboration and transdisciplinary integration in the context of team science. Overall, the hypothesized factor structureswith some minor modifications-were validated. A total of four scales were developed, and measured the following: perceived satisfaction with collaboration, the impact of collaboration on the research process, trust and respect in a collaborative setting, and transdisciplinary integration. All scales were found to have adequate internal consistency (i.e., Cronbach ␣'s were all Ͼ0.70); to be correlated with most intermediate markers of ELI; and to show some ability to detect some group differences. One of the key findings from this study is that the hypothesized factors were verified, with minor modifications (i.e., correlated-error terms were added to these solutions). Having some correlated-error terms suggests that there might be some redundancies among these items that might be important to re-examine in future administrations (e.g., collaboration Item 7, involvement of collaborators from outside the center, and Item 8, involvement of collaborators from diverse disciplines). However, it is important to note that the negatively worded items on both scales created some problems: not loading on the scale or creating spurious correlated-error terms. It is well-known that having a subset of negatively worded items leads to a methodologic artifact-either having an extraneous factor or having correlated-error terms among all negatively worded items (as observed in this paper). 29 Certainly the presence of such methodologic artifacts calls into question the common measurement practice of mixing positively and negatively worded items in the scale. 29 Because these items address an important area, they were maintained to maximize the content validity of the scale. It should be noted that the internal consistency of the scale was not adversely affected by keeping the negatively worded items in the scale.
Associations among the scales with intermediate markers of progress were presented to further evaluate the construct validity of these scales. These results suggest that those who perceived higher levels of satisfaction with collaboration and those who had an overall positive view of transdisciplinary integration also perceived that their center was making good progress in creating new methods, new science and models, and new interventions. The lack of association with the publications index is not unexpected, as cross-sectional associations were examined in Year 3 of the initiative, and the number of publications is expected to be limited at this early stage of the transdisciplinary effort. In fact, the results found a restricted range of publications (0 -6 total) for the initiative.
It has been suggested that empirical efforts to link specific facets of team-based science (e.g., processes of cross-disciplinary collaboration and intellectual integration generated through center-based working groups, retreats, and training programs) with more tangible scientific and societal outcomes may require longitudinal studies that extend over 1 or more decades. 30 Team science initiatives are structurally complex, and several years are required to establish and coordinate the efforts of multiple investigators and trainees working within and across several (often geographically dispersed) centers. 10 Therefore, the results reported here must be supplemented in future years by longer-term investigations that track the scientific and societal contributions of team initiatives sustained over 1 or more decades; and must incorporate comparison groups comprising individuals or small groups of scholars working on similar scientific questions-but from outside the framework of "big science." In closing, it should be noted that this study was limited in its ability to examine the predictive validity of these scales, as only cross-sectional data were available. Furthermore, the stability (test-retest reliability) of these scales was not assessed. Therefore, much more work is needed to further assess the utility of these scales for detecting changes over time (e.g., in the collaborative effectiveness and productivity of transdisciplinary centers); for detecting stability; and for elucidating the pathways by which team science initiatives generate longer-term impacts on scientific progress and population health as suggested by the ELI logic model. Another potential limitation of this study was that TTURC researchers may have reacted to the demand characteristics of the study by both responding in a manner that would make them appear to be working in more of a transdisciplinary manner and responding in a positive way to this type of collaborative work, especially given the financial incentive of TTURC initiatives. Nonetheless, with these caveats, this paper provides valid tools that can be utilized to examine the underlying processes of team science-an important initial step toward advancing the science-of-team-science field.
No financial disclosures were reported by the authors of this paper.