In recent years, learning-based computer-aid diagnostic (CAD) models have attracted significant attentions, and demonstrated their potential to make real-world impact. The prosperity of the models is currently being fueled by leveraging the rapid advancements in artificial intelligence (AI). They were shown to be capable of resolving various clinical tasks, like stratifying cancerous patients, predicting the prognosis results, and automatically allocating lesion locations. However, most models primarily rely on a single modality of information, such as imaging data or clinical data. In comparison, the real-world clinical specialists generally conclude and make diagnoses by incorporating imaging observations with anatomical knowledge and other clinical information, such as laboratory test results and demographic data. This suggests that despite the advancements in current learning-based CAD models, we posit that existing models can be further improved by integrating imaging data with broader clinical insights.

In this dissertation, we explore the potentials of incorporating imaging information together with clinical information into the learning-based CAD models, with focus on applications associated with magnetic resonance imaging (MRI). We begin by demonstrating how the integration of radiological imaging features with clinical information helps enhance the performance of traditional machine-learning-based CAD models, and showing its generalizability onto different diseases. We then delve into deep-learning-based CAD models, and how it can be benefited by integrating both information on patient stratification. Finally, we discuss about the incorporation of domain-specific clinical knowledge into imaging-based deep learning network designs, using anatomical priors to reduce false positive detections and boost model efficacy on lesion allocation. Through extensive experimentation across diverse patient populations, our research not only validates the efficacy of integrating both imaging and clinical information in learning-based CAD model designs, but also highlights their potentials to facilitate the deployment of improved learning-based CAD models in real-world clinical settings in the foreseeable future.