Pre-trained models have demonstrated remarkable capabilities in language understanding and generation, opening new possibilities in healthcare. They show promise in mining scientific literature, analyzing large-scale healthcare data, identifying patterns in emerging diseases, and automating clinical workflows—essentially functioning as research assistants. However, general-purpose pre-trained models—typically trained on web-scale corpora—lack the clinical grounding needed for reliable deployment in healthcare. To be effective, these models must be optimized for domain-specific needs. This thesis addresses three core challenges in adapting and utilizing pre-trained models for healthcare: (i) the lack of sufficient data for fine-tuning, (ii) evolving healthcare data, and (iii) the need to ensure transparency and traceability of AI-generated content.
To address data scarcity, we propose a three-level optimization framework that fine-tunes a pre-trained model to generate high-quality synthetic data for a target task with limited data. The framework begins by adapting the pre-trained model to a related, abundant dataset, assigning a learnable weight to each training sample. These weights are iteratively updated based on feedback from (another) downstream model trained on the generated data, enabling the framework to upweight samples that contribute more to downstream performance. This feedback inherently improves the fine-tuning of the pre-trained model, leading to the generation of data that enhances downstream task performance. We demonstrate the effectiveness of this approach on a long-COVID article classification task—a challenging low-resource setting.
For the second challenge—adapting to evolving healthcare data—we propose a bi-level optimization framework that fine-tunes a model on new data by updating only a sparse subset of parameters selected for task-specific adaptation. Rest of the model is regularized to remain close to values learned from previously seen sources, helping to mitigate forgetting. To identify which parameters to update, we assign a learnable score to each one and jointly optimize these scores and their corresponding weights in a two-stage process. We impose a sparsity constraint on the scores to ensure that large updates are limited to a small subset of parameters. We evaluate this framework on an early sepsis prediction task using patient data from four real-world hospitals.
To enable traceability of AI-generated content, we propose a watermarking algorithm applied at inference time that perturbs the model’s logits to bias generation toward a subset of vocabulary tokens determined by a secret key. To ensure this biasing does not degrade generation quality, we introduce a multi-objective optimization framework that jointly learns how many tokens to bias and by how much—balancing watermark detectability with semantic coherence. The approach improves detectability, preserves text quality, and enhances robustness against a range of watermark removal attacks compared to prior methods.
Together, these contributions offer a principled framework for adapting and securely utilizing pre-trained models in real-world healthcare settings.
