Skip to main content
eScholarship
Open Access Publications from the University of California

Scholarly Works (8 results)

  • Thesis
  • Peer Reviewed
UC San Diego Electronic Theses and Dissertations (2022)

The retina, the light sensitive tissue lining the interior of the eye, is the only part of the central nervous system (CNS) that can be imaged at micron resolution in vivo. Retinal diseases including age-related macular degeneration, diabetes retinopathy, and vascular occlusions are important causes of vision loss and have systemic implications for millions of patients. Retinal imaging is of great significance to diagnosing and monitoring both retinal diseases and systematic diseases that manifest in the retina. A variety of imaging devices have been developed, including color fundus (CF) photography, infrared reflectance (IR), fundus autofluorescence (FAF), dye-based angiography, optical coherence tomography (OCT), and OCT angiography (OCT-A). Each imaging modality is particularly useful for observing certain aspects of the retina, and can be utilized for visualization of specific diseases.

In this dissertation, we propose deep learning based methods for retinal image processing, including multimodal retinal image registration, OCT motion correction, and OCT retinal layer segmentation. We present our established work on a deep learning framework for multimodal retinal image registration, a comprehensive study of the correlation between subjective and objective evaluation metrics for multimodal retinal image registration, convolutional neural networks for correction of axial and coronal motion artifacts in 3D OCT volumes, and joint motion correction and 3D OCT layer segmentation network.

The dissertation not only proposes novel approaches in image processing, enhances the observation of retinal diseases, but will also provide insights on observing systematic diseases through the retina, including diabetes, cardiovascular disease, and preclinical Alzheimer’s Disease. The proposed deep learning based retinal image processing approaches would build a connection between ophthalmology and image processing literature, and the findings may provide a good insight for researchers who investigate retinal image registration, retinal image segmentation and retinal disease detection.

    Cover page: Deep Learning Based Multimodal Retinal Image Processing
    • Article
    • Peer Reviewed
    UC San Diego Previously Published Works (2022)
    Optical Coherence Tomography (OCT) is a widely used non-invasive high resolution 3D imaging technique for biological tissues and plays an important role in ophthalmology. OCT retinal layer segmentation is a fundamental image processing step for OCT-Angiography projection, and disease analysis. A major problem in retinal imaging is the motion artifacts introduced by involuntary eye movements. In this paper, we propose neural networks that jointly correct eye motion and retinal layer segmentation utilizing 3D OCT information, so that the segmentation among neighboring B-scans would be consistent. The experimental results show both visual and quantitative improvements by combining motion correction and 3D OCT layer segmentation comparing to conventional and deep-learning based 2D OCT layer segmentation.
      Cover page: JOINT MOTION CORRECTION AND 3D SEGMENTATION WITH GRAPH-ASSISTED NEURAL NETWORKS FOR RETINAL OCT.
      • Article
      • Peer Reviewed
      UC San Diego Previously Published Works (2021)
      A sequence of images is usually captured to observe the change of health status in medical diagnosis. However, an image sequence taken over year usually suffers from severe deformation, making it time-consuming for physicians to match corresponding patterns. In this paper, we propose a coarse-to-fine pipeline for retinal image registration based on convolutional neural network. By leveraging the three components of the pipeline: feature matching, outlier rejection, and local registration, we recover the deformation and accurately align multi-temporal image sequences. Experimental results show that the proposed network is robust to severe deformation as well as illumination and contrast variations. With the proposed registration pipeline, the change of image patterns over time can be identified through visual analysis.
        Cover page: A Convolutional Neural Network Pipeline For Multi-Temporal Retinal Image Registration
        • Article
        • Peer Reviewed
        UC San Diego Previously Published Works (2021)
        Multi-modal retinal image registration between 2D Ultra-Widefield (UWF) and narrow-angle (NA) images has not been well-studied, since most existing methods mainly focus on NA image alignment. The stereographic projection model used in UWF imaging causes strong distortions in peripheral areas, which leads to inferior alignment quality. We propose a distortion correction method that remaps the UWF images based on estimated camera view points of NA images. In addition, we set up a CNN-based registration pipeline for UWF and NA images, which consists of the distortion correction method and three networks for vessel segmentation, feature detection and matching, and outlier rejection. Experimental results on our collected dataset shows the effectiveness of the proposed pipeline and the distortion correction method.
          Cover page: Perspective Distortion Correction for Multi-Modal Registration between Ultra-Widefield and Narrow-Angle Retinal Images
          • Article
          • Peer Reviewed
          UC San Diego Previously Published Works (2022)
          Multi-modal retinal image registration plays an important role in the ophthalmological diagnosis process. The conventional methods lack robustness in aligning multi-modal images of various imaging qualities. Deep-learning methods have not been widely developed for this task, especially for the coarse-to-fine registration pipeline. To handle this task, we propose a two-step method based on deep convolutional networks, including a coarse alignment step and a fine alignment step. In the coarse alignment step, a global registration matrix is estimated by three sequentially connected networks for vessel segmentation, feature detection and description, and outlier rejection, respectively. In the fine alignment step, a deformable registration network is set up to find pixel-wise correspondence between a target image and a coarsely aligned image from the previous step to further improve the alignment accuracy. Particularly, an unsupervised learning framework is proposed to handle the difficulties of inconsistent modalities and lack of labeled training data for the fine alignment step. The proposed framework first changes multi-modal images into a same modality through modality transformers, and then adopts photometric consistency loss and smoothness loss to train the deformable registration network. The experimental results show that the proposed method achieves state-of-the-art results in Dice metrics and is more robust in challenging cases.
            Cover page: Two-Step Registration on Multi-Modal Retinal Images via Deep Neural Networks
            • Article
            • Peer Reviewed
            UC San Diego Previously Published Works (2021)
            Optical Coherence Tomography (OCT) is a powerful technique for non-invasive 3D imaging of biological tissues at high resolution that has revolutionized retinal imaging. A major challenge in OCT imaging is the motion artifacts introduced by involuntary eye movements. In this paper, we propose a convolutional neural network that learns to correct axial motion in OCT based on a single volumetric scan. The proposed method is able to correct large motion, while preserving the overall curvature of the retina. The experimental results show significant improvements in visual quality as well as overall error compared to the conventional methods in both normal and disease cases.
              Cover page: Learning to Correct Axial Motion in Oct for 3D Retinal Imaging
              • Article
              • Peer Reviewed
              UC San Diego Previously Published Works (2023)

              Background

              Retinitis pigmentosa (RP) represents a group of progressive, genetically heterogenous blinding diseases. Recently, relationships between measures of retinal function and structure are needed to help identify outcome measures or biomarkers for clinical trials. The ability to align retinal multimodal images, taken on different platforms, will allow better understanding of this relationship. We investigate the efficacy of artificial intelligence (AI) in overlaying different multimodal retinal images in RP patients.

              Methods

              We overlayed infrared images from microperimetry on near-infra-red images from scanning laser ophthalmoscope and spectral domain optical coherence tomography in RP patients using manual alignment and AI. The AI adopted a two-step framework and was trained on a separate dataset. Manual alignment was performed using in-house software that allowed labelling of six key points located at vessel bifurcations. Manual overlay was considered successful if the distance between same key points on the overlayed images was ≤1/2°.

              Results

              Fifty-seven eyes of 32 patients were included in the analysis. AI was significantly more accurate and successful in aligning images compared to manual alignment as confirmed by linear mixed-effects modelling (p < 0.001). A receiver operating characteristic analysis, used to compute the area under the curve of the AI (0.991) and manual (0.835) Dice coefficients in relation to their respective 'truth' values, found AI significantly more accurate in the overlay (p < 0.001).

              Conclusion

              AI was significantly more accurate than manual alignment in overlaying multimodal retinal imaging in RP patients and showed the potential to use AI algorithms for future multimodal clinical and research applications.
                Cover page: Efficacy and accuracy of artificial intelligence to overlay multimodal images from different optical instruments in patients with retinitis pigmentosa
                • Article
                • Peer Reviewed
                UC San Diego Previously Published Works (2020)

                Purpose

                The purpose of this study was to evaluate the ability to align two types of retinal images taken on different platforms; color fundus (CF) photographs and infrared scanning laser ophthalmoscope (IR SLO) images using mathematical warping and artificial intelligence (AI).

                Methods

                We collected 109 matched pairs of CF and IR SLO images. An AI algorithm utilizing two separate networks was developed. A style transfer network (STN) was used to segment vessel structures. A registration network was used to align the segmented images to each. Neither network used a ground truth dataset. A conventional image warping algorithm was used as a control. Software displayed image pairs as a 5 × 5 checkerboard grid composed of alternating subimages. This technique permitted vessel alignment determination by human observers and 5 masked graders evaluated alignment by the AI and conventional warping in 25 fields for each image.

                Results

                Our new AI method was superior to conventional warping at generating vessel alignment as judged by masked human graders (P < 0.0001). The average number of good/excellent matches increased from 90.5% to 94.4% with AI method.

                Conclusions

                AI permitted a more accurate overlay of CF and IR SLO images than conventional mathematical warping. This is a first step toward developing an AI that could allow overlay of all types of fundus images by utilizing vascular landmarks.

                Translational relevance

                The ability to align and overlay imaging data from multiple instruments and manufacturers will permit better analysis of this complex data helping understand disease and predict treatment.
                  Cover page: Artificial Intelligence for Automated Overlay of Fundus Camera and Scanning Laser Ophthalmoscope Images
                  Top