This dissertation explores structural damage identification of civil infrastructure by an image-based approach. The underlying basis for the research is that optical images of structures or structural components provide pixels that can be viewed as spatially high- resolution s̀ensors', which convey the topographical appearance characteristics of structural integrity in a highly collaborative manner. With images captured at different times, the temporal variations of the images can capture the changes of structural integrity, i.e. structural damage. Two scales of structural damage are addressed in this dissertation, namely, the geospatial scale of structural damage in urban areas and the local scale of structural damage in structural components. Accordingly, satellite imagery and digital camera imagery are used in the two scales of damage identification. The methodology in this dissertation for solving the identification problem do not fall into the traditional photogrammetric category. Instead, modern computer vision and machine learning methods are relied upon. For the urban structural damage identification, the task is formulated as a damage extraction and classification problem. Accordingly, two sets of solution frameworks, each of which includes a damage feature extraction method and a multi-level structural damage classification method, are proposed. For the image-based local structural damage identification, two problems are studied; one is the geometric quantification of damage in images, the other is the monitoring of the inception or propagation of damage using multi-temporal images. Two different model-based solutions are proposed to solve these problems. The methods proposed in the study of urban damage identification using satellite imagery can be used to perform urban damage classification for individual urban buildings or urban sub-areas in the immediate aftermath of a large-scale disaster that strikes a built urban area. Therefore, a preliminary damage assessment can be quickly produced, which may facilitate rapid disaster response. Based on the study of image-based local damage identification, the resulting damage-related information may be integrated into mechanics-based predicative modeling or vibration-based structural health monitoring, and the proposed methods can be extended to analyze images from different types of non-destructive imaging devices