UC Davis

In recent times, the exponential surge in firmware demand and shorter Time-to-Market have underscored the necessity for proficient and precise firmware security analysis. But firmware security analysis has never been an easy job due to its complexity, lack of source code and limited resources. Currently, various static and dynamic firmware analysis tools are being developed. However, a prevalent concern lies in the absence of a standard benchmark that incorporates diverse characteristics, which is essential for effectively testing the tools from various angles. This thesis aims to bridge this gap by introducing FirmBench, a comprehensive benchmark specifically designed for binary analysis. FirmBench is meticulously curated to offer a wide array of scenarios and challenges, thoroughly evaluating the capabilities of security analysis tools. This research aims to contribute significantly to the advancement of firmware security analysis, providing developers with a reliable and versatile benchmark for assessing the effectiveness of their tools. FirmBench stands out from existing benchmarks due to its unique features, including the provision of source code, structured metadata, utilization of complex peripheral libraries and inclusion of samples with specific CVEs injected into framework of SDKs. FirmBench was put through numerous analyses using a wide array of static and dynamic tools. Through our characterization process, we have demonstrated that FirmBench is suitable for evaluating various state-of-the-art static and dynamic analysis tools. Furthermore, the benchmark enables testing tools with distinct combinations of operating systems (OS), architectures, micro-controller units (MCUs) and peripherals. This research has also yielded significant insights, such as implications stemming from the absence of centralized benchmark and the challenges associated with using existing datasets for binary analysis.