The emergence of pervasive sensing, high bandwidth communications and inexpensive data storage and computation systems makes it possible to drastically change how we design, monitor and regulate very large-scale physical and human networks. Even small performance gains in the way we operate these networks translate into large savings. There are many critical challenges to create functional monitoring systems, such as data reliability, computational efficiency and proper system design, including choices of sensors, communication protocols, and analysis approaches.

In this dissertation introduces a framework to design a monitoring system, deploy it, maintain it and process the incoming heterogeneous sources of information, resulting in new applications. The framework is applied to urban traffic monitoring and road infrastructure sensing. We develop various state of the art statistical inference algorithms, compute performance guarantees and study some of the fundamental limits of the proposed ideas. We illustrate the methodology using experimental deployments we have built and are currently in use.

The California Department of Transportation (Caltrans) freeway sensor network has two components: the sensor system of 25,000 inductive loop sensors grouped into 8,000 vehicle detector stations (VDS) and covering 30,500 freeway direction-miles; and the communication network over which the sensor measurements are transported to Caltrans Traffic Management Centers.

The sensor network is virtually the only source of data for use in traffic operations, performance measurement, planning and traveler information. However, the value of these data is greatly reduced by the poor reliability of the sensor network: On a typical day in 2005, only 60 percent of the statewide sensor network provided reliable measurements.

This is the report of an empirical study of the reliability of the sensor network based on two data sets. The first set, obtained from PeMS, consists of a daily summary of the quality of data received from each loop sensor in Caltrans Districts 4, 7 and 11 during the 27 month observation period January 2005–March 2007. The second data set consists of reports of field inspections of more than 4,000 loops each in Districts 4 and 7 during December 2005–December 2006 as part of Caltrans’ Detector Fitness Program.

The study proposes and calculates three metrics of system performance: productivity is the fraction of days that sensors provide reliable measurements; stability is the frequency with which sensors switch from being reliable to becoming unreliable; and lifetime and fixing time—the number of consecutive days that sensors are continuously working or failed, respectively. Productivity measures the performance of the sensor system; stability measures the reliability of the communication network; lifetime and fixing time provide more detailed views of both components of the sensor network.

These metrics are used to evaluate the differences in system performance in Districts 4, 7 and 11. Productivity in District 11 is much better than in Districts 4 and 7; District 4 is slightly worse than District 7. A significant part of the productivity difference is due to the large number of sensors in Districts 4 and 7 that never worked during the 27 month observation period.

The stability metric shows that the communication network in all three Districts suffer short-term outages; again, District 4 is the worst and District 11 is the best. The outages are likely due to the communication network technology, including protocols, that is used in the different Districts.

The metrics are also used to evaluate the effectiveness of the Detector Fitness Program (DFP). The DFP is unlikely to be cost-effective: two-thirds of the visited loops show no improvement in system performance, the remaining one-third show marginal improvement. Simple suggestions for a more effective design of the DFP are offered.

Lastly, the report proposes a statistical model of sensor failure that could be used in a scientific approach to the maintenance and replacement of the sensor system.