Demand Response (DR) serves to reduce the demand for electricity especially during times when supply is scarce or expensive. The program entails calling the recruited consumers to reduce their energy consumption from an established baseline. Baseline is an estimate of the counter-factual against which load reductions are measured to determine payments to consumers. This creates an incentive for consumers to inflate their baseline so that their payments are inflated. There have been reported cases where consumers have inflated their baseline to increase their payments. To address this, we propose a novel self-reported baseline mechanism for DR programs in both retail and wholesale electricity markets. Also by comparing with mechanisms used in practice by System Operator's like CAISO (California Independent System Operator) we show that the proposed mechanism is either reliable or requires less payments for the same level of load curtailment provided by consumers.

Finally we consider a repeated DR setting where the DR events repeat. We propose a baseline DR mechanism and a pricing policy that learns consumer's response to payments and exploits the learnt information such that the cumulative losses in savings, compared to the oracle determined optimal pricing mechanism, does not grow at an uncontrolled rate. In particular we show that it grows at a sub-linear rate.

Motivated by environmental and energy security concerns, many states and countries have enacted legislation calling for increased renewable adoption in the electricity sector. Unlike conventional fossil fuel-based electricity sources, renewable resources such as wind and solar power are characterized by intermittent, uncertain, and non-dispatchable output. Successfully addressing this variability in renewable generation is a prerequisite for deep renewable penetration. The prevailing paradigm in most power systems, exemplified by programs such as the Participant Intermittent Renewable Program (PIRP) in California, is one where the system operator accepts all renewable generation and absorbs the attendant variability through operating reserves. This approach works today at modest renewable generation levels but will result in untenable increases in reserve requirements tomorrow when renewables serve a sizeable fraction of total electric load. Hence, meeting ambitious renewable penetration targets will require the implementation of a number of variability mitigation strategies across the entire spectrum of power system operations. In this dissertation, we identify and explore areas in which control and optimization techniques can help provide some of these solutions.

One such mitigation strategy is the coordinated aggregation of deferrable loads and storage, in which load is tailored to match variable supply. We investigate methods of exploiting these demand-side capabilities by developing and evaluating algorithms for the real-time scheduling of these flexible resources. We note that the benefits of coordinated aggregation can be achieved at modest levels of both deferrable load participation and flexibility. We also provide an analytical framework for understanding how these resources influence the costs of meeting load requirements incurred in wholesale electricity markets. Specifically, we explore the interplay between deferrable load scheduling and cost-minimizing procurements of bulk power and reserve capacity made in the day-ahead forward market.

Another solution we consider, specific to wind power, involves curtailing generator output in certain situations. We explore how a wind power producer - subject to financial penalties for imbalances from contracted amounts - might leverage power curtailment capability to mitigate financial risk arising from price and production uncertainty. In particular, we analytically quantify the economic benefit derived from curtailment as an explicit function of expected prices, and compute empirical estimates of this curtailment benefit using price data from the various power system operators.

Transportation systems are the backbone of our modern economy. The ease of urban mobility is crucial for the success of cities. The automobile, which revolutionized urban mobility in the 20th century, is now struggling to keep up with demands for safety, efficiency, and equity. With more than 36,000 lives lost annually to traffic crashes in the US, and the average American spending 97 hours every year stuck in traffic, there is an imminent need for innovative solutions.

Two recent technologies show promise in delivering the next leap in urban mobility: (a) transportation network services, and (b) automated vehicles. Transportation network services (TNS) are businesses that connect people spread across the transportation network for efficient provision of goods and services. Examples include ride-hailing services such as Uber, and delivery services such as Instacart. Automated vehicles (AVs), popularly known as self-driving cars, obviate the need for a human to be in the driver’s seat. The combination of AVs and TNS is expected to significantly reduce traffic fatalities, alleviate congestion, and improve accessibility.

Despite their transformative potential, the path to a bright AV-TNS future of urban mobility seems unclear. Companies have realized after years of effort that designing safe AVs is much more difficult than originally anticipated. Recent studies have found that TNS companies have increased congestion in major US cities. Moreover, a significant proportion of TNS workers are unable to earn the prevailing minimum wage in their jurisdictions. This thesis is an attempt to understand why the promised AV-TNS future has remained elusive.

We explore two broad themes along this path: (a) economics of transportation network services, and (b) automated vehicle safety. Our work on these themes constitute the two halves of this thesis. In the first half, we introduce a unified framework to analyze the economics of transportation network services. Using this framework, we analyze the impact of recent labor regulations on the ride-hailing ecosystem and suggest alternatives that can achieve better outcomes. Adapting this framework to the case of delivery services, we demonstrate how the trade-off between delivery times and the cost of delivery, mediated by the extent of order pooling, dictates their economic viability. In the second half, we use a combination of theoretical models and empirical data to argue why connectivity is indispensable for safe AV deployment. Recognizing that some crashes are inevitable without connectivity, we propose a risk assessment framework that leverages human driving data along with on-road AV testing data to develop insights into their safety capabilities. We discuss several applications of this framework that inform safe AV deployment.

Detection of process sensitive patterns known as hotspots is critical to maximising yield in integrated circuit layouts. Design rule checking is a common tool in the industry for fast hotspot detection, but it is becoming impractical as we approach smaller technology nodes, due to an exponentially growing rule book which must be manually maintained.

A pattern matching based system for hotspot detection is explored in this dissertation which has the advantage of being automatically generated while still performing detection with a runtime comparable to design rule checking. Instead of scanning a design layout for certain combinations of distances found in a library of rules, we use a pattern matching tool to find occurrences in the layout of patterns which match or closely resemble known bad configurations.

In order for such a system to detect hotspots effectively, we require a comprehensive catalogue of hotspot patterns for the pattern matcher to use. Of course, it is not feasible to simply store all known hotspots patterns in the catalogue, as the memory requirements and pattern matching runtimes would be enormous. Instead, the catalogue must contain a smaller number of hotspot classes, where each class represents a set of geometrically similar patterns which embody the same type of hotspot. The pattern matching tool is able to find matches to a hotspot class by matching to a particular representative pattern of the class, and allowing for small geometric deviations from this representative.

To build such a catalogue, a large diverse set of hotspot clips must be analysed and reduced to a small number of hotspot classes. To perform this reduction meaningfully, a deep understanding of what it means for two hotspots to be the same "type'" of hotspot is required. We perform an in-depth analysis of the process sensitivity of patterns. From this we design a distance metric which quantifies the extent to which the process sensitivities of two hotspots are caused by similar geometry of surrounding features.

Equipped with this distance metric, clustering algorithms are used to group a dataset of clips into a small number of clusters containing similar hotspots. Because the number of clips to analyse must be very large in order to build a comprehensive catalogue, it is important to make this process as computationally efficient as possible. To this end, the distance metric and the clustering algorithm are designed and tailored for maximum efficiency.

To test our hotspot detection system, we build a catalogue of hotspot classes based on a dataset of clips extracted from a device layout. A pattern matching tool then uses our catalogue to detect hotspots, and the results are presented.

This dissertation is concerned with the distributed control of nonlinear multi-agent consensus problems. The main objective of the consensus study is to design distributed algorithms that rely on only local interaction to achieve global group behavior. We propose a distributed sliding mode control (DSMC) framework for nonlinear heterogeneous multi-agent systems under different information exchange topologies. The DSMC constructs the topological sliding surface and reaching law via a so-called "topological structured function". The control law obtained by matching the topological sliding surface and topological reaching law is naturally distributed. Under this framework, topological diversity is explicitly incorporated. Also, the consensus problem is significantly simplified by mapping N interconnected higher-order dynamics into an N-th order sliding variable. The DSMC framework supports both leaderless consensus and consensus with a leader. For both cases, we show asymptotic stability when the topology contains a spanning tree, and further prove finite-time convergence under the undirected topologies. We also extend the DSMC framework to MIMO systems.

To demonstrate the usage and show the effectiveness of DSMC, we discuss two major applications with simulation results. The first application is for heterogeneous platoon systems. The control objective is to regulate vehicles to travel at a common speed while maintaining desired inter-vehicle gaps. The information flow topology dictates the pattern of communication between vehicles in the platoon. The second application is for flocking of nonholonomic unicycle agents.

Contemporary power systems are characterized by increasing penetrations of renewable and distributed energy resources (DERs), which present both opportunities and challenges for reliable system operation. Uncertainty in renewable generation must be managed over both shorter and longer timescales, necessitating greater flexibility in the supply and demand of electricity. Fortunately, this rise in renewables has been accompanied by the growth of flexible energy resources such as energy storage and aggregated demand flexibility. However, these assets are generally not adequately integrated into existing electricity market structures leading to many benefits of their flexibility going unrealized. Additionally, distributed energy resources, such as rooftop solar and electric vehicles, present a challenge to the traditional top-down management of power systems, requiring new decentralized control and market architectures to ensure their safe operation and to maximize their value.

This dissertation tackles a range of questions relating to the integration of flexible and distributed resources into electricity markets in the presence of uncertainty, using techniques from control theory, game theory, multiparametric programming, and distributed optimization.

The main contributions of this dissertation encompass novel analysis and design of markets, both centralized and decentralized, that explicitly integrate distributed and flexible resources, numerical methods providing exact solutions to multi-leader follower equilibrium problems among strategic bidders, and insights into the role of virtual bidders in managing demand uncertainty in two-stage markets.

Chapter 1 introduces the context for the dissertation, describing trends driving the energy industry and the challenges and opportunities they pose. Current practice and existing literature are described with regard to uncertainty, flexibility, and decentralized markets and control. Chapter 2 lays out the mathematical background for the dissertation, describing models for power flow, electricity markets, and market participants, in addition to the theory of multiparametric programming. Chapter 3 demonstrates the equivalence of virtual bidding and stochastic optimization in two-stage markets under certain conditions. It also characterizes the equilibrium of a crowd of virtual bidders with heterogeneous beliefs and explores simple learning strategies for virtual bidders to reach equilibrium. Chapter 4 studies robust Cournot-Bertrand equilibria among generation firms. Firms anticipate demand and renewable uncertainty and optimize their actions to ensure their profits are robust to this uncertainty. The chapter concludes by discussing the effects on market outcomes of this behavior. Chapter 5 addresses the more general topic of strategic bidding by generation firms in electricity markets. This question involves solving non-convex bilevel optimization problems, which form part of the wider class of multi-leader follower problems. Both the single firm problem, constituting an MPEC, and the multi-firm equilibrium problem, constituting an EPEC, are solved exactly using techniques from multiparametric programming. This latter contribution, in particular, represents a significant advance over current methods, generalizing the solution of supply function equilibrium problems to cases with asymmetric players and transmission constraints. Chapter 6 analyzes a number of regulatory models for the integration of energy storage into wholesale electricity markets, including Open Access Storage, and investigates the implications and incentives of a variety of storage ownership structures under each model. Chapter 7 demonstrates the potential of load flexibility for congestion relief, defining the notion of a congestion-free dispatch. Chapter 8 describes the design of a decentralized coordination mechanism for DERs in a microgrid using blockchain technology. The implementation uses ADMM with a smart contract on the blockchain serving as the ADMM coordinator. Chapter 9 illustrates the design of an open-gate forward market with specific flexibility products for DERs, allowing users to hedge better against uncertainty and correctly express and maximize the value of their flexibility. A number of algorithms are presented to address the combinatorial scheduling and pricing problem involved with flexible loads.

Global warming now poses one of the most serious challenges to the well-being of humanity at large. The projected increase in the Earth's mean surface and ocean temperatures will have a severe impact on human health in both the developed and developing regions of the world. As the burning of fossil fuels contributes significantly to worldwide greenhouse gas emissions, there has been a concerted policy reform effort in both the US and abroad to transform the electricity sector by increasing the displacement of conventional fossil fuel-based thermal generation with clean renewable generation such as wind and solar. California, for example, has set a target of 33% renewable energy penetration by the year 2020. Wind and solar energy will play a key role in realizing such aggressive targets. However, at these deep penetration levels, the inherent variability of wind and solar power production poses serious engineering and market challenges. These are due to the uncertainty, intermittency, and uncontrollability of wind and solar power. They are essentially random -- a stark contrast to conventional thermal power generation.

How is variability in wind an solar power production dealt with today? Today, wind and solar energy are assimilated into the grid through legislative mandates, feed-in tariffs, lenient imbalance penalty pricing, guaranteed grid access, tax relief, and/or construction subsidies. Specifically, in California, the Participating Intermittent Renewable Program (PIRP) legislation compels the independent system operator (ISO) to accept all produced wind power subject to certain contractual constraints. This amounts to a system take-all-wind modus operandi in which wind power is treated as a negative load and the subsequent increase in the variability of net-load is absorbed by a portfolio of reserve generation capacity, whose cost is allocated amongst the load serving entities (LSE). Moreover, this socialisation of added reserve costs amongst the LSEs constitutes an implicit subsidy for variability costs to participating wind power producers. We submit to the reader that the current extra-market approach to renewable energy integration will become untenable as wind and solar energy penetration increases. Under this system-take-all-power regime, the attendant intermittency and limited forecastability of wind and solar power production will lead to a significant increase in the reserve generation requirements necessary to maintain system balance -- an unacceptable consequence. It is too expensive. Ergo, it will rapidly become infeasible to continue the implicit subsidization of the variability costs among the load serving entities. Moreover, it severely mitigates the net greenhouse gas benefit of renewable energy, as regulating reserves are normally supplied by fast-acting, fossil fuel based thermal generators such as natural gas turbines. The current strategy cannot scale. Clearly, strategies that mitigate the need for additional reserve requirements will be an essential means to supporting deep integration of variable renewable energy. Throughout this dissertation, we will focus explicitly on wind, however, much of the analysis is directly applicable to solar power generation as well.

How will variability be dealt with tomorrow? In the near term, we argue that wind power producers will be forced to participate in conventional electricity markets alongside traditional dispatchable generation, where they will face ex-post financial penalties for deviations from contracts offered ex-ante in forward markets -- thus eliminating the implicit subsidy for variability costs. In response to the financial risk emanating from uncertainty in wind power production, a rational wind power producer will be forced to curtail its projected output, thus decreasing the amount of variability that has to be compensated for with reserve generation by the system operator. However, such a removal of the implicit subsidy for variability cost may result in significant profit loss to the wind power producer. Consequently, it will become necessary for the wind power producer to develop and evaluate strategies that aid in the mitigation of wind power output variability. In this dissertation, we quantify, within the setting of a perfectly competitive market, the maximal expected profit achievable by a wind power producer through optimal bidding. Moreover, as wind is an inherently variable source of energy, we explore the sensitivity of optimal expected profit to uncertainty in the underlying wind process and quantify the marginal economic value of various firming mechanisms that aid in the mitigation of power output variability. Specifically, we appraise the benefit of improved forecasting and quantify the added value of recourse opportunities afforded by the co-location of an energy storage system and/or fast-acting thermal generation with the wind power producer. Further, we explore the extent to which a group of N independent wind power producers can exploit the statistical benefits of aggregation and risk sharing by forming a willing coalition to pool their variable power to jointly offer the aggregate output as single entity into a forward energy market.

An interesting way to study biological systems is from the perspective of control theory. Most of these systems can only operate within a narrow range of conditions, and external disturbances typically conspire to push them out of their normal operating domains. This has necessitated the development of mechanisms for physiological and environmental sensing to ensure that vital internal variables are regulated. Without these feedback control systems, life itself would be impossible due to daily and seasonal variations in environmental conditions. Survival therefore depends largely on the effectiveness of these control systems, and it is commonly assumed that natural selection has led to the evolution of optimal control strategies. This framework of evolutionary optimality is used in this dissertation to examine sporulation, a bacterial survival strategy employed by Bacillus subtilis.

Sporulation is the process of forming a morphologically-distinct, dormant structure that is able to withstand severe environmental insults. This dormant cell structure, called an endospore (or spore), is typically formed in response to nutrient limitation or other environmental signals associated with low food supply. A signal transduction pathway integrates intra- and extra-cellular signals to arrive at a decision to sporulate, effectively acting as a control mechanism that allows survival in the face of environmental nutrient limitation and other serious disturbances. In the framework of evolutionary optimality, this control strategy is hypothesized to be optimal in the sense of maximizing a fitness reward function for the B. subtilis colony.

In this dissertation, sporulation is analyzed from this perspective to uncover some interesting characteristics of the decision policy. It is shown that sporulation provides a higher fitness than a simpler bacterial survival strategy with many of the same benefits (called dormancy in this study). The particular environment in which this is verified features a catastrophic event several generations into the future, after which only the survival structures remain. This result agrees with the morphological differences between a spore and a simple dormant cell. Moreover, the optimal sporulation policy qualitatively agrees with experimental data, while the optimal dormancy strategy is significantly different from observed behavior in literature studies. These results offer a candidate evolutionary justification for the existence of sporulation.

Fitness-maximizing sporulation policies are then studied in the contexts of simple population-level models with food-per-growing-cell rate dependencies. Since the results for a single population model are not consistent with expected results, a second model is postulated based on resource competition between two bacterial populations. The competitive exclusion principle is shown to hold for the proposed model structure, which provides an extension of previous resource competition results due to the sporulation model's "storage-like" spore states. For specialized cases of this competing populations model (birth and death rates equal for both populations), an approximation of the steady state solution is derived and input-output stability is analytically proved using perturbation methods. Both of these are notable due to the model's non-unique steady state solutions and the nonlinearity of the proposed model. Though the steady state approximation imposes nutrient influx changes to be small, it is intended to model adiabatic system trajectories for slowly-changing nutrient conditions. In response to these changes, a game theoretic analysis yields two policies that cannot be invaded by rare mutants: 100% steady state sporulation efficiency if nutrient influx decreases on average, and 0% steady state sporulation efficiency if nutrient influx increases on average. Evolutionary dynamics are introduced to model changes between these two optimal policies, and the nutrient influx is assumed to randomly switch between positive and negative values based on a two-state Markov chain. A non-equilibrium policy is derived for these modeling and environmental conditions, which is much closer to experimental data than the optimal policies. The "choice" of the non-equilibrium policy over the optimal policies is then examined in a prospect theory framework, and it is shown that the preference of the non-optimal policy could be explained with appropriate shifts of reference: when nutrient influx increases on average, the bacterium expects the worst (pessimistic), and when nutrient influx decreases on average, the bacterium expects the best (optimistic). While the non-equilibrium policy appears to fall within the class of bet-hedging (risk-spreading) strategies, this is not the case as the probability of population extinction is not decreased.

The work presented in this dissertation yields some interesting results on B. subtilis behavior, some of which may generalize to other organisms that are capable of entering an inactive state.

Climate change is one of the most urgent problems faced by humanity, and rising sea levels, extreme weather events and desertification pose a severe threat to human life as we know it. Greenhouse gas emissions resulting from human activities have caused long-term global warming, and are primarily caused by the burning of fossil fuels to generate energy, e.g., for electricity, heat or transport. It is essential that we move to cleaner sources of energy across the board to prevent further greenhouse gas emissions, and this move is driven by three main trends.

First, the rise of distributed energy resources through rooftop solar, backup batteries and electric vehicles has led to the creation of a new class of consumers which have electricity production capability. These resources tend to be variable, and are owned and operated independently. Second, the move to variable clean energy production will lead to power system operators having an increased need for demand side flexibility in order to accommodate the supply-side variability. Flexible consumers can bid in their flexibility into markets for profit, or use it to reduce their emissions impact. Third, the increasing share of electric vehicles will led to an intersection of the transportation and power networks, where electric vehicles will be able to use their batteries as `mobile' storage in the power network.

This dissertation addresses some key challenges associated with each of these trends, and proposes solutions for them.