The goal of this dissertation is to investigate the formulation and analysis of a trust- region interior-point method for solving nonconvex optimization problems with a mixture of equality and inequality constraints. The proposed method is based on minimizing a merit function that may be interpreted as a shifted primal-dual penalty-barrier function. The method generates a sequence of iterates with limit points that are either infeasible stationary points or complementary approximate Karush-Kuhn-Tucker points, i.e., every limit point satisfies reasonable stopping criteria and is a Karush-Kuhn-Tucker point under a regularity condition that is the weakest constraint qualification associated with sequential optimality conditions. Under suitable additional assumptions, the method is equivalent to a shifted variant of the primal-dual path-following method in the neighborhood of a solution.The proposed method has an inner/outer iteration structure. The outer iteration specifies the form of the merit function. The inner iteration optimizes the merit function with fixed parameters using a trust-region method. The algorithm for solving the trust-region subproblem involves a procedure based on the application of a one-dimensional Newton’s method. Methods are proposed for treating the so-called ”hard case” in which no root of the one dimensional equation exists.

Trust-region methods are amongst the most commonly used methods in unconstrained mathematical optimization. Their impressive performance and sound theoretical guarantees make them suitable for a wide range of problem types. However, the computational complexity of existing methods for solving the trust-region subproblem prevents trust-region methods from being widely used in large-scale problems in both unconstrained and constrained settings. This dissertation introduces and analyzes three novel methods for solving the trust-region subproblem for large-scale constrained optimization problems. Convergence rates and proofs are presented where applicable. Furthermore, a trust-region approach is developed for the recently introduced all-shifted primal-dual penalty-barrier method for solving nonconvex, constrained optimization problems.

The three trust-region algorithms introduced are the shifted and inverted generalized Lanczos trust region algorithm, the locally optimal preconditioned conjugate gradient trust region, and the Jacobi-Davidson QZ trust region algorithm. Each new method exhibits improved performance over the existing standard methods and is best suited for problems too large for the traditional methods to handle efficiently. Furthermore, each method exhibits particular benefits for differently scaled problems.

Projected-search methods for bound-constrained optimization are based on searching along a continuous path obtained by projecting a search direction onto the feasible region. These methods have the potential to change the direction of the search path multiple times along the boundary of the feasible region at the cost of computing a single direction. However, as the objective function is only piecewise differentiable along the path, conventional projected-search methods are limited at using a simple backtracking procedure to obtain a step that satisfies an “Armijo-like” sufficient decrease condition. To extend the benefits of Wolfe line search for unconstrained optimization to projected-search methods, a new quasi-Wolfe step is introduced. Two general classes of projected-search methods that use the new quasi-Wolfe search are then formulated and analyzed. These methods may be broadly categorized as either active-set methods or interior methods. Additionally, a new quasi-Newton projected-search method UBOPT is proposed for unconstrained and bound-constrained optimization. The method computes quasi-Newton directions in a sequence of subspaces, and employs the framework of the class of projected-search active-set methods.

Furthermore, a new interior method is proposed for general nonlinearly constrained optimization, combining a shifted primal-dual interior method with a projected-search method for bound-constrained optimization. The method involves the computation of an approximate Newton direction for a primal-dual penalty-barrier function that incorporates shifts on both the primal and dual variables. The shifts allow the method to be safely “warm started” from a good approximate solution and eliminate the ill-conditioning of the associated linear equations that may occur when the variables are close to zero. The approximate Newton direction is used in conjunction with a new projected-search algorithm that employs a flexible non-monotone quasi-Armijo line search for the minimization of each penalty-barrier function. Numerical results demonstrate that the new method requires significantly fewer iterations than a conventional interior method, thereby reducing the number of times that the search direction need be computed.

This dissertation focuses on cubic regularization methods for the globalization of Newton's method, with applications in unconstrained and constrained optimization. In recent years, cubic regularization algorithms have emerged as popular alternatives to trust-region and line-search methods for unconstrained optimization. The goal of this research is to tackle some of the challenges associated with cubic regularization and extend the methods to solve constrained problems. The first part of the dissertation is dedicated to enhancing the efficiency of cubic regularization methods for unconstrained optimization without sacrificing their favorable convergence properties. A nonmonotone adaptive cubic regularization approach is proposed that combines adaptive cubic regularization with a line search. In particular, a sufficient decrease in the objective function is obtained by performing a nonmonotone line-search based on satisfying certain strong Wolfe conditions. This is an alternative to repeatedly solving the cubic subproblem with varying regularization parameters and requires lower computational cost and fewer iterations. In addition, two hybrid algorithms are developed that substitute cubic regularization with Newton's method when the objective function is well-behaved or a sufficient descent direction is readily obtainable from a conventional Newton method. By judiciously determining when to utilize cubic regularization, the efficiency of Newton's method can be balanced against the robustness of cubic regularization. In the second part of the dissertation, a novel primal-dual interior-point method for general nonlinear constrained optimization is proposed that minimizes a sequence of shifted primal-dual penalty-barrier functions. The method combines cubic regularization and a line-search to ensure global convergence. The proposed method calculates the cubic regularized step by factoring a sequence of matrices with diagonally-modified primal-dual structure, enabling the use of off-the-shelf linear equation software. This approach allows the extension of the method to large-scale problems while maintaining computational efficiency. Finally, the performance of the proposed algorithms for both unconstrained and constrained optimization is illustrated by extensive numerical results obtained from problems in the CUTEst test collection.

The main goal of this dissertation is to study the formulation and analysis of primal-dual path-following methods for nonlinear programming (NLP), which involves the minimization or maximization of a nonlinear objective function subject to constraints on the variables. Two important types of nonlinear program are problems with nonlinear equality constraints and problems with nonlinear inequality constraints. In this dissertation, two new methods are proposed for nonlinear programming. The first is a new primal-dual path-following augmented Lagrangian method (PDAL) for solving a nonlinear program with equality constraints only. The second is a new primal-dual path-following shifted penalty-barrier method (PDPB) for solving a nonlinear program with a mixture of equality and inequality constraints. The method of PDPB may be regarded as an extension of PDAL to handle nonlinear inequality constraints.

Algorithms PDAL and PDPB are iterative methods that share the same "two-level" structure involving outer and inner iterations. In the outer iteration of PDAL, the optimality conditions are perturbed to define a "path-following trajectory'' parameterized by a set of Lagrange multiplier estimates and a penalty parameter. The iterates are constructed to closely follow the trajectory towards a constrained local minimizer of the nonlinear program. If an outer iterate deviates significantly from the trajectory, then an inner iteration is invoked in which a primal-dual augmented Lagrangian merit function is minimized to force the iterates back to a neighborhood of the trajectory.

A similar approach is used to handle the inequality constraints in PDPB. In this case, the trajectory is followed towards a local solution of the mixed-constraint nonlinear program. This trajectory is parameterized by a set of Lagrange multiplier estimates and penalty and barrier parameters associated with the equality and inequality constraints. If an iterate moves away from the trajectory, a primal-dual shifted penalty-barrier merit function is minimized using a trust-region method. By introducing slack variables, global convergence can be achieved from any starting point without the need for an initial strictly feasible point. Furthermore, numerical experiments indicate that when minimizing the shifted barrier function, the trust-region method requires fewer matrix factorizations and iterations than a comparable line-search method.

The class of stabilized sequential quadratic programming (SQP) methods for nonlinearly constrained optimization solve a sequence of related quadratic programming (QP) subproblems formed from a two-norm penalized quadratic model of the Lagrangian function subject to shifted, linearized constraints. While these methods have been shown to exhibit superlinear local convergence even when the constraint Jacobian is rank deficient at the solution, they generally have no global convergence theory. To address this issue, primal-dual SQP methods (pdSQP) employ a certain primal-dual augmented Lagrangian merit function and solve a subproblem that involves the minimization of a quadratic model of the merit function subject to simple bound constraints. The model of the merit function is constructed so that the resulting primal-dual subproblem is equivalent to the stabilized SQP subproblem. When used in conjunction with a flexible line-search, the merit function guarantees convergence from any starting point, while the connection with the stabilized subproblem allows pdSQP to retain the superlinear local convergence that is characteristic of stabilized SQP methods.

A new dynamic convexification framework is developed that is applicable for nonconvex general standard form, stabilized, and primal-dual bound-constrained QP subproblems. Dynamic convexification involves three distinct stages: pre-convexification, concurrent convexification and post-convexification. New techniques are derived and analyzed for the implicit modification of symmetric indefinite factorizations and for the imposition of temporary artificial constraints, both of which are suitable for pre-convexification. Concurrent convexification works synchronously with the active-set method used to solve the subproblem, and computes minimal modifications needed to ensure that the QP iterates are uniformly bounded. Finally, post-convexification defines an implicit modification that ensures the solution of the subproblem yields a descent direction for the merit function.

A new exact second-derivative primal-dual SQP method (dcpdSQP) is formulated for large-scale nonconvex optimization. Convergence analysis is presented that demonstrates guaranteed global convergence. Extensive numerical testing indicates that the performance of the proposed method is comparable or better than conventional full convexification while significantly reducing the number of factorizations required.