Deep neural networks have demonstrated significant achievements across various fields, yet their memory and time complexities present obstacles for implementing them on resource-constrained devices. Compressing deep neural networks using tensor decomposition can decrease both memory usage and computational costs. The performance of a low-rank tensorized network depends on the choices of hyperparameters including the tensor rank and geometry. Previous studies have concentrated on identifying optimal tensor ranks. This thesis studies the effect of tensor geometry used for folding data for low-rank tensor compression. It is demonstrated that tensor geometry significantly affects compression efficiency of the tensorized data and model parameters. Consequently, a novel mathematical formulation is developed to optimize tensor geometry. The tensor geometry optimization model is adopted for efficient deployment of low-rank neural networks. The presented tensor geometry optimization model is combinatorial and thus challenging to solve. Therefore, surrogate and relaxed versions of the model are developed and various methods including integer linear programming, graph optimization, and random search algorithms are applied to solve the presented optimization model. The proposed tensor geometry optimization achieved a notable reduction in both the memory and time complexities of neural networks while maintaining accuracy. The developed methods can be applied for hardware-software co-design of artificial intelligence (AI) accelerators particularly on resource-constrained devices.