UCLA

Deep Learning has transformed our interaction with the world significantly, with numerous breakthroughs in the fields of computer vision, natural language understanding, autonomous driving and more. We have witnessed great success of large models in capturing intrinsic patterns and representations present in the training data. However, despite prosperous development, we are faced with challenges such as computational constraints and the lack of high-quality annotated data when scaling up models. Consequently, it is necessary to design both algorithm and data efficient methods to address these issues. For algorithm efficiency, we explore techniques including meta-learning and dataset distillation to reduce training time. On the other hand, for data efficiency, we show performance improvement by pre-training on unannotated noisy datasets followed by finetuning. Specifically, we investigate representation learning and language modeling, two prevalent frameworks to enhance the utilization of pre-training data. In the meanwhile, we propose an automatic data annotation pipeline to further enable model and data codevelopment. With all the efforts in efficient deep learning, we make it feasible and practical to train a well-performed model efficiently.