UC San Diego

Deep neural networks (DNNs) have enabled the creation of high-quality synthetic media. The intent of generating synthetic videos can be harmless as they can be used for tasks such as advertisement campaigns, and face replacement in movies and animated films. However, DNNs can also be trained to synthesize facially manipulated videos called Deepfakes, which may be used maliciously to defame famous personalities, spread misinformation and influence elections based on false facts.

My research focuses on the responsible use of deep learning for media synthesis. On the synthesis side, I develop methods for natural and expressive speech synthesis methods for new speakers in data-limited settings. My work enables the creation of a digital voice clone of a person that can either be generated using text or a reference speech from a different speaker. Another area of my research focuses on the robust detection of Deepfake videos. We investigate the limitations of current DeepFake detection methods and demonstrate that they can be easily bypassed using adversarially crafted DeepFake videos. To address these limitations of Deepfake detectors, we propose FaceSigns, a proactive method for proving media authenticity using semi-fragile neural watermarks. FaceSigns can embed recoverable watermark data into real images and videos at the time of their capture, which can withstand a set of benign image and video transforms while being fragile to malicious tampering such as face swapping.