UC San Diego

The evolution of big data processing and analysis has led to data-parallel frameworks such as Hadoop, MapReduce, Spark, and Hive, which are capable of analyzing large streams of data such as server logs, web transactions, and user reviews. Videos are one of the biggest sources of data and dominate the Internet traffic. Video processing on a large scale is critical and challenging as videos possess spatial and temporal features, which are not taken into account by the existing data-parallel frameworks. There are a broad range of users who want to apply sophisticated video processing pipelines such as transcoding, feature extraction, classification, scene cut detection and digital compositing to video content

Parallel video processing poses several significant research challenges to the existing data processing frameworks. Current systems are capable of processing videos but with higher resource startup times, a small degree of parallelism, low average resource utilization, coarse-grained billing, and higher latency. This research proposes a low latency software run-time for processing a single video efficiently by orchestrating cloud-based microservices. The system leverages lightweight microservices provided by Amazon Web Services Lambda framework.