Skip to main content
eScholarship
Open Access Publications from the University of California

UC San Diego

UC San Diego Previously Published Works bannerUC San Diego

Is Memory Disaggregation Feasible? A Case Study with Spark SQL

  • Author(s): Rao, Pramod S
  • Porter, George
  • et al.
Abstract

This paper explores the feasibility of entirely disaggregated memory from compute and storage for a particular, widely deployed workload, Spark SQL analytics queries.  We measure the empirical rate at which records are processed and calculate the effective memory bandwidth utilized based on the sizes of the columns accessed in the query.  Our findings contradict conventional wisdom: not only is memory disaggregation possible under this workload, but achievable with already available, commercial network technology.  Beyond this finding, we also recommend changes that can be made to Spark SQL to improve its ability to support memory disaggregation.

Many UC-authored scholarly publications are freely available on this site because of the UC's open access policies. Let us know how this access is important for you.

Main Content
Current View