UC Irvine

Modern Large Language Models (LLMs) offer exceptional inference accuracy but remain severely restricted by high resource and power requirements. Post-Training Quantization (PTQ) mitigates these memory bottlenecks by compressing weights to sub-4-bit regimes; however, existing Hadamard rotation-based methods that enable nearly lossless 4-bit weight and activation quantization (W4A4) does not push weight quantization below INT4, thereby limiting the memory size reduction potential of Hadamard transform. Furthermore, many GPU-based mixed-precision approaches lack native hardware support for integer to floating point multiplication and custom low-bit multiplication kernels. This requires runtime upcasting of integer weights and fails to leverage the resource saving potentials of low-bit representations during computation, leaving custom FPGA accelerators as the most efficient platform for low-bit mixed precision execution. To address these problems, this paper presents a projection-wise quantization method and a custom FPGA accelerator design that supports this quantization method. The proposed quantization scheme for Llama 2-7B compresses weights to 3.56 bits and achieves 10.87% reduction in weight storage relative to W4A4 QuaRot with a minimal perplexity increase of 2% and obtains a better Compression-to-Degradation Ratio than the current state-of-the-art SliM-LLM. In addition, the mixed precision accelerator design with custom Table Lookup-based matrix multiplication kernels for 3-bit weight operations achieves nearly 50% reduction in LUT usage compared to standard Multiply-and-Accumulate (MAC) units. Finally, to mitigate the additional latency overhead of online Hadamard operation, the proposed accelerator design incorporates a 2-stage pipelined online Hadamard transform unit that reduces the latency of attention projection Hadamard transformations by 13.7%.