TurboSparse-LLM Performance: Outperforming Mixtral and Gemma with Extreme Sparsity

Table of Links

Abstract and 1. Introduction

2. Related Work and Background

3. Analysis

   3.1 Limitations about Existing ReLUficatio

   3.2 dReLU

4. Are Neurons in Expert still Sparsely Activated?

5. dReLU Sparsification

6. Experiments Results

   6.1 Downstream Tasks Performance

   6.2 Sparsity of Sparsified Models

7. Practical Inference Speedup Evaluation

   7.1 Experiments Setting

   7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference

   7.4 Deploy LLMs on mobile phones

8. Conclusion and References

A. Appendix / supplemental material

B. Limitation

C. Broader Impact

6 Experiments Results

6.1 Downstream Tasks Performance

We measure our sparsified models’ performance on tasks included in OpenLLM Leaderboard which include 25-shot Arc-Challenge [13], 10-shot Hellaswag [65], 5-shot MMLU [22], 0-shot TruthfulQA [35], 5-shot Winogrande [51] and 8-shot GSM8K [14]. In addition, we also follow Llama 2’s evaluation task included commonsense reasoning tasks. We report the average of PIQA [8], SCIQ [26], ARC easy [13], OpenBookQA [41]. We compare our models to several external open-source LLMs, including Gemma-2B [58], Mistral-7B [24] and Mixtral-47B [25].

Table 6 shows the results from different models. TurboSparse-Mistral-7B outperforms Gemma-2B by far, while only activating 3B parameters. TurboSparse-Mixtral-47B outperforms the original Mixtral-47B with only 4.5B parameters activated. The results demonstrate that LLMs with ReLU based intrinsic activation sparsity can keep the same or better performance while hold the significant FLOPs reduction.

:::info
Authors:

(1) Yixin Song, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University;

(2) Haotong Xie, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University;

(3) Zhengyan Zhang, Department of Computer Science and Technology, Tsinghua University;

(4) Bo Wen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University;

(5) Li Ma, Shanghai Artificial Intelligence Laboratory;

(6) Zeyu Mi, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University Mi yzmizeyu@sjtu.edu.cn);

(7) Haibo Chen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University.

:::

:::info
This paper is available on arxiv under CC BY 4.0 license.

:::