Apparate: Early-Exit Models for ML Latency and Throughput Optimization – Implementation

This content originally appeared on HackerNoon and was authored by Writings, Papers and Blogs on Text Models

:::info Authors:

(1) Yinwei Dai, Princeton University (Equal contributions);

(2) Rui Pan, Princeton University (Equal contributions);

(3) Anand Iyer, Georgia Institute of Technology;

(4) Ravi Netravali, Georgia Institute of Technology.

:::

Table of Links

Abstract and 1 Introduction

2 Background and Motivation and 2.1 Model Serving Platforms

2.2 Early-Exit Models

2.3 Challenges

3 Design

3.1 Preparing Models with Early Exits

3.2 Accuracy-Aware Threshold Tuning

3.3 Latency-Focused Ramp Adjustments

4 Implementation

5 Evaluation and 5.1 Methodology

5.2 Overall Results

5.3 Comparison with Existing EE Strategies

5.4 Microbenchmarks

6 Additional Related Work

7 Conclusion, References, Appendix

4 IMPLEMENTATION

Apparate is implemented as a layer atop TensorFlowServing [39] and Clockwork [22] (using PyTorch [7]) and includes the components described in 3 written as Python modules in ∼7500 lines of code. Although we chose these platforms for our current implementation, we note that Apparate is not limited to them and its techniques can be implemented in any inference platform. Importantly, Apparate entirely leverages the scheduling and queuing mechanisms of the underlying framework. Original models are ingested in the ONNX format [6] and compiled for performance. Ramp training (during bootstrapping) uses the first 10% of each dataset following a 1:9 split for training and validation; the remaining 90% of each dataset is used for evaluation.

\

:::info This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.

:::

\

This content originally appeared on HackerNoon and was authored by Writings, Papers and Blogs on Text Models

Print Share Comment Cite Upload Translate Updates

APA

Writings, Papers and Blogs on Text Models | Sciencx (2024-10-02T18:00:20+00:00) Apparate: Early-Exit Models for ML Latency and Throughput Optimization – Implementation. Retrieved from https://www.scien.cx/2024/10/02/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-implementation/

MLA

" » Apparate: Early-Exit Models for ML Latency and Throughput Optimization – Implementation." Writings, Papers and Blogs on Text Models | Sciencx - Wednesday October 2, 2024, https://www.scien.cx/2024/10/02/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-implementation/

HARVARD

Writings, Papers and Blogs on Text Models | Sciencx Wednesday October 2, 2024 » Apparate: Early-Exit Models for ML Latency and Throughput Optimization – Implementation., viewed ,<https://www.scien.cx/2024/10/02/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-implementation/>

VANCOUVER

Writings, Papers and Blogs on Text Models | Sciencx - » Apparate: Early-Exit Models for ML Latency and Throughput Optimization – Implementation. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2024/10/02/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-implementation/

CHICAGO

" » Apparate: Early-Exit Models for ML Latency and Throughput Optimization – Implementation." Writings, Papers and Blogs on Text Models | Sciencx - Accessed . https://www.scien.cx/2024/10/02/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-implementation/

IEEE

" » Apparate: Early-Exit Models for ML Latency and Throughput Optimization – Implementation." Writings, Papers and Blogs on Text Models | Sciencx [Online]. Available: https://www.scien.cx/2024/10/02/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-implementation/. [Accessed: ]

rf:citation

» Apparate: Early-Exit Models for ML Latency and Throughput Optimization – Implementation | Writings, Papers and Blogs on Text Models | Sciencx | https://www.scien.cx/2024/10/02/apparate-early-exit-models-for-ml-latency-and-throughput-optimization-implementation/ |

Please log in to upload a file.

There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.