ENOT-AutoDL/gpt2-tensorrt
By ENOT-AutoDL · ↓ 0 · ♥ 4 · Updated 2023-06-08
What's in this repo
This page lists the files in ENOT-AutoDL/gpt2-tensorrt, including gpt2-xl-i8.onnx, gpt2-xl.onnx.
- 2 ONNX models (.onnx files)
- 6 files, including 2 .data, 2 .onnx, 1 .md
- Last updated 2023-06-08
ONNX files
How to use
These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.
README
GPT2
This repository contains GPT2 onnx models compatible with TensorRT:
- gpt2-xl.onnx - GPT2-XL onnx for fp32 or fp16 engines
- gpt2-xl-i8.onnx - GPT2-XL onnx for int8+fp32 engines
Quantization of models was performed by the ENOT-AutoDL framework. Code for building of TensorRT engines and examples published on github.
Metrics:
GPT2-XL
| TensorRT INT8+FP32 | torch FP16 | |
|---|---|---|
| Lambada Acc | 72.11% | 71.43% |
Test environment
- GPU RTX 4090
- CPU 11th Gen Intel(R) Core(TM) i7-11700K
- TensorRT 8.5.3.1
- pytorch 1.13.1+cu116
Latency:
GPT2-XL
| Input sequance length | Number of generated tokens | TensorRT INT8+FP32 ms | torch FP16 ms | Acceleration |
|---|---|---|---|---|
| 64 | 64 | 462 | 1190 | 2.58 |
| 64 | 128 | 920 | 2360 | 2.54 |
| 64 | 256 | 1890 | 4710 | 2.54 |
Test environment
- GPU RTX 4090
- CPU 11th Gen Intel(R) Core(TM) i7-11700K
- TensorRT 8.5.3.1
- pytorch 1.13.1+cu116
How to use
Example of inference and accuracy test published on github:
git clone https://github.com/ENOT-AutoDL/ENOT-transformers
Model info
Created: 2023-06-05
Updated: 2023-06-08
View on Hugging Face