ENOT-AutoDL/gpt-j-6B-tensorrt-int8
By ENOT-AutoDL · ↓ 0 · ♥ 7 · Updated 2023-06-08
What's in this repo
This page lists the files in ENOT-AutoDL/gpt-j-6B-tensorrt-int8, including gptj-i8.onnx.
- 1 ONNX models (.onnx files)
- 7 files, including 3 .engine, 1 .md, 1 .data, 1 .onnx
- Last updated 2023-06-08
ONNX files
| File | Size | Download |
|---|---|---|
| gptj-i8.onnx | — | Download |
How to use
These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.
README
INT8 GPT-J 6B
GPT-J 6B is a transformer model trained using Ben Wang's Mesh Transformer JAX. "GPT-J" refers to the class of model, while "6B" represents the number of trainable parameters.
This repository contains GPT-J 6B onnx model suitable for building TensorRT int8+fp32 engines. Quantization of model was performed by the ENOT-AutoDL framework. Code for building of TensorRT engines and examples published on github.
Metrics:
| TensorRT INT8+FP32 | torch FP16 | torch FP32 | |
|---|---|---|---|
| Lambada Acc | 78.46% | 79.53% | - |
| Model size (GB) | 8.5 | 12.1 | 24.2 |
Test environment
- GPU RTX 4090
- CPU 11th Gen Intel(R) Core(TM) i7-11700K
- TensorRT 8.5.3.1
- pytorch 1.13.1+cu116
Latency:
| Input sequance length | Number of generated tokens | TensorRT INT8+FP32 ms | torch FP16 ms | Acceleration |
|---|---|---|---|---|
| 64 | 64 | 1040 | 1610 | 1.55 |
| 64 | 128 | 2089 | 3224 | 1.54 |
| 64 | 256 | 4236 | 6479 | 1.53 |
| 128 | 64 | 1060 | 1619 | 1.53 |
| 128 | 128 | 2120 | 3241 | 1.53 |
| 128 | 256 | 4296 | 6510 | 1.52 |
| 256 | 64 | 1109 | 1640 | 1.49 |
| 256 | 128 | 2204 | 3276 | 1.49 |
| 256 | 256 | 4443 | 6571 | 1.49 |
Test environment
- GPU RTX 4090
- CPU 11th Gen Intel(R) Core(TM) i7-11700K
- TensorRT 8.5.3.1
- pytorch 1.13.1+cu116
How to use
Example of inference and accuracy test published on github:
git clone https://github.com/ENOT-AutoDL/ENOT-transformers
Model info
Created: 2023-03-28
Updated: 2023-06-08
View on Hugging Face