jojo1899/Phi-3.5-mini-instruct-onnx-int4

By jojo1899 · ↓ 0 · ♥ 0 · Updated 2024-11-07

What's in this repo

This page lists the files in jojo1899/Phi-3.5-mini-instruct-onnx-int4, including model.onnx.

1 ONNX models (.onnx files)
8 files, including 4 .json, 1 .md, 1 .onnx, 1 .data
Last updated 2024-11-07

ONNX files

File	Size	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

This is an INT4 quantized version of the Phi-3.5-mini-instruct model. The Python packages used in creating this model are as follows:

onnx==1.16.1
onnxruntime-directml==1.20.0
onnxruntime-genai-directml==0.4.0
torch==2.5.1
transformers==4.45.2

This quantized model is created using the following command:

python -m onnxruntime_genai.models.builder -m microsoft/Phi-3.5-mini-instruct -e dml -p int4 --extra_options {"int4_block_size"=128} -o ./Phi-3.5-mini-instruct_onnx_int4

onnxruntime_genai.models.builder quantizes the model using MatMul4BitsQuantizer from matmul_4bits_quantizer.py of onnxruntime/quantization/ with the "DEFAULT" method.

Model info

onnx license:mit region:us

↓ 0 ♥ 0

Created: 2024-11-07

Updated: 2024-11-07

View on Hugging Face

Files (8)

.gitattributes

README.md

genai_config.json

model.onnx ONNX

model.onnx.data

special_tokens_map.json

tokenizer.json

tokenizer_config.json

Related models

More models

No task type declared.