zephyr-7b-alpha DeepSparse 版本

使用方法

pip install deepsparse-nightly[llm]

from deepsparse import TextGeneration
model = TextGeneration(model="hf:mgoin/zephyr-7b-alpha-ds")
out = model("Once upon a time,", max_new_tokens=100)
print(out.generations[0].text)
### there was a young woman named Lily. She was a kind and gentle soul, with a heart full of love and compassion. Lily had always been fascinated by the natural world, and she spent most of her free time exploring the forests and fields around her home.\n\nOne day, as she was wandering through the woods, Lily stumbled upon a small clearing. In the center of the clearing, she saw a beautiful butterfly fluttering its wings. The butterfly was unlike any she had

如何从 zephyr-7b-alpha 导出

安装包含此 PR 的 SparseML

git clone https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha
sparseml.transformers.export_onnx --model_path ./zephyr-7b-alpha --task text-generation --sequence_length 512 --trust_remote_code
cp deployment/model.onnx deployment/model-orig.onnx
python ~/onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx