xs_blenderbot_onnx (仅 168 mb)

facebook/blenderbot_small-90M 模型 (350 mb) 的 onnx 量化版本

更快的 CPU 推理

简介

使用前：

• 从此仓库的文件中下载 blender_model.py 脚本

• pip install onnxruntime

您可以使用 HuggingFace 的 generate 函数及其所有参数来使用该模型

用法

使用文本生成管道

>>>from blender_model import TextGenerationPipeline

>>>max_answer_length = 100
>>>response_generator_pipe = TextGenerationPipeline(max_length=max_answer_length)
>>>utterance = "Hello, how are you?"
>>>response_generator_pipe(utterance)
i am well. how are you? what do you like to do in your free time?

或者您可以直接调用模型

>>>from blender_model import OnnxBlender
>>>from transformers import BlenderbotSmallTokenizer
>>>original_repo_id = "facebook/blenderbot_small-90M"
>>>repo_id = "remzicam/xs_blenderbot_onnx"
>>>model_file_names = [
    "blenderbot_small-90M-encoder-quantized.onnx",
    "blenderbot_small-90M-decoder-quantized.onnx",
    "blenderbot_small-90M-init-decoder-quantized.onnx",
]
>>>model=OnnxBlender(original_repo_id, repo_id, model_file_names)
>>>utterance = "Hello, how are you?"
>>>inputs = tokenizer(utterance,
                    return_tensors="pt")
>>>outputs= model.generate(**inputs,
                        max_length=max_answer_length)
>>>response = tokenizer.decode(outputs[0],
                        skip_special_tokens = True)
>>>print(response)
i am well. how are you? what do you like to do in your free time?

致谢

为了创建这个模型，我采用了 https://github.com/siddharth-sharma7/fast-Bart 仓库中的代码。