在 DEEPSPARSE 上运行的 DISTILBERT 嗡嗡作响。🚀🚀🚀

这个模型是 👇

    ███████╗ ██████╗   █████╗  ██████╗  ███████╗ ███████╗
    ██╔════╝ ██╔══██╗ ██╔══██╗ ██╔══██╗  ██╔════╝ ██╔════╝
    ███████╗ ██████╔╝ ███████║ ██████╔╝  ███████╗ █████╗  
    ╚════██║ ██╔═══╝  ██╔══██║ ██╔══██╗ ╚════██║█ █╔══╝  
    ███████║ ██║      ██║  ██║ ██║  ██ ║███████║ ███████╗
    ╚══════╝ ╚═╝      ╚═╝  ╚═╝ ╚═╝  ╚═ ╝╚══════╝ ╚══════╝

看起来像这样 👇

推理端点，除异常值外（4毫秒），在 2 个 vCPU 上的平均延迟：

访问推理端点的处理器

class EndpointHandler:

    def __init__(self, path=""):

        self.pipeline = Pipeline.create(task="text-classification", model_path=path)

    def __call__(self, data: Dict[str, Any]) -> Dict[str, str]:
        """
        Args:
            data (:obj:): prediction input text
        """
        inputs = data.pop("inputs", data)

        start = perf_counter()
        prediction = self.pipeline(inputs)
        end = perf_counter()
        latency = end - start

        return {
            "labels": prediction.labels, 
            "scores": prediction.scores,
            "latency (secs.)": latency
        }

̷͈̍ ̵̳͒R̶̙̓i̸̟͘c̴̻̆k̸̑͜ÿ̷̳́ ̸̪̚ ̷͖̀