ONNX INT8 量化版 Whisper V3 Turbo

本模型是 openai/whisper-large-v3-turbo 的 ONNX Runtime 优化版本， 采用动态量化技术量化为 INT8，以提升推理性能。

量化详情

• 框架：通过 Optimum 使用 ONNX Runtime
• 量化类型：动态 INT8 量化
• 配置：AVX512 优化，按张量量化

已量化的组件：

• 编码器模型
• 解码器模型
• 带过去状态的解码器模型（用于 KV 缓存）

性能优势

• 更小的模型体积：相比 FP32 减少约 4 倍（INT8 对比 32 位浮点数）
• 更快的推理速度：更低的内存带宽需求和优化的整数运算
• 更低的内存占用：推理过程中减少显存/内存使用
• 极小的精度损失：动态量化在获得性能提升的同时保持模型质量

为了使用该模型，我创建了以下代码片段：

class QuantizedONNXWhisperModel():
    \"\"\"
    Quantized Whisper v3 Turbo model using ONNX Runtime (via Optimum).
    \"\"\"

    def __init__(self):
        super().__init__()
        self.model = None
        self.processor = None
        self.pipe = None
        self.model_name = \"Vmpletsos/whisper-large-v3-turbo-onnx-int8\"
        self.device = \"cpu\"

        self._load_model()
        self._is_initialized = True
        logger.info(\"ONNX Whisper model ready\")

    def _load_model(self):
        \"\"\"Load the ONNX model and processor.\"\"\"
        logger.info(f\"Loading ONNX model: {self.model_name} on {self.device}...\")

        # Load the ONNX Model
        self.model = ORTModelForSpeechSeq2Seq.from_pretrained(
            self.model_name,
            provider=\"CPUExecutionProvider\",
            encoder_file_name=\"encoder_model_quantized.onnx\",
            decoder_file_name=\"decoder_model_quantized.onnx\",
            decoder_with_past_file_name=\"decoder_with_past_model_quantized.onnx\",
            use_io_binding=False,  # Disable IO binding which can cause issues
        )

        # Load the Processor from the actual base model
        self.processor = AutoProcessor.from_pretrained(\"openai/whisper-large-v3-turbo\")

        self.pipe = pipeline(
            \"automatic-speech-recognition\",
            model=self.model,
            tokenizer=self.processor.tokenizer,
            feature_extractor=self.processor.feature_extractor,
            chunk_length_s=30,
            batch_size=16,
            return_timestamps=True,
            device=-1,
        )

        logger.info(f\"Model loaded successfully.\")

    async def transcribe(self, filepath: Path, **kwargs) -> TranscriptionResult:
        \"\"\"
        Transcribe audio file using ONNX Runtime pipeline.
        \"\"\"
        # Run transcription
        result = self.pipe(
            str(filepath),
            generate_kwargs={\"task\": \"transcribe\"},
        )

        full_text = result[\"text\"]

        if \"chunks\" in result:
            for chunk in result[\"chunks\"]:
                start, end = chunk.get(\"timestamp\", (0.0, 0.0))
                text = chunk.get(\"text\", \"\")
                logger.debug(f\"[{start}s -> {end}s] {text}\")

        return full_text
        )