vmanvs/halugate-sentinel-onnx

HaluGate Sentinel (Quantized ONNX Versions)

HaluGate Sentinel is a high-efficiency binary classifier designed as a "Stage 0" gatekeeper for LLM pipelines. It analyzes incoming user prompts and decides whether they require factual grounding (RAG/Tooling) or can be handled by a creative/reasoning engine directly.

This repository contains optimized ONNX versions of the model, specifically tuned for deployment in browser environments via Transformers.js and edge devices via ONNX Runtime.

🚀 Model Variants

Through rigorous testing, we have found that 4-bit and 1-bit/BNB4 quantization significantly outperforms standard 8-bit quantization for this architecture due to outlier handling.

Note on Quantization Performance: ModernBERT architectures often exhibit "outlier" activations. In our tests, 8-bit global quantization (INT8) caused significant confidence degradation. Using block-wise 4-bit (Q4) or BitsAndBytes (BNB4) isolates these outliers, resulting in performance that nearly matches the original FP32 precision.

📊 Classification Labels

The model outputs two classes based on the prompt's intent:

1. FACT_CHECK_NEEDED (Label 1): Information-seeking queries that rely on world knowledge (e.g., "What is the current price of Bitcoin?").
2. NO_FACT_CHECK_NEEDED (Label 0): Creative, coding, opinion, or pure reasoning tasks (e.g., "Write a poem about a cat" or "How do I sort a list in Python?").

✅ Tests

The user is requested to test the model for their use case, here are some light-weight tests done on the model for reference:

Query: Write a nursery rhyme about a cat.
Result: NO_FACT_CHECK_NEEDED (100.00%)

Query: How do I sort a list in Python?
Result: NO_FACT_CHECK_NEEDED (99.97%) | | fp16 | Query: What is the current price of Bitcoin?
Result: FACT_CHECK_NEEDED (88.68%)

Query: Write a nursery rhyme about a cat.
Result: NO_FACT_CHECK_NEEDED (100.00%)

Query: How do I sort a list in Python?
Result: NO_FACT_CHECK_NEEDED (99.97%) | | q4 | Query: What is the current price of Bitcoin?
Result: FACT_CHECK_NEEDED (99.13%)

Query: Write a nursery rhyme about a cat.
Result: NO_FACT_CHECK_NEEDED (100.00%)

Query: How do I sort a list in Python?
Result: NO_FACT_CHECK_NEEDED (99.99%) | | bnb4 | Query: What is the current price of Bitcoin?
Result: FACT_CHECK_NEEDED (99.49%)

Query: Write a nursery rhyme about a cat.
Result: NO_FACT_CHECK_NEEDED (100.00%)

Query: How do I sort a list in Python?
Result: NO_FACT_CHECK_NEEDED (99.91%) | | q4f16 | Query: What is the current price of Bitcoin?
Result: FACT_CHECK_NEEDED (99.14%)

Query: Write a nursery rhyme about a cat.
Result: NO_FACT_CHECK_NEEDED (100.00%)

Query: How do I sort a list in Python?
Result: NO_FACT_CHECK_NEEDED (99.99%) | | uint8 | Query: What is the current price of Bitcoin?
Result: FACT_CHECK_NEEDED (98.92%)

Query: Write a nursery rhyme about a cat.
Result: NO_FACT_CHECK_NEEDED (99.97%)

Query: How do I sort a list in Python?
Result: NO_FACT_CHECK_NEEDED (95.22%) | | q8, int8
(avoid) | Query: What is the current price of Bitcoin?
Result: NO_FACT_CHECK_NEEDED (55.46%)

Query: Write a nursery rhyme about a cat.
Result: NO_FACT_CHECK_NEEDED (100.00%)

Query: How do I sort a list in Python?
Result: NO_FACT_CHECK_NEEDED (78.03%) |

📈 Performance Notes

• Precision Stability: Unlike many models where 8-bit is the standard, HaluGate Sentinel shows improved confidence scores in q4f16 and bnb4 formats. This is likely due to the "Block-wise" quantization techniques preserving ModernBERT's internal outlier activations better than global 8-bit scaling.
• Context Window: Supports up to 8192 tokens (ModernBERT backbone).

⚠️ Limitations

• English Only: The model was primarily trained and validated on English datasets.
• Borderline Queries: Philosophical or hybrid prompts (e.g., "Is time travel possible?") may show lower confidence scores. We recommend implementing a "default-to-safe" (Fact Check Needed) policy for scores below 0.70.

🛠 Usage (Transformers.js / JavaScript)

import { pipeline } from '@huggingface/transformers';

// Load the 4-bit version for optimal performance
const classifier = await pipeline('text-classification', 'vmanvs/halugate-sentinel-onnx', {
	device: 'webgpu',
    dtype: 'q4f16', // or 'bnb4' for maximum compression
});

const result = await classifier("Who won the 2020 world series?");
console.log(result);
// Output: [{ label: 'FACT_CHECK_NEEDED', score: 0.991... }]

🐍 Usage (Python / ONNX Runtime)

Python

import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("vmanvs/halugate-sentinel-onnx")
session = ort.InferenceSession("model_q4f16.onnx")

inputs = tokenizer("How do you implement a binary tree?", return_tensors="np")
outputs = session.run(None, dict(inputs))
# Process logits with softmax...
def softmax(x): 
		"""Compute softmax values for each sets of scores in x.""" 
		e_x = np.exp(x - np.max(x, axis=-1, keepdims=True)) 
		return e_x / e_x.sum(axis=-1, keepdims=True)

📝 Citation

This model is a derivation of HaluGate Sentiel by LLM Semantic Router Team.

Plaintext

@misc{halugate2025,
  author = {LLM Semantic Router Team},
  title = {HaluGate Sentinel: A Frontline Switch for Hallucination Mitigation},
  year = {2025},
  publisher = {Hugging Face},
  journal = {Hugging Face Repository},
}