madhavgohel/bert-token-onnx

🧠 IntextIQ: Intext Intelligence – Smart Way to Find What Matters

📖 Overview

IntextIQ is an intelligent token extraction and memory-augmented reasoning framework designed to enhance Large Language Models (LLMs) in long-context dialogue scenarios.
The system identifies and retains semantically important tokens from user inputs, stores them efficiently, and retrieves them through a Retrieval-Augmented Generation (RAG) pipeline for improved contextual understanding and factual consistency.

This approach enables LLMs to “remember what matters” — maintaining coherent, factual, and contextually aware conversations even across extended interactions.

🚀 Key Features

• Token-Level Semantic Extraction
  Extracts meaningful tokens using a fine-tuned DistilBERT model trained on conversational data generated via Mistral-22B and GPT-OSS-20B.

• Memory-Augmented RAG Integration
  Seamlessly integrates token retrieval with the RAG framework to provide explicit contextual memory for LLMs.

• Groq LLM Integration
  Combines Groq’s high-speed inference with IntextIQ’s semantic memory to generate more factual and contextually relevant responses.

• BIO Tagging for Token Labeling
  Implements BIO (Begin–Inside–Outside) tagging for accurate semantic segmentation of text sequences.

• Interactive Gradio Interface
  User-friendly interface for inputting text, viewing extracted tokens, and visualizing RAG-based response generation.

🧩 System Architecture

The overall architecture consists of six major components:

1. Input Message – Raw user input.
2. IntextIQ Model (Token Extractor) – Extracts key tokens using fine-tuned DistilBERT.
3. Embedding Model – Converts tokens into high-dimensional semantic vectors.
4. RAG System – Retrieves relevant context from memory or external data sources.
5. LLM – Generates enhanced output using the augmented context.
6. Output Layer – Returns factually grounded and contextually coherent responses.

⚙️ Tech Stack

📊 Dataset Information

Two datasets were generated using synthetic conversational data representing average human dialogue:

The second dataset was used for fine-tuning due to improved data quality and contextual diversity.

🧠 Model Workflow

1. Preprocessing
   Text cleaning, normalization, and BIO tagging.
2. Token Extraction
   Fine-tuned DistilBERT identifies important tokens.
3. Embedding Generation
   Each token converted into a Safetensor embedding vector.
4. RAG Integration
   Retrieved token embeddings are supplied to the Groq LLM during response generation.
5. Response Generation
   LLM generates enhanced, contextually grounded answers.
6. Visualization
   Gradio interface displays extracted tokens and RAG outputs.

🧪 Evaluation Metrics

Key Results:
IntextIQ achieves substantial improvement in factual grounding and context continuity, confirming the efficacy of token-level retrieval in RAG-based dialogue systems.

🔍 Performance Summary

• 26.4% improvement in Context Relevance
• 21.6% increase in Factual Consistency
• 13% boost in Semantic Coherence (ROUGE-L)
• 66% reduction in Hallucination Rate

These metrics validate that the semantic compression and retrieval approach improves LLM accuracy, coherence, and contextual reasoning in extended dialogues.

📈 Future Extensions

• Incorporate RLHF (Reinforcement Learning with Human Feedback) for adaptive token weighting.
• Extend memory to persistent multi-session knowledge graphs.
• Develop domain-specific variants (e.g., legal, educational, healthcare).
• Explore multimodal token extraction for images and voice data.
• Deploy optimized versions using quantization or distillation for edge inference.