xpuenabler/OpenVLA-OFT-LIBERO-130-QNN-CPU

OpenVLA-OFT LIBERO-10 — QNN FP32 CPU Inference

This repository contains QNN FP32 CPU models and inference scripts for running OpenVLA-OFT on the LIBERO-10 benchmark using Qualcomm AI Engine Direct (QNN SDK).

All inference runs on CPU using libQnnCpu.so.

Benchmark Results

Evaluated on LIBERO-10 tasks 0–9, 3 trials each, max 300 steps.

Repository Structure

.
├── README.md
├── inference/
│   ├── infer_libero_episode_qnn_fp32_cpu_v9.py  # QNN single-episode inference
│   ├── infer_libero_episode_onnx_cpu.py          # ONNX single-episode inference
│   └── eval_libero_batch.py                       # Batch evaluation (ONNX + QNN)
├── conversion/
│   ├── make_llm_v9.py        # Generate LLM backbone v9 ONNX (ScatterND+Trilu fix)
│   ├── convert_llm_v9.sh     # Convert LLM v9 ONNX → QNN
│   ├── convert_ve.sh         # Convert Vision Encoder ONNX → QNN
│   └── convert_ah.sh         # Convert Action Head ONNX → QNN
├── onnx/
│   ├── vision_encoder.onnx   # Vision Encoder ONNX model (894 KB)
│   └── action_head.onnx      # Action Head ONNX model (3.9 KB)
├── qnn_models/               # Compiled QNN shared libraries (LFS)
│   ├── libvision_encoder_sliced.so   # Vision Encoder QNN (2.9 GB)
│   ├── libaction_head.so             # Action Head QNN (502 MB)
│   └── libllm_backbone.so            # LLM Backbone QNN (25 GB)
└── eval_results/
    ├── onnx_results.json    # ONNX batch evaluation results
    └── qnn_v9_results.json  # QNN v9 batch evaluation results

Prerequisites

Software

• QNN SDK: Qualcomm AI Engine Direct 2.43.0 (qairt/2.43.0.260128)
• Python 3.10 (for QNN conversion): miniforge3/envs/qnn
• Python 3.11 (for inference): miniforge3/envs/openvla-oft
• LIBERO installed in the inference environment
• OpenVLA-OFT RLinf codebase at /home/work/nota-data/gmkim/RLinf

Base Model

Download the original OpenVLA-OFT LIBERO-130 checkpoint from HuggingFace:

huggingface-cli download xpuenabler/RLinf-OpenVLAOFT-LIBERO-130 \
  --local-dir /path/to/models/RLinf-OpenVLAOFT-LIBERO-130

Update MODEL_PATH in all scripts to point to this directory.

QNN Model Libraries

The compiled .so files are stored in qnn_models/ via Git LFS:

git lfs install
git lfs pull

Place them as follows:

qnn_workspace/
├── vision_encoder_sliced_fp32/model_lib_x86/x86_64-linux-clang/libvision_encoder_sliced.so
├── llm_backbone_fp32_v9/model_lib_x86_bfd/libs/x86_64-linux-clang/libllm_backbone.so
└── ah_transpose_fix/model_lib_x86/x86_64-linux-clang/libaction_head.so

Or update MODEL_LIBS paths in infer_libero_episode_qnn_fp32_cpu_v9.py to point directly to the downloaded .so files.

Reproducing a Success Episode

The following reproduces Task 1 (cream cheese + butter → basket), Trial 38 which was verified as SUCCESS.

Step 1: Set up environment variables

export TMPDIR=/path/to/tmp_py         # Required: /tmp may be full
export PYTHONNOUSERSITE=1             # Required: prevent sys-Python conflicts
export PYTHONPATH=/path/to/RLinf:/path/to/RLinf/venv-openvla-oft/lib/python3.11/site-packages:/path/to/RLinf/venv-openvla-oft/libero

Step 2: Run QNN inference

TMPDIR=/home/work/nota-data/gmkim/tmp_py \
PYTHONNOUSERSITE=1 \
PYTHONPATH=/home/work/nota-data/gmkim/RLinf:\
/home/work/nota-data/gmkim/RLinf/venv-openvla-oft/lib/python3.11/site-packages:\
/home/work/nota-data/gmkim/RLinf/venv-openvla-oft/libero \
  /home/work/nota-data/gmkim/miniforge3/envs/openvla-oft/bin/python3.11 \
  inference/infer_libero_episode_qnn_fp32_cpu_v9.py \
  --task-id 1 \
  --trial-id 38 \
  --max-steps 300

Expected output:

EPISODE SUMMARY (QNN FP32 CPU v9)
Task [1]: put both the cream cheese box and the butter in the basket
Result:  SUCCESS
Steps:   231 / 300

Running with ONNX (reference)

TMPDIR=/home/work/nota-data/gmkim/tmp_py \
PYTHONNOUSERSITE=1 \
PYTHONPATH=/home/work/nota-data/gmkim/RLinf:\
/home/work/nota-data/gmkim/RLinf/venv-openvla-oft/lib/python3.11/site-packages:\
/home/work/nota-data/gmkim/RLinf/venv-openvla-oft/libero \
  /home/work/nota-data/gmkim/miniforge3/envs/openvla-oft/bin/python3.11 \
  inference/infer_libero_episode_onnx_cpu.py \
  --task-id 1 \
  --trial-id 38 \
  --max-steps 300

Batch Evaluation

# Evaluate ONNX (30 episodes, tasks 0-9, trials 0-2)
python inference/eval_libero_batch.py --model onnx

# Evaluate QNN v9 (same episodes)
python inference/eval_libero_batch.py --model qnn

# Compare results
python inference/eval_libero_batch.py --compare

Checkpoint files are saved automatically to eval_results/ so interrupted runs can be resumed.

ONNX → QNN Conversion

If you want to re-convert the models from ONNX:

Vision Encoder

bash conversion/convert_ve.sh

Action Head

bash conversion/convert_ah.sh

LLM Backbone (v9)

The LLM backbone requires two steps:

Step 1: Generate v9 ONNX (replaces ScatterND+Trilu with QNN-compatible ops):

# make_llm_v9.py reads onnx/llm_backbone_v7.onnx (included in this repo, 785KB header)
# External weight data (~25 GB) must be present in the same directory,
# exported from the original OpenVLA-OFT checkpoint.
TMPDIR=/home/work/nota-data/gmkim/tmp_py \
PYTHONNOUSERSITE=1 \
  /home/work/nota-data/gmkim/miniforge3/envs/qnn/bin/python3.10 \
  conversion/make_llm_v9.py

Note on external data: llm_backbone_v7.onnx stores model weights as external data files (embed_tokens.weight, onnx__MatMul_*, etc., ~25 GB total). make_llm_v9.py uses load_external_data=False (modifies graph structure only), so the 785KB .onnx header in this repo is sufficient for graph transformation. However, qnn-onnx-converter (Step 2) requires the full weights — export them first from the original model checkpoint using conversion/make_llm_v9.py after a full ONNX export.

Step 2: Convert to QNN + compile .so:

bash conversion/convert_llm_v9.sh

Note: convert_llm_v9.sh handles the full pipeline:

1. qnn-onnx-converter → llm_backbone.cpp + llm_backbone.bin (~25 GB tar)
2. Build directory setup from QNN SDK templates
3. Patches Makefile: -fuse-ld=lld → -fuse-ld=bfd -nostartfiles (required for >2 GB data)
4. make → libllm_backbone.so (~25 GB, takes ~1 hour)

Technical Notes

v9 LLM Backbone

The original ONNX LLM backbone contains ScatterND and Trilu ops that QNN's CPU backend cannot execute. v9 replaces them with a static subgraph:

• ScatterND → Replaced with Cast + Concat + Sub + Mul + Reshape + Add using precomputed causal mask
• Trilu → Replaced with Identity on a precomputed upper-triangular constant

Cosine similarity between v9 QNN output and original ONNX: cos = 0.962

Tensor Layout Differences (ONNX vs QNN)

All transpositions are handled automatically in the inference scripts.

LLM Backbone Input Specification (v9)

Citation

@article{kim2024openvla,
  title={OpenVLA-OFT: Efficient Fine-Tuning of Open-Source Vision-Language-Action Models},
  author={Kim, Moojin and others},
  year={2024}
}