tarekziade/vit-base-patch16-224-distilgpt2

By tarekziade · ↓ 8 · ♥ 0 · Updated 2024-07-24

What's in this repo

This page lists the files in tarekziade/vit-base-patch16-224-distilgpt2, including decoder_model.onnx, decoder_model_merged.onnx, decoder_model_merged_quantized.onnx and more.

8 ONNX models (.onnx files)
22 files, including 8 .json, 8 .onnx, 2 .bin, 1 .md, 1 .txt, 1 .safetensors
Last updated 2024-07-24

ONNX files

File	Size	Download
onnx/decoder_model.onnx	—	Download
onnx/decoder_model_merged.onnx	—	Download
onnx/decoder_model_merged_quantized.onnx	—	Download
onnx/decoder_model_quantized.onnx	—	Download
onnx/decoder_with_past_model.onnx	—	Download
onnx/decoder_with_past_model_quantized.onnx	—	Download
onnx/encoder_model.onnx	—	Download
onnx/encoder_model_quantized.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

tags:

image-to-text
image-captioning license: apache-2.0 metrics:
rouge datasets:
Mozilla/flickr30k-transformed-captions widget:
src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/savanna.jpg example_title: Savanna
src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/football-match.jpg example_title: Football Match
src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/airport.jpg example_title: Airport base_model:
google/vit-base-patch16-224-in21k

model-index:

name: mozilla/distilvit results:
- task: type: image-to-text name: Image To Text dataset: name: Mozilla/flickr30k-transformed-captions type: Mozilla/flickr30k-transformed-captions metrics:
  - name: ROUGE-1 type: rouge value: 43.006 verified: true
  - name: ROUGE-2 type: rouge value: 16.9939 verified: true
  - name: ROUGE-L type: rouge value: 38.8923 verified: true
  - name: ROUGE-LSUM type: rouge value: 38.8877 verified: true
  - name: loss type: loss value: 0.19939416646957397
  - name: gen_len type: gen_len value: 11.327256736227712 verified: true

distilvit

This model is a work in progress. Fine-tuned version of those base models:

a VIT model for the image encoder: https://huggingface.co/google/vit-base-patch16-224-in21k
a Distilled GPT-2 model for the text decoder: https://huggingface.co/distilbert/distilgpt2

This model was trained on:

Flickr30k : https://huggingface.co/datasets/nlphuji/flickr30k
COCO 2017: https://cocodataset.org

You can get that checkpoint using the 3083a3cef6e3c8dd90df3f088074bbe836b0f403 commit.

It was then further fine-tuned on :

For the latter, the dataset was annotated by our team to correct the alt text generated by the model, using the checkvite tool.

You can find the code used to create the model here: https://github.com/mozilla/distilvit

Model info

image-to-text transformers

transformers pytorch onnx safetensors vision-encoder-decoder image-text-to-text image-to-text image-captioning dataset:Mozilla/flickr30k-transformed-captions base_model:google/vit-base-patch16-224-in21k base_model:quantized:google/vit-base-patch16-224-in21k license:apache-2.0

↓ 8 ♥ 0

Created: 2024-06-19

Updated: 2024-07-24

View on Hugging Face

Files (22)

.gitattributes

README.md

config.json

generation_config.json

merges.txt

model.safetensors

onnx/decoder_model.onnx ONNX

onnx/decoder_model_merged.onnx ONNX

onnx/decoder_model_merged_quantized.onnx ONNX

onnx/decoder_model_quantized.onnx ONNX

onnx/decoder_with_past_model.onnx ONNX

onnx/decoder_with_past_model_quantized.onnx ONNX

onnx/encoder_model.onnx ONNX

onnx/encoder_model_quantized.onnx ONNX

preprocessor_config.json

pytorch_model.bin

quantize_config.json

special_tokens_map.json

tokenizer.json

tokenizer_config.json

training_args.bin

vocab.json

Related models

More image-to-text models

Browse all image-to-text models