说明文档

CLIP 变体

CLIP 模型由 OpenAI 的研究人员开发，旨在了解哪些因素有助于计算机视觉任务的鲁棒性。该模型的开发也是为了测试模型以零样本方式泛化到任意图像分类任务的能力。它并非为通用模型部署而开发——要部署像 CLIP 这样的模型，研究人员首先需要仔细研究其在特定部署环境中的能力。

有关限制和偏见的更多详细信息，请参阅原始 CLIP 模型卡片。

本仓库包含转换为多种其他变体的 OpenAI CLIP 模型，详情见下文。

免责声明与许可

我对这些转换没有进行太多测试。我简要尝试了 float16 版本，它们似乎与原始 float32 非常相似，但正如预期的那样，qint8/quint8 版本的相似度下降更多。我无法尝试 qint8，因为它似乎不支持某些操作，但为了完整性我还是将其包含在内。从简要测试来看，quint8 版本似乎运行正常。

转换代码的许可证是 MIT，模型的许可证与 OpenAI 模型的原始许可证相同（🤷‍♂️）。我与 OpenAI 没有任何关联。

致谢

OpenAI CLIP
OpenAI CLIP JavaScript by josephrocca
CLIP-ONNX by Lednik7
将模型从 PyTorch 导出到 ONNX 并使用 ONNX Runtime 运行
imgbeddings by minimaxir
... 可能还有更多

示例

参见 example.py

❯ python .\example.py
Loading visual model: models/clip-vit-base-patch32-visual-float16.onnx
Visual inference ready, input size 224, type tensor(float16)
Images shape: (2, 3, 224, 224)
Embeddings shape: (2, 512)

Loading textual model: models/clip-vit-base-patch32-textual-float16.onnx
Textual inference ready, input size 77, type tensor(int32)
Texts shape: (14, 77)
Embeddings shape: (14, 512)

flowers.jpg
  similarity  bar chart    text
------------  -----------  ---------------------------------------------------------------
    0.294922  >>>>>>>>     a close up photo of a cherry blossom
    0.267578  >>>>>>>>     cherry blossom
    0.249878  >>>>>>>      flowers
    0.242554  >>>>>>>      a photo taken on a bright and sunny day
    0.228882  >>>>>>       bees
    0.222778  >>>>>>       plant
    0.216187  >>>>>>       a photo taken on a dark and cloudy day
    0.201538  >>>>>>       ruhrgebiet
    0.196655  >>>>>        processing plant
    0.192139  >>>>>        a photo taken at midnight
    0.18689   >>>>>        industry
    0.177856  >>>>>        cars
    0.176636  >>>>>        dogs and cats
    0.111267  >>>          a large industrial plant with many pipes, walkways and railings

heavy-industry.jpg
  similarity  bar chart    text
------------  -----------  ---------------------------------------------------------------
    0.336182  >>>>>>>>>>   a large industrial plant with many pipes, walkways and railings
    0.316895  >>>>>>>>>    processing plant
    0.302002  >>>>>>>>>    industry
    0.27417   >>>>>>>>     ruhrgebiet
    0.254883  >>>>>>>      plant
    0.22876   >>>>>>       a photo taken on a dark and cloudy day
    0.219482  >>>>>>       a photo taken on a bright and sunny day
    0.211304  >>>>>>       a photo taken at midnight
    0.198608  >>>>>        cars
    0.190552  >>>>>        flowers
    0.181885  >>>>>        bees
    0.180542  >>>>>        cherry blossom
    0.174438  >>>>>        dogs and cats
    0.14917   >>>>         a close up photo of a cherry blossom

参数

目前唯一支持的格式是开放神经网络交换 (ONNX)。

所有目前可用的 OpenAI 模型都已完成转换。部分 ID 取自 Hugging Face 上的 OpenAI 模型，其他则按照相同格式自行命名。

模型名称	模型 ID
RN50	resnet-50
RN101	resnet-101
RN50x4	resnet-50x4
RN50x16	resnet-50x16
RN50x64	resnet-50x64
RN50	resnet-50
RN50	resnet-50
RN50	resnet-50
ViT-B/16	vit-base-patch16
ViT-B/32	vit-base-patch32
ViT-L/14	vit-large-patch14
ViT-L/14@336px	vit-large-patch14-336

由于 CLIP 是一个多模态模型，原始模型被拆分为两个独立的"模式"，一个用于处理图像，另一个用于处理文本。

模式
visual（视觉）
textual（文本）

模型还被转换为多种数据类型。

数据类型
float16
qint8
quint8

变体

路径	模型 ID	模式	数据类型	可用	大小 (MB)
models/clip-resnet-50-visual-float32.onnx	resnet-50	visual	float32	✅	153
models/clip-resnet-50-visual-float16.onnx	resnet-50	visual	float16	✅	77
models/clip-resnet-50-visual-qint8.onnx	resnet-50	visual	qint8	✅	39
models/clip-resnet-50-visual-quint8.onnx	resnet-50	visual	quint8	✅	39
models/clip-resnet-50-textual-float32.onnx	resnet-50	textual	float32	✅	255
models/clip-resnet-50-textual-float16.onnx	resnet-50	textual	float16	✅	128
models/clip-resnet-50-textual-qint8.onnx	resnet-50	textual	qint8	✅	64
models/clip-resnet-50-textual-quint8.onnx	resnet-50	textual	quint8	✅	64
models/clip-resnet-101-visual-float32.onnx	resnet-101	visual	float32	✅	225
models/clip-resnet-101-visual-float16.onnx	resnet-101	visual	float16	✅	112
models/clip-resnet-101-visual-qint8.onnx	resnet-101	visual	qint8	✅	57
models/clip-resnet-101-visual-quint8.onnx	resnet-101	visual	quint8	✅	57
models/clip-resnet-101-textual-float32.onnx	resnet-101	textual	float32	✅	254
models/clip-resnet-101-textual-float16.onnx	resnet-101	textual	float16	✅	127
models/clip-resnet-101-textual-qint8.onnx	resnet-101	textual	qint8	✅	64
models/clip-resnet-101-textual-quint8.onnx	resnet-101	textual	quint8	✅	64
models/clip-resnet-50x4-visual-float32.onnx	resnet-50x4	visual	float32	✅	348
models/clip-resnet-50x4-visual-float16.onnx	resnet-50x4	visual	float16	✅	174
models/clip-resnet-50x4-visual-qint8.onnx	resnet-50x4	visual	qint8	✅	88
models/clip-resnet-50x4-visual-quint8.onnx	resnet-50x4	visual	quint8	✅	88
models/clip-resnet-50x4-textual-float32.onnx	resnet-50x4	textual	float32	✅	365
models/clip-resnet-50x4-textual-float16.onnx	resnet-50x4	textual	float16	✅	183
models/clip-resnet-50x4-textual-qint8.onnx	resnet-50x4	textual	qint8	✅	92
models/clip-resnet-50x4-textual-quint8.onnx	resnet-50x4	textual	quint8	✅	92
models/clip-resnet-50x16-visual-float32.onnx	resnet-50x16	visual	float32	✅	669
models/clip-resnet-50x16-visual-float16.onnx	resnet-50x16	visual	float16	✅	335
models/clip-resnet-50x16-visual-qint8.onnx	resnet-50x16	visual	qint8	✅	169
models/clip-resnet-50x16-visual-quint8.onnx	resnet-50x16	visual	quint8	✅	169
models/clip-resnet-50x16-textual-float32.onnx	resnet-50x16	textual	float32	✅	495
models/clip-resnet-50x16-textual-float16.onnx	resnet-50x16	textual	float16	✅	248
models/clip-resnet-50x16-textual-qint8.onnx	resnet-50x16	textual	qint8	✅	124
models/clip-resnet-50x16-textual-quint8.onnx	resnet-50x16	textual	quint8	✅	124
models/clip-resnet-50x64-visual-float32.onnx	resnet-50x64	visual	float32	✅	1681
models/clip-resnet-50x64-visual-float16.onnx	resnet-50x64	visual	float16	✅	840
models/clip-resnet-50x64-visual-qint8.onnx	resnet-50x64	visual	qint8	✅	424
models/clip-resnet-50x64-visual-quint8.onnx	resnet-50x64	visual	quint8	✅	424
models/clip-resnet-50x64-textual-float32.onnx	resnet-50x64	textual	float32	✅	812
models/clip-resnet-50x64-textual-float16.onnx	resnet-50x64	textual	float16	✅	406
models/clip-resnet-50x64-textual-qint8.onnx	resnet-50x64	textual	qint8	✅	204
models/clip-resnet-50x64-textual-quint8.onnx	resnet-50x64	textual	quint8	✅	204
models/clip-resnet-50-visual-float32.onnx	resnet-50	visual	float32	✅	153
models/clip-resnet-50-visual-float16.onnx	resnet-50	visual	float16	✅	77
models/clip-resnet-50-visual-qint8.onnx	resnet-50	visual	qint8	✅	39
models/clip-resnet-50-visual-quint8.onnx	resnet-50	visual	quint8	✅	39
models/clip-resnet-50-textual-float32.onnx	resnet-50	textual	float32	✅	255
models/clip-resnet-50-textual-float16.onnx	resnet-50	textual	float16	✅	128
models/clip-resnet-50-textual-qint8.onnx	resnet-50	textual	qint8	✅	64
models/clip-resnet-50-textual-quint8.onnx	resnet-50	textual	quint8	✅	64
models/clip-resnet-50-visual-float32.onnx	resnet-50	visual	float32	✅	153
models/clip-resnet-50-visual-float16.onnx	resnet-50	visual	float16	✅	77
models/clip-resnet-50-visual-qint8.onnx	resnet-50	visual	qint8	✅	39
models/clip-resnet-50-visual-quint8.onnx	resnet-50	visual	quint8	✅	39
models/clip-resnet-50-textual-float32.onnx	resnet-50	textual	float32	✅	255
models/clip-resnet-50-textual-float16.onnx	resnet-50	textual	float16	✅	128
models/clip-resnet-50-textual-qint8.onnx	resnet-50	textual	qint8	✅	64
models/clip-resnet-50-textual-quint8.onnx	resnet-50	textual	quint8	✅	64
models/clip-resnet-50-visual-float32.onnx	resnet-50	visual	float32	✅	153
models/clip-resnet-50-visual-float16.onnx	resnet-50	visual	float16	✅	77
models/clip-resnet-50-visual-qint8.onnx	resnet-50	visual	qint8	✅	39
models/clip-resnet-50-visual-quint8.onnx	resnet-50	visual	quint8	✅	39
models/clip-resnet-50-textual-float32.onnx	resnet-50	textual	float32	✅	255
models/clip-resnet-50-textual-float16.onnx	resnet-50	textual	float16	✅	128
models/clip-resnet-50-textual-qint8.onnx	resnet-50	textual	qint8	✅	64
models/clip-resnet-50-textual-quint8.onnx	resnet-50	textual	quint8	✅	64
models/clip-vit-base-patch16-visual-float32.onnx	vit-base-patch16	visual	float32	✅	345
models/clip-vit-base-patch16-visual-float16.onnx	vit-base-patch16	visual	float16	✅	173
models/clip-vit-base-patch16-visual-qint8.onnx	vit-base-patch16	visual	qint8	✅	87
models/clip-vit-base-patch16-visual-quint8.onnx	vit-base-patch16	visual	quint8	✅	87
models/clip-vit-base-patch16-textual-float32.onnx	vit-base-patch16	textual	float32	✅	254
models/clip-vit-base-patch16-textual-float16.onnx	vit-base-patch16	textual	float16	✅	127
models/clip-vit-base-patch16-textual-qint8.onnx	vit-base-patch16	textual	qint8	✅	64
models/clip-vit-base-patch16-textual-quint8.onnx	vit-base-patch16	textual	quint8	✅	64
models/clip-vit-base-patch32-visual-float32.onnx	vit-base-patch32	visual	float32	✅	352
models/clip-vit-base-patch32-visual-float16.onnx	vit-base-patch32	visual	float16	✅	176
models/clip-vit-base-patch32-visual-qint8.onnx	vit-base-patch32	visual	qint8	✅	89
models/clip-vit-base-patch32-visual-quint8.onnx	vit-base-patch32	visual	quint8	✅	89
models/clip-vit-base-patch32-textual-float32.onnx	vit-base-patch32	textual	float32	✅	254
models/clip-vit-base-patch32-textual-float16.onnx	vit-base-patch32	textual	float16	✅	127
models/clip-vit-base-patch32-textual-qint8.onnx	vit-base-patch32	textual	qint8	✅	64
models/clip-vit-base-patch32-textual-quint8.onnx	vit-base-patch32	textual	quint8	✅	64
models/clip-vit-large-patch14-visual-float32.onnx	vit-large-patch14	visual	float32	✅	1216
models/clip-vit-large-patch14-visual-float16.onnx	vit-large-patch14	visual	float16	✅	608
models/clip-vit-large-patch14-visual-qint8.onnx	vit-large-patch14	visual	qint8	✅	306
models/clip-vit-large-patch14-visual-quint8.onnx	vit-large-patch14	visual	quint8	✅	306
models/clip-vit-large-patch14-textual-float32.onnx	vit-large-patch14	textual	float32	✅	495
models/clip-vit-large-patch14-textual-float16.onnx	vit-large-patch14	textual	float16	✅	248
models/clip-vit-large-patch14-textual-qint8.onnx	vit-large-patch14	textual	qint8	✅	124
models/clip-vit-large-patch14-textual-quint8.onnx	vit-large-patch14	textual	quint8	✅	124
models/clip-vit-large-patch14-336-visual-float32.onnx	vit-large-patch14-336	visual	float32	✅	1217
models/clip-vit-large-patch14-336-visual-float16.onnx	vit-large-patch14-336	visual	float16	✅	609
models/clip-vit-large-patch14-336-visual-qint8.onnx	vit-large-patch14-336	visual	qint8	✅	307
models/clip-vit-large-patch14-336-visual-quint8.onnx	vit-large-patch14-336	visual	quint8	✅	307
models/clip-vit-large-patch14-336-textual-float32.onnx	vit-large-patch14-336	textual	float32	✅	495
models/clip-vit-large-patch14-336-textual-float16.onnx	vit-large-patch14-336	textual	float16	✅	248
models/clip-vit-large-patch14-336-textual-qint8.onnx	vit-large-patch14-336	textual	qint8	✅	124
models/clip-vit-large-patch14-336-textual-quint8.onnx	vit-large-patch14-336	textual	quint8	✅	124

mlunar/clip-variants

作者 mlunar

↓ 0 ♥ 14

创建时间: 2022-10-03 09:53:35+00:00

更新时间: 2022-10-04 19:50:07+00:00

在 Hugging Face 上查看

文件 (88)

.gitattributes

.gitignore

LICENSE

README.md

cliponnx/__init__.py

cliponnx/bpe_simple_vocab_16e6.txt.gz

cliponnx/models.py

cliponnx/simple_tokenizer.py

convert.py

example.py

flowers.jpg

heavy-industry.jpg

models/clip-resnet-101-textual-float16.onnx ONNX

models/clip-resnet-101-textual-float32.onnx ONNX

models/clip-resnet-101-textual-qint8.onnx ONNX

models/clip-resnet-101-textual-quint8.onnx ONNX

models/clip-resnet-101-visual-float16.onnx ONNX

models/clip-resnet-101-visual-float32.onnx ONNX

models/clip-resnet-101-visual-qint8.onnx ONNX

models/clip-resnet-101-visual-quint8.onnx ONNX

models/clip-resnet-50-textual-float16.onnx ONNX

models/clip-resnet-50-textual-float32.onnx ONNX

models/clip-resnet-50-textual-qint8.onnx ONNX

models/clip-resnet-50-textual-quint8.onnx ONNX

models/clip-resnet-50-visual-float16.onnx ONNX

models/clip-resnet-50-visual-float32.onnx ONNX

models/clip-resnet-50-visual-qint8.onnx ONNX

models/clip-resnet-50-visual-quint8.onnx ONNX

models/clip-resnet-50x16-textual-float16.onnx ONNX

models/clip-resnet-50x16-textual-float32.onnx ONNX

models/clip-resnet-50x16-textual-qint8.onnx ONNX

models/clip-resnet-50x16-textual-quint8.onnx ONNX

models/clip-resnet-50x16-visual-float16.onnx ONNX

models/clip-resnet-50x16-visual-float32.onnx ONNX

models/clip-resnet-50x16-visual-qint8.onnx ONNX

models/clip-resnet-50x16-visual-quint8.onnx ONNX

models/clip-resnet-50x4-textual-float16.onnx ONNX

models/clip-resnet-50x4-textual-float32.onnx ONNX

models/clip-resnet-50x4-textual-qint8.onnx ONNX

models/clip-resnet-50x4-textual-quint8.onnx ONNX

models/clip-resnet-50x4-visual-float16.onnx ONNX

models/clip-resnet-50x4-visual-float32.onnx ONNX

models/clip-resnet-50x4-visual-qint8.onnx ONNX

models/clip-resnet-50x4-visual-quint8.onnx ONNX

models/clip-resnet-50x64-textual-float16.onnx ONNX

models/clip-resnet-50x64-textual-float32.onnx ONNX

models/clip-resnet-50x64-textual-qint8.onnx ONNX

models/clip-resnet-50x64-textual-quint8.onnx ONNX

models/clip-resnet-50x64-visual-float16.onnx ONNX

models/clip-resnet-50x64-visual-float32.onnx ONNX

models/clip-resnet-50x64-visual-qint8.onnx ONNX

models/clip-resnet-50x64-visual-quint8.onnx ONNX

models/clip-vit-base-patch16-textual-float16.onnx ONNX

models/clip-vit-base-patch16-textual-float32.onnx ONNX

models/clip-vit-base-patch16-textual-qint8.onnx ONNX

models/clip-vit-base-patch16-textual-quint8.onnx ONNX

models/clip-vit-base-patch16-visual-float16.onnx ONNX

models/clip-vit-base-patch16-visual-float32.onnx ONNX

models/clip-vit-base-patch16-visual-qint8.onnx ONNX

models/clip-vit-base-patch16-visual-quint8.onnx ONNX

models/clip-vit-base-patch32-textual-float16.onnx ONNX

models/clip-vit-base-patch32-textual-float32.onnx ONNX

models/clip-vit-base-patch32-textual-qint8.onnx ONNX

models/clip-vit-base-patch32-textual-quint8.onnx ONNX

models/clip-vit-base-patch32-visual-float16.onnx ONNX

models/clip-vit-base-patch32-visual-float32.onnx ONNX

models/clip-vit-base-patch32-visual-qint8.onnx ONNX

models/clip-vit-base-patch32-visual-quint8.onnx ONNX

models/clip-vit-large-patch14-336-textual-float16.onnx ONNX

models/clip-vit-large-patch14-336-textual-float32.onnx ONNX

models/clip-vit-large-patch14-336-textual-qint8.onnx ONNX

models/clip-vit-large-patch14-336-textual-quint8.onnx ONNX

models/clip-vit-large-patch14-336-visual-float16.onnx ONNX

models/clip-vit-large-patch14-336-visual-float32.onnx ONNX

models/clip-vit-large-patch14-336-visual-qint8.onnx ONNX

models/clip-vit-large-patch14-336-visual-quint8.onnx ONNX

models/clip-vit-large-patch14-textual-float16.onnx ONNX

models/clip-vit-large-patch14-textual-float32.onnx ONNX

models/clip-vit-large-patch14-textual-qint8.onnx ONNX

models/clip-vit-large-patch14-textual-quint8.onnx ONNX

models/clip-vit-large-patch14-visual-float16.onnx ONNX

models/clip-vit-large-patch14-visual-float32.onnx ONNX

models/clip-vit-large-patch14-visual-qint8.onnx ONNX

models/clip-vit-large-patch14-visual-quint8.onnx ONNX

poetry.lock

poetry.toml

pyproject.toml

variants.py