KairongLiu/CLTMPSE

By KairongLiu · ↓ 0 · ♥ 1 · Updated 2024-11-29

What's in this repo

This page lists the files in KairongLiu/CLTMPSE, including model.onnx.

1 ONNX models (.onnx files)
141 files, including 29 .json, 18 .bin, 15 .tgt, 13 .pkl, 11 .src, 11 .txt
Last updated 2024-11-29

ONNX files

File	Size	Download
foundation/E5/onnx/model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Chinese-Lao-Thai-Multilingual-Parallel-Sentence-Pair-Extraction(CLTMPSE)

#step1. preprocess 构造音标词典和词替换规则，构造训练数据：

open utils/IPA_sim_statistic_analysis.py
set statistic_conclusion_exist=False (row 96)
run , then generate IPA_lo_dict and IPA_th_dict
set statistic_conclusion_exist=True
run, then generate same_list

#step2. train 开始训练：

set params
run main.py

--do_train --evaluate_during_training --train_data_file train.json --eval_data_file valid.json --test_data_file test_lo.json --data_dir data --lang replace_word_level --mode 73 --data_dir data --output_dir ./models --num_train_epochs 20 --max_seq_length 200 --gradient_accumulation_steps 1 --warmup_steps 500 --train_batch_size 32 --eval_batch_size 16

#step3. test 测试:

set params
run main.py

test lo

--do_test --test_data_file test_lo.json --data_dir data --lang replace_word_level --mode 73 --data_dir data --output_dir ./models --max_seq_length 200 --eval_batch_size 16

test th

--do_test --test_data_file test_th.json --data_dir data --lang replace_word_level --mode 73 --data_dir data --output_dir ./models --max_seq_length 200 --eval_batch_size 16

Model info

onnx safetensors license:apache-2.0 region:us

↓ 0 ♥ 1

Created: 2024-11-28

Updated: 2024-11-29

View on Hugging Face

Files (141)

.gitattributes

2ipa.py

README.md

__pycache__/model.cpython-39.pyc

data/.DS_Store

data/join/test.json

data/join/test.src

data/join/test.tgt

data/join/test_lo.json

data/join/test_lo_binarized.pkl

data/join/test_th.json

data/join/test_th_binarized.pkl

data/join/train.json

data/join/train.src

data/join/train.tgt

data/join/valid.json

data/join/valid.src

data/join/valid.tgt

data/join/valid_binarized.pkl

data/lo/test.json

data/lo/test.src

data/lo/test.tgt

data/lo/test_binarized.pkl

data/lo/train.json

data/lo/train.json.debug

data/lo/train.src

data/lo/train.tgt

data/lo/train_dict.tgt

data/lo/valid.json

data/lo/valid.src

data/lo/valid.tgt

data/lo/valid_binarized.pkl

data/lo/valid_dict.tgt

data/replace_word_level/test_lo.json

data/replace_word_level/test_lo_binarized.pkl

data/replace_word_level/test_th.json

data/replace_word_level/test_th_binarized.pkl

data/replace_word_level/train.json

data/replace_word_level/train.src

data/replace_word_level/train.tgt

data/replace_word_level/train_binarized-debug.pkl

data/replace_word_level/valid.json

data/replace_word_level/valid.src

data/replace_word_level/valid.tgt

data/replace_word_level/valid_binarized.pkl

data/replace_word_levelIPA

data/replace_word_level_IPAtest_lo

data/replace_word_level_IPAtest_th

data/replace_word_level_IPAvalid

data/test_lo.json

data/test_th.json

data/th/_Binarized--debug.pkl

data/th/test.json

data/th/test.src

data/th/test.tgt

data/th/test_binarized.pkl

data/th/train.json

data/th/train.json--debug

data/th/train.src

data/th/train.tgt

data/th/train_binarized.pkl

data/th/train_dict.tgt

data/th/valid.json

data/th/valid.src

data/th/valid.tgt

data/th/valid_binarized.pkl

data/th/valid_dict.tgt

data/triple/data_lo.txt

data/triple/data_th.txt

data/triple/data_zh.txt

data/triple/triple_binarized.pklIPA

data/triple/triple_binarized.pklword

data/tripleIPA2

do_train.7327 .log

foundation/.DS_Store

foundation/E5/.DS_Store

foundation/E5/1_Pooling/config.json

foundation/E5/README.md

foundation/E5/config.json

foundation/E5/gitattributes.txt

foundation/E5/model.safetensors

foundation/E5/modules.json

foundation/E5/onnx/config.json

foundation/E5/onnx/model.onnx ONNX

foundation/E5/onnx/sentencepiece.bpe.model

foundation/E5/onnx/special_tokens_map.json

foundation/E5/onnx/tokenizer.json

foundation/E5/onnx/tokenizer_config.json

foundation/E5/pytorch_model.bin

foundation/E5/sentence_bert_config.json

foundation/E5/special_tokens_map.json

foundation/E5/tokenizer.json

foundation/E5/tokenizer_config.json

main.py

model.py

models/.DS_Store

models/checkpoint-best-aver.7327/optimizer.pt

models/checkpoint-best-aver.7327/pytorch_model.bin

models/checkpoint-best-aver.7327/scheduler.pt

models/checkpoint-best-aver.7327/training_0.bin

models/checkpoint-best-aver.7327/training_10.bin

models/checkpoint-best-aver.7327/training_11.bin

models/checkpoint-best-aver.7327/training_13.bin

models/checkpoint-best-aver.7327/training_15.bin

models/checkpoint-best-aver.7327/training_17.bin

models/checkpoint-best-aver.7327/training_2.bin

models/checkpoint-best-aver.7327/training_3.bin

models/checkpoint-best-aver.7327/training_4.bin

models/checkpoint-best-aver.7327/training_8.bin

models/checkpoint-best-aver.7327/training_9.bin

models/checkpoint-best-aver.7333/optimizer.pt

models/checkpoint-best-aver.7333/pytorch_model.bin

models/checkpoint-best-aver.7333/scheduler.pt

models/checkpoint-best-aver.7333/training_0.bin

models/checkpoint-best-aver.7333/training_1.bin

models/checkpoint-last.7327/idx_file.txt

models/checkpoint-last.7327/optimizer.pt

models/checkpoint-last.7327/pytorch_model.bin

models/checkpoint-last.7327/scheduler.pt

models/checkpoint-last.7327/step_file.txt

models/checkpoint-last.7333/idx_file.txt

models/checkpoint-last.7333/optimizer.pt

models/checkpoint-last.7333/pytorch_model.bin

models/checkpoint-last.7333/scheduler.pt

models/checkpoint-last.7333/step_file.txt

models/predictions_7327_test_lo.json_.txt

models/predictions_7327_test_th.json_.txt

readme.txt

replace_word_level_IPAtest_lo

test_th.json

utils/IPA_lo_dict

utils/IPA_sim_statistic_analysis.py

utils/IPA_th_dict

utils/__pycache__/acc_and_f1.cpython-38.pyc

utils/__pycache__/acc_and_f1.cpython-39.pyc

utils/__pycache__/attn.cpython-38.pyc

utils/__pycache__/attn.cpython-39.pyc

utils/acc_and_f1.py

utils/attn.py

utils/same_list

微信图片_20240401182606.png

Related models

More from KairongLiu

No other models by this author.

More models

No task type declared.