Model Conversion and Quantization Script

This repository contains a Python script to convert Hugging Face models to ONNX and ORT formats, perform quantization, and generate README files for the converted models. The script automates the process of optimizing models for deployment, making it easier to use models in different environments.

日本語版READMEはこちら

• Model Conversion and Quantization Script
  • Features
  • Requirements
  • Usage
  • Example Usage
  • Notes
  • License
  • Contribution

Features

• Model Conversion: Convert Hugging Face models to ONNX and ORT formats.
• Model Optimization: Optimize the ONNX models for better performance.
• Quantization: Perform FP16, INT8, and UINT8 quantization on the models.
• README Generation: Automatically generate English and Japanese README files for the converted models.
• Hugging Face Integration: Optionally upload the converted models to Hugging Face Hub.

Requirements

• Python 3.11 or higher

• Install required packages using requirements.txt:

  pip install -r requirements.txt

Alternatively, you can install the packages individually:

pip install torch transformers onnx onnxruntime onnxconverter-common onnxruntime-tools onnxruntime-transformers huggingface_hub

Usage

1. Clone the Repository

   git clone https://github.com/yourusername/model_conversion.git
   cd model_conversion

2. Install Dependencies

   Ensure that you have Python 3.11 or higher installed. Install the required packages using requirements.txt:

   pip install -r requirements.txt

3. Run the Conversion Script

   The script convert_model.py converts and quantizes the model.

   python convert_model.py --model your-model-name --output_dir output_directory

   • Replace your-model-name with the name or path of the Hugging Face model you want to convert.
   • The --output_dir argument specifies the output directory. If not provided, it defaults to the model name.

   Example:

   python convert_model.py --model bert-base-uncased --output_dir bert_onnx

4. Upload to Hugging Face (Optional)

   To upload the converted models to Hugging Face Hub, add the --upload flag.

   python convert_model.py --model your-model-name --output_dir output_directory --upload

   Make sure you are logged in to Hugging Face CLI:

   huggingface-cli login

Example Usage

After running the conversion script, you can use the converted models as shown below:

importonnxruntimeasortimportnumpyasnpfromtransformersimportAutoTokenizerimportos# Load the tokenizertokenizer=AutoTokenizer.from_pretrained('your-model-name')
# Prepare inputstext='Replace this text with your input.'inputs=tokenizer(text, return_tensors='np')
# Specify the model paths# Test both the ONNX model and the ORT modelmodel_paths= [
'onnx_models/model_opt.onnx', # ONNX model'ort_models/model.ort'# ORT format model
]
# Run inference with each modelformodel_pathinmodel_paths:
print(f'\n===== Using model: {model_path} =====')
# Get the model extensionmodel_extension=os.path.splitext(model_path)[1]
# Load the modelifmodel_extension=='.ort':
# Load the ORT format modelsession=ort.InferenceSession(model_path, providers=['CPUExecutionProvider'])
else:
# Load the ONNX modelsession=ort.InferenceSession(model_path)
# Run inferenceoutputs=session.run(None, dict(inputs))
# Display the output shapesforidx, outputinenumerate(outputs):
print(f'Output {idx} shape: {output.shape}')
# Display the results (add further processing if needed)print(outputs)

Notes

• Ensure that your ONNX Runtime version is 1.15.0 or higher to use ORT format models.
• Adjust the providers parameter based on your hardware (e.g., 'CUDAExecutionProvider' for GPUs).

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

Contribution

Contributions are welcome! Please open an issue or submit a pull request for improvements.