CUDAExecutionProvider Not Available / GPU Not Visible on NVIDIA T4

[anshoomehra]

Describe the bug
Do not see CUDAExecutionProvider or GPU available from ONNX Runtime even though onnxruntime-gpu is installed.

Urgency
In critical stage of project & hence urgent.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux lab-am-vm 4.19.0-16-cloud-amd64 Set up CI with Azure Pipelines #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux

• ONNX Runtime installed from (source or binary): PIP Install

• ONNX Runtime version:
  onnx==1.9.0
  onnxruntime==1.7.0
  onnxruntime-gpu==1.7.0

• Python version: 3.7.10

• CUDA/cuDNN version: release 11.0, V11.0.194

• GPU model and memory: NVIDIA T4, 16G

To Reproduce

• Describe steps/code to reproduce the behavior.

import onnxruntime as ort

ort.get_device()
>>> 'CPU'

ort.get_available_providers()
>>> ['CPUExecutionProvider']

Force setting code to CUDAExecutionProvider results in obvious error & inline with above telemetry:

import onnxruntime as ort

model_path = 'models/bart_large_cnn_fl/encoder/model.onnx'

providers = [
    ('CUDAExecutionProvider', {
        'device_id': 0,
        'arena_extend_strategy': 'kNextPowerOfTwo',
        'cuda_mem_limit': 2 * 1024 * 1024 * 1024,
        'cudnn_conv_algo_search': 'EXHAUSTIVE',
        'do_copy_in_default_stream': True,
    }),
    'CPUExecutionProvider',
]

session = ort.InferenceSession(model_path, providers=providers)

Error MSG

Expected behavior
Expecting device as GPU and provider as CUDAExecutionProvider

[hariharans29]

Did you install both onnxruntime and onnxruntime_gpu on your machine ? If so, can you try after uninstalling onnxruntime with pip uninstall onnxruntime ?

[anshoomehra]

Hariharan Seshadri (@hariharans29) yes, I did uninstall and reinstall everything clean still the same issue. Sequence of install onnx, onnxruntime, onnxruntime-gpu…

onnx==1.9.0
onnxruntime==1.7.0
onnxruntime-gpu==1.7.0

I also just kept onnxruntime-gpu, uninstalling onnxruntime and import fails ..

[hariharans29]

You don't need onnx/onnxruntime installed if you plan to use onnxruntime_gpu.

Can you uninstall everything and just install onnxruntime_gpu ?

[hariharans29]

If there is an import failure with just onnxruntime_gpu installed, atleast it is on the right track. I suspect you are missing some dependencies listed here - https://www.onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements. Make sure these are installed before trying the import again.

[anshoomehra]

If there is an import failure with just onnxruntime_gpu installed, atleast it is on the right track. I suspect you are missing some dependencies listed here - https://www.onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements. Make sure these are installed before trying the import again.

Hariharan Seshadri (@hariharans29) Yes the import fails with just onnxruntime_gpu, let me check on dependencies & revert back if that solves the issue ..

appreciate your time & prompt responses - thanks

[anshoomehra]

Hariharan Seshadri (@hariharans29) I tried uninstalling onnxruntime and imports start to fail .. Should the import paths remain the same post uninstall??

from onnxruntime import (
    ExecutionMode,
    GraphOptimizationLevel,
    InferenceSession,
    SessionOptions,
)

Also, I tried looking for dependencies, these does not look like Python Libs, can you please guide on how can I verify the existence and versions of these?

[anshoomehra]

Hariharan Seshadri (@hariharans29) It worked!! I guess the issue was below :

I was installing onnx, onnxruntime and onnxruntime-gpu in this sequence and later uninstalled onnxruntime only which created import issues. Instead, I uninstalled onnx as well this time and just installed onnxruntime-gpu from sctrach and that made it work, so for some reason onnx was conflicting the libs it seems ...

Thank you so much for helping me get to this stage :-)

I see one issue though with this new method, one of the imports for quantize libs still fail, listed below - any suggestions on how to resolve this ?

[hariharans29]

Yufeng Li (@yufenglee) - would you know about these quantization libs ?

Anshoo Mehra (@anshoomehra) - The libraries listed above are CUDA RT, CuDNN libs. There is documentation on the web that shoul help you check for these. They are dependencies for our GPU builds.

[anshoomehra]

Hariharan Seshadri (@hariharans29) I am running into one another error, not sure if you would suggest open new issue for this?

I am able to export the model successfully with 'CUDAExecutionProvider', however running inference on GPU seems not load all variables to GPU resulting in the below error, any other steps I may need apart from switching devices to GPU for model and tokenizer to make it fully run on GPU?

[hariharans29]

Please open a new issue with more details and a complete repro script

[anshoomehra]

Hariharan Seshadri (@hariharans29) logged new issue as per your suggestion.

We can close this post Yufeng Li (@yufenglee) resolves the quantization libraries import issue with onnxruntime-gpu.

Thanks again for all your help.

[FrancescoSaverioZuppichini]

I have the same issue, using nvidia container nvcr.io/nvidia/pytorch:22.08-py3

Installing pip install onnxruntime-gpu==1.12 fixed the issue!

[EDIT] it looks like it's random, it's not always working

[Ahmet0691]

from optimum.onnxruntime import ORTModelForQuestionAnswering
from transformers import pipeline, AutoTokenizer
import time
import torch
onnx_path = "C:\\Users\\Users\\PycharmProjects\\pythonProject10\\onnx"
model = ORTModelForQuestionAnswering.from_pretrained(onnx_path,file_name="model_quantized.onnx", device =0)
tokenizer = AutoTokenizer.from_pretrained(onnx_path)
nlp = pipeline("question-answering",model=model, tokenizer=tokenizer, device=0)
question = input (str("any questions : "))
with open('text.txt', 'r', encoding='utf8') as file:
context = file.read()
start_time = time.time()
result = nlp(question= question, context=context)
end_time = time.time()
print(result)
print("Latency:", end_time - start_time)

**Error Massage** Traceback (most recent call last): File "C:\Users\Users\PycharmProjects\pythonProject10\quentiz.py", line 8, in nlp = pipeline("question-answering",model=model, tokenizer=tokenizer, device=0) File "C:\Users\Users\.virtualenvs\pythonProject10\lib\site-packages\transformers\pipelines\__init__.py", line 903, in pipeline return pipeline_class(model=model, framework=framework, task=task, **kwargs) File "C:\Users\Users\.virtualenvs\pythonProject10\lib\site-packages\transformers\pipelines\question_answering.py", line 269, in __init__ **kwargs, File "C:\Users\Users\.virtualenvs\pythonProject10\lib\site-packages\transformers\pipelines\base.py", line 780, in __init__ self.model = self.model.to(self.device) File "C:\Users\Users\.virtualenvs\pythonProject10\lib\site-packages\optimum\onnxruntime\modeling_ort.py", line 269, in to validate_provider_availability(provider) # raise error if the provider is not available File "C:\Users\Users\.virtualenvs\pythonProject10\lib\site-packages\optimum\onnxruntime\utils.py", line 197, in validate_provider_availability f"Asked to use {provider}, but `onnxruntime-gpu` package was not found. Make sure to install `onnxruntime-gpu` package instead of `onnxruntime`." ImportError: Asked to use CUDAExecutionProvider, but `onnxruntime-gpu` package was not found. Make sure to install `onnxruntime-gpu` package instead of `onnxruntime`.

Process finished with exit code 1

I uninstalled OnnxRuntime and removed OnnxRuntime-GPU, then reinstalled, tried these steps multiple times but could not find a solution. Please Help..