GitHub - vossr/Chat-With-RTX-python-api: Chat With RTX Python API · GitHub

Name		Name	Last commit message	Last commit date
readme.md		readme.md
rtx_api_2_11.py		rtx_api_2_11.py
rtx_api_3_5.py		rtx_api_3_5.py
rtx_api_4_24.py		rtx_api_4_24.py
rtx_api_july_2024.py		rtx_api_july_2024.py
rtx_server_0_5_2025.py		rtx_server_0_5_2025.py
rtx_server_july_2024.py		rtx_server_july_2024.py
start_server.bat		start_server.bat
test.py		test.py
test_chat.py		test_chat.py

Repository files navigation

Python API for Chat With RTX

Usage

.\start_server.bat

importrtx_api_july_2024asrtx_apiresponse=rtx_api.send_message("write fire emoji")
print(response)

Speed

Chat With RTX builds int4 (W4A16 AWQ) tensortRT engines for LLMs

Model	On 4090
Mistral	457 char/sec
Llama2	315 char/sec
ChatGLM3	385 char/sec
Gemma	407 char/sec

Update History of Chat With RTX
3.2024 Removed youtube video transcript fetch
4.2024 Added Whisper Speech to text model
7.2024 Electron app ui

LICENSE: CC0

About

Chat With RTX Python API

chat-with-rtx llama2-13b llm llm-inference mistral-7b nvidia-chat-with-rtx tensorrt tensorrt-llm

67 stars

4 watching

Report repository

Contributors

Languages