OpenAI

Based on OpenAI-DotNet

A OpenAI package for the Unity Game Engine to use chat-gpt, GPT-4, GPT-3.5-Turbo and Dall-E though their RESTful API (currently in beta). Independently developed, this is not an official library and I am not affiliated with OpenAI. An OpenAI API account is required.

All copyrights, trademarks, logos, and assets are the property of their respective owners.

This repository is available to transfer to the OpenAI organization if they so choose to accept it.

Installing

Requires Unity 2021.3 LTS or higher.

The recommended installation method is though the unity package manager and OpenUPM.

Via Unity Package Manager and OpenUPM

• Open your Unity project settings
• Add the OpenUPM package registry:
  • Name: OpenUPM
  • URL: https://package.openupm.com
  • Scope(s):
    • com.openai
    • com.utilities

• Open the Unity Package Manager window
• Change the Registry from Unity to My Registries
• Add the OpenAI package

Via Unity Package Manager and Git url

• Open your Unity Package Manager
• Add package from git url: https://github.com/RageAgainstThePixel/com.openai.unity.git#upm

  Note: this repo has dependencies on other repositories! You are responsible for adding these on your own.

  • com.utilities.async
  • com.utilities.rest
  • com.utilities.audio
  • com.utilities.encoder.wav

---

Documentation

Table of Contents

• Authentication
• Azure OpenAI
  • Azure Active Directory Authentication
• OpenAI API Proxy
• Models
  • List Models
  • Retrieve Models
  • Delete Fine Tuned Model
• Completions
  • Streaming
• Chat
  • Chat Completions
  • Streaming
  • Functions 🆕
• Edits
  • Create Edit
• Embeddings
  • Create Embedding
• Audio
  • Create Transcription
  • Create Translation
• Images
  • Create Image
  • Edit Image
  • Create Image Variation
• Files
  • List Files
  • Upload File
  • Delete File
  • Retrieve File Info
  • Download File Content
• Fine Tuning
  • Create Fine Tune Job
  • List Fine Tune Jobs
  • Retrieve Fine Tune Job Info
  • Cancel Fine Tune Job
  • List Fine Tune Events
  • Stream Fine Tune Events
• Moderations
  • Create Moderation

Authentication

There are 4 ways to provide your API keys, in order of precedence:

1. Pass keys directly with constructor
2. Unity Scriptable Object
3. Load key from configuration file
4. Use System Environment Variables

Pass keys directly with constructor

⚠️ We recommended using the environment variables to load the API key instead of having it hard coded in your source. It is not recommended use this method in production, but only for accepting user credentials, local testing and quick start scenarios.

varapi=newOpenAIClient("sk-apiKey");

Or create a OpenAIAuthentication object manually

varapi=newOpenAIClient(newOpenAIAuthentication("sk-apiKey","org-yourOrganizationId"));

Unity Scriptable Object

You can save the key directly into a scriptable object that is located in the Assets/Resources folder.

You can create a new one by using the context menu of the project pane and creating a new OpenAIConfiguration scriptable object.

⚠️ Beware checking this file into source control, as other people will be able to see your API key. It is recommended to use the OpenAI-DotNet-Proxy and authenticate users with your preferred OAuth provider.

Load key from configuration file

Attempts to load api keys from a configuration file, by default .openai in the current directory, optionally traversing up the directory tree or in the user's home directory.

To create a configuration file, create a new text file named .openai and containing the line:

Organization entry is optional.

Json format

{
"apiKey": "sk-aaaabbbbbccccddddd",
"organization": "org-yourOrganizationId"
}

Deprecated format

OPENAI_KEY=sk-aaaabbbbbccccddddd
ORGANIZATION=org-yourOrganizationId

You can also load the configuration file directly with known path by calling static methods in OpenAIAuthentication:

• Loads the default .openai config in the specified directory:

varapi=newOpenAIClient(OpenAIAuthentication.Default.LoadFromDirectory("path/to/your/directory"));

• Loads the configuration file from a specific path. File does not need to be named .openai as long as it conforms to the json format:

varapi=newOpenAIClient(OpenAIAuthentication.Default.LoadFromPath("path/to/your/file.json"));

Use System Environment Variables

Use your system's environment variables specify an api key and organization to use.

• Use OPENAI_API_KEY for your api key.
• Use OPENAI_ORGANIZATION_ID to specify an organization.

varapi=newOpenAIClient(OpenAIAuthentication.Default.LoadFromEnvironment());

Azure OpenAI

You can also choose to use Microsoft's Azure OpenAI deployments as well.

You can find the required information in the Azure Playground by clicking the View Code button and view a URL like this:

https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/chat/completions?api-version={api-version}

• your-resource-name The name of your Azure OpenAI Resource.
• deployment-id The deployment name you chose when you deployed the model.
• api-version The API version to use for this operation. This follows the YYYY-MM-DD format.

To setup the client to use your deployment, you'll need to pass in OpenAISettings into the client constructor.

varauth=newOpenAIAuthentication("sk-apiKey");varsettings=newOpenAISettings(resourceName:"your-resource-name",deploymentId:"deployment-id",apiVersion:"api-version");varapi=newOpenAIClient(auth,settings);

Azure Active Directory Authentication

Authenticate with MSAL as usual and get access token, then use the access token when creating your OpenAIAuthentication. Then be sure to set useAzureActiveDirectory to true when creating your OpenAISettings.

Tutorial: Desktop app that calls web APIs: Acquire a token

// get your access token using any of the MSAL methodsvaraccessToken=result.AccessToken;varauth=newOpenAIAuthentication(accessToken);varsettings=newOpenAISettings(resourceName:"your-resource",deploymentId:"deployment-id",apiVersion:"api-version",useActiveDirectoryAuthentication:true);varapi=newOpenAIClient(auth,settings);

OpenAI API Proxy

Using either the OpenAI-DotNet or com.openai.unity packages directly in your front-end app may expose your API keys and other sensitive information. To mitigate this risk, it is recommended to set up an intermediate API that makes requests to OpenAI on behalf of your front-end app. This library can be utilized for both front-end and intermediary host configurations, ensuring secure communication with the OpenAI API.

Front End Example

In the front end example, you will need to securely authenticate your users using your preferred OAuth provider. Once the user is authenticated, exchange your custom auth token with your API key on the backend.

Follow these steps:

1. Setup a new project using either the OpenAI-DotNet or com.openai.unity packages.
2. Authenticate users with your OAuth provider.
3. After successful authentication, create a new OpenAIAuthentication object and pass in the custom token with the prefix sess-.
4. Create a new OpenAISettings object and specify the domain where your intermediate API is located.
5. Pass your new auth and settings objects to the OpenAIClient constructor when you create the client instance.

Here's an example of how to set up the front end:

varauthToken=awaitLoginAsync();varauth=newOpenAIAuthentication($"sess-{authToken}");varsettings=newOpenAISettings(domain:"api.your-custom-domain.com");varapi=newOpenAIClient(auth,settings);

This setup allows your front end application to securely communicate with your backend that will be using the OpenAI-DotNet-Proxy, which then forwards requests to the OpenAI API. This ensures that your OpenAI API keys and other sensitive information remain secure throughout the process.

Back End Example

In this example, we demonstrate how to set up and use OpenAIProxyStartup in a new ASP.NET Core web app. The proxy server will handle authentication and forward requests to the OpenAI API, ensuring that your API keys and other sensitive information remain secure.

1. Create a new ASP.NET Core minimal web API project.
2. Add the OpenAI-DotNet nuget package to your project.
   • Powershell install: Install-Package OpenAI-DotNet-Proxy
   • Manually editing .csproj: <PackageReference Include="OpenAI-DotNet-Proxy" />
3. Create a new class that inherits from AbstractAuthenticationFilter and override the ValidateAuthentication method. This will implement the IAuthenticationFilter that you will use to check user session token against your internal server.
4. In Program.cs, create a new proxy web application by calling OpenAIProxyStartup.CreateDefaultHost method, passing your custom AuthenticationFilter as a type argument.
5. Create OpenAIAuthentication and OpenAIClientSettings as you would normally with your API keys, org id, or Azure settings.

publicpartialclassProgram{privateclassAuthenticationFilter:AbstractAuthenticationFilter{publicoverridevoidValidateAuthentication(IHeaderDictionaryrequest){// You will need to implement your own class to properly test// custom issued tokens you've setup for your end users.if(!request.Authorization.ToString().Contains(userToken)){thrownewAuthenticationException("User is not authorized");}}}publicstaticvoidMain(string[]args){varauth=OpenAIAuthentication.LoadFromEnv();varsettings=newOpenAIClientSettings(/* your custom settings if using Azure OpenAI */);varopenAIClient=newOpenAIClient(auth,settings);varproxy=OpenAIProxyStartup.CreateDefaultHost<AuthenticationFilter>(args,openAIClient);proxy.Run();}}

Once you have set up your proxy server, your end users can now make authenticated requests to your proxy api instead of directly to the OpenAI API. The proxy server will handle authentication and forward requests to the OpenAI API, ensuring that your API keys and other sensitive information remain secure.

Models

List and describe the various models available in the API. You can refer to the Models documentation to understand what models are available and the differences between them.

Also checkout model endpoint compatibility to understand which models work with which endpoints.

To specify a custom model not pre-defined in this library:

varmodel=newModel("model-id");

The Models API is accessed via OpenAIClient.ModelsEndpoint

List models

Lists the currently available models, and provides basic information about each one such as the owner and availability.

varapi=newOpenAIClient();varmodels=awaitapi.ModelsEndpoint.GetModelsAsync();foreach(varmodelinmodels){Debug.Log(model.ToString());}

Retrieve model

Retrieves a model instance, providing basic information about the model such as the owner and permissions.

varapi=newOpenAIClient();varmodel=awaitapi.ModelsEndpoint.GetModelDetailsAsync("text-davinci-003");Debug.Log(model.ToString());

Delete Fine Tuned Model

Delete a fine-tuned model. You must have the Owner role in your organization.

varapi=newOpenAIClient();varresult=awaitapi.ModelsEndpoint.DeleteFineTuneModelAsync("your-fine-tuned-model");Assert.IsTrue(result);

Completions

Given a prompt, the model will return one or more predicted completions, and can also return the probabilities of alternative tokens at each position.

The Completions API is accessed via OpenAIClient.CompletionsEndpoint

varapi=newOpenAIClient();varresult=awaitapi.CompletionsEndpoint.CreateCompletionAsync("One Two Three One Two",temperature:0.1,model:Model.Davinci);Debug.Log(result);

To get the CompletionResult (which is mostly metadata), use its implicit string operator to get the text if all you want is the completion choice.

Completion Streaming

Streaming allows you to get results are they are generated, which can help your application feel more responsive, especially on slow models like Davinci.

varapi=newOpenAIClient();awaitapi.CompletionsEndpoint.StreamCompletionAsync(result =>{foreach(varchoiceinresult.Completions){Debug.Log(choice);}},"My name is Roger and I am a principal software engineer at Salesforce. This is my resume:",maxTokens:200,temperature:0.5,presencePenalty:0.1,frequencyPenalty:0.1,model:Model.Davinci);

Chat

Given a chat conversation, the model will return a chat completion response.

The Chat API is accessed via OpenAIClient.ChatEndpoint

Chat Completions

Creates a completion for the chat message

varapi=newOpenAIClient();varmessages=newList<Message>{newMessage(Role.System,"You are a helpful assistant."),newMessage(Role.User,"Who won the world series in 2020?"),newMessage(Role.Assistant,"The Los Angeles Dodgers won the World Series in 2020."),newMessage(Role.User,"Where was it played?"),};varchatRequest=newChatRequest(messages,Model.GPT3_5_Turbo);varresult=awaitapi.ChatEndpoint.GetCompletionAsync(chatRequest);Debug.Log($"{result.FirstChoice.Message.Role}: {result.FirstChoice.Message.Content}");

Chat Streaming

varapi=newOpenAIClient();varmessages=newList<Message>{newMessage(Role.System,"You are a helpful assistant."),newMessage(Role.User,"Who won the world series in 2020?"),newMessage(Role.Assistant,"The Los Angeles Dodgers won the World Series in 2020."),newMessage(Role.User,"Where was it played?"),};varchatRequest=newChatRequest(messages,Model.GPT3_5_Turbo,number:2);awaitapi.ChatEndpoint.StreamCompletionAsync(chatRequest, result =>{foreach(varchoiceinresult.Choices.Where(choice =>!string.IsNullOrWhiteSpace(choice.Delta?.Content))){// Partial response contentDebug.Log(choice.Delta.Content);}foreach(varchoiceinresult.Choices.Where(choice =>!string.IsNullOrWhiteSpace(choice.Message?.Content))){// Completed response contentDebug.Log($"{choice.Message.Role}: {choice.Message.Content}");}});

Chat Functions

Only available with the latest 0613 model series!

varapi=newOpenAIClient();varmessages=newList<Message>{newMessage(Role.System,"You are a helpful weather assistant."),newMessage(Role.User,"What's the weather like today?"),};foreach(varmessageinmessages){Debug.Log($"{message.Role}: {message.Content}");}// Define the functions that the assistant is able to use:varfunctions=newList<Function>{newFunction(nameof(WeatherService.GetCurrentWeather),"Get the current weather in a given location",newJObject{["type"]="object",["properties"]=newJObject{["location"]=newJObject{["type"]="string",["description"]="The city and state, e.g. San Francisco, CA"},["unit"]=newJObject{["type"]="string",["enum"]=newJArray{"celsius","fahrenheit"}}},["required"]=newJArray{"location","unit"}})};varchatRequest=newChatRequest(messages,functions:functions,functionCall:"auto",model:"gpt-3.5-turbo-0613");varresult=awaitapi.ChatEndpoint.GetCompletionAsync(chatRequest);messages.Add(result.FirstChoice.Message);Debug.Log($"{result.FirstChoice.Message.Role}: {result.FirstChoice.Message.Content} | Finish Reason: {result.FirstChoice.FinishReason}");varlocationMessage=newMessage(Role.User,"I'm in Glasgow, Scotland");messages.Add(locationMessage);Debug.Log($"{locationMessage.Role}: {locationMessage.Content}");chatRequest=newChatRequest(messages,functions:functions,functionCall:"auto",model:"gpt-3.5-turbo-0613");result=awaitapi.ChatEndpoint.GetCompletionAsync(chatRequest);messages.Add(result.FirstChoice.Message);if(!string.IsNullOrWhiteSpace(result.FirstChoice.Message.Content)){// It's possible that the assistant will also ask you which units you want the temperature in.Debug.Log($"{result.FirstChoice.Message.Role}: {result.FirstChoice.Message.Content} | Finish Reason: {result.FirstChoice.FinishReason}");varunitMessage=newMessage(Role.User,"celsius");messages.Add(unitMessage);Debug.Log($"{unitMessage.Role}: {unitMessage.Content}");chatRequest=newChatRequest(messages,functions:functions,functionCall:"auto",model:"gpt-3.5-turbo-0613");result=awaitapi.ChatEndpoint.GetCompletionAsync(chatRequest);}Debug.Log($"{result.FirstChoice.Message.Role}: {result.FirstChoice.Message.Function.Name} | Finish Reason: {result.FirstChoice.FinishReason}");Debug.Log($"{result.FirstChoice.Message.Function.Arguments}");varfunctionArgs=JsonConvert.DeserializeObject<WeatherArgs>(result.FirstChoice.Message.Function.Arguments.ToString());varfunctionResult=WeatherService.GetCurrentWeather(functionArgs);messages.Add(newMessage(Role.Function,functionResult));Debug.Log($"{Role.Function}: {functionResult}");// System: You are a helpful weather assistant.// User: What's the weather like today?// Assistant: Sure, may I know your current location? | Finish Reason: stop// User: I'm in Glasgow, Scotland// Assistant: GetCurrentWeather | Finish Reason: function_call// {// "location": "Glasgow, Scotland",// "unit": "celsius"// }// Function: The current weather in Glasgow, Scotland is 20 celsius

Edits

Given a prompt and an instruction, the model will return an edited version of the prompt.

The Edits API is accessed via OpenAIClient.EditsEndpoint

Create Edit

Creates a new edit for the provided input, instruction, and parameters using the provided input and instruction.

varapi=newOpenAIClient();varrequest=newEditRequest("What day of the wek is it?","Fix the spelling mistakes");varresult=awaitapi.EditsEndpoint.CreateEditAsync(request);Debug.Log(result);

Embeddings

Get a vector representation of a given input that can be easily consumed by machine learning models and algorithms.

Related guide: Embeddings

The Edits API is accessed via OpenAIClient.EmbeddingsEndpoint

Create Embeddings

Creates an embedding vector representing the input text.

varapi=newOpenAIClient();varresult=awaitapi.EmbeddingsEndpoint.CreateEmbeddingAsync("The food was delicious and the waiter...",Models.Embedding_Ada_002);Debug.Log(result);

Audio

Converts audio into text.

The Audio API is accessed via OpenAIClient.AudioEndpoint

Create Transcription

Transcribes audio into the input language.

varapi=newOpenAIClient();varrequest=newAudioTranscriptionRequest(audioClip,language:"en");varresult=awaitapi.AudioEndpoint.CreateTranscriptionAsync(request);Debug.Log(result);

Create Translation

Translates audio into into English.

varapi=newOpenAIClient();varrequest=newAudioTranslationRequest(audioClip);varresult=awaitapi.AudioEndpoint.CreateTranslationAsync(request);Debug.Log(result);

Images

Given a prompt and/or an input image, the model will generate a new image.

The Images API is accessed via OpenAIClient.ImagesEndpoint

Create Image

Creates an image given a prompt.

varapi=newOpenAIClient();varresults=awaitapi.ImagesEndPoint.GenerateImageAsync("A house riding a velociraptor",1,ImageSize.Small);foreach(var(path,texture)inresults){Debug.Log(path);// path == file://path/to/image.pngAssert.IsNotNull(texture);// texture == The preloaded Texture2D}

Edit Image

Creates an edited or extended image given an original image and a prompt.

varapi=newOpenAIClient();varresults=awaitapi.ImagesEndPoint.CreateImageEditAsync(Path.GetFullPath(imageAssetPath),Path.GetFullPath(maskAssetPath),"A sunlit indoor lounge area with a pool containing a flamingo",1,ImageSize.Small);foreach(var(path,texture)inresults){Debug.Log(path);// path == file://path/to/image.pngAssert.IsNotNull(texture);// texture == The preloaded Texture2D}

Create Image Variation

Creates a variation of a given image.

varapi=newOpenAIClient();varresults=awaitapi.ImagesEndPoint.CreateImageVariationAsync(Path.GetFullPath(imageAssetPath),1,ImageSize.Small);foreach(var(path,texture)inresults){Debug.Log(path);// path == file://path/to/image.pngAssert.IsNotNull(texture);// texture == The preloaded Texture2D}

Alternatively, the endpoint can directly take a Texture2D with Read/Write enabled and Compression set to None.

varapi=newOpenAIClient();varresults=awaitapi.ImagesEndPoint.CreateImageVariationAsync(imageTexture,1,ImageSize.Small);// imageTexture is of type Texture2Dforeach(var(path,texture)inresults){Debug.Log(path);// path == file://path/to/image.pngAssert.IsNotNull(texture);// texture == The preloaded Texture2D}

Files

Files are used to upload documents that can be used with features like Fine-tuning.

The Files API is accessed via OpenAIClient.FilesEndpoint

List Files

Returns a list of files that belong to the user's organization.

varapi=newOpenAIClient();varfiles=awaitapi.FilesEndpoint.ListFilesAsync();foreach(varfileinfiles){Debug.Log($"{file.Id} -> {file.Object}: {file.FileName} | {file.Size} bytes");}

Upload File

Upload a file that contains document(s) to be used across various endpoints/features. Currently, the size of all the files uploaded by one organization can be up to 1 GB. Please contact us if you need to increase the storage limit.

varapi=newOpenAIClient();varfileData=awaitapi.FilesEndpoint.UploadFileAsync("path/to/your/file.jsonl","fine-tune");Debug.Log(fileData.Id);

Delete File

Delete a file.

varapi=newOpenAIClient();varresult=awaitapi.FilesEndpoint.DeleteFileAsync(fileData);Assert.IsTrue(result);

Retrieve File Info

Returns information about a specific file.

varapi=newOpenAIClient();varfileData=awaitGetFileInfoAsync(fileId);Debug.Log($"{fileData.Id} -> {fileData.Object}: {fileData.FileName} | {fileData.Size} bytes");

Download File Content

Downloads the specified file.

varapi=newOpenAIClient();vardownloadedFilePath=awaitapi.FilesEndpoint.DownloadFileAsync(fileId);Debug.Log(downloadedFilePath);Assert.IsTrue(File.Exists(downloadedFilePath));

Fine Tuning

Manage fine-tuning jobs to tailor a model to your specific training data.

Related guide: Fine-tune models

The Files API is accessed via OpenAIClient.FineTuningEndpoint

Create Fine Tune Job

Creates a job that fine-tunes a specified model from a given dataset.

Response includes details of the enqueued job including job status and the name of the fine-tuned models once complete.

varapi=newOpenAIClient();varrequest=newCreateFineTuneRequest(fileData);varfineTuneJob=awaitapi.FineTuningEndpoint.CreateFineTuneJobAsync(request);Debug.Log(fineTuneJob.Id);

List Fine Tune Jobs

List your organization's fine-tuning jobs.

varapi=newOpenAIClient();varfineTuneJobs=awaitapi.FineTuningEndpoint.ListFineTuneJobsAsync();foreach(varjobinfineTuneJobs){Debug.Log($"{job.Id} -> {job.Status}");}

Retrieve Fine Tune Job Info

Gets info about the fine-tune job.

varapi=newOpenAIClient();varresult=awaitapi.FineTuningEndpoint.RetrieveFineTuneJobInfoAsync(fineTuneJob);Debug.Log($"{result.Id} -> {result.Status}");

Cancel Fine Tune Job

Immediately cancel a fine-tune job.

varapi=newOpenAIClient();varresult=awaitapi.FineTuningEndpoint.CancelFineTuneJobAsync(fineTuneJob);Assert.IsTrue(result);

List Fine Tune Events

Get fine-grained status updates for a fine-tune job.

varapi=newOpenAIClient();varfineTuneEvents=awaitapi.FineTuningEndpoint.ListFineTuneEventsAsync(fineTuneJob);Debug.Log($"{fineTuneJob.Id} -> status: {fineTuneJob.Status} | event count: {fineTuneEvents.Count}");

Stream Fine Tune Events

varapi=newOpenAIClient();awaitapi.FineTuningEndpoint.StreamFineTuneEventsAsync(fineTuneJob, fineTuneEvent =>{Debug.Log($" {fineTuneEvent.CreatedAt} [{fineTuneEvent.Level}] {fineTuneEvent.Message}");});

Moderations

Given a input text, outputs if the model classifies it as violating OpenAI's content policy.

Related guide: Moderations

The Moderations API can be accessed via OpenAIClient.ModerationsEndpoint

Create Moderation

Classifies if text violates OpenAI's Content Policy.

varapi=newOpenAIClient();varresponse=awaitapi.ModerationsEndpoint.GetModerationAsync("I want to kill them.");Assert.IsTrue(response);