generative-ts

a typescript library for building LLM applications+agents

Install

To install everything:

npm i generative-ts

You can also do more granular installs of scoped packages if you want to optimize your builds further (see packages)

Usage

AWS Bedrock

API docs: createAwsBedrockModelProvider

import{AmazonTitanTextApi,createAwsBedrockModelProvider}from"generative-ts";// Bedrock supports many different APIs and models. See API docs (above) for full list.consttitanText=createAwsBedrockModelProvider({api: AmazonTitanTextApi,modelId: "amazon.titan-text-express-v1",// If your code is running in an AWS Environment (eg, Lambda) authorization will happen automatically. Otherwise, explicitly pass in `auth`});constresponse=awaittitanText.sendRequest({$prompt:"Brief history of NY Mets:"// all other options for the specified `api` available here});console.log(response.results[0]?.outputText);

Cohere

API docs: createCohereModelProvider

import{createCohereModelProvider}from"generative-ts";constcommandR=createCohereModelProvider({modelId: "command-r-plus",// Cohere defined model ID// you can explicitly pass auth here, otherwise by default it is read from process.env});constresponse=awaitcommandR.sendRequest({$prompt:"Brief History of NY Mets:",preamble: "Talk like Jafar from Aladdin",// all other Cohere /generate options available here});console.log(response.text);

Google Cloud VertexAI

API docs: createVertexAiModelProvider

import{createVertexAiModelProvider}from"@packages/gcloud-vertex-ai";constgemini=awaitcreateVertexAiModelProvider({modelId: "gemini-1.0-pro",// VertexAI defined model ID// you can explicitly pass auth here, otherwise by default it is read from process.env});constresponse=awaitgemini.sendRequest({$prompt:"Brief History of NY Mets:",// all other Gemini options available here});console.log(response.data.candidates[0]);

Groq

API docs: createGroqModelProvider

import{createGroqModelProvider}from"generative-ts";constllama3=createGroqModelProvider({modelId: "llama3-70b-8192",// Groq defined model ID// you can explicitly pass auth here, otherwise by default it is read from process.env});constresponse=awaitllama3.sendRequest({$prompt:"Brief History of NY Mets:"// all other OpenAI ChatCompletion options available here (Groq uses the OpenAI ChatCompletion API for all the models it hosts)});console.log(response.choices[0]?.message.content);

Huggingface Inference

API docs: createHuggingfaceInferenceModelProvider

import{createHuggingfaceInferenceModelProvider,HfTextGenerationTaskApi}from"generative-ts";// Huggingface Inference supports many different APIs and models. See API docs (above) for full list.constgpt2=createHuggingfaceInferenceModelProvider({api: HfTextGenerationTaskApi,modelId: "gpt2",// you can explicitly pass auth here, otherwise by default it is read from process.env});constresponse=awaitgpt2.sendRequest({$prompt:"Hello,"// all other options for the specified `api` available here});console.log(response[0]?.generated_text);

LMStudio

API docs: createLmStudioModelProvider

import{createLmStudioModelProvider}from"generative-ts";constllama3=createLmStudioModelProvider({modelId: "lmstudio-community/Meta-Llama-3-70B-Instruct-GGUF",// a ID of a model you have downloaded in LMStudio});constresponse=awaitllama3.sendRequest({$prompt:"Brief History of NY Mets:"// all other OpenAI ChatCompletion options available here (LMStudio uses the OpenAI ChatCompletion API for all the models it hosts)});console.log(response.choices[0]?.message.content);

Mistral

API docs: createMistralModelProvider

import{createMistralModelProvider}from"generative-ts";constmistralLarge=createMistralModelProvider({modelId: "mistral-large-latest",// Mistral defined model ID// you can explicitly pass auth here, otherwise by default it is read from process.env});constresponse=awaitmistralLarge.sendRequest({$prompt:"Brief History of NY Mets:"// all other Mistral ChatCompletion API options available here});console.log(response.choices[0]?.message.content);

OpenAI

API docs: createOpenAiChatModelProvider

import{createOpenAiChatModelProvider}from"generative-ts";constgpt=createOpenAiChatModelProvider({modelId: "gpt-4-turbo",// OpenAI defined model ID// you can explicitly pass auth here, otherwise by default it is read from process.env});constresponse=awaitgpt.sendRequest({$prompt:"Brief History of NY Mets:",max_tokens: 100,// all other OpenAI ChatCompletion options available here});console.log(response.choices[0]?.message.content);

Custom HTTP Client

todo;

Supported Providers and Models

See Usage for how to use each provider.

Provider	Models	Model APIs
AWS Bedrock	Multiple hosted models	Native model APIs
Cohere	Command / Command R+	Cohere /generate and /chat
Google Vertex AI	Gemini x.y	Gemini; OpenAI in preview
Groq	Multiple hosted models	OpenAI ChatCompletion
Huggingface Inference	Open-source	Huggingface Inference APIs
LMStudio (localhost)	Open-source (must be downloaded)	OpenAI ChatCompletion
Mistral	Mistral x.y	Mistral ChatCompletion
OpenAI	GPT x.y	OpenAI ChatCompletion
Azure (coming soon)
Replicate (coming soon)
Anthropic (coming soon)
Fireworks (coming soon)

It's also easy to add your own TODO LINK

Packages

If you're using a modern bundler, just install generative-ts to get everything. Modern bundlers support tree-shaking, so your final bundle won't include unused code. (Note: we distribute both ESM and CJS bundles for compatibility.) If you prefer to avoid unnecessary downloads, or you're operating under constraints where tree-shaking isn't an option, we offer scoped packages under @generative-ts/ with specific functionality for more fine-grained installs.

Package	Description
generative-ts	Everything	Includes all scoped packages listed below
@generative-ts/core	Core functionality (zero dependencies)	Interfaces, classes, utilities, etc
@generative-ts/gcloud-vertex-ai	Google Cloud VertexAI ModelProvider	Uses Application Default Credentials (ADC) to properly authenticate in GCloud environments
@generative-ts/aws-bedrock	AWS Bedrock ModelProvider	Uses aws4 to properly authenticate when running in AWS environments

Report Bugs / Submit Feature Requests

Please submit all issues here: https://github.com/Econify/generative-ts/issues

Contributing

To get started developing, optionally fork and then clone the repository and run:

nvm use
npm ci

To run examples and integration/e2e tests, create an .env file by running cp .env.example .env and then add values where necessary

Publishing

The "main" generative-ts package and the scoped @generative-ts packages both are controlled by the generative-ts npm organization. Releases are published via circleci job upon pushes of tags that have a name starting with release/. The job requires an NPM token that has publishing permissions to both generative-ts and @generative-ts. Currently this is a "granular" token set to expire every 30 days, created by @jnaglick, set in a circleci context.