Skip to content

fix(chat): write the resolved model to chats.model_id at provision time#830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sweetmantech merged 4 commits into from
Aug 13, 2026
Merged

fix(chat): write the resolved model to chats.model_id at provision time #830

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8dd9e31
fix(chat): write the resolved model to chats.model_id at provision time
sweetmantech Aug 12, 2026
24d5188
style: prettier formatting on two test files
sweetmantech Aug 12, 2026
9bb063b
fix(chat): default chat model is kimi-k3; single constant, no aliases…
sweetmantech Aug 13, 2026
bb879e6
refactor(const): one DEFAULT_MODEL for everything, value kimi-k3
sweetmantech Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion app/api/chat/runs/route.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -32,7 +32,7 @@ export async function OPTIONS() {
* - prompt: String prompt (mutually exclusive with messages)
* - messages: Array of UIMessages (mutually exclusive with prompt)
* - artistId: Optional UUID of the artist account
* - model: Optional model ID override (default anthropic/claude-haiku-4.5)
* - model: Optional model ID override (default moonshotai/kimi-k3)
* - topic: Optional session title
* - accountId: Optional accountId override (requires org API key)
*
Expand Down
2 changes: 1 addition & 1 deletion lib/agents/generalAgent/__tests__/getGeneralAgent.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -117,7 +117,7 @@ describe("getGeneralAgent", () => {

const result = await getGeneralAgent(body);

expect(result.model).toBe("openai/gpt-5.4-nano");
expect(result.model).toBe("moonshotai/kimi-k3");
});

it("uses custom model when specified in body", async () => {
Expand Down
2 changes: 1 addition & 1 deletion lib/chat/__tests__/handleChatWorkflowStream.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -286,7 +286,7 @@ describe("handleChatWorkflowStream", () => {
mockStartedRun();
await handleChatWorkflowStream(makeRequest());
const startArgs = vi.mocked(start).mock.calls[0]?.[1]?.[0] as { modelId: string };
expect(startArgs.modelId).toBe("anthropic/claude-haiku-4.5");
expect(startArgs.modelId).toBe("moonshotai/kimi-k3");
});

// Bundle A.4 — forward the Privy JWT from the validated body into
Expand Down
5 changes: 2 additions & 3 deletions lib/chat/handleChatWorkflowStream.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
import { DEFAULT_MODEL } from "@/lib/const";
import { NextRequest, NextResponse } from "next/server";
import { createUIMessageStreamResponse, type UIMessageChunk } from "ai";
import { start, getRun } from "workflow/api";
Expand All@@ -22,8 +23,6 @@ import { discoverSkills } from "@/lib/skills/discoverSkills";
import { getSandboxSkillDirectories } from "@/lib/skills/getSandboxSkillDirectories";
import generateUUID from "@/lib/uuid/generateUUID";

const DEFAULT_MODEL_ID = "anthropic/claude-haiku-4.5";

/**
* Handles POST /api/chat/workflow.
*
Expand DownExpand Up@@ -90,7 +89,7 @@ export async function handleChatWorkflowStream(request: NextRequest): Promise<Re
await updateSession(validated.sessionId, buildActiveLifecycleUpdate(session.sandbox_state));
void persistLatestUserMessage(validated.chatId, validated.messages as never);

const modelId = chat.model_id ?? DEFAULT_MODEL_ID;
const modelId = chat.model_id ?? DEFAULT_MODEL;

// Connect the sandbox up-front so we can (a) read the real working
// directory and (b) discover project-level skills. The connected
Expand Down
9 changes: 7 additions & 2 deletions lib/chat/runs/__tests__/handleStartChatRun.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -48,7 +48,7 @@ const validated = {
orgId: null,
messages: [{ id: "m1", role: "user", parts: [{ type: "text", text: "go" }] }],
artistId: undefined,
modelId: "anthropic/claude-haiku-4.5",
modelId: "test/model-y",
};

const provisioned = {
Expand DownExpand Up@@ -86,7 +86,12 @@ describe("handleStartChatRun", () => {
});

expect(provisionRunSession).toHaveBeenCalledWith(
expect.objectContaining({ accountId: "acc-1", title: "Scheduled generation" }),
expect.objectContaining({
accountId: "acc-1",
title: "Scheduled generation",
// Provenance (chat#1956): the resolved model reaches the chat insert.
modelId: "test/model-y",
}),
);
// the minted key is injected as recoupAccessToken AND threaded as ephemeralKeyId
expect(buildRunAgentInput).toHaveBeenCalledWith(
Expand Down
18 changes: 16 additions & 2 deletions lib/chat/runs/__tests__/provisionRunSession.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -56,7 +56,7 @@ describe("provisionRunSession", () => {
});

it("installs global skills into the sandbox before discovering them", async () => {
await provisionRunSession({ accountId: "account-1", title: "t" });
await provisionRunSession({ accountId: "account-1", title: "t", modelId: "test/model-x" });

// Headless runs must PROVISION skills, not just discover them (chat#1822).
expect(installSessionGlobalSkills).toHaveBeenCalledWith({
Expand All@@ -69,10 +69,24 @@ describe("provisionRunSession", () => {
expect(installOrder).toBeLessThan(discoverOrder);
});

// Provenance thread-through (chat#1956): the resolved model must reach the
// chat insert, not stop at the workflow input.
it("forwards modelId to createSessionWithInitialChat", async () => {
await provisionRunSession({ accountId: "account-1", title: "t", modelId: "test/model-x" });

expect(createSessionWithInitialChat).toHaveBeenCalledWith(
expect.objectContaining({ modelId: "test/model-x" }),
);
});

it("still completes the run when skill install fails (best-effort)", async () => {
vi.mocked(installSessionGlobalSkills).mockRejectedValueOnce(new Error("install boom"));

const result = await provisionRunSession({ accountId: "account-1", title: "t" });
const result = await provisionRunSession({
accountId: "account-1",
title: "t",
modelId: "test/model-x",
});

expect(result.session).toEqual(updated);
expect(discoverSkills).toHaveBeenCalled();
Expand Down
12 changes: 11 additions & 1 deletion lib/chat/runs/__tests__/validateChatRunRequest.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -50,7 +50,17 @@ describe("validateChatRunRequest", () => {

const noModel = await validateChatRunRequest(req({ prompt: "hi" }));
if (noModel instanceof NextResponse) throw new Error("unexpected error");
expect(noModel.modelId).toBe("anthropic/claude-haiku-4.5");
expect(noModel.modelId).toBe("moonshotai/kimi-k3");

// cubic P2: an empty or whitespace model must normalize to the default,
// never persist as "" (provenance) or reach the workflow as "".
const emptyModel = await validateChatRunRequest(req({ prompt: "hi", model: "" }));
if (emptyModel instanceof NextResponse) throw new Error("unexpected error");
expect(emptyModel.modelId).toBe("moonshotai/kimi-k3");

const blankModel = await validateChatRunRequest(req({ prompt: "hi", model: " " }));
if (blankModel instanceof NextResponse) throw new Error("unexpected error");
expect(blankModel.modelId).toBe("moonshotai/kimi-k3");
});

it("rejects when neither prompt nor messages is provided (400)", async () => {
Expand Down
1 change: 1 addition & 0 deletions lib/chat/runs/handleStartChatRun.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -46,6 +46,7 @@ export async function handleStartChatRun(request: NextRequest): Promise<Response
accountId,
title: DEFAULT_RUN_SESSION_TITLE,
artistId,
modelId,
});

const { rawKey, keyId } = await mintEphemeralAccountKey(accountId);
Expand Down
4 changes: 4 additions & 0 deletions lib/chat/runs/provisionRunSession.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -37,16 +37,20 @@ export async function provisionRunSession({
accountId,
title,
artistId,
modelId,
}: {
accountId: string;
title: string;
artistId?: string;
/** Resolved model for this run — written to the chat row for provenance (chat#1956). */
modelId: string;
}): Promise<ProvisionedRunSession> {
const created = await createSessionWithInitialChat({
accountId,
title,
chatTitle: "Scheduled generation",
artistId,
modelId,
});
if (created.ok === false) {
throw new Error(
Expand Down
10 changes: 6 additions & 4 deletions lib/chat/runs/validateChatRunRequest.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -6,9 +6,7 @@ import { validateAuthContext } from "@/lib/auth/validateAuthContext";
import { errorResponse } from "@/lib/networking/errorResponse";
import { validationErrorResponse } from "@/lib/zod/validationErrorResponse";
import { generateUUID } from "@/lib/uuid/generateUUID";

/** Default model for headless generation when the caller omits `model`. */
export const DEFAULT_RUN_MODEL_ID = "anthropic/claude-haiku-4.5";
import { DEFAULT_MODEL } from "@/lib/const";

/**
* Body schema for `POST /api/chat/runs` (the durable-workflow re-point,
Expand DownExpand Up@@ -78,11 +76,15 @@ export async function validateChatRunRequest(
? [{ id: generateUUID(), role: "user", parts: [{ type: "text", text: trimmedPrompt }] }]
: (messages as UIMessage[]);

// `||` not `??`: an empty/whitespace model must fall back too, or "" would
// be persisted as provenance and sent to the workflow as the model id.
const trimmedModel = typeof model === "string" ? model.trim() : "";

return {
accountId: auth.accountId,
orgId: auth.orgId,
messages: uiMessages,
artistId,
modelId: model ?? DEFAULT_RUN_MODEL_ID,
modelId: trimmedModel || DEFAULT_MODEL,
};
}
9 changes: 8 additions & 1 deletion lib/const.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -9,7 +9,14 @@ export const SMART_ACCOUNT_ADDRESS = "0xbAf31935ED514e8F7da81D0A730AB5362DEEEEb7
export const USDC_ADDRESS = "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913" as Address;
export const PAYMASTER_URL = `https://api.developer.coinbase.com/rpc/v1/base/${process.env.PAYMASTER_KEY}`;
export const IMAGE_GENERATE_PRICE = "0.15";
export const DEFAULT_MODEL = "openai/gpt-5.4-nano";
/**
* THE default model, everywhere a caller picks none: interactive chats,
* headless runs (persisted to `chats.model_id` at provision time, chat#1956),
* evals, catalog batch analysis, and the general agent's fallback.
* kimi-k3 chosen 2026-08-12: best recall and lowest cost in a 4-model A/B on
* a production roster-brief task (api#830 review thread).
*/
export const DEFAULT_MODEL = "moonshotai/kimi-k3";
export const LIGHTWEIGHT_MODEL = "openai/gpt-4o-mini";
export const PRIVY_PROJECT_SECRET = process.env.PRIVY_PROJECT_SECRET;
/** Base URL for the public API documentation site */
Expand Down
3 changes: 3 additions & 0 deletions lib/sessions/__tests__/createSessionHandler.persistence.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -68,6 +68,9 @@ describe("createSessionHandler — persistence", () => {
const chatArgs = vi.mocked(insertChat).mock.calls[0][0];
expect(chatArgs.session_id).toBe("sess_1");
expect(chatArgs.title).toBe("New chat");
// Provenance (chat#1956): every writer sets model_id explicitly so the
// chats.model_id column default can be dropped.
expect(chatArgs.model_id).toBe("moonshotai/kimi-k3");
});

it("uses auth.accountId for personal sessions", async () => {
Expand Down
15 changes: 14 additions & 1 deletion lib/sessions/__tests__/createSessionWithInitialChat.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -16,7 +16,13 @@ vi.mock("@/lib/supabase/sessions/insertSession", () => ({ insertSession: vi.fn()
vi.mock("@/lib/supabase/sessions/deleteSessionById", () => ({ deleteSessionById: vi.fn() }));
vi.mock("@/lib/supabase/chats/insertChat", () => ({ insertChat: vi.fn() }));

const args = { accountId: "acc-1", title: "T", chatTitle: "New chat", artistId: "art-1" };
const args = {
accountId: "acc-1",
title: "T",
chatTitle: "New chat",
artistId: "art-1",
modelId: "test/model-x",
};

describe("createSessionWithInitialChat", () => {
beforeEach(() => {
Expand All@@ -38,6 +44,13 @@ describe("createSessionWithInitialChat", () => {
);
});

// Provenance: the chat row must record the model that will run (chat#1956) —
// never rely on the chats.model_id column default to fill it in.
it("writes the caller's modelId onto the inserted chat row", async () => {
await createSessionWithInitialChat(args);
expect(insertChat).toHaveBeenCalledWith(expect.objectContaining({ model_id: "test/model-x" }));
});

it("uses workspaceAccountId for the repo when provided", async () => {
await createSessionWithInitialChat({ ...args, workspaceAccountId: "org-9" });
expect(ensurePersonalRepo).toHaveBeenCalledWith({ accountId: "org-9" });
Expand Down
3 changes: 3 additions & 0 deletions lib/sessions/chats/__tests__/createSessionChatHandler.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -95,6 +95,9 @@ describe("createSessionChatHandler", () => {
expect(insertArgs.id).toBe("chat_requested");
expect(insertArgs.session_id).toBe("sess_1");
expect(insertArgs.title).toBe("New chat");
// Provenance (chat#1956): every writer sets model_id explicitly so the
// chats.model_id column default can be dropped.
expect(insertArgs.model_id).toBe("moonshotai/kimi-k3");
});

it("creates a chat with a generated id when no id is provided", async () => {
Expand Down
2 changes: 1 addition & 1 deletion lib/sessions/chats/__tests__/getSessionChatsHandler.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@ describe("getSessionChatsHandler", () => {
defaultModelId: string;
};
expect(body.chats).toEqual(summaries);
expect(body.defaultModelId).toBe("openai/gpt-5.4-nano");
expect(body.defaultModelId).toBe("moonshotai/kimi-k3");
expect(getChatSummaries).toHaveBeenCalledWith({
sessionId: "sess_1",
accountId,
Expand Down
2 changes: 2 additions & 0 deletions lib/sessions/chats/createSessionChatHandler.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -5,6 +5,7 @@ import { validateCreateSessionChatRequest } from "@/lib/sessions/chats/validateC
import { selectChats } from "@/lib/supabase/chats/selectChats";
import { insertChat } from "@/lib/supabase/chats/insertChat";
import { toChatResponse } from "@/lib/sessions/toChatResponse";
import { DEFAULT_MODEL } from "@/lib/const";

const INITIAL_CHAT_TITLE = "New chat";

Expand DownExpand Up@@ -53,6 +54,7 @@ export async function createSessionChatHandler(
id: requestedChatId ?? generateUUID(),
session_id: sessionId,
title: INITIAL_CHAT_TITLE,
model_id: DEFAULT_MODEL,
});

if (!chatRow) {
Expand Down
2 changes: 2 additions & 0 deletions lib/sessions/createSessionHandler.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -9,6 +9,7 @@ import {
import { failedToCreateSession } from "@/lib/sessions/failedToCreateSession";
import { toSessionResponse } from "@/lib/sessions/toSessionResponse";
import { toChatResponse } from "@/lib/sessions/toChatResponse";
import { DEFAULT_MODEL } from "@/lib/const";

const INITIAL_CHAT_TITLE = "New chat";

Expand DownExpand Up@@ -49,6 +50,7 @@ export async function createSessionHandler(request: NextRequest): Promise<NextRe
title,
chatTitle: INITIAL_CHAT_TITLE,
artistId: body.artistId,
modelId: DEFAULT_MODEL,
});

if (result.ok === false) {
Expand Down
10 changes: 9 additions & 1 deletion lib/sessions/createSessionWithInitialChat.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,12 +35,15 @@ export async function createSessionWithInitialChat({
title,
chatTitle,
artistId,
modelId,
}: {
accountId: string;
workspaceAccountId?: string;
title: string;
chatTitle: string;
artistId?: string;
/** Model recorded on the chat row — provenance, never left to a column default (chat#1956). */
modelId: string;
}): Promise<CreateSessionWithChatResult> {
const cloneUrl = await ensurePersonalRepo({ accountId: workspaceAccountId ?? accountId });
if (!cloneUrl) return { ok: false, reason: "repo" };
Expand All@@ -50,7 +53,12 @@ export async function createSessionWithInitialChat({
);
if (!session) return { ok: false, reason: "insert" };

const chat = await insertChat({ id: generateUUID(), session_id: session.id, title: chatTitle });
const chat = await insertChat({
id: generateUUID(),
session_id: session.id,
title: chatTitle,
model_id: modelId,

@cubic-dev-aicubic-dev-aiBotAug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When a run request supplies model: "", this line persists an empty model_id instead of a usable default. Reject or normalize empty model IDs before provisioning so the persisted provenance and the model sent to the workflow are valid.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/sessions/createSessionWithInitialChat.ts, line 60:
<comment>When a run request supplies `model: ""`, this line persists an empty `model_id` instead of a usable default. Reject or normalize empty model IDs before provisioning so the persisted provenance and the model sent to the workflow are valid.</comment>
<file context>
@@ -50,7 +53,12 @@ export async function createSessionWithInitialChat({
+ id: generateUUID(),
+ session_id: session.id,
+ title: chatTitle,
+ model_id: modelId,
+ });
if (!chat) {
</file context>

@cubic-dev-aicubic-dev-aiBotAug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This PR centralizes the invariant that "every chat writer persists a model explicitly" and the follow-up DB PR drops the chats.model_id column default. There is another insertChat caller — scripts/backfill/migrateRoom.ts — that constructs a chat row without model_id. If that idempotent backfill is re-run after the column default is dropped, it will insert chats with a NULL model_id, which breaks the same provenance invariant this change is meant to guarantee. Consider setting model_id explicitly in migrateRoom.ts (or confirming the DB migration backfills existing and skips new NULL inserts) before dropping the default.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/sessions/createSessionWithInitialChat.ts, line 60:
<comment>This PR centralizes the invariant that "every chat writer persists a model explicitly" and the follow-up DB PR drops the `chats.model_id` column default. There is another `insertChat` caller — `scripts/backfill/migrateRoom.ts` — that constructs a chat row without `model_id`. If that idempotent backfill is re-run after the column default is dropped, it will insert chats with a NULL `model_id`, which breaks the same provenance invariant this change is meant to guarantee. Consider setting `model_id` explicitly in `migrateRoom.ts` (or confirming the DB migration backfills existing and skips new NULL inserts) before dropping the default.</comment>
<file context>
@@ -50,7 +53,12 @@ export async function createSessionWithInitialChat({
+ id: generateUUID(),
+ session_id: session.id,
+ title: chatTitle,
+ model_id: modelId,
+ });
if (!chat) {
</file context>

@sweetmantechsweetmantechAug 13, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid concern, deliberate non-fix: migrateRoom.ts backfills legacy rooms whose actual model is unknown. After recoupable/database#56, an omitted model_id inserts NULL — which for a legacy room is the honest value ("not recorded"), exactly the semantics the migration's column comment defines. Writing DEFAULT_CHAT_MODEL_ID there would fabricate provenance for historical rows, the precise failure chat#1956 exists to end. The invariant is better stated as: every writer of NEW chats records the model that will run; the backfill records what it knows, which is nothing.

});
if (!chat) {
const rolledBack = await deleteSessionById(session.id);
if (!rolledBack) {
Expand Down
Loading