Uh oh!
There was an error while loading. Please reload this page.
feat(prompt): --environment flag - #330
Conversation
Latest downloadable build artifacts for this PR commit
Available artifact names
|
7d5482d to
dae65e7Comparea65b244 to
5399912CompareThere was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:1a1b66369d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| project_id: &str, | ||
| prompt_id: &str, | ||
| ) -> Result<Vec<String>> { | ||
| let body = prompt_versions_request(project_id, prompt_id); |
There was a problem hiding this comment.
are we properly paginating here?
There was a problem hiding this comment.
No.
(lines numbers from the branch cedric/endpoint-check-reject-unknown-slug for braintrust)
Audit logs don't support pagination:braintrust/api-ts/src/btql.ts:1610:
if (body.audit_log) {
// Audit log queries do not support pagination.
boundQuery.limit = undefined;
boundFilter.limit = undefined;
}
And cursors are not supported braintrust/api-ts/src/btql.ts:1971:
if (!useBrainstore && body.audit_log) {
adjustBoundQueryForAuditLog({
boundQuery: boundFilter,
objectType: objectType,
testingOnlyAllowQueryFullAuditLog:
body._testing_only_allow_query_full_audit_log,
});
if (cursorParams) {
throw new BadRequestError("Audit log queries do not support cursors");
}
}
Btw the python SDK doesn't paginate either when querying audit logs braintrust-sdk-python/py/src/braintrust/logger.py:6146:
resp = _state.api_conn().post(
"btql",
json={
"query": query,
"audit_log": True,
"use_columnstore": False,
"brainstore_realtime": True,
},
headers={"Accept-Encoding": "gzip"},
)
.... all according to the slop
There was a problem hiding this comment.
I don't think this code branch is related to audit logs in any way right? I was more asking about pagination for prompts list. Why are audit logs relevant?
There was a problem hiding this comment.
The llms are really convinced that to fetch the specific version of a prompt, you need audit logs:
### 1. Where bt requests audit logs src/prompts/api.rs list_prompt_versions posts the request to /btql: ```rust client.post_with_headers("/btql", &body, &headers).await?; prompt_versions_request enables audit mode:
"audit_log": true,The response is treated as mutation history by filtering:
row.pointer("/audit_data/action")for upsert and merge.
2. Where the backend receives it
The /btql route is registered in:
api/app.py
@app.route("/btql", methods=["OPTIONS", "POST"], **REQUEST_KWARGS)@js_apidefbtql(): passIt is processed by:
api-ts/src/btql.ts
exportasyncfunctionrunBtqlRequest(...){constparsed=requestSchema.safeParse(ctxData);// ... }The request schema defines the flag:
audit_log: z.boolean().optional(),3. Why audit logs are used to list versions
The backend switches from the normal object schema to the historical audit schema:
constbindSchema=body.audit_log ? useBrainstore ? BRAINSTORE_AUDIT_LOG_LOGICAL_SCHEMA : BRAINTRUST_AUDIT_LOG_LOGICAL_SCHEMA : BRAINTRUST_LOGICAL_SCHEMA;It similarly selects the audit table:
consttableType=body.audit_log ? "audit_log" : "main";A normal project_prompts(...) query returns prompt state, optionally at a known version. It does not provide an operation for enumerating
every version ID.
Audit rows contain each historical mutation’s _xact_id and audit_data.action, allowing bt to discover all upsert and merge versions.
There is no backend rule specifically requiring prompt versions to use audit logs. This is the current SDK/CLI design because the backend
lacks a dedicated prompt-version-listing endpoint.
There was a problem hiding this comment.
However I do agree that needing to use audit logs for this is confusing and probably bad design.
There was a problem hiding this comment.
Fixing this is a follow up though.
control which environment is shown or assigned to the prompts
4bb697c to
0d4b76aCompareUh oh!
There was an error while loading. Please reload this page.
Adds
bt prompts ...:versionssubcommand(un)assignsubcommands--environmentflag forlist/view/assign/unassignandbt functions view--versionflag forviewUsage in the examples bellow.
Prompt version ids have a long form (19 characters) and a short form (16 characters), both are valid when requested.
In case you were unaware of the existence of environments like me, they are in https://www.braintrust.dev/app/your-org/p/your-project/configuration/org/environments
Examples:
Details
https://linear.app/braintrustdata/issue/SDK-244/environment-tags-in-the-bt-cli
See also the backend PR, and the environment management pr.