Braintrust Review 2026: An Eval Platform Billed by the Gigabyte, With Your Own Data Plane Only on Enterprise
Braintrust is where a team runs evals on its LLM app and agents, compares experiments side by side, and scores production traces with the same scorers. It is closed-source SaaS with open-source SDKs: Starter is free, Pro costs $249 a month, and your own data plane needs Enterprise. The bill counts gigabytes sent and scores, not traces or seats. Below: costs at three workloads, why Starter with a card often beats Pro, what breaks first, the May 2026 key incident, and the contract.
There are no affiliate links in this review. Every Braintrust link goes straight to braintrust.dev, its documentation, its legal pages, its trust center or its GitHub organization.
| Made by | Braintrust Data, Inc., San Francisco; $80M Series B led by ICONIQ, Feb 17, 2026 |
| What it is | Evals, experiments, tracing, prompt playgrounds, an investigation agent (Loop) and an LLM gateway |
| Open source | SDKs (Apache-2.0) and the autoevals scorer library (MIT); the platform itself is closed |
| Plans | Starter $0; Pro $249 a month; Enterprise custom |
| Usage | $4 or $3 per GB of data sent, $2.50 or $1.50 per 1,000 scores, after 1 GB and 10k (Starter) or 5 GB and 50k (Pro) |
Braintrust is the most complete eval workflow in this group. Start on Starter with a card, move to Pro for retention and roles rather than price, and treat self-hosting as an Enterprise purchase.
“Discover patterns in production, turn them into evals, and improve quality with every release.” That is the home-page promise; the eval half is what the few developers who post about it single out.
The pricing is the part to understand before you sign up. Braintrust does not count traces, steps or seats; it counts the bytes you send (inputs, outputs, prompts, metadata, spans, datasets, attachments) and the scores you record. Its own calculator assumes about 96 KB per request. At that size, a five-person team sending 200,000 traces a month and scoring 20,000 of them pays about $97.80 on Starter with on-demand billing switched on, or $291.60 on Pro, against $130.40 on Langfuse Core and $1,145 on LangSmith Plus at the rates on their pricing pages the same day. Pro's lower usage rates only win on price beyond roughly 138 to 238 GB a month, so most teams buy Pro for 30-day retention, roles, environments and the click-through DPA, not to save money.
The catches are concrete. Starter keeps logs for 14 days and has only one permission group (Owner). Deleting data never lowers the bill, because both processed data and Pro's retention storage are measured when data arrives. There is no hard spending cap on Pro. BYOC and self-hosted data planes, audit logs, SAML SSO, retention automations and the BAA are Enterprise only, and a self-hosted data plane more than three months out of date is unsupported. Security has had a rough year. In May 2026 an attacker used a compromised employee credential to reach a backup holding customers' decrypted AI provider keys. Braintrust said org-level provider secrets were “potentially exposed”, told every customer to rotate them, and says its forensic analysis with CrowdStrike found the attacker's activity “limited to Anthropic API keys”; 30 customers later reported suspicious activity on stored keys. In July it published two high-severity advisories for its Python and TypeScript SDKs, which could copy provider API keys into trace metadata; fixed versions exist, but keys already stored stay stored. The lesson for buyers: keep long-lived provider keys out of Braintrust where you can. And the website Terms forbid publishing benchmarks or comparative tests of the service without permission.
Developers who post about it are positive about the eval workflow and lukewarm on whether they need a platform at all. There is no Trustpilot review and no independent star rating to quote.
Not a hands-on test: JAVIS has not run Braintrust, opened an account or paid for a plan. This review is built from Braintrust's own pages, legal documents, repositories and public evidence, all as of 24 Oct 2026; the full list is in the evidence section.
Your code sends traces and runs evals through the SDK; Braintrust stores them in a data plane and scores, compares and investigates them.
Braintrust splits into a control plane (the web app, sign-in, project names and settings), which Braintrust always runs, and a data plane (traces, experiment rows, datasets, prompts and completions), which runs in Braintrust's US or EU region, or in your own cloud on Enterprise. You instrument with SDKs for Python, TypeScript, Go, Ruby, C# and Java, framework integrations (LangChain, LangGraph, Pydantic AI, Agno and others), or OpenTelemetry. Evals combine a dataset, a task and scorers, from the open-source autoevals library, your own code or an LLM judge. On top sit online scoring of production logs, human review, dashboards, alerts, Topics (clustering logs into themes), Patterns and Loop.
Well funded, used by names you know, and installed about a quarter as often as Langfuse.
led by ICONIQ, 17 Feb 2026
braintrust, 24 Sep to 23 Oct 2026
braintrust, same window
GitHub, 24 Oct 2026, 06:28 UTC
Braintrust launched on Hacker News on 28 Sep 2023 (Show HN item 37692239, 8 points), founded by Ankur Goyal, who previously led AI at Figma; the post named Zapier, Coda and Replit as early partners. On 17 Feb 2026 the company announced an $80M Series B led by ICONIQ, with Andreessen Horowitz, Greylock, Elad Gil and basecase returning, and named Notion, Replit, Cloudflare, Ramp and Dropbox as customers. Axios Pro reported an $800M post-money valuation the same day; Braintrust's own post gives no valuation.
Downloads are the hardest adoption signal, with the usual warning that CI servers and dependency installs inflate them. Summing PyPI's daily counts (mirrors excluded) for 24 Sep to 23 Oct 2026 gives 7,426,240 for braintrust and 4,646,102 for autoevals, against 28,337,080 for langfuse, 85,110,347 for LangChain's langsmith (a dependency of LangChain itself) and 659,316 for arize-phoenix. On npm, braintrust had 6,121,584 downloads in the same window and autoevals 2,157,274.
GitHub tells a smaller story, because the platform is not open source. The braintrustdata organization's most-starred repository is autoevals (1,065 stars, 94 forks, MIT), followed by braintrust-proxy (411 stars), the open-source AI proxy that Braintrust's docs now mark as deprecated. The Python SDK repository (Apache-2.0) was created on 20 Feb 2026 and has 22 stars, but the braintrust package itself has been on PyPI since 24 May 2023, with 294 releases by 24 Oct 2026, so the low count reflects a young repository, not a young SDK; the TypeScript repository has 29. Stars measure attention, not customers.
autoevals repository on GitHub, 24 Oct 2026.No star rating exists. The few developers who post about it like the eval workflow and wonder whether they need a platform.
Trustpilot has an unclaimed profile for braintrust.dev with 0 reviews as of 24 Oct 2026 (the other “Braintrust” listed there, braintrust.io, is a different company). Braintrust has no mobile app, and no G2 or Product Hunt score is quoted here. Hacker News comment search for “braintrust” since 1 Jan 2025 shows 96 results on 24 Oct 2026 (02:02 UTC), and most are passing mentions in tool lists, job posts or other companies with the same name. The opinions below are the ones with substance, each one person's account with its comment id and date.
What people like
“I highly rate Braintrust.” Found datasets easy to set up and calls it a game changer for knowing whether prompts work, for LLM classification.
Refers to Braintrust in passing as “a really solid eval tool/platform”.
“We've been using Braintrust for evals at Zapier and it's been really great.” Zapier was named as a launch partner, so read it as a partner's view.
Mixed: useful, but is a platform needed?
The value came from running evals across models; “Braintrust makes this pretty easy, but If I was to do it again I would vibecode the same functionality.”
Lists alternatives as “Braintrust (closed), Promptfoo (open ...) and Laminar (open ...)”: closed source is the first thing some buyers note.
The founder of a competing tool says Braintrust takes the path of a “full development platform for LLMs”, against tools that stay in your IDE. A rival's framing.
What the issue trackers and advisories show
Braintrust's open-source SDKs carry the public paper trail. On 24 Oct 2026 the Python SDK had 51 open issues and 14 open pull requests (189 closed issues), and the TypeScript SDK 52 open issues and 24 open pull requests (251 closed), re-checked at 06:28 UTC. On 17 Jul 2026 (UTC) Braintrust published two advisories rated high (CVSS 7.7): its LiteLLM instrumentation in the Python SDK (versions 0.2.0 up to 0.28.0) and its Claude Agent SDK instrumentation in the TypeScript SDK (from 0.4.0, fixed in 3.23.1) could copy provider API keys passed in code into span metadata, where project members could see them. Braintrust's help article says it was an implementation bug, not a supply-chain compromise, and that upgrading does not remove keys already stored.
Read together: a handful of vocal developers like how easy evals are and object that the platform is closed; it is not a satisfaction survey.
Free to start, $249 a month for Pro, and every plan metered by gigabytes, scores and model credits.
Starter
No card required
- 1 GB data, then $4/GB
- 10k scores, then $2.50 per 1,000
- 14-day log retention
- $10 model credits
- Owner role only
Pro
Unlimited users
- 5 GB data, then $3/GB
- 50k scores, then $1.50 per 1,000
- 30-day retention, extendable to 180
- $100 model credits
- Roles, environments, custom dashboards
Enterprise
Annual invoice
- BYOC or self-hosted data plane
- SAML/OIDC SSO, audit logs
- Retention automations, S3 export
- SOC 2 report, BAA, uptime SLA
| Limit | Starter | Pro | Enterprise |
|---|---|---|---|
| Users, projects | Unlimited | Unlimited | Unlimited |
| Log retention | 14 days | 30 days; paid to 180 | To 365 |
| Experiments kept | To 365 days | To 365 days | To 365 days |
| Roles | Owner only | 4 built-in | Custom |
| Review scorers | 1 per project | Unlimited | Unlimited |
| Single sign-on | SAML/OIDC | ||
| Gateway built-in models | No | Yes | Yes |
| Data plane | US or EU | US or EU | + BYOC, self-host |
| DPA / BAA | No / No | Yes / No | Custom / Yes |
How the price has moved
The Pro fee has not moved: Internet Archive copies of braintrust.dev/pricing show $249 a month in July 2025, January 2026, March 2026, May 2026 and September 2026. What changed is the metering around it. In July 2025 the free plan offered 1 million trace spans and five users; by January 2026 users were unlimited; by May 2026 spans had disappeared as a metric and Starter gained on-demand overage at $4/GB and $2.50 per 1,000 scores; by the September 2026 copy the cards showed model credits ($10 and $100) and Pro's extra retention at $0.50 per GB per month, where July 2025 had said “$3/GB retained”. Features also moved between tiers: the billing FAQ says organizations that signed up before 16 Mar 2026 are on “legacy plans” that keep features such as custom permission groups, export automations and custom charts.
Two page-level contradictions to know. The plan cards say Pro has unlimited users, while the pricing FAQ calls Pro “Best suited for small teams of up to 5 people”; we read that as advice, not a cap. And the startup programme (6 to 12 months of free Pro) needs at least $100K raised on the pricing FAQ, plus “Series A or earlier” in the docs.
Not an affiliate link. It opens Braintrust's official website.
Three workloads, priced on the same assumptions: Braintrust is cheap when traces are small, and Pro is rarely the cheaper plan.
Everything below is arithmetic on the published rates with invented but plausible traffic. The unknown that matters most is how many bytes each trace carries. We use Braintrust's own calculator rule (“8,000 tokens per request”, “4 bytes per token and an overhead of 3x”), which works out to 96,000 bytes a trace, and show a light 24 KB and a deep 384 KB reading where it changes the answer. We count 1 GB as 1,000,000,000 bytes; the calculator's own 0.73 GB for 65 million tokens matches a 1,073,741,824-byte gigabyte, which would make every Braintrust figure here about 7% lower. Langfuse and LangSmith are priced from their pricing pages the same day, using the traffic assumptions of our Langfuse review.
8,000 tokens × 4 bytes × 3
200k traces, worked example 2
same traffic
a month, depending on scores
Worked example 1: a solo developer
A chatbot with 15,000 traces a month (four steps each) and one LLM judge on 20% of them, so 3,000 scores. At 96 KB a trace that is 1.44 GB. Starter includes 1 GB, so with a card on file the overage is 0.44 × $4 = $1.76; the scores fit inside 10,000. Without a card, Braintrust's change-plan page says Starter limits are enforced and “usage above those limits will be blocked”, so plan to add a card (Braintrust charges a $1 validation fee) or trim payloads. At 24 KB a trace the month costs $0. The same traffic is 78,000 units on Langfuse, over its free 50,000, so Core at $29; on LangSmith Developer it is 10,000 traces over the free 5,000 × $0.005 = $50.
Worked example 2: a five-person team in production
An agent with 200,000 traces a month (six steps each) and a judge on 10% of them: 20,000 scores. At 96 KB that is 19.2 GB. Starter with on-demand billing: 18.2 GB × $4 = $72.80, plus 10,000 extra scores × $2.50 per 1,000 = $25, so $97.80. Pro: $249 + 14.2 GB × $3 = $42.60, scores inside 50,000, so $291.60. At 24 KB a trace (4.8 GB) the bills are $40.20 and $249. Langfuse Core charges $130.40 for the same traffic (1,420,000 units); LangSmith Plus with five seats charges $195 + 190,000 × $0.005 = $1,145, reading its “includes 10K traces total” as one allowance per organization, or $945 if each seat brought 10,000. Starter is cheapest here, but it keeps logs for 14 days and every member is an Owner.
Worked example 3: a heavy month with deep agents
500,000 traces with 20 steps each and 50,000 scores. At 96 KB that is 48 GB: Starter on demand $288 (47 × $4 + 40,000 scores × $2.50 per 1,000), Pro $378 ($249 + 43 × $3). Deep traces are where the metering matters: at 384 KB a trace (192 GB) Pro costs $810 and Starter $864. Langfuse Core charges $766.75 for this traffic because it counts every step (10,550,000 units), and LangSmith Plus $2,645 ($2,445 per-seat reading). So Braintrust is cheaper than Langfuse for deep but compact traces, and dearer once traces pass about 355 KB each.
What decides your bill
Bytes per trace, not traces. A retrieval step that logs ten full documents, a tool that returns a web page, or an image attachment can multiply the size of a trace without adding a step. The chart below holds the heavy workload fixed and varies only the trace size.
When Pro pays for itself, and when it does not
Pro's fee buys lower rates ($3 against $4 per GB, $1.50 against $2.50 per 1,000 scores) and bigger allowances. Setting Starter's on-demand bill equal to Pro's gives a break-even of 238 GB a month if you record no more than 10,000 scores, 213 GB at 20,000 scores and 138 GB at 50,000. Below that, Starter with a card is cheaper. Braintrust's own docs say Pro has “better economics as your usage scales”, which is true only past those volumes. The reasons to pay $249 are the features: 30-day retention (or up to 180 days), the Engineer and Viewer roles instead of everyone being an Owner, environments, custom dashboards, sandbox evals, built-in models through the Gateway, the click-through DPA, and 50 concurrent Loop threads.
Retention, credits and the bills that do not shrink
Keeping logs longer on Pro costs $0.50 per GB per month, and the billing FAQ says retention storage “bills the data you ingest into each month you keep beyond the base window, not what's currently stored”. Our reading: the team in example 2 extending to 90 days keeps each month's 19.2 GB for two extra months, so about 2 × 19.2 × $0.50 = $19.20 a month once steady. Deleting data, a retention policy or a shorter window reduce neither processed data nor retention storage; “The only way to lower retention storage is to ingest less.” Model credits run out separately: on Starter without a card, Topics and built-in models pause when the $10 is spent. There is no hard spending cap: spend alerts (up to three thresholds, by email or Slack) “do not interrupt or limit your usage”.
Keeping your traces in your own cloud is possible, but it is an Enterprise contract and a serious piece of infrastructure.
Braintrust's deployment docs offer four models. SaaS (any plan): Braintrust runs both planes in its US or EU region. BYOC (Enterprise): Braintrust runs the data plane inside your cloud account. Self-hosted (Enterprise): you run the data plane from Braintrust's Terraform modules (AWS on ECS and EC2; GCP and Azure on Kubernetes with Helm). Non-standard self-hosted: anything that cannot use those artifacts, by review only. In every case the control plane, with project names, settings, hashed API keys, org-level provider secrets and login details, stays with Braintrust; the SDK and your browser talk to your data plane directly.
What a production self-hosted data plane asks for
The hardware guide is sized for production, not a laptop: PostgreSQL with 8+ vCPUs, 64 GB+ of RAM and 1,000 GB+ of storage at 15,000+ IOPS; Redis with 4 GB+ on a non-burstable instance; Brainstore readers (at least two) and fast readers (two more by default on recent modules) at 16 vCPUs and 32 GB each, and a writer at 32 vCPUs and 64 GB, all on NVMe storage with at least 150,000 IOPS (the docs say not to use EBS volumes). That is at least five Brainstore nodes plus PostgreSQL and Redis before the API service. Braintrust ships data plane releases one to two times a month, and its support table says a data plane “More than 3 months out of date” is “Unsupported”. By default the data plane sends health checks, system metrics and billing usage back to Braintrust's control plane.
Match your situation to one line.
Experiments are the heart of it; the 2026 additions point an agent at your traces.
An eval in Braintrust is three things: data (a dataset, or rows pulled from production logs), a task (your prompt, model or agent) and scores. The SDK's Eval() runs them and records an experiment; the UI compares experiments row by row and score by score, flags regressions, and lets you group by input or filter to what changed. Scorers can come from autoevals (an MIT-licensed library of LLM-judge and heuristic scorers in Python and TypeScript), from your own code, or from human review. The same scorers can run online on a sample of production logs. A 15 Oct 2026 blog post describes “workflow evals”, which send eval tasks and judges through OpenAI and Anthropic batch APIs at those providers' 50% batch discount, with results that can take up to 24 hours.
Loop, Patterns and Topics
Loop is Braintrust's built-in agent: you ask it in plain language to investigate logs and experiments, and it can create or edit prompts, scorers, datasets, dashboards and automations, with your approval for changes. It runs on built-in models billed against your model credits. Patterns uses Loop on a schedule to record recurring problems with evidence and a suggested fix; the docs label it public preview. Topics clusters logs into themes and is billed at $0.06 per million input tokens and $0.40 per million output tokens after credits. Loop can also read connected GitHub repositories (public preview); write access to push fixes is opt-in. These features are new, and none of their output has been verified by JAVIS.
Measure bytes first, then build the eval loop.
braintrust 0.28.0 or later and TypeScript 3.23.1 or later, the versions that fix the credential advisories; pass provider keys through environment variables, not call arguments.autoevals or code scorers, and run experiments in CI.An illustrative workflow built from Braintrust's documentation, 24 Oct 2026, not a record of a JAVIS run.
Retention, the free plan's hard limits, old SDKs and a self-hosted data plane that falls behind.
Nothing in Braintrust has a forced cutover date as of 24 Oct 2026. What fails is quieter: data ages out, limits block ingestion, and preview features change. Here is what teams hit, in roughly the order they hit it.
| Trigger | What happens | Fix |
|---|---|---|
| Starter without a card passes 1 GB or 10k scores | Starter limits are enforced and usage above them is blocked, per the change-plan docs; alerts arrive at 80%, 90% and 100% | Add a card for on-demand billing, or trim payloads |
| Debugging something older than 14 days on Starter (30 on Pro) | Logs outside the window are not shown and not returned by SQL or the API | Pro, or extend Pro retention at $0.50/GB/month |
| Dataset rows that reference a full trace | The row stores a reference, so it depends on the trace staying inside the retention window (July 2026 changelog) | Copy the span instead, or extend retention |
| Model credits run out on Starter | Topics, built-in models and Loop automations pause until next month or until a card is added | Card, Pro, or your own provider keys |
| Calling a built-in model through the Gateway on Starter | Refused with “Native inference requires a Pro or Enterprise plan” | Pro, or a provider key |
| Org-level AI provider keys stored in Braintrust | “Potentially exposed” in the May 2026 incident; Braintrust says the attacker’s activity was “limited to Anthropic API keys contained in the migration backup”; it asked every customer to delete and rotate keys | Federated identity for Anthropic, assume-role for Bedrock, provider-side spend limits |
| Python SDK below 0.28.0 with LiteLLM, or TypeScript below 3.23.1 with the Claude Agent SDK | Keys passed in call arguments may already sit in span metadata; upgrading does not remove them | Upgrade, run the advisory's detection query, rotate keys |
| Code still on the old AI proxy | Deprecated and “will no longer be regularly maintained”; hosted calls already route to the Gateway | Migrate to the Gateway, which is itself public preview with no SLA |
| Downgrading an account opened before 16 Mar 2026 | Legacy status is lost permanently, with it the grandfathered features | Check what you use before downgrading |
| Self-hosted data plane more than 3 months old | Unsupported; new features stay gated until you upgrade | Upgrade monthly or at least quarterly |
| Self-hosted on GCP or Azure moving to data plane v2.17.0 | Python scorers move from 3.11 to 3.13; bundled compiled dependencies fail to import | Rebuild and push scorers with Python 3.13 |
| AWS Terraform module v6.0 to v6.4.0 with the ECS cutover | LLM calls from user-authored scorers and tools went to the hosted Gateway, outside your AWS account | Module v6.5.0 or later (v6.10.0 is current) |
| AWS Terraform module v6.10.0 | Loop sandboxes lose internet egress; the Loop runtime's default maximum capacity rises from 4 to 50 | Remove the egress setting; pin capacity if it bounds your cost |
Upgrade safety checklist
- Pin SDK versions at or above Python 0.28.0 and TypeScript 3.23.1; the latest on 24 Oct 2026 are 0.48.0 on PyPI and 3.39.0 on npm (both 21 Oct 2026).
- Search existing traces for leaked keys with the query in the advisories, then rotate any key found.
- Export anything you must keep before it ages out of 14 or 30 days; experiments last up to 365 days, logs do not.
- Move proxy URLs to the Gateway and decide whether a preview service without an SLA can sit in your request path.
- Set spend alerts at, say, 50%, 80% and 100% of budget; they warn but never stop usage.
- Upgrade one major Terraform module version at a time (v4 to v5 to v6); go to v6.5.0 or later, never v6.0.0.
- Carry custom Lambda environment variables into the ECS variables before the first v6 apply, and review
terraform planfor any database or bucket deletion. - Before v6.10.0, remove internet egress settings for Loop sandboxes and set
loop_runtime_max_capacityif you relied on the old limit of 4. - On GCP and Azure, rebuild Python scorers for 3.13 before data plane v2.17.0.
- Allow outbound traffic to
pulse.braintrust.devbefore data plane v2.20.0, which sends telemetry there.
Compiled from Braintrust's self-hosting upgrade guides, self-hosting releases page, billing docs and advisories, 24 Oct 2026. A checklist from the docs, not a JAVIS migration run.
Six SDK languages and OpenTelemetry in; a hosted MCP server, a SQL interface and a preview Gateway out.
Native SDKs cover Python, TypeScript, Go, Ruby, C# and Java, with integrations for OpenAI, Anthropic, LangChain, LangGraph, Pydantic AI, Agno, Spring AI, Cloudflare and others, plus OpenTelemetry ingestion that maps OpenInference attributes. Coding-agent sessions from Claude Code, Codex, Cursor and similar tools can be traced too. The hosted MCP server lets Claude Code, Cursor, Codex, VS Code and other MCP clients query logs, author scorers, run evals and edit datasets; in the EU region it lives at api-eu.braintrust.dev/mcp. Data can be queried with SQL; queries time out after 30 seconds, and long-running async queries are Enterprise only. Browser automation is not a Braintrust feature; it traces what your agents do.
The Gateway gives one OpenAI-compatible endpoint for many providers, with caching and logging. The hosted Gateway is “in public preview and free to use”, and Braintrust “does not provide a service-level agreement (SLA)” for it; the status page shows the AI Gateway at 99.44% uptime for July to October 2026. Limits to plan around: 10,000 function executions per 10 seconds and 20 MB per span.
Audited, regional and contractually barred from training on your data, but stored provider keys were exposed in May 2026 and the strongest controls are Enterprise only.
Braintrust's trust center lists SOC 2 Type II reports for 5 May to 5 Aug 2025 and 6 Aug 2025 to 31 Jul 2026 (the newer one announced on 24 Sep 2026), plus web application and API penetration test reports added on 13 Aug 2026; all are behind an access request. The docs list SOC 2 attestation (the report) as an Enterprise feature. The home page says HIPAA compliant, but the BAA you need for health data is Enterprise only. Hosted organizations pick a US or EU data plane at creation; the Terms say the region “cannot be changed after an organization is created”.
report on request
CVSS 7.7, 17 Jul 2026
self-serve MSA section 8
activity limited to Anthropic keys
The May 2026 incident
Braintrust disclosed the incident on its trust center on 5 May 2026 and closed it out on 30 May. Its own timeline: suspicious behavior was flagged on Saturday 2 May; on 3 May the team found unauthorized access to “a temporary EC2 instance within one internal AWS account”; on 4 May it confirmed impact to one customer; on 5 May it rotated secrets and told customers to delete and rotate AI provider keys; from 7 May more customers reported suspicious activity. The conclusion says “An employee credential compromise” led to the instance used during a control-plane database migration, where AI provider credentials were “temporarily exposed while being moved between storage systems”, and that the attacker accessed a backup “which contained decrypted AI provider credentials”. Working with CrowdStrike, Braintrust says it found the attacker's activity “limited to Anthropic API keys”, with no wider access to customer data; one customer was identified at first and “30 customers reported suspicious activity tied to AI provider keys stored in Braintrust”. TechCrunch reported the incident on 6 May 2026, quoting a spokesperson who then said there was “no evidence of a breach at this time”.
Since then Braintrust has shipped Anthropic workload identity federation and AWS Bedrock assume-role authentication, so those two providers can be used without storing long-lived keys, and shows when each key was last configured. One architectural point is worth knowing even if you self-host: Braintrust's data-location table puts organization-level LLM provider secrets (encrypted) in the control plane, which Braintrust always runs, and project-level secrets in the data plane. Our reading is that a self-hosted customer who wants provider keys to stay in its own account should use project-level providers or federated identity; confirm that with Braintrust before relying on it.
The SDK advisories
The two advisories are the specific risk to check. GHSA-xwg7-7579-vg67 (CVE-2026-85276) covers the Python braintrust package from 0.2.0 until 0.28.0: its LiteLLM wrappers recorded most keyword arguments, including api_key, Azure tokens, AWS keys and Vertex credentials, in span metadata. GHSA-h8xp-g5cq-35xw (CVE-2026-85273) covers the npm braintrust package from 0.4.0 (released 2 Oct 2025) until 3.23.1, for keys passed to the Claude Agent SDK's query() options. Braintrust's help article on the LiteLLM case says keys loaded only from environment variables were not captured; the TypeScript advisory says users with access to an affected project “may have been able to view” stored keys.
Incident-readiness checklist for Braintrust customers
- Rotate org-level AI provider keys: open the org-level AI provider settings, “delete or revoke the existing secret”, add a new one and confirm the settings show it was configured today (trust center, 6 May 2026).
- Where supported, stop storing long-lived keys: Braintrust now offers Anthropic workload identity federation and AWS Bedrock assume-role (trust center, 30 May 2026).
- Check SDK versions against the two advisories: Python
braintrustbelow 0.28.0 with LiteLLM instrumentation, npmbraintrustbelow 3.23.1 with the Claude Agent SDK wrapper. Upgrade, run the trace search query in each advisory, and rotate any key it finds. - If you cannot upgrade at once, both advisories offer three workarounds: supply provider keys through the environment instead of call arguments, set a masking function before the Braintrust logger starts (in Python from 0.2.5), or disable the affected instrumentation (LiteLLM in Python, the Claude Agent SDK wrapper in TypeScript) until you upgrade.
- Subscribe to updates on trust.braintrust.dev; Braintrust says it now uses a centralized customer security distribution list for incidents.
- Scope keys narrowly: a separate provider key per Braintrust project or environment, with spend limits set at the provider, so one leak has a ceiling.
- Prefer project-level providers if you self-host; the docs place project-level secrets in your data plane and org-level ones in Braintrust's control plane.
- Decide what never gets logged: remove credentials and personal data before upload with the SDK masking function. Braintrust's ingestion redaction is in private preview, and its docs say that for EU-region organizations the text it scans is sent to a model endpoint outside the EU.
- Watch provider usage dashboards for spikes; in May, customers spotted the problem through unusual AI provider usage.
Vendor steps from trust.braintrust.dev (6, 8 and 30 May 2026), GitHub advisories GHSA-xwg7-7579-vg67 and GHSA-h8xp-g5cq-35xw, and the “Protect sensitive data” docs, 24 Oct 2026. A checklist, not a JAVIS security audit.
California law, no refunds, no training on your data, and two documents that disagree on cancellation.
Self-serve buyers deal with Braintrust Data, Inc. under the website Terms of Service (last updated 7 Jul 2026) and, for Pro bought online, a “Terms of Service (Self-Serve MSA)” that covers “Braintrust Pro Services ordered online”; a DPA applies on request. The points a buyer should know, quoted or paraphrased closely:
The MSA says Braintrust “will not use any Customer Data to train, develop, or improve artificial intelligence or machine learning models”. Usage metrics that do not identify you belong to Braintrust. Data “may be irretrievably deleted” if the account is 90 days or more delinquent.
The MSA runs month to month until either side gives 30 days' notice. The Terms say subscriptions auto-renew and you “must cancel your subscription 30 days before the subscription period renewal date” by emailing support; the docs instead describe an in-app downgrade at the end of the billing period.
None: the Terms say no refunds and “no credits for partially used periods”; the MSA says all fees are non-refundable. Help exporting your data is billable.
MSA: capped at fees paid or payable in the prior 12 months. Terms: the greater of $100 or what you paid in the past six months.
California law. The Terms add binding individual arbitration and a class-action waiver, with an opt-out; the MSA names courts in San Francisco.
The Terms bar performing or disclosing “any benchmark, comparative test, performance or availability test, or evaluation of the Services” without permission. The MSA lets Braintrust use your name and logo in marketing.
The DPA gives you 10 days to object to a new subprocessor; its list names AWS (US or EU), Vercel, Supabase, Clerk, Cloudflare, Datadog, Sentry and Upstash (US) and Baseten (US or EU), and says the control plane may transfer to the US. The privacy notice linked from every page is dated 21 Sep 2023 and still refers to braintrustdata.com; it does not cover data you upload.
What to ask before you sign
| Ask Braintrust | Why, and where it comes from |
|---|---|
| Which data region, and can it change later? | Terms 2.6: the region “cannot be changed after an organization is created”; the Gateway is global and separate from it (organizations docs). |
| Where do our provider keys live, and can we avoid storing them? | Docs put org-level secrets in the control plane; the 30 May 2026 post lists federated options only for Anthropic and Bedrock. |
| A breach-notice deadline in hours | The DPA says “without undue delay” and gives no hour count. |
| The current subprocessor list | The DPA exhibit names nine; the trust center added OpenAI, Anthropic, Google (4 Aug) and TypeSafe AI (15 Oct 2026). |
| Does the uptime SLA cover the Gateway? | The SLA is Enterprise only; Braintrust “does not provide a service-level agreement (SLA) for the hosted Gateway”, which ran 99.44% in Jul to Oct 2026 (Gateway docs, status page). |
| Retention, export and deletion terms | Retention automations and S3 export are Enterprise only; the MSA bills export help and allows deletion after 90 days delinquent. |
| Which cancellation route applies | Terms 2.3: email 30 days before renewal; billing docs: in-app downgrade at period end. |
| Permission to publish internal comparisons | Terms 3.2(j) bar disclosing benchmarks or comparative tests without permission. |
| If self-hosting: telemetry, upgrades, access | Telemetry is on by default; data planes over 3 months old are unsupported; the docs say customers who grant Braintrust remote access have experienced much faster resolutions (self-hosting docs). |
We have read these pages as a buyer would, not as lawyers. Have your counsel review the order form, MSA, DPA and any BAA before tracing personal, health or confidential data.
Teams that live in evals and experiments, send compact traces, and can afford Enterprise if they need their own data plane.
- Product and ML teams running offline evals and comparing models every week
- Teams with many people who should see results: unlimited users on every plan
- Agents with many small steps, where per-step pricing elsewhere adds up
- Companies wanting an EU data region on a self-serve plan
- Buyers who want a written promise of no training on their data
- Teams that require open source or cheap self-hosting
- Apps whose traces carry large documents, images or tool outputs
- Small teams needing more than 14 days of logs without paying $249
- Health-data workloads without an Enterprise budget (BAA)
- Anyone needing a hard spend cap above the free tier
- Teams that cannot accept storing provider keys with a vendor after the May 2026 incident
Nine things to check before you rely on it.
The bill follows payload size; one verbose tool can double it.
14 days on Starter, 30 on Pro; longer costs extra or needs Enterprise.
Alerts only; deleting data does not lower the bill.
BYOC, self-hosting, SAML, audit logs, retention automations, BAA.
Only SDKs and autoevals are open source.
Patterns, Loop's GitHub link and the hosted Gateway are public preview.
Stored keys potentially exposed in May (activity limited to Anthropic keys in a backup, per Braintrust); two SDK advisories in July.
Cancellation route, user guidance, subprocessor lists and a 2023 privacy notice disagree.
No Trustpilot reviews; few Hacker News opinions. JAVIS did not test it hands-on.
Priced from each vendor's page on 24 Oct 2026; the units differ, so the plans are not like-for-like.
| Product and when to pick it | What the pricing page says | JAVIS review |
|---|---|---|
| Braintrust Eval-first teams, compact traces | Starter $0 (1 GB, 10k scores, 14 days); Pro $249 a month (5 GB then $3/GB, 50k scores then $1.50 per 1,000, 30 days); Enterprise custom | This article |
| Langfuse Open source, self-host free | Hobby $0 (50k units); Core $29 (100k units, then $8 per 100k, lower with volume); Pro $199; Enterprise $2,499; self-host free under MIT | Langfuse review |
| LangSmith LangChain and LangGraph users | Developer $0 (1 seat, 5k base traces); Plus $39 a seat (10k base traces); then $0.005 per base trace | LangChain review |
| Arize AX Observability with the open-source Phoenix | AX Free (25k spans, 1 GB, 15 days); AX Pro $50 a month (50k spans, 10 GB, 30 days); Enterprise adds self-hosting; overage rates not on the plan page | Not reviewed yet |
| Helicone Gateway-style request logging | Hobby free (10,000 requests, 1 GB); Pro $79 a month plus usage; Team $799 plus usage | Not reviewed yet |
The comparison that matters most is with Langfuse. Langfuse is open source, self-hostable for free and cheaper for shallow or bulky traces; Braintrust has the more developed experiment and eval tooling, the Loop agent, and is cheaper for deep agents with small steps. LangSmith costs the most in all three of our examples because it charges per seat and per trace; it is the natural pick inside the LangChain stack. Arize AX and Helicone price spans and requests, and their overage rates are not on their plan pages, so we did not compute them.
Ownership is moving: ClickHouse bought Langfuse in January 2026, and Helicone's pricing page says “Helicone is no longer part of Mintlify.” If you are still choosing what to build with, our reviews of Langflow and Haystack cover two frameworks whose pipelines you would evaluate here.
This review has no affiliate links, and Braintrust did not see it.
JAVIS has no approved Braintrust affiliate link, so every Braintrust link here goes straight to braintrust.dev, its documentation, its GitHub organization, its trust center or its legal pages (Terms, Self-Serve MSA, DPA), without tracking parameters.
Hands-on status: not tested hands-on, as stated in the JAVIS take; sources are listed in the evidence section. Independent review evidence is limited, as described above.
Starter is free without a card; start with the evaluation quickstart in the docs.
Open the Braintrust docsMeasure a week of gigabytes and scores, then compare Starter with a card against Pro.
Open the pricing pageBuilding or evaluating LLM apps? These reviews cover the nearest ground.
Each card opens a published JAVIS review.
Questions people ask before committing
Is Braintrust free?
The Starter plan is free with no card: 1 GB of processed data, 10,000 scores, 14-day log retention, $10 of model credits and unlimited users a month. Above that you add a card for on-demand billing ($4 per GB, $2.50 per 1,000 scores) or upgrade.
Can I self-host Braintrust?
Only on Enterprise. You can run the data plane in your own AWS, GCP or Azure account from Braintrust's Terraform modules, or have Braintrust run it there (BYOC); the control plane always stays with Braintrust.
Does Braintrust train on my data?
The self-serve MSA says Braintrust will not use customer data to train, develop or improve AI or machine learning models. Built-in AI features such as Loop and Topics send selected data to the subprocessors that run those models.
Has Braintrust had a security incident?
Yes. In May 2026 an attacker reached a backup with customers' decrypted AI provider keys after an employee credential was compromised; Braintrust says the activity was limited to Anthropic API keys, that 30 customers reported suspicious key activity, and that it asked all customers to rotate stored keys. Separately, two SDK advisories in July 2026 covered keys copied into trace metadata.
Is Braintrust better than Langfuse?
For eval-heavy teams with compact traces, often yes; for open source, free self-hosting or bulky traces, Langfuse. See the cost section and our Langfuse review.
Where this came from, and when.
Every price, limit and date in this review comes from the pages below, all as of 24 Oct 2026. Expand for the full list, the scoring method and what the review does not cover.
Show sources, scoring method and scope
Pricing: braintrust.dev/pricing (cards, comparison table, FAQ and usage calculator with its calculation note), the plans-and-limits docs, the billing overview, FAQ, change-plan and monitor-usage docs, and the data-retention docs, 24 Oct 2026. Price history from Internet Archive copies of braintrust.dev/pricing dated 1 Jul 2025, 31 Jan 2026, 3 Mar 2026, 1 May 2026 and 14 Sep 2026, and the May and July 2026 changelog entries.
Product: Braintrust home page, docs on deployment options, organizations and regions, self-hosting (overview, hardware, upgrades, AWS module v6), Gateway, AI proxy, Loop, Patterns, Topics, MCP, autoevals and experiments, the product changelog and the self-hosting releases page (latest data plane v2.18.0), the September 2026 blog post on Loop, Patterns and Debugger, and the October 2026 workflow-evals post, all as of 24 Oct 2026.
Security and legal: trust.braintrust.dev (SOC 2 Type II periods, updates of 4 Aug, 13 Aug, 24 Sep and 15 Oct 2026), status.braintrust.dev (July to October 2026), the Terms of Service (last updated 7 Jul 2026), the Self-Serve MSA, the DPA and the privacy notice (21 Sep 2023), GitHub advisories GHSA-xwg7-7579-vg67 and GHSA-h8xp-g5cq-35xw (published 17 Jul 2026 UTC), the trust-center incident posts of 6 May, 8 May and 30 May 2026, and TechCrunch's report of 6 May 2026 (Lorenzo Franceschi-Bicchierai), 24 Oct 2026.
Repository and market: the braintrustdata GitHub organization on 24 Oct 2026, read at 01:57 UTC and re-checked at 06:28 UTC: autoevals 1,065 stars and 94 forks; braintrust-proxy 411 stars; Python SDK 22 stars, 51 open issues, 14 open pull requests; TypeScript SDK 29 stars, 52 open issues (51 at the first reading), 24 open pull requests. PyPI daily counts without mirrors summed for 24 Sep to 23 Oct 2026; npm counts for the same window; latest versions braintrust 0.48.0 (PyPI) and 3.39.0 (npm), both 21 Oct 2026. Series B announcement on the Braintrust blog (17 Feb 2026); valuation as reported by Axios Pro.
Independent evidence: Hacker News story 37692239 (28 Sep 2023); comments 38351007 (20 Nov 2023), 40987269 (17 Jul 2024), 44456615 (3 Jul 2025), 46557487 (9 Jan 2026), 47333821 (11 Mar 2026), 47409778 (17 Mar 2026) and 48813080 (7 Jul 2026). Story 48042494 (6 May 2026, 4 points) links the TechCrunch incident report. Comment search for “braintrust” since 1 Jan 2025: 96 results on 24 Oct 2026, 02:02 UTC. Trustpilot: unclaimed profile for braintrust.dev with 0 reviews, 24 Oct 2026. G2 and Product Hunt scores are not quoted.
Images: thirty-two screenshots of public pages taken on 24 Oct 2026 (Braintrust pricing, docs, legal, trust center and status pages, three GitHub pages and Trustpilot), and eight official images from Braintrust's docs and blog, each labelled as vendor material. Related cards use JAVIS's own featured images.
Video: two videos from the official Braintrust YouTube channel (@BraintrustData).
Competitors: Langfuse (including its pricing-calculator source for graduated tiers), LangSmith, Arize and Helicone pricing pages, 24 Oct 2026. Cost examples are arithmetic on published rates with invented traffic; trace sizes follow Braintrust's calculator rule with light and deep variants.
Store ratings: none; Braintrust has no mobile app.
JAVIS Fit Score (72/100): an editorial composite scored out of 20 on five criteria. Price clarity 15: public rates and a calculator, but a byte-based meter that is hard to forecast, tier changes in March 2026 and unpublished Enterprise prices. Capability breadth 18: experiments, scorers, online scoring, human review, Loop, Patterns, Topics, Gateway, MCP and six SDK languages, with several features in preview. Independent evidence 11: no star rating, few Hacker News opinions, a closed platform and two high-severity SDK advisories in 2026. Security and data control 13: SOC 2 Type II, US or EU regions, a no-training clause and BYOC, but the May 2026 incident in which stored AI provider keys were potentially exposed (Braintrust says the attacker’s activity was limited to Anthropic API keys in a migration backup), two SDK advisories, and SAML, audit logs, retention automations and the BAA only on Enterprise. Cost control and fit 15: cheap for compact traces, a free tier that blocks rather than bills, but alerts instead of caps on paid usage. It is judgment, not a measurement.
What this review does not cover: real ingestion speed or query performance, how many bytes your own traces carry, the quality of Loop's and Patterns' findings, Enterprise prices and contract terms, the gated SOC 2 and penetration-test reports, the forensic report behind the May 2026 incident, and a technical security review.
