LLM observability is the practice of recording every step of an LLM or agent run so you can replay a bad answer and find out why it went wrong. Tracing tells you what happened. Evaluation tells you whether the output was any good. Monitoring tells you when production is quietly getting worse. Most buying guides collapse those three jobs into a single feature grid, and that collapse is where teams waste money.
Two constraints matter more than the feature list. All 127 attributes in the OpenTelemetry GenAI attribute registry are marked Development and none is Stable. An LLM judge also needs calibration before you can trust its verdicts.
LLM observability is three jobs, and most lists conflate them
Tracing is a record of one execution. Evaluation is a score against a criterion. Monitoring is a continuous watch over live traffic. Buy for the one you are missing, not the one with the prettiest dashboard.
Tracing is the easiest of the three and the least differentiating. Every platform on my list captures prompts, responses, latency and token counts. What differs is the span model for agents: whether a tool call, a retrieval and a planning step are first-class spans you can filter on, or whether they are dumped into one opaque request blob.
Evaluation is where the real separation happens. A platform that does offline evaluation well lets you build a fixed dataset of real production traces, change a prompt, and re-run it. That is what catches a regression before a customer does. A platform that only does online evaluation is telling you about production after the fact, which is better than nothing and worse than a gate.
Monitoring is the alerting layer, and it is the one most likely to be missing entirely. It needs score distributions over time, not single scores, because a model that quietly gets 10% worse will not trip a per-request threshold.
All 127 attributes in the OpenTelemetry GenAI attribute registry are marked Development. Zero are Stable. There is no stable OpenTelemetry GenAI attribute to build a vendor-neutral contract on.
Pricing, standards and performance claims
Prices and features reflect vendor pages checked on 16 October 2026. The standards discussion uses the OpenTelemetry GenAI conventions repository and OpenInference documentation; judge-reliability findings are attributed to research papers.
No ingest, query-latency or dashboard tests were run for this comparison. Performance claims remain attributed to their vendors or named studies.
The open-standard question: are your traces actually portable?
Short answer: you can emit OpenTelemetry GenAI spans to most of these backends, but you should not treat that as a portability guarantee, because the conventions are still in development.
This is the gap that almost every comparison page skips. Here is what is actually true right now.
The GenAI semantic conventions were moved out of the main OpenTelemetry semantic conventions repository and now live in a standalone repository, open-telemetry/semantic-conventions-genai. That repository was created on 5 May 2026. It has no release tags, and its README still lists the Schema URL field as “TODO”. The main semconv site now serves a page that reads “Moved” for every GenAI page.
Inside that repository, the agent and framework span conventions cover create_agent, invoke_agent, invoke_workflow, plan, execute_tool, embeddings, retrieval, memory operations and agent skills. Every gen_ai.operation.name value and every gen_ai.provider.name value in the enum is marked Development. The only Stable item anywhere in that family is the _OTHER error fallback.
The second standard you will hear about is OpenInference, the Arize-originated convention implemented by Phoenix. Its Python package openinference-semantic-conventions was at version 0.1.41 on 1 October 2026, and the Arize-ai/openinference repository had been pushed on 16 October 2026. Phoenix is built on OpenTelemetry and powered by OpenInference instrumentation, and its documentation says traces work with existing tooling “without a proprietary format”.
So the honest summary is this: OpenInference is de facto at the 0.x level and OpenTelemetry GenAI is at Development. Neither is a contract. What that means practically is that you should instrument against the standard, keep your own thin layer of your own fields, and test your exit path before you need it.
What actually breaks in production
The failures that hurt most are the ones that do not throw an exception. They produce a 200 response and a plausible-looking answer.
Here are the ones I would instrument for, in rough order of how quietly they cause damage:
- Silent quality drift after a prompt or model change. Nothing errors. The answer is just worse. Detecting it requires score distributions over time, not per-request thresholds.
- Truncated output. A
max_tokenslimit or a streaming disconnect cuts the response mid-sentence. If your output is JSON, it fails to parse and you catch it. If it is prose, it ships. - Retrieval returning nothing useful. The agent gets an empty or off-topic context and answers from its own priors with total confidence.
- Tool calls with wrong arguments. The HTTP call succeeds. The ID was wrong. The agent confidently reports the wrong result.
- Token and cost explosion. A retry loop or a runaway agent that keeps calling tools until it hits a step limit. This is the failure that reaches your invoice rather than your customer.
- Refusal regressions. A safety update or a system prompt change quietly makes the model refuse valid requests. Nobody notices because refusals do not page anyone.
- Eval regressions that reach customers. The test suite exists, but it runs against synthetic inputs that no longer resemble production traffic.
The Dynatrace-commissioned State of SRE and Platform Engineering 2026 report, based on a survey of 919 senior IT leaders fielded between October 2025 and January 2026, puts a number on the gap: only 40% of platform engineers reported embedding observability across all deployment stages, while 37% named integrating with existing tools as their single biggest barrier. That is a vendor-commissioned study and I am treating it as directional, but the direction is consistent with everything else on this list.
Agent-specific tracing: spans per step, replay and determinism
For a multi-step agent, the question is not “can I see the trace” but “can I see which step failed and re-run just that step.”
The OpenTelemetry agent conventions give you a useful vocabulary here, even in development status. An agent run maps to invoke_agent, an internal loop step to an internal span, a planning phase to plan, and each tool call to execute_tool. Metric names exist for gen_ai.invoke_agent.duration, gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls. When those three metrics are populated for your traces, you can alert on “inference calls per request went up 40%” which is the earliest available signal of a loop bug.
On replay and determinism, be honest about what is achievable. Agents are stochastic unless you pin everything: model version, temperature, seed and tool responses. The gen_ai.request.seed attribute exists for this. Even fully pinned, external tool state will have moved. The practical pattern is not full replay but counterfactual re-scoring: capture the real trace, then re-run only your deterministic evaluators over the captured prompt and response. That gives you a regression gate without pretending you can rewind the world.
On sample size, the arithmetic matters more than intuition. For a binary pass rate near 50 percent, detecting a five-point difference at 95 percent confidence needs roughly 400 samples; a ten-point difference needs roughly 100. Run the same fixed dataset against both versions and use a paired test, because outputs on identical inputs are correlated and a naive two-proportion test will overstate your confidence.
The 11 platforms compared
All prices below were read from the vendor’s own pricing page on 16 October 2026. Free tiers are included because for most teams the free tier is the real starting point. Unpublished unit prices are marked.
| Platform | Entry price (published) | Billable unit | Self-host | Licence of OSS core | Open standard ingest |
|---|---|---|---|---|---|
| LangSmith (LangChain) | $0/seat Developer, 1 seat, 5k base traces/mo; $39/seat/mo Plus, 10k base traces/mo | Traces. Per-trace price not published; usage calculator and sales only | Enterprise only (self-hosted, hybrid, or cloud) | SDKs open source; platform is SaaS | OTLP/HTTP supported, including a collector-based redaction gateway |
| Langfuse | Hobby $0 (50k units/mo, 30 days, 2 users); Core $29/mo (100k units, 90 days); Pro $199/mo (100k units, 3 years); Enterprise $2,499/mo | Units = traces + observations + scores; then $8 per 100k units | Yes, MIT licensed, unlimited usage, no feature gates | MIT | OpenTelemetry (Java, Go, custom) and LiteLLM proxy listed |
| Arize AX | AX Free $0 (25k spans/mo, 1 GB/mo, 15 days); AX Pro $50/mo (50k spans, 10 GB, 30 days); Enterprise custom | Spans plus GB ingest | Yes (Enterprise) | Phoenix OSS is Elastic License 2.0, free to self-host with no feature limits | Built on OpenTelemetry with OpenInference instrumentation |
| Braintrust | Starter $0 ($10 model credits, 1 GB processed, 10k scores, 14-day retention); Pro $249/mo (5 GB, 50k scores, 30-day retention) | Processed data in GB; then $3/GB. Scores then $1.50/1k | Enterprise (hosted or on-prem) | SDK open source | Native OTLP trace and log ingestion documented |
| Datadog Agent Observability | Free up to 40k LLM spans/mo, 15-day retention, unlimited evals; Pro $240/mo on-demand, $200 month-to-month, $160 on annual commitment | LLM spans only; then $5.00/10k on-demand, $4.20 month-to-month, $3.50 annual | Datadog SaaS | Not applicable | OpenTelemetry supported; bring any custom environment via OTLP |
| Comet Opik | Open Source $0; Free Cloud $0 (up to 10 members, 25k spans/mo, 60-day retention); Pro Cloud $19/mo (50 members, 100k spans/mo, 60 days) | Spans per month | Yes, same codebase as hosted | Apache-2.0 | Native OTLP endpoint documented, HTTP transport only for now |
| Traceloop / OpenLLMetry | Free forever up to 50k spans/mo, up to 5 seats, 24-hour retention; Enterprise above 50k spans/mo | Spans per month | Yes, on-prem, air-gapped supported | OpenLLMetry is Apache-2.0 | OpenLLMetry is the instrumentation layer |
| Confident AI (DeepEval) | Free $0 (2 seats, 1 project, 5 test runs per week, 1 GB-month of spans); Starter $200/mo (5 projects, 5 GB-months); Team $2,000/mo (75 GB-months) | GB-months of trace spans ingested or retained; then $1/GB-month | Yes, on-prem on Enterprise | DeepEval is open source | Tracing plus native support for external evaluators |
| Helicone | Hobby $0 (10k requests/mo, 1 GB, 7-day retention); Pro $79/mo; Team $799/mo | Requests plus GB storage | On-prem on Enterprise | SDK open source | Proxy-based, with OpenTelemetry support |
| New Relic AI Monitoring | 100 GB/month data ingest free, then $0.40/GB; Data Plus $0.60/GB | GB of data ingested, plus Compute Capacity Units for AI features | Yes | Not applicable | OpenTelemetry core product |
| Honeycomb | Free up to 20M events/mo and 100M metrics data points; Pro starting $150/mo up to 750M events/mo; Enterprise custom | Events plus metrics data points | Private Cloud on Enterprise | Not applicable | Aggregates the OpenTelemetry spans you already send |
Three things in that table deserve emphasis. LangSmith is the only major platform on this list that does not publish a per-trace price; it points you at a calculator and sales. Datadog bills LLM spans only, explicitly not tool spans or retrieval spans, so an orchestration-heavy agent costs less there than the raw span count suggests. And Arize AX spans and Phoenix OSS are now a portfolio question rather than a product question, because Dynatrace completed its acquisition of Arize on 1 October 2026.
What it costs: the arithmetic nobody shows
Per-span pricing is a volume tax. The mistake teams make is costing per request when the vendor bills per observation.
Langfuse publishes the cleanest definition of any of these. From its billable units documentation: Units = Count of Traces + Count of Observations + Count of Scores. Any trace, observation or score stored counts, whether it came from your app or from a Langfuse feature such as an LLM-as-a-judge evaluator or an experiment. Langfuse’s own worked example on that page shows 20,070 traces plus 119,500 observations plus 561 scores equalling 140,131 units in a month. That is roughly six observations per trace on real usage, which is the multiplier that breaks naive budgets.
Using that ratio, and assuming one judge score per trace, here is the arithmetic:
- Fix your request volume and your spans-per-request first. Measure them in staging. Everything below is linear in this number, and it is the one input that is entirely yours.
- Multiply by roughly eight billable units per agent run. One trace plus about six observations plus one score. On 10,000 requests a month that is about 80,000 Langfuse units.
- Subtract the included allowance before pricing anything. Langfuse Core includes 100,000 units at $29/mo, so 10,000 requests a month fits inside the cheapest paid tier.
- Then apply the overage rate. $8 per additional 100,000 units, on every paid Langfuse tier.
- Check whether retention silently changes your unit count. Scoring a trace creates another unit. Turning on online evaluators can auto-upgrade Langfuse traces and raise your bill in ways that are not obvious from the toggle you flipped.
| Monthly agent requests | Approx. Langfuse units | Langfuse monthly cost |
|---|---|---|
| 10,000 | ~80,000 | $29 (Core, inside the 100k allowance) |
| 50,000 | ~400,000 | $53 ($29 + 3 x $8) |
| 500,000 | ~4,000,000 | $341 ($29 + 39 x $8) |
For Datadog, the same volume modelled with an assumption of two LLM spans per agent request (my assumption, not Datadog’s; measure yours) gives roughly 20,000 LLM spans at 10,000 requests a month, which sits inside the free 40k allowance. At 50,000 requests a month that becomes 100,000 spans, of which 60,000 are billable at 6 x 10k: $240 + 6 x $5.00 = $270/month on-demand, or $160 + 6 x $3.50 = $181/month on an annual commitment. At 500,000 requests a month it becomes 1,000,000 spans and 960,000 billable, or $720/month on-demand. Note that at higher retention the maths changes materially: Datadog lists 30-day retention at $6.50, 60-day at $8 and 90-day at $9 per 10k LLM spans per month.
Two practical warnings. Braintrust and Confident AI meter bytes, not counts, so your bill tracks payload size and will grow silently if you start logging full conversation history. And retention is priced separately almost everywhere: LangSmith charges a lower rate for its 14-day base traces than for its 180-day extended traces, and it auto-upgrades a trace to extended retention when an online evaluator or an automation rule touches it.
LLM-as-a-judge needs its own calibration
A judge is a measuring instrument. You would not deploy a latency monitor without calibrating it, and most teams deploy an LLM judge without doing exactly that.
The documented biases are real and they are not hypothetical:
- Position bias. In the 2023 paper Large Language Models are not Fair Evaluators, reordering two candidate responses was enough to make Vicuna-13B beat ChatGPT on 66 of 80 tested queries when GPT-4 was the evaluator.
- Self-preference. LLM Evaluators Recognize and Favor Their Own Generations (April 2024) found GPT-4 and Llama 2 have non-trivial accuracy at recognising their own outputs, and that self-recognition capability correlates linearly with the strength of self-preference bias.
- Style bias. A 2026 study of nine debiasing strategies across five judge models found style bias was the dominant bias at 0.10-0.76, favouring markdown over plain prose, while position bias stayed at or below 0.04.
- Verbosity bias. Here the sources genuinely disagree, and you should know it. One 2026 study of 21 judges measured verbosity bias as small, under 0.011, under a single pairwise rubric. The debiasing study found it heterogeneous: Gemini 2.5 Flash, Gemini 2.5 Pro and Llama preferred longer answers (+0.24 to +0.44), Claude preferred concise (-0.12) and GPT-4o was neutral (-0.04). The difference is almost entirely the measurement setup.
The finding I would put in front of any team about to gate a release on a judge is this. A large-scale 2026 evaluation of 21 judges from nine providers, spanning 118 runs and roughly 541,000 individual judgements, found that the gap between exact-match agreement and Cohen’s kappa was 33 to 41 percentage points on MT-Bench, and that judge rankings shifted by up to 14 positions depending on which benchmark you used. Two judges that were actually deployed in production showed test-retest reliability above 0.95 while still showing position bias above 0.10. Consistency is not correctness.
A more recent reliability audit, submitted on 8 October 2026, stress-tested six frontier models and found that verdicts changed across identical replications at temperature zero, and that position-order swaps flipped the majority of verdicts on challenging tasks. Its most useful finding for practitioners is also its most actionable one: the most deterministic judge achieved perfect consistency by trivially repeating incorrect verdicts, agreeing with ground truth only 51% of the time. The same work found that switching from pairwise win-rate to holistic rubric scoring improved trustworthiness more than any single prompting intervention.
Practically, the calibration loop is:
- Hand-label 200 to 300 real production outputs, stratified so the sample is not all easy cases.
- Run your judge twice on the same set and measure order consistency by swapping slots.
- Report Cohen’s kappa, not exact match. If you report only exact match you will overstate yourself by 33 to 41 points on MT-Bench-like data.
- Measure style sensitivity by reformatting outputs between markdown and plain prose and re-scoring.
- Prefer holistic rubric scoring over pairwise win-rate, and re-run the whole measurement whenever you change the judge model or its prompt.
The cost argument for doing this properly is favourable too. The debiasing study found Gemini 2.5 Flash with a combined debiasing strategy reached the highest agreement of any configuration it tested (71.0%, kappa 0.549) at roughly $0.001 per evaluation, about 15x cheaper than the best frontier configuration it tested (Claude Sonnet 4 at 69.5%, roughly $0.015). A cheaper judge that you have calibrated beat a frontier judge that you had not.
Vendor risk became a real decision factor in 2026
Five independent acquisitions of observability or evaluation companies closed between January and October 2026. If your roadmap depends on one of these products, that is an input you should price.
- ClickHouse acquired Langfuse, announced around 16 January 2026 alongside a $400M Series D at a reported $15 billion valuation.
- Mintlify acquired Helicone, announced 3 March 2026 by Helicone’s founders.
- ServiceNow acquired Traceloop, reported in March 2026; Traceloop’s own site now reads “Traceloop is joining Service now!”
- Cisco acquired Galileo, announced 9 April 2026 and completed 22 May 2026.
- Dynatrace acquired Arize for $915 million, announced 13 August 2026 and completed 1 October 2026.
The Arize case is the most instructive. Dynatrace’s own release states that Arize will continue supporting both Phoenix, its open-source platform, and AX, its enterprise platform, with capabilities integrated into Dynatrace over time. The practical lesson is that the open-source core survives acquisition; the enterprise roadmap does not belong to you. If that matters to you, weight MIT and Apache-2.0 licences heavily. Note that Phoenix is Elastic License 2.0, which is source-available but not OSI-approved open source, while Langfuse’s core is MIT and Opik is Apache-2.0.
Legal, privacy and compliance considerations
Your traces contain prompts and completions. In most regulated contexts, that makes them personal data, and retention configuration becomes a legal decision rather than a product setting.
- Retention is the primary control. Published defaults range widely: 15 days on Datadog’s free tier and 30 days on Arize AX Free, 30 days on Langfuse Hobby, 60 days on Opik Cloud, and 14-day base versus 180-day extended on LangSmith. Pick the shortest window you can defend.
- Data residency is uneven. Langfuse lists US, EU and Japan regions on Hobby, Core and Pro, adding a HIPAA-compliant US region on Enterprise, and AWS PrivateLink on Enterprise only. Arize offers SaaS or self-hosted on Enterprise. Others do not publish a residency matrix on their pricing pages.
- Self-hosting is a compliance control as much as a cost one. Langfuse’s MIT core, Phoenix under ELv2 and Opik under Apache-2.0 all let traces stay inside your own network.
- Compliance attestations are enterprise-tier features. Langfuse lists SOC 2 Type II and ISO 27001 reports plus HIPAA compliance from the Pro tier upward, and AWS PrivateLink on Enterprise. Opik Enterprise lists SOC 2, ISO 27001, ISO 9001, HIPAA and GDPR. Helicone lists SOC-2 Type II and HIPAA on Team.
If you are handling health, financial or legal data, get counsel involved rather than relying on a vendor’s marketing page. This is one of those areas where a confident blog answer is worth less than a short conversation with whoever owns your compliance obligations.
The bottom line
If you are starting from nothing, start with an OpenTelemetry-native, self-hostable option and prove the workflow before you pay for anything: Langfuse’s MIT core or Phoenix under ELv2 both do this. If you already run Datadog or New Relic and your agent traffic is modest, the marginal cost of adding their AI product is usually lower than the operational cost of a second vendor.
Where I would push back on the consensus: do not buy a platform because it scores well in a feature grid. Buy it for the one of the three jobs you are actually missing. And before you trust any automated verdict, calibrate the judge, because the published evidence says the instrument is noisier than the dashboards imply.
FAQ
What is LLM observability?
It is the collection and analysis of the individual steps inside an LLM or agent execution: the system prompt, retrieved context, every tool call and its arguments, the model’s response, token counts, latency and cost. Unlike classic APM, where a trace shows whether a service returned 200, an LLM trace has to answer a subjective question (was this answer any good) which is why observability alone does not tell you that quality has degraded.
What is the difference between tracing, evaluation and monitoring?
Tracing records what happened on a specific run. Evaluation scores output against a criterion, offline against a dataset or online against live traffic. Monitoring watches production continuously so you get alerted when a score distribution shifts. A platform can be strong at one and weak at the other two, and most buying mistakes come from not separating them.
Are OpenTelemetry GenAI semantic conventions stable yet?
No. They are marked Development and live in a separate repository, open-telemetry/semantic-conventions-genai, created on 5 May 2026. The repository has no release tags and its README still lists the Schema URL as TODO. In the GenAI attribute registry, 127 of 127 attributes are marked Development and zero are Stable.
Can I move my LLM traces from one observability vendor to another?
Partially. If your instrumentation emits OpenTelemetry GenAI spans, most backends will accept the payload, but the conventions themselves are unstable, and each backend maps fields to its own schema on ingest. Treat portability as “I can re-emit spans elsewhere”, not “I can lift a database and keep every query”. Verify the documented export path before you commit.
How much does LLM observability cost per month?
For 10,000 agent requests a month generating roughly 60,000 observations, published entry tiers range from free (Opik Cloud free tier at 25,000 spans, Langfuse Hobby at 50,000 units) to $29 (Langfuse Core, 100,000 units) to $79 (Helicone Pro). At 500,000 requests a month the same architectures land between roughly $340 and $720 a month, because most vendors meter spans or units rather than requests.
Is LLM-as-a-judge reliable enough to gate a release?
Not without calibration. A study of 21 judges from nine providers across roughly 541,000 judgements found that Cohen’s kappa sits 33 to 41 percentage points below raw exact-match agreement on MT-Bench, and that two judges deployed in production showed test-retest reliability above 0.95 while still showing position bias above 0.10. Run a minimum viable validation protocol against human labels before you trust a verdict.
Which platforms support self-hosting?
Langfuse’s OSS edition is MIT licensed with unlimited self-hosted usage and no feature gates. Arize Phoenix is free to self-host under the Elastic License 2.0 with no feature limits. Comet Opik is Apache-2.0 with the same codebase as the hosted product. Traceloop and OpenLIT also self-host, and LangSmith offers self-hosted and hybrid deployment only on Enterprise.
What are the most common production failures in LLM apps?
In rough order of how quietly they hurt: silent quality drift after a prompt or model swap, truncated outputs that still parse as valid, retrieval returning nothing relevant, tool calls with wrong arguments that succeed at the HTTP level, token and cost explosion from runaway loops, refusal regressions after a safety update, and eval regressions that reach customers because the test suite never ran against real traffic.
How many samples do I need to call an eval change significant?
More than most teams run. For a binary pass rate near 50 percent you need roughly 400 samples to resolve a five-point difference at 95 percent confidence, and roughly 100 samples for a ten-point difference. The practical answer is to run the same fixed dataset on both versions and use a paired test, because agent outputs on identical inputs are correlated.
Should I pick a vendor I might get bought?
Price in the risk. Five independent acquisitions closed between January and October 2026: ClickHouse bought Langfuse, Mintlify bought Helicone, ServiceNow bought Traceloop, Cisco bought Galileo, and Dynatrace bought Arize for $915 million. Two of those (Arize and Helicone) were listed on every comparison page I reviewed eight months ago. The open-source licence is the part that survives an acquisition.
Sources
- OpenTelemetry GenAI semantic conventions repository: status of the GenAI conventions, their relocation out of the main semconv repo, and the missing Schema URL
- OpenTelemetry GenAI attribute registry: 127 attributes, all Development, none Stable
- OpenTelemetry GenAI agent and framework spans: invoke_agent, plan, execute_tool, invoke_workflow span definitions
- OpenTelemetry GenAI metrics conventions: gen_ai.invoke_agent.inference_calls, tool_calls, duration metrics
- OpenTelemetry semantic conventions, GenAI index: confirms the “Moved” notice in the main repository
- LangSmith pricing: seat pricing, included base traces, base versus extended retention, LSU definition
- LangSmith usage and billing: 14-day base versus 180-day extended retention, auto-upgrade behaviour, 14 September 2026 retention cap change
- Langfuse pricing: Hobby, Core, Pro and Enterprise tiers, $8 per 100k units, Teams add-on
- Langfuse billable units: units = traces + observations + scores, and the published worked example of 20,070 traces / 119,500 observations / 561 scores
- Langfuse self-hosted pricing: MIT licence, unlimited usage, Enterprise priced additively to ClickHouse
- Langfuse joins ClickHouse: ClickHouse has acquired Langfuse
- Arize pricing: AX Free, AX Pro at $50/month, AX Enterprise
- Arize Phoenix self-hosting: free to self-host with no feature limits, built on OpenTelemetry and OpenInference
- Arize Phoenix licence: Elastic License 2.0
- Braintrust pricing: Starter and Pro tiers, processed data and score metering, S3 export on Enterprise
- Braintrust OpenTelemetry ingestion: native OTLP trace and log ingest
- Datadog pricing: Agent Observability: 40k free LLM spans, Pro platform fee, per-10k span rates, retention add-ons, LLM-spans-only billing
- Datadog Agent Observability documentation: trace and span model, 15-day default retention, 15-month metrics retention
- Comet Opik pricing: Open Source, Free Cloud at 25k spans, Pro Cloud at $19/month
- Opik OpenTelemetry integration: native OTLP endpoint, HTTP transport only
- Traceloop pricing: free tier at 50k spans, OpenLLMetry Apache-2.0, ServiceNow notice
- Confident AI pricing: Free, Starter at $200/month, Team at $2,000/month, $1 per GB-month of trace spans
- Helicone pricing: Hobby, Pro at $79/month, Team at $799/month, retention per tier
- Helicone joins Mintlify: acquisition announced 3 March 2026, 14.2 trillion tokens processed
- New Relic pricing: 100 GB free ingest, $0.40/GB and $0.60/GB Data Plus, Compute Capacity Units for AI features
- Honeycomb pricing: Free at 20M events/month, Pro from $150/month
- Honeycomb AI Ecosystem: aggregates OpenTelemetry spans already sent; LLM cost tracking described as an estimate from a public price table
- OpenLIT pricing: Apache-2.0, unlimited self-hosted usage, Enterprise contact, cloud forthcoming
- Giskard pricing: open-source tier and unpriced Professional tier
- promptfoo pricing: Community free forever with 10k red-teaming probes per month, Enterprise and On-Premise custom
- promptfoo assertions reference: is-json, is-valid-openai-function-call, javascript, llm-rubric and related assertions for detecting truncation and malformed tool calls
- promptfoo CI/CD integration: GitHub Actions, GitLab CI, CircleCI, Jenkins and library integrations for gating releases
- Amazon CloudWatch Omni launch post: 22 September 2026, OpenTelemetry-based ingest, telemetry-based pricing, 30-day trial and $1,000 ingest credits
- Vertex AI pricing: Gen AI Evals: model-based metrics billed on autorater tokens, computation-based metrics at $0.00003 per 1k input characters and $0.00009 per 1k output characters, Gemini 2.0 Flash autorater
- Azure AI Foundry evaluation concepts: risk and safety evaluations billed by consumption
- Dynatrace completes acquisition of Arize: completed 1 October 2026, Phoenix and AX continue to be supported
- Dynatrace to acquire Arize: $915 million, announced 13 August 2026
- Dynatrace: As AI scales across enterprises, breaking points emerge: State of SRE and Platform Engineering 2026, 919 IT leaders surveyed October 2025 to January 2026
- Cisco announces intent to acquire Galileo: announced 9 April 2026, completed 22 May 2026
- Reliability without Validity (arXiv:2606.19544): 21 judges, nine providers, ~541,000 judgements, kappa deflation of 33-41 points, rankings shifting up to 14 positions
- All Verdicts are Not Equal (arXiv:2610.12083): submitted 8 October 2026, verdict instability at temperature zero, most deterministic judge agreeing with ground truth only 51% of the time
- BabelJudge (arXiv:2606.22329): position, verbosity, order inconsistency and cross-lingual degradation; Swahili order consistency of 0.480
- Judging the Judges (arXiv:2604.23178): nine debiasing strategies, style bias 0.10-0.76, heterogeneous verbosity bias, Gemini 2.5 Flash at $0.001 per evaluation
- Accounting for Bias Enables Sustainable LLM Evaluation (arXiv:2609.31184): leaderboards cannot fix bias by adding comparisons
- Pair Difficulty Matters (arXiv:2609.37577): why position bias, transitivity and pairwise agreement mislead judge selection
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685): the original position, verbosity and self-enhancement bias findings
- Large Language Models are not Fair Evaluators (arXiv:2305.17926): reordering responses made Vicuna-13B beat ChatGPT on 66 of 80 queries
- LLM Evaluators Recognize and Favor Their Own Generations (arXiv:2404.13076): self-recognition and self-preference bias
- What Is Your Agent’s GPA? (arXiv:2510.08847): agent Goal-Plan-Action evaluation, 95% error coverage and 86% localisation on TRAIL/GAIA
