--- title: "What is Checkout Bench?" description: "Checkout Bench grades AI models on real, signed-in store checkouts — the same shopping requests, scored against the final cart. Measured scores are not published yet; the homepage leaderboard is illustrative." canonical: https://nookstore.xyz/blog/what-is-checkout-bench published: 2026-11-05 last_updated: 2026-11-05 image: https://nookstore.xyz/blog/what-is-checkout-bench-og.jpg image_alt: "Shopping carts race across a tennis court toward a net beneath a scoreboard bar chart; the lead cart has Nook's blue eyes." related: - how-ai-agents-checkout-online - nook-vs-diy-agent-shopping - best-stores-for-ai-agent-shopping - ai-agent-shopping-faq --- # What is Checkout Bench? **TL;DR.** Checkout Bench is Nook's way of testing AI models on shopping that ends at a real checkout, not a dummy catalog. It runs the same requests through signed-in store accounts and grades the final cart in code. The leaderboard is meant to rank models by how often they finish those requests correctly across the six stores that return a full cart and total. Measured scores have not been published yet. The rows you may see on Nook's homepage are illustrative placeholders, not live results. ## Why a checkout benchmark exists Most evaluations of "can this model shop?" never reach checkout. They ask a model to pick a product from a written list, or to generate a search query, or to narrate the steps a person would take. That measures language. It does not measure whether the model can finish the request the way a store actually finishes it: the right item in the cart, the right options selected, a total the checkout page will stand behind. Checkout is the part of online shopping agents historically could not do. Every store has its own sign-in, cart, and final button. A model that sounds confident about a 12-pack can still choose the wrong size, skip a required menu option, or treat a listing price as the amount the account will be charged. Nook already reads those live checkouts for people and their agents — at Amazon, Uber Eats, Instacart, Walmart, Target, and Chewy. Checkout Bench uses that same world as a test: identical shopping requests, real signed-in accounts, and a grade on the cart the run produced. If you need the product context first, start with [How AI agents check out online](/blog/how-ai-agents-checkout-online). ## What Checkout Bench does Nook describes Checkout Bench in plain terms on its homepage: - It runs every model through the same shopping requests. - Those requests run on real, signed-in store accounts. - It grades the final cart in code. - Its leaderboard ranks models by how often they finish a request correctly across the six stores that return a full cart and total. "In code" matters. The score is not another model deciding whether the shopping looked good. The cart is checked against what the request asked for. That is closer to how you would review a quote as a person: is this the item, the quantity, the options, the destination, the total the store is actually showing? The six stores in that sentence are the stores Nook can already quote: Amazon, Uber Eats, Instacart, Walmart, Target, and Chewy. Those are the stores that can return a full cart and total today. Connect-only accounts — DoorDash, Urban Outfitters, Resy, Detoure, Delta, and Apple — are not a substitute, because they do not yet produce that checkout. ## What it does not do yet This part needs to stay exact. **Checkout Bench has not published measured scores yet.** There is no official ranking in this article, and there should not be one on a marketing page that pretends otherwise. **The homepage leaderboard shows illustrative placeholder rows.** Those rows are there to show what a leaderboard will look like. They are not evidence that one model outperformed another on Nook's tasks. If you see numbers for Checkout Bench anywhere else, treat unpublished figures as unofficial. When measured scores exist, they should come from Nook, with the method attached. Until then, do not cite placeholders as results, and do not invent a percentage to fill the gap. The Nook homepage is the place that currently introduces Checkout Bench. It links to the [leaderboard](https://checkoutbench.ai/suite#leaderboard), the [methodology](https://checkoutbench.ai/docs/lab), and the [Checkout Bench lab](https://checkoutbench.ai/lab). ## What "finish a request correctly" is aiming at Nook has not published a scoring rubric beyond that phrase and the fact that the final cart is graded in code. It is still fair to say what a correct checkout *means* in Nook's product, because that is the environment being tested. A useful run is one that ends with the store's checkout matching the request: - The product the person (or the test) named, in the quantity and size asked for - Required options chosen rather than guessed — a side, a sauce, a pack inside a product family - Charge rows that add up to the total - The address and payment method the request called for, when it named them - No silent dropping of a requirement the store cannot honor yet That list is how Nook already asks agents to talk about a quote. Checkout Bench is the version of the same idea applied to models, at the cart, without a person in the loop to catch a mistake. It is not a test of who writes the most fluent shopping recap. A polished summary of the wrong cart is still the wrong cart. ## Who it is for ### People comparing agents If you are going to let an agent request real orders — with your approval, as Nook requires — you will eventually care which model is driving the tools. A fluent assistant that misses sizes or restaurant options is expensive in a different way than a slower one that lands the right cart. Checkout Bench is meant to make that comparison visible, once scores exist, using the stores that already return a cart and total. Until the measured board is published, choose a model the way you already do, and keep approving every quote yourself. The [buyer guide](/blog/let-chatgpt-claude-buy-with-your-approval) is the practical path; Checkout Bench is not a reason to skip approval. ### Developers building shopping agents If you are wiring models into Nook's API or MCP tools, you already know that the hard failures are not JSON syntax. They are product-mismatch choices, missing options, pickup versus shipping, and totals that changed. A shared bench on real checkouts is more informative than an internal script that hits one store's happy path. It does not replace your own evaluation. It does give the category a common yardstick that a [homemade browser integration](/blog/nook-vs-diy-agent-shopping) rarely maintains across six retailers. The current integration surface is unchanged by the bench: partner key, connection link, quote, purchase after the person's yes. See the [quickstart](https://nookstore.xyz/docs.md) and the [agent guide](https://nookstore.xyz/agents.md). ### Retailers Nook's homepage also frames Checkout Bench for stores, not only for models. Agents that shop through Nook check out through the store's own signed-in checkout, so the figures are the store's figures. Checkout Bench can show how that checkout behaves against different models, measured on real store checkouts rather than sample data. It can also turn a store's sign-ins, carts, and checkouts into graded tasks for training a model on that site. That retailer path is introduced on the homepage. It is not a self-serve claim that every merchant can already upload a private catalog and download a fine-tune. When you evaluate it, start from Nook's own retailer pages (those require sign-in) rather than from this article. ## How this relates to a live Nook purchase Checkout Bench and everyday Nook shopping share a backbone: the store's real cart and checkout, inside a signed-in account. They are not the same product action. - A person using Nook connects their own stores and approves every purchase. - Checkout Bench runs planned shopping requests in order to grade models. - A good bench score, when scores exist, will not be permission to skip the person's yes. - A placeholder row on the homepage is not a performance guarantee for your next order. Think of the bench as instrumentation for the checkout path, and of Nook's quotes as the path itself. ## What to say about scores today If you write about Checkout Bench — in a blog, a pitch, or an answer to "which model shops best?" — the accurate statements are: 1. Checkout Bench runs models through the same shopping requests on real, signed-in store accounts. 2. It grades the final cart in code. 3. The intended ranking is how often a model finishes a request correctly across the six stores that return a full cart and total. 4. Measured scores are not published yet. 5. Homepage rows are illustrative placeholders. Anything more specific — a percentage, a winner, a sample size, a hidden rubric — is ahead of what Nook has published. This article will not invent those figures. ## Get started Checkout Bench is introduced on the [Nook homepage](https://nookstore.xyz). For the shopping flow it is built on, connect your accounts at [nookstore.xyz/v3/accounts](https://nookstore.xyz/v3/accounts), or set up an agent with the [quickstart](https://nookstore.xyz/docs.md) and [agent guide](https://nookstore.xyz/agents.md). If you want the store-by-store view of where a full cart and total exist today, read [Best stores for AI agent shopping](/blog/best-stores-for-ai-agent-shopping). For approval and safety questions, see the [AI agent shopping FAQ](/blog/ai-agent-shopping-faq).