Prompt Bloat: A Love Story, by Frontier Labs

To find out what the romance costs, we pointed OpenAI's prompt generator and Anthropic's Workbench generator at the same real job: turn bank descriptors like KLARNA* HMSVERIGE SE SEK into clean merchant names. We fed each the training examples, ran the prompt it wrote on a held-out set, on the same model, and counted what it spent.

Both prompts worked. Both also cost about ten times more per call than the task needs, and bought no extra accuracy for the money.

The receipt is not pretty. OpenAI's prompt got 60% of merchants right and cost about $0.24 per 1,000 calls. Anthropic's got 67% and cost about $0.33 per 1,000 calls. Then we handed each of those prompts to EigenPrompt and let it search. Starting from OpenAI's, it found one that was more accurate, 64%, for $0.018 per 1,000 calls, thirteen times cheaper. Starting from Anthropic's, it held the same 67% accuracy for $0.034, just under ten times cheaper.

The answers came out the same. The cost did not: the optimized prompts ran for about a tenth of the price. That gap between a prompt that looks finished and one that has actually been measured is the bloat in the title, and it bills you on every call.

The rules we set

A benchmark is only worth reading if it is fair, so here is the whole protocol in one place. None of it is subtle, and all of it is reproducible.

• The task. Map a raw bank descriptor (INPUT) to a canonical merchant name (EXPECTED_OUTPUT), from the public entity-resolution dataset. The dataset and both generated starting prompts are published in the eigenprompt-resources repository.
• The split. The generators and the optimizer only ever saw the training rows. Every number below is scored on held-out rows none of them touched.
• The scorer. Exact match: the predicted merchant equals the labeled one, or it does not. Same evaluator for every prompt.
• The model. Every prompt runs on the same target model, groq/openai/gpt-oss-120b, at the same settings. We are comparing prompts, not models.
• The generators. OpenAI's prompt generator and Anthropic's prompt generator in Workbench, each given the task and the training examples. We tested the prompt each tool returns, not the models behind them, and not OpenAI's separate dataset-backed optimizer.
• The optimizer. EigenPrompt in Standard mode, started from each generator's own prompt, so the only thing that changes is the search.

If a number ever came from grading a prompt on rows it had already seen, it would not be in this post. None of them did.

OpenAI's prompt: a fine first draft, at 13× the price

Here is the prompt OpenAI's generator wrote, verbatim:

Identify the primary (canonical) merchant name based on a raw bank transaction description string.

- Transaction: {{INPUT}}
- Output ONLY the canonical merchant name (e.g., 'Amazon', 'Starbucks', 'Netflix'). If no merchant is clearly indicated or cannot be determined, output 'Unknown'.

Guidelines:
- Ignore all extraneous information such as numbers, location details, payment platforms, or codes unless essential for identifying the merchant.
- Normalize variants or abbreviations to standardized merchant names ("AMZN MKTP US" → "Amazon Marketplace", "GOOG" → "Google" etc.).
- When payment facilitators (e.g., "PayPal", "Square", "Stripe") are present, prioritize identifying the end merchant, not the payment service, unless that's the only info available.
- Use step-by-step reasoning to extract, match, and standardize the merchant. Only provide the canonical name after all reasoning.
- If the transaction string lacks a discernible merchant, or if the entity appears to be an individual person or generic payment (e.g. "PAYPAL *JOHN SMITH", "POS DEBIT TEMP AUTH HOLD"), output 'Unknown'.

**Reasoning and Output Order:**
- ALWAYS perform any step of extraction, cleaning, or reasoning BEFORE giving the merchant name.
- The output MUST only be the merchant name (or 'Unknown'), never any explanation or reasoning trace.
- (The provided examples all supply conclusions only, not reasoning, but for best results insist that all reasoning happens before the answer—never show the answer first in any internal process.)

**Output Format:**
- Output: a single line, plain text, containing ONLY the canonical merchant name, or 'Unknown'

**Examples:**
(The following demonstrate expected canonicalization, abbreviation expansion, and 'Unknown' fallback.)

Input:
AMZN MKTP US*2J8X9K4L
Output:
Amazon Marketplace

Input:
GOOGLGOOGL*YOUTUBE
Output:
YouTube

Input:
SQ *JOES COFFEE SHOP
Output:
Joe's Coffee Shop

Input:
POS DEBIT TEMP AUTH HOLD 0000
Output:
Unknown

Input:
PAYPAL *JOHN SMITH 4029357733
Output:
Unknown

Input:
PAYPAL *AIRBNB 4155552000
Output:
Airbnb

---

**REMINDER:**
Only output the canonical merchant name, or 'Unknown'—never any intermediate text, reasoning, or commentary. Identify the merchant as accurately as possible from the transaction description before producing your answer.

It took about thirty seconds to generate, and it looks finished: guidelines, few-shot examples, an output contract, a bolded closing reminder. Keep an eye on the line that insists on "step-by-step reasoning" before every answer. It will show up on the bill.

On the held-out set, this prompt scored 60% and cost about $0.24 per 1,000 calls. On the chart below it is the gold diamond, marooned on the right-hand edge. Everything in blue is what EigenPrompt found starting from that exact prompt.

The whole blue frontier sits to the left of the baseline. In plain terms, every prompt EigenPrompt found is cheaper than the one we started with. Two stand out. The cheapest prompt that still beats the original's accuracy reaches 64% for $0.018 per 1,000 calls, thirteen times cheaper than OpenAI's for a slightly better score. The most accurate climbs to 81%, twenty-one points above the generated prompt, and still costs about $0.082, a third of the original's price. That top prompt is compact and blunt about what it wants:

Extract the primary merchant name from the transaction description.

- Transaction: {{INPUT}}
- Return only the canonical merchant name (e.g., Amazon, Starbucks) or **Unknown** if it cannot be identified.

Rules:
* Ignore numbers, locations, and payment‑platform codes unless essential.
* Convert known abbreviations to standard names (e.g., “AMZN MKTP” → “Amazon Marketplace”, “GOOG” → “Google”).
* When a payment facilitator (PayPal, Square, Stripe) appears, name the underlying merchant; if none is present, output the facilitator.
* If the description refers to a person or generic hold, output **Unknown**.

Process:
* Perform all extraction and reasoning internally before answering.
* Output a single line containing only the merchant name or **Unknown**, with no additional text.

Fourteen lines, where the generator's draft used fifty-seven. The tell is the last two. "Perform all extraction and reasoning internally" and "output a single line" are not accuracy instructions. They are cost instructions, and we'll come back to why. The search tried 44 variants and finished in about 35 minutes, for roughly a dollar of API spend.

Anthropic's prompt: more accurate, and even pricier

Anthropic's generator produced the better starting prompt of the two: 67% on the held-out set, seven points ahead of OpenAI's. It was also the more expensive, at about $0.33 per 1,000 calls, the priciest starting point in the study. Here it is in full. The <scratchpad> block near the end politely asks the model to think through six questions before every single answer:

You will be identifying the primary merchant name from a bank transaction description. Bank transaction descriptions often contain various codes, numbers, locations, and payment processor information alongside the actual merchant name. Your task is to extract the canonical merchant name in a clean, recognizable format.

Here is the transaction description:

<transaction>
{{INPUT}}
</transaction>

Bank transaction descriptions typically follow certain patterns:

1. **Payment processors/platforms**: Prefixes like "SQ *", "PP*", "PAYPAL *", "STRIPE*", "SHOP PAY*", "ADYEN*", "BRAINTREE*", "KLARNA*", "AFTERPAY", "AFFIRM*" indicate the payment processor, not the merchant. The actual merchant name follows the asterisk or prefix.

2. **Digital service abbreviations**: Companies like Amazon (AMZN), Microsoft (MSFT), Apple (APLPAY, APPLE.COM/BILL), Google (GOOGL, GOOGLE), Netflix (NFLX), Meta (META PAY) often use abbreviated forms.

3. **Store/location identifiers**: Hash symbols (#) followed by numbers indicate store locations, not the merchant name. State abbreviations and city names are location information, not part of the merchant name.

4. **Delivery platforms**: Services like "DOORDASH*", "UBER EATS *", "GRUBHUB*", "INSTACART*" are the primary merchant (the delivery service), even though a restaurant name may follow.

5. **Parent company indicators**: Some transactions include parent company names (e.g., "DARDEN", "BLOOMIN", "BRINKER INTL") but the restaurant brand is the primary merchant.

6. **Generic/unclear transactions**: Descriptions like "POS DEBIT", "TEMP AUTH HOLD", "CARD PURCHASE", "ACH WEB PMT", or person-to-person payments without clear merchant context should be classified as "Unknown".

Common patterns to recognize:
- "AMZN MKTP" or "AMZN Mktp" = Amazon Marketplace
- "NFLX" = Netflix
- "GOOGL*YOUTUBE" = YouTube
- "SQ *[NAME]" = the name after the asterisk is the merchant
- "PAYPAL *[NAME]" = if it's a business name, use that; if it's a person's name, output "Unknown"
- Store chains followed by # and numbers = just the store chain name
- Apostrophes in brand names (Dunkin', McDonald's, Chick-fil-A, etc.) should be preserved

<scratchpad>
Before providing your answer, think through:
1. Is there a payment processor prefix? If so, what comes after it?
2. Are there any recognizable brand abbreviations or names?
3. Is this a delivery service transaction? If so, the delivery service is the merchant.
4. Are there store numbers, locations, or other metadata that should be excluded?
5. What is the canonical, customer-facing name of this merchant?
6. If the transaction is too generic or appears to be person-to-person, should this be "Unknown"?
</scratchpad>

After your analysis, output only the canonical merchant name. This should be:
- The recognizable brand name (e.g., "Starbucks", "Target", "Netflix")
- Properly capitalized and formatted as the brand presents itself
- Free of store numbers, locations, phone numbers, and transaction codes
- If the merchant cannot be determined or the transaction is too generic, output "Unknown"

Your final response should contain only the merchant name, nothing else.

Same shape, same story. Starting from Anthropic's prompt, EigenPrompt held the original's 67% accuracy for $0.034 per 1,000 calls, within a rounding error of ten times cheaper for an identical score. Push for quality instead and the frontier reaches 76% at $0.061, nine points up and still five and a half times cheaper than the start. The winning prompt leans on structure, not length:

Extract the canonical merchant name from the transaction.
<transaction>{{INPUT}}</transaction>

Rules:
- Strip processor prefixes (SQ*, PAYPAL*, STRIPE*, etc.).
- Expand abbreviations (AMZN, NFLX, APLPAY, GOOGL).
- Remove store numbers (#), locations, and phone numbers.
- For delivery apps (DoorDash, Uber Eats), the app is the merchant.
- Prefer specific brand over parent company.
- Exclude personal names and generic terms (POS DEBIT, P2P).

Output only the properly capitalized brand name. If unidentifiable, output "Unknown".

And it did not get there by getting longer: twelve lines, down from forty-eight. It strips the processor prefixes, expands the abbreviations that actually occur in the data, and stops. Sixty-one variants, about half an hour, a couple of dollars.

Where the bloat actually hides

So how is there a 10× gap on a task this small? These are one-line classification calls: the descriptor is short, the answer is two or three words.

We have cost numbers, not a token-by-token autopsy, so we won't pretend to know the exact split. But with both prompts on the page, you can see the mechanism. Anthropic's draft carries a <scratchpad> block that walks the model through six questions before every answer. OpenAI's insists on "step-by-step reasoning" ahead of each response. The winning prompts spend their closing lines undoing exactly that, capping the model's output: "reason internally," "output only the brand name," "a single line, no additional text." On a reasoning model like gpt-oss-120b, the expensive part of a call is not the prompt you wrote. It is the thinking the prompt sets in motion. A prompt that invites the model to deliberate buys reasoning tokens on every request, needed or not, and for "is this DoorDash or Chipotle," they are mostly not needed.

None of it looks like waste on the page. A scratchpad reads as rigor; few-shot examples read as diligence. The cost shows up only on the invoice, which is how it clears review and slips into production unnoticed.

This is about the prompt, not the model

None of this is a knock on GPT or Claude, or a claim that one lab writes better prompts than the other. The generators did the hard part. They turn a blank page into a working prompt in seconds, and both prompts here work. Anthropic's even started ahead on accuracy.

Our point is narrower and more useful. A generated prompt is a first draft: an untested guess about what your model needs, measured against neither your data nor your budget. Ship it and you have quietly committed to paying an order of magnitude more per call than the task requires, and nobody on the team ever decided to. Prompt generators solve the blank-page problem. They were never built to solve the measurement problem.

Ship the frontier, not the first draft

There is no single best prompt, only a frontier of trade-offs between accuracy and cost. The prompt a generator hands you is one point on that curve, and usually not the one you would pick if you could see the rest. EigenPrompt draws the whole curve. It rewrites and tests prompt after prompt against your own labeled data, on the model you actually pay for, scores each on accuracy and on dollars, and returns the set that nothing else beats on both. Then you choose the point that fits: cheapest-that-clears-the-bar, or most-accurate-you-can-afford.

Nobody sat down and out-wrote the labs. The search measured its way through dozens of variants of their drafts and kept only the points nothing else could beat.

For this task, starting from prompts the best generators could write, that meant the same answers at a fraction of the price. Your task is different, and your numbers will be too. The move is the same: don't ship the first draft. Measure it, then ship the point on the frontier you actually want to pay for.

We went deeper on this exact dataset in the full entity-resolution case study, which walks through Efficient, Standard, and Advanced modes and the label-quality ceiling that caps every prompt.