Data-Driven Prompt Optimization

We asked the OpenAI and Anthropic prompt generators to write the same merchant-extraction prompt, then ran both on a held-out test. Each was a reasonable first draft, and each cost roughly 10x more per call than it needed to for the accuracy it delivered. Starting from those same prompts, EigenPrompt found versions that matched the accuracy for about a tenth of the cost, and more accurate ones that were still several times cheaper.

A screen-by-screen guide to reproducing the Standard EigenPrompt entity-resolution run: exact-match scoring, the entity_resolution dataset, the baseline prompt, target and optimizer models, mode settings, caps, and the result screens to inspect when the run finishes.

Bank transaction descriptors hide the merchant behind processors, app stores, and payment rails. We pointed EigenPrompt at a plain merchant-extraction prompt and compared what Efficient, Standard, and Advanced optimization each bought. Standard went from 64% to 81% while cutting per-call cost 41%; Advanced reached 83%.

EigenPrompt is a data-driven prompt optimizer that automatically rewrites and tests your prompt to find the best trade-offs between accuracy and cost. Pointed at an LLM-based support-ticket router, it took the prompt from 76% to 92% accuracy at sending tickets to the right desk, turned up a cheaper version that still beat the original, and flagged the mislabeled and ambiguous tickets that were capping the score. Clean those up and accuracy reaches 97%.

A clear, up-to-date glossary of prompt optimization and prompt engineering terms — from eval leakage and prompt caching to reasoning tokens, tool calling, and DSPy.