How We Doubled LLM Categorization Accuracy (Without Blowing Up Costs)

The story described in this article comes from a cloud automation workflow designed to analyze AWS environments and generate actionable insights for engineering teams. Within that system, large language models were introduced to automatically categorize findings from cloud account scans into structured recommendations.

The following sections outline how we made LLM-based categorization of cloud scan findings something engineers can trust in production, with clean structure, predictable behavior, and sensible costs.

TL;DR

By separating long-lived reference content from per-request data and right-sizing batch size, we significantly improved model performance without increasing costs. This post focuses on the LLM categorization strategy itself: prompt split and small batches.

Context: Project Setup and Baseline Design

At TrackIt, we developed a tool called the WAFR Automation Tool, designed to classify findings from an AWS account scanning service into AWS Well-Architected best practices. It uses Claude 4 Sonnet through Amazon Bedrock, combined with an internal mapping layer. Each finding is mapped to a specific pillar, question, and best practice in line with the Well-Architected framework.

Initially, each request included the entire reference corpus plus approximately 400 findings, and we expected a single JSON array in response. This design introduced instability and occasional failures. The workflow runs on Claude 4 Sonnet via Amazon Bedrock, but the approach and lessons are model agnostic.

Testing: “Let’s See What We’ve Got”

To measure how well the setup worked, we built a small evaluation harness (a lightweight test setup to measure accuracy and consistency):

• Dataset: 100 pre-labeled findings from a single account (our gold set).
• Metric: strict exact match on pillar → question → best practice (no partial credit).

Observations: Baseline accuracy was low, and outputs were volatile with 400-item mega-batches.  Some runs categorized findings correctly, while others returned incomplete or inconsistent JSO. That’s the thing about large batches: one shaky segment can nudge the whole result off course.

Understanding Why the Original Setup Did Not Work

• Token bloat → attention dilution. Re-sending the full Well-Architected corpus every time inflated inputs and distracted the model from the nuances of individual findings. The Well-Architected framework is extensive, covering six pillars with detailed guidance, which is valuable for architects but demanding on token budgets.
• Mega-batching (400 findings) → cascading errors. Asking the model to process hundreds of findings in a single call increased cognitive load. A minor error or formatting issue in one section could easily cascade through the rest of the response, producing inconsistent or incomplete results.

The Fix: Small Moves, Big Wins

1. Separate Static from Dynamic (and let Bedrock handle caching)

• Static prompt: the structured reference content from the AWS Well-Architected Framework (the six pillars, questions, and best practices) along with our mapping instructions, which form a stable and reusable context.
• Dynamic prompt: only the current, small set of findings.

On Amazon Bedrock, we enabled prompt caching so the large static segment would not be recomputed or billed for every request. According to AWS documentation, caching can significantly reduce input-token cost and latency when multiple requests reuse the same prompt prefix, in some cases by up to 90% in cost and 85% in latency. Results vary depending on workload, but the improvement was clearly visible in our setup.

2. Shrink the batch from 400 to 10 findings per call

Smaller batches gave the model more room to reason, reduced cross-item interference, and produced cleaner, more consistent JSON. While this meant sending more calls overall, caching the static knowledge offset most of the additional overhead and kept costs within a practical range for daily use.

Results: Higher Accuracy and More Stable Runs

• Accuracy: went from low to high on the same gold set (a manually verified reference used to measure model accuracy).
• Stability: less variance across runs; far fewer “why did that happen?” moments.
• Cost & latency: more calls but smaller payloads, with prompt caching reducing repeated input work enough to keep both within a comfortable range.

Tip: think of the prompt like a backpack. Keep the map (static knowledge) inside once, and only refresh the snacks (fresh findings) as you go.

Field Guide: Spotting—and Fixing—This in Your Own Pipeline

Red flags you might already see

• A single oversized prompt that mixes a large block of reference text with transient inputs.
• 100+ items per call “to be efficient,” yet accuracy remains low and JSON is inconsistent.
• Results that vary between runs with no clear explanation.

Quick diagnostics (platform-agnostic)

• Build a 50–200 item gold set from your real data.
• Measure exact match for your target schema; track run-to-run variance.
• Monitor invalid-JSON rates closely and treat them as a key reliability signal.

Low-risk experiments that generalize

• Split static vs. dynamic context. If your platform supports it, cache the static prefix (e.g., Bedrock’s prompt caching) so you don’t keep paying for the same long preface. 
• Right-size the batch. Try 5–20 items per call, then tune upward if accuracy and variance stay healthy.
• Enforce strict JSON. Parse and validate; fail fast when structure breaks. If you need stronger guarantees later, Bedrock’s Converse approach supports structured outputs aligned to a JSON schema.

Conclusion: Two Small Changes, Big Impact

The improvement came from two straightforward changes: caching the stable knowledge and keeping batches small. These steps turned a fragile mapping workflow into one engineers can reliably use in production. Beyond accuracy, the pipeline now behaves more predictably across runs and scales better under load without unexpected cost spikes.

For teams working on Well-Architected mappings or any classification task that blends stable reference content with per-request inputs, this approach applies broadly across models and platforms. Start by separating long‑lived context from per‑call data, measure on a small gold set, and adjust batch size until outputs stabilize. If your requirements tighten, layer in stricter JSON validation or schema‑guided responses without changing the core approach.

Want to dig deeper?

• AWS’s Well-Architected overview and the six pillars.
• Amazon Bedrock prompt caching (what it is, why it helps)
• Structured outputs with Bedrock Converse (for teams that want schema-tight JSON)
• Background on Claude 4 Sonnet availability in Bedrock