Evaluating Generative AI Models on Amazon Bedrock

You're a SaaS company looking to build an AI-powered customer support system. You've identified a few LLMs that can summarize customer queries and classify them by issue type. But which one should you choose?

Several models could do the job, but one might be more accurate, another faster, and a third more cost-effective. How do you determine which model offers the right balance for your application?

Choosing the Right Model

Selecting the right large language model (LLM) for a production workload involves balancing several factors, including output quality, inference cost, and latency. With a growing range of models available from different providers, choosing the right one requires a structured evaluation process that accounts for the specific technical requirements and business priorities of each use case.

Existing evaluation frameworks such as DeepEval provide capabilities for assessing model performance. However, these tools can require technical expertise to configure and operate, making them less accessible to business stakeholders and other non-technical users who also need to participate in model selection.

The Gen AI Model Evaluator Tool

The Gen AI Model Evaluator Tool was developed by TrackIt to make this evaluation process more accessible. The tool allows technical and non-technical users to evaluate inference models available through Amazon Bedrock using their own datasets, without requiring the entire process to be managed through code. It currently supports text summarization and classification tasks, with the ability to upload structured data in CSV and JSON formats, as well as unstructured documents such as PDFs and Word files.

The tool supports deterministic metrics, LLM-as-a-Judge evaluations, and a hybrid approach combining both methods. Evaluation results can be assessed against three key criteria: accuracy, cost, and latency. Users can also assign custom weights to each of these criteria to reflect the priorities of a particular application and compare models accordingly.

1. Weight Customization

The tool allows users to assign custom weights to accuracy, cost, and latency based on their application’s priorities. These metrics measure output quality, price per 1,000 tokens, and response time, respectively. The weights must add up to 100% and determine how models are ranked, making it easier to compare inference options based on specific requirements.

2. Model Selection

Three or more Amazon Bedrock models can be selected for evaluation against the same dataset. The interface displays each model’s provider, context window, and cost per 1,000 tokens, making it easier to compare available options. Custom Bedrock model IDs can also be added, allowing evaluations to include models beyond the default selection.

3. Dataset Upload 

Both structured datasets in CSV and JSON formats and unstructured documents in PDF, DOC, and DOCX formats are supported. Multiple documents can be uploaded together to create an evaluation dataset. For unstructured inputs, the tool preprocesses the documents and transforms their content into a structured format suitable for evaluation, making it possible to assess model performance using existing documentation without manually preparing a dataset from scratch.

4. Task Selection and Dataset Preprocessing

After uploading the dataset, users can choose between summarization and classification, depending on the intended evaluation task. For document-based datasets, the tool also offers two chunking strategies: processing each detected section as a separate sample or treating each uploaded file as a single sample. The preprocessing workflow parses the documents and generates synthetic outputs to prepare the dataset for evaluation.

5. Deterministic Evaluation Metrics

The tool offers a range of programmatic metrics for evaluating model outputs against reference results. For summarization, these include BLEU, ROUGE, METEOR, Levenshtein similarity, and BERTScore, which assess different aspects of textual overlap and similarity. Users can select the metrics relevant to their evaluation, with most offering fast, local scoring and minimal computational overhead.

6. LLM-as-a-Judge Metrics

The tool also supports LLM-as-a-Judge evaluation, using an additional Amazon Bedrock model to assess the quality of generated outputs. Available metrics include reasoning, which evaluates coherence and logical consistency, and faithfulness, which checks whether outputs remain grounded in the source material. These evaluations provide a more qualitative assessment of model performance, complementing the deterministic metrics.

7. Dataset Review

Before evaluation begins, users can review the processed dataset, including the source documents and their generated summaries. This step provides an opportunity to verify the prepared samples and confirm the dataset before running the evaluation. If changes are needed, the dataset can be re-uploaded before proceeding.

8. Evaluation Results and Model Recommendations

Once the evaluation is complete, the tool presents a recommended model based on the configured weights for accuracy, cost, and latency. The results dashboard summarizes key performance indicators, including the weighted accuracy score, total inference cost, response latency, and throughput. This provides a quick overview of how each model performs against the selected priorities.

The dashboard also includes a detailed accuracy breakdown, showing individual metric scores and their contribution to the final weighted accuracy. Additional charts compare cost, latency, time to first token, and throughput across the selected models, helping teams understand the trade-offs behind the recommendation and make more informed model selection decisions.

Conclusion

Evaluating LLM inference models requires balancing output quality, cost, and latency against the needs of a specific application. TrackIt's Gen AI Model Inference Evaluation Tool simplifies this process by combining configurable model rankings, flexible dataset inputs, and both deterministic and LLM-as-a-Judge evaluation methods in a single interface.

The tool is available at genai-model-inference-evaluation-tool.trackit.io.

TrackIt can design and implement tailored evaluation tools that reflect specific workloads, performance metrics, and infrastructure. Contact TrackIt to discuss building a custom evaluation solution aligned with existing systems and business requirements.