Foundation Evals Lab
A closer look at what your model actually does.
View on GitHub
About the project
Foundation Evals Lab is a native macOS evaluation workbench for Apple’s Foundation Models. It gives prompts, test cases, scoring, and run history a shared place, so you can investigate model behaviour and compare changes with a saved baseline.
Build repeatable evaluations
Organise cases, instructions, and scoring into suites you can run again as you change your prompts or model settings.
Look beyond the answer
Inspect responses, scores, timing, and execution traces to understand how a run reached its result.
Compare with a baseline
Keep run history and compare saved evaluations to see where behaviour improves or regresses.
Who is Foundation Evals Lab for?
Foundation Evals Lab is for developers testing prompts, instructions, and tools built around Apple’s Foundation Models. Saved suites and baseline comparisons make it useful when you need to understand whether a change improves the responses your app depends on.
Install Foundation Evals Lab
- Download the latest macOS DMG from the GitHub releases page.
- Open the disk image, drag Foundation Evals into Applications, then eject the disk image.
- Launch the app and check that the on-device model is ready. Xcode is only needed if you choose to build from source.
Run your first evaluation
- Open Suite Editor, enter shared instructions, and add cases with prompts and expected answers where appropriate.
- Choose exact-text, contains-text, AI-rubric, or collect-only scoring, then set the number of repetitions.
- Run the suite and inspect each response, score, explanation, and execution trace.
- Use Run History to select a saved baseline and compare results, or export a JSON report.
Requirements and limitations
- macOS 27 or later and a Mac that supports Apple Intelligence.
- For the default on-device provider, enable Apple Intelligence and download its model.
- Xcode 27 is required to build the app from source; it is not required for the downloaded app.
An evaluation measures the cases you provide. Review AI-rubric explanations and use verified reference answers for factual tasks; a few repetitions do not establish statistical significance. On-device inference runs locally, while configured HTTP providers and tools receive the content needed for their calls. Exported traces can contain prompts, responses, and tool data.
Read the setup, provider, and data-handling guide ↗