LLM Eval & Testing

Arize AI provides an end-to-end ML observability platform, expanding to include deep capabilities for LLM evaluation and monitoring.

Vellum provides a platform for prompt engineering, LLM deployment, and evaluation with built-in analytics and monitoring.

Humanloop offers tools for fine-tuning, evaluating, and deploying large language models with human feedback.

Langfuse is an open-source observability and evaluation platform for LLM applications, tracking traces, metrics, and user feedback.

An open-source tool and platform for testing and evaluating LLM prompts and models.

Promptlayer acts as an API wrapper for all LLMs, providing logging, analytics, and prompt management for developers.

Patronus AI offers an automated LLM evaluation platform to detect flaws such as hallucinations, toxicity, and bias before deployment.

Helicone (an open-source project supported by Braintrust) provides an observability platform for LLMs, including logging, caching, and analytics.

Giskard is an open-source platform for ML model testing, including an expanding focus on evaluating and debugging LLMs for security and robustness.