Testing LLM Apps with Trace-based Tests

This page summarizes the projects mentioned and recommended in the original post on dev.to

CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
coderabbit.ai
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • opentelemetry-collector

    OpenTelemetry Collector

    To add telemetry to our app you will use OpenTelemetry, an open-source observability framework for generating, capturing, and collecting telemetry data such as logs, metrics, and traces from software services and applications.

  • CodeRabbit

    CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

    CodeRabbit logo
  • tracetest

    🔭 Tracetest - Build integration and end-to-end tests in minutes, instead of days, using OpenTelemetry and trace-based testing.

    For this article, I’ll show a simplified version of the code, to show how we can trigger an LLM task, expose it via API and then instrument it. You can see the complete source code, here.

  • openllmetry

    Open-source observability for your LLM application, based on OpenTelemetry

    To instrument our app you will use the CLI tool opentelemetry-instrument that automatically sets up auto-instrumentation in your code without needing to do boilerplate configuration, and OTel Python SDK and OpenLLMetry to do manual instrumentation and specific instrumentation for LLM SDKs, like the file ./app/telemetry.py in the example:

  • liboai

    A C++17 library to access the entire OpenAI API.

    This class of AI algorithms has been widely used in enterprise-level applications due to its capability of contextual understanding, being scalable, and handling large volumes of text data, among other features. With simple integration APIs like OpenAI, Google Gemini, Hugging Face, and Antrophic, and good frameworks that deal with external providers and in-house models, like LangChain, developers can implement interesting applications that use LLMs for internal tasks.

  • jaeger

    CNCF Jaeger, a Distributed Tracing Platform

    By running the make start/on-docker command at beginning of the section, you already started the API and also an Observability stack with an OpenTelemetry Collector and Jaeger.

  • deepeval

    The LLM Evaluation Framework

    This test does a call to our API and performs some simple assertions to check if you received a proper output, a valid JSON with a summary field in it. You also could do some tests to see if the text is relevant and have low accuracies (like the Python deepeval lib does).

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Amazon EKS Monitoring with OpenTelemetry [Step By Step Guide]

    5 projects | dev.to | 5 Dec 2023
  • Ending Microservices Chaos: How Architecture Governance Keeps Your Microservices on Track

    1 project | dev.to | 22 Oct 2024
  • Desbloqueie a Observabilidade: Guia prático com OpenTelemetry e Prometheus

    4 projects | dev.to | 16 Oct 2024
  • Prometheus vs CloudWatch for Cloud Native Applications (Updated in 2024)

    2 projects | dev.to | 9 Oct 2024
  • OpenTelemetry and vendor neutrality: how to build an observability strategy

    10 projects | news.ycombinator.com | 18 Sep 2024

Did you konow that Go is
the 4th most popular programming language
based on number of metions?