{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "470c58c216122960",
      "metadata": {},
      "source": [
        "# Evaluating Documentation Agent\n",
        "\n",
        "In this recipe, we will be using taking our [Documentation Agent](https://github.com/Mirascope/mirascope/tree/dev/examples/cookbook/agents/documentation_agent/agent.py) example and running evaluations on the LLM call. We will be exploring various different evaluations we can run to ensure quality and expected behavior.\n",
        "\n",
        "<div class=\"admonition tip\">\n",
        "<p class=\"admonition-title\">Mirascope Concepts Used</p>\n",
        "<ul>\n",
        "<li><a href=\"../../../learn/prompts/\">Prompts</a></li>\n",
        "<li><a href=\"../../../learn/calls/\">Calls</a></li>\n",
        "<li><a href=\"../../../learn/chaining/\">Chaining</a></li>\n",
        "<li><a href=\"../../../learn/evals/\">Evals</a></li>\n",
        "</ul>\n",
        "</div>\n",
        "\n",
        "<div class=\"admonition note\">\n",
        "<p class=\"admonition-title\">Check out the Documentation Agent Cookbook</p>\n",
        "<p>\n",
        "We will be using our <code>DocumentationAgent</code> for our evaluations. For a detailed explanation regarding this code snippet, refer to the <a href=\"../../../../cookbook/agents/documentation_agent/\">Documentation Agent Cookbook</a>.\n",
        "</p>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d0a34080",
      "metadata": {},
      "source": [
        "## Setup\n",
        "\n",
        "To set up our environment, first let's install all of the packages we will use:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cad80e33",
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install \"mirascope[openai]\"\n",
        "# for testing and llama index\n",
        "!pip install  ipytest pytest llama-index"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1ed75379",
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"YOUR_API_KEY\"\n",
        "# Set the appropriate API key for the provider you're using"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "940afcdf21ebb022",
      "metadata": {},
      "source": [
        "## Basic Evaluations\n",
        "\n",
        "We will first test the functionality of our LLM Rerank function to ensure it performs as intended. We have prepared a list of mock documents, each with an assigned semantic score, simulating retrieval from our vector store. The LLM Rerank function will then reorder these documents based on their relevance to the query, rather than relying solely on their semantic scores.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "576da54628220bfe",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-09-30T14:15:48.358365Z",
          "start_time": "2024-09-30T14:15:45.467608Z"
        }
      },
      "outputs": [],
      "source": [
        "import ipytest\n",
        "import pytest\n",
        "from mirascope.core import openai, prompt_template\n",
        "from pydantic import BaseModel, Field\n",
        "\n",
        "ipytest.autoconfig()\n",
        "\n",
        "documents = [\n",
        "    {\"id\": 0, \"text\": \"Bob eats burgers every day.\", \"semantic_score\": 0.8},\n",
        "    {\"id\": 1, \"text\": \"Bob's favorite food is not pizza.\", \"semantic_score\": 0.9},\n",
        "    {\"id\": 2, \"text\": \"I ate at In-N-Out with Bob yesterday\", \"semantic_score\": 0.5},\n",
        "    {\"id\": 3, \"text\": \"Bob said his favorite food is burgers\", \"semantic_score\": 0.9},\n",
        "]\n",
        "\n",
        "\n",
        "class Relevance(BaseModel):\n",
        "    id: int = Field(..., description=\"The document ID\")\n",
        "    score: int = Field(..., description=\"The relevance score (1-10)\")\n",
        "    document: str = Field(..., description=\"The document text\")\n",
        "    reason: str = Field(..., description=\"A brief explanation for the assigned score\")\n",
        "\n",
        "\n",
        "@openai.call(\n",
        "    \"gpt-4o-mini\",\n",
        "    response_model=list[Relevance],\n",
        "    json_mode=True,\n",
        ")\n",
        "@prompt_template(\n",
        "    \"\"\"\n",
        "    SYSTEM:\n",
        "    Document Relevance Assessment\n",
        "    Given a list of documents and a question, determine the relevance of each document to answering the question.\n",
        "\n",
        "    Input\n",
        "        - A question\n",
        "        - A list of documents, each with an ID and content summary\n",
        "\n",
        "    Task\n",
        "        - Analyze each document for its relevance to the question.\n",
        "        - Assign a relevance score from 1-10 for each document.\n",
        "        - Provide a reason for each score.\n",
        "\n",
        "    Scoring Guidelines\n",
        "        - Consider both direct and indirect relevance to the question.\n",
        "        - Prioritize positive, affirmative information over negative statements.\n",
        "        - Assess the informativeness of the content, not just keyword matches.\n",
        "        - Consider the potential for a document to contribute to a complete answer.\n",
        "\n",
        "    Important Notes\n",
        "        - Exclude documents with no relevance less than 5 to the question.\n",
        "        - Be cautious with negative statements - they may be relevant but are often less informative than positive ones.\n",
        "        - Consider how multiple documents might work together to answer the question.\n",
        "        - Use the document title and content summary to make your assessment.\n",
        "\n",
        "    Documents:\n",
        "    {documents}\n",
        "\n",
        "    USER: \n",
        "    {query}\n",
        "    \"\"\"\n",
        ")\n",
        "def llm_query_rerank(documents: list[dict], query: str): ...\n",
        "\n",
        "\n",
        "@pytest.mark.parametrize(\n",
        "    \"query,documents,top_n_ids\",\n",
        "    ((\"What is Bob's favorite food\", documents, {3, 0}),),\n",
        ")\n",
        "def test_llm_query_rerank(query: str, documents: list[dict], top_n_ids: set[int]):\n",
        "    \"\"\"Tests that the LLM query rerank ranks more relevant documents higher.\"\"\"\n",
        "    response = llm_query_rerank(documents, query)\n",
        "    results = sorted(response, key=lambda x: x.score or 0, reverse=True)\n",
        "    assert all(result.score > 5 for result in results)\n",
        "    assert top_n_ids.issuperset({result.id for result in results[: len(top_n_ids)]})\n",
        "\n",
        "\n",
        "ipytest.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "37eaf5f0637e453c",
      "metadata": {},
      "source": [
        "\n",
        "Our tests:\n",
        "\n",
        "* Ensures that all returned documents have a relevancy score above 5 out of 10, indicating a minimum threshold of relevance.\n",
        "* Checks that the top-ranked documents (as many as specified in `top_n_ids`) are within the set of expected documents, allowing for some flexibility in the exact ordering.\n",
        "\n",
        "The test accommodates the non-deterministic nature of LLM-based reranking. While we can't expect identical results in every run, especially when multiple documents are similarly relevant, we can at least verify that the output falls within our boundaries.\n",
        "\n",
        "## Evaluating Code Snippets and General Q&A\n",
        "\n",
        "The example documents we are using for our `DocumentationAgent` are the Mirascope docs. Since Mirascope is a python library, users are likely to ask about both code implementation and conceptual understanding. Therefore, our evaluation process needs to address these two distinct scenarios.\n",
        "\n",
        "### Evaluating Code Snippet\n",
        "\n",
        "To ensure the accuracy and functionality of the LLM-generated code, we implement a two-step verification process:\n",
        "\n",
        "* Syntax Validation: We create a general Python code tester to verify the syntactic correctness of the generated code.\n",
        "* Import Verification: Since syntax validation alone is insufficient, we incorporate an additional check for proper imports. This step confirms all modules and dependencies are valid and no \"magic\" imports exist.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "217bb1bd95d224ce",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2024-09-30T14:19:26.294715Z",
          "start_time": "2024-09-30T14:19:23.757790Z"
        }
      },
      "outputs": [],
      "source": [
        "import ast\n",
        "import importlib.util\n",
        "\n",
        "\n",
        "def check_syntax(code_string: str) -> bool:\n",
        "    try:\n",
        "        compile(code_string, \"<string>\", \"exec\")\n",
        "        return True\n",
        "    except SyntaxError as e:\n",
        "        print(f\"Syntax error: {e}\")\n",
        "        return False\n",
        "\n",
        "\n",
        "def is_importable(code_string: str) -> bool:\n",
        "    try:\n",
        "        tree = ast.parse(code_string)\n",
        "    except SyntaxError:\n",
        "        return False\n",
        "\n",
        "    for node in ast.walk(tree):\n",
        "        if isinstance(node, ast.Import | ast.ImportFrom):\n",
        "            module_name = (\n",
        "                node.module if isinstance(node, ast.ImportFrom) else node.names[0].name\n",
        "            )\n",
        "            if not check_module(module_name):\n",
        "                return False\n",
        "\n",
        "            if isinstance(node, ast.ImportFrom):\n",
        "                for alias in node.names:\n",
        "                    if not check_attribute(module_name, alias.name):\n",
        "                        return False\n",
        "\n",
        "    return True\n",
        "\n",
        "\n",
        "def check_module(module_name):\n",
        "    try:\n",
        "        spec = importlib.util.find_spec(module_name)\n",
        "        return spec is not None\n",
        "    except (ImportError, AttributeError, ValueError):\n",
        "        return False\n",
        "\n",
        "\n",
        "def check_attribute(module_name, attribute):\n",
        "    try:\n",
        "        spec = importlib.util.find_spec(module_name)\n",
        "        if spec is None:\n",
        "            return False\n",
        "        module = importlib.util.module_from_spec(spec)\n",
        "        if spec.loader:\n",
        "            spec.loader.exec_module(module)\n",
        "        return hasattr(module, attribute)\n",
        "    except (ImportError, AttributeError):\n",
        "        return False\n",
        "\n",
        "\n",
        "@pytest.mark.parametrize(\n",
        "    \"import_str,expected\",\n",
        "    [\n",
        "        (\"from mirascope.core import openai\", True),\n",
        "        (\"import math\", True),\n",
        "        (\"from datetime import datetime\", True),\n",
        "        (\"import non_existent_module\", False),\n",
        "        (\"from os import path, nonexistent_function\", False),\n",
        "        (\"from sys import exit, nonexistent_variable\", False),\n",
        "        (\"from openai import OpenAI\", True),\n",
        "        (\"from mirascope.core import openai\", True),\n",
        "    ],\n",
        ")\n",
        "def test_is_importable(import_str: str, expected: bool):\n",
        "    assert is_importable(import_str) == expected\n",
        "\n",
        "\n",
        "@pytest.mark.parametrize(\n",
        "    \"syntax,expected\",\n",
        "    [(\"print('Hello, World!')\", True), (\"print('Hello, World!'\", False)],\n",
        ")\n",
        "def test_check_syntax(syntax: str, expected: bool):\n",
        "    assert check_syntax(syntax) == expected\n",
        "\n",
        "\n",
        "ipytest.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a374eb8058fba3ec",
      "metadata": {},
      "source": [
        "Now that we have our `check_syntax` and `is_importable` tests working, we can test our LLM output:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "235b87ab1980a5dc",
      "metadata": {},
      "outputs": [],
      "source": [
        "from typing import Literal, cast\n",
        "\n",
        "from llama_index.core import (\n",
        "    QueryBundle,\n",
        "    load_index_from_storage,\n",
        ")\n",
        "from llama_index.core.indices.vector_store.base import VectorStoreIndex\n",
        "from llama_index.core.retrievers import VectorIndexRetriever\n",
        "from llama_index.core.storage import StorageContext\n",
        "\n",
        "storage_context = StorageContext.from_defaults(persist_dir=\"storage\")\n",
        "loaded_index = load_index_from_storage(storage_context)\n",
        "\n",
        "\n",
        "def get_documents(query: str) -> list[str]:\n",
        "    \"\"\"The get_documents tool that retrieves Mirascope documentation based on the\n",
        "    relevance of the query\"\"\"\n",
        "    query_bundle = QueryBundle(query)\n",
        "    retriever = VectorIndexRetriever(\n",
        "        index=cast(VectorStoreIndex, loaded_index),\n",
        "        similarity_top_k=10,\n",
        "    )\n",
        "    retrieved_nodes = retriever.retrieve(query_bundle)\n",
        "    choice_batch_size = 5\n",
        "    top_n = 2\n",
        "    results: list[Relevance] = []\n",
        "    for idx in range(0, len(retrieved_nodes), choice_batch_size):\n",
        "        nodes_batch = [\n",
        "            {\n",
        "                \"id\": idx + id,\n",
        "                \"text\": node.node.get_text(),  # pyright: ignore[reportAttributeAccessIssue]\n",
        "                \"document_title\": node.metadata[\"document_title\"],\n",
        "                \"semantic_score\": node.score,\n",
        "            }\n",
        "            for id, node in enumerate(retrieved_nodes[idx : idx + choice_batch_size])\n",
        "        ]\n",
        "        results += llm_query_rerank(nodes_batch, query)\n",
        "    results = sorted(results, key=lambda x: x.score or 0, reverse=True)[:top_n]\n",
        "\n",
        "    return [result.document for result in results]\n",
        "\n",
        "\n",
        "class Response(BaseModel):\n",
        "    classification: Literal[\"code\", \"general\"] = Field(\n",
        "        ..., description=\"The classification of the question\"\n",
        "    )\n",
        "    content: str = Field(..., description=\"The response content\")\n",
        "\n",
        "\n",
        "class DocumentationAgent(BaseModel):\n",
        "    @openai.call(\"gpt-4o-mini\", response_model=Response, json_mode=True)\n",
        "    @prompt_template(\n",
        "        \"\"\"\n",
        "        SYSTEM:\n",
        "        You are an AI Assistant that is an expert at answering questions about Mirascope.\n",
        "        Here is the relevant documentation to answer the question.\n",
        "\n",
        "        First classify the question into one of two types:\n",
        "            - General Information: Questions about the system or its components.\n",
        "            - Code Examples: Questions that require code snippets or examples.\n",
        "\n",
        "        For General Information, provide a summary of the relevant documents if the question is too broad ask for more details. \n",
        "        If the context does not answer the question, say that the information is not available or you could not find it.\n",
        "\n",
        "        For Code Examples, output ONLY code without any markdown, with comments if necessary.\n",
        "        If the context does not answer the question, say that the information is not available.\n",
        "\n",
        "        Examples:\n",
        "            Question: \"What is Mirascope?\"\n",
        "            Answer:\n",
        "            A toolkit for building AI-powered applications with Large Language Models (LLMs).\n",
        "            Explanation: This is a General Information question, so a summary is provided.\n",
        "\n",
        "            Question: \"How do I make a basic OpenAI call using Mirascope?\"\n",
        "            Answer:\n",
        "            from mirascope.core import openai, prompt_template\n",
        "\n",
        "\n",
        "            @openai.call(\"gpt-4o-mini\")\n",
        "            def recommend_book(genre: str) -> str:\n",
        "                return f'Recommend a {genre} book'\n",
        "\n",
        "            response = recommend_book(\"fantasy\")\n",
        "            print(response.content)\n",
        "            Explanation: This is a Code Examples question, so only a code snippet is provided.\n",
        "\n",
        "        Context:\n",
        "        {context:list}\n",
        "\n",
        "        USER:\n",
        "        {question}\n",
        "        \"\"\"\n",
        "    )\n",
        "    def _call(self, question: str) -> openai.OpenAIDynamicConfig:\n",
        "        documents = get_documents(question)\n",
        "        return {\"computed_fields\": {\"context\": documents}}\n",
        "\n",
        "    def _step(self, question: str):\n",
        "        answer = self._call(question)\n",
        "        print(\"(Assistant):\", answer.content)\n",
        "\n",
        "    def run(self):\n",
        "        while True:\n",
        "            question = input(\"(User): \")\n",
        "            if question == \"exit\":\n",
        "                break\n",
        "            self._step(question)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3a603d20f5ef8269",
      "metadata": {},
      "outputs": [],
      "source": [
        "@pytest.mark.parametrize(\n",
        "    \"query,expected\",\n",
        "    [\n",
        "        (\"How do I make a basic OpenAI call using Mirascope?\", None),\n",
        "        (\"What is Mirascope?\", \"a toolkit for building AI-powered applications\"),\n",
        "    ],\n",
        ")\n",
        "def test_documentation_agent_code(query: str, expected: str):\n",
        "    documentation_agent = DocumentationAgent()\n",
        "    response = documentation_agent._call(query)\n",
        "    if response.classification == \"code\":\n",
        "        assert check_syntax(response.content) and is_importable(response.content)\n",
        "    else:\n",
        "        assert expected in response.content\n",
        "\n",
        "\n",
        "ipytest.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c44dcff1e93df2ce",
      "metadata": {},
      "source": [
        "\n",
        "### Evaluating General Q&A\n",
        "\n",
        "For non-code responses generated by the LLM, our primary goal is to verify that the LLM is sourcing its responses directly from the Mirascope documentation rather than relying on its broader knowledge base. Here we require that the LLM's response contains a sentence verbatim.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "efa6ac0f037c69f9",
      "metadata": {},
      "outputs": [],
      "source": [
        "@pytest.mark.parametrize(\n",
        "    \"query,expected\",\n",
        "    [\n",
        "        (\"What is Mirascope?\", \"a toolkit for building AI-powered applications\"),\n",
        "    ],\n",
        ")\n",
        "def test_documentation_agent_general(query: str, expected: str):\n",
        "    documentation_agent = DocumentationAgent()\n",
        "    response = documentation_agent._call(query)\n",
        "    if response.classification == \"general\":\n",
        "        assert expected in response.content\n",
        "\n",
        "\n",
        "ipytest.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e154e4cb11d13cce",
      "metadata": {},
      "source": [
        "\n",
        "When adapting this recipe to your specific use-case, consider the following:\n",
        "\n",
        "* Update the Few-Shot examples to match your documents.\n",
        "* Experiment with other providers for LLM Reranking or use multiple LLM Rerankers and average out the scores.\n",
        "* Add history to the `DocumentationAgent` and have the LLM generate the query for `get_documents` for a more relevant semantic search.\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 2
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython2",
      "version": "2.7.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}