{ "cells": [ { "cell_type": "markdown", "id": "d9b7d2edd30ecc1d", "metadata": {}, "source": [ "# LLM Validation With Retries\n", "\n", "This recipe demonstrates how to leverage Large Language Models (LLMs) -- specifically Anthropic's Claude 3.5 Sonnet -- to perform automated validation on any value. We'll cover how to use **LLMs for complex validation tasks**, how to integrate this with Pydantic's validation system, and how to leverage [Tenacity](https://tenacity.readthedocs.io/en/latest/) to automatically **reinsert validation errors** back into an LLM call to **improve results**.\n", "\n", "
\n", "

Mirascope Concepts Used

\n", "\n", "
\n", "\n", "
\n", "

Background

\n", "

\n", "While traditional validation tools like type checkers or Pydantic are limited to hardcoded rules (such as variable types or arithmetic), LLMs allow for much more nuanced and complex validation. This approach can be particularly useful for validating natural language inputs or complex data structures where traditional rule-based validation falls short.\n", "

\n", "
" ] }, { "cell_type": "markdown", "id": "7fb27b941602401d91542211134fc71a", "metadata": {}, "source": [ "## Setup\n", "\n", "Let's start by installing Mirascope and its dependencies:" ] }, { "cell_type": "code", "execution_count": null, "id": "acae54e37e7d407bbb7b55eff062a284", "metadata": {}, "outputs": [], "source": [ "!pip install \"mirascope[anthropic,tenacity]\"" ] }, { "cell_type": "code", "execution_count": null, "id": "9a63283cbaf04dbcab1f6479b197f3a8", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"ANTHROPIC_API_KEY\"] = \"YOUR_API_KEY\"\n", "# Set the appropriate API key for the provider you're using" ] }, { "cell_type": "markdown", "id": "3b29386fb53eaab3", "metadata": {}, "source": [ "## Basic LLM Validation\n", "\n", "Let's start with a simple example of using an LLM to check for spelling and grammatical errors in a text snippet:" ] }, { "cell_type": "code", "execution_count": 1, "id": "97f8099cef79eb38", "metadata": { "ExecuteTime": { "end_time": "2024-09-30T08:32:38.440282Z", "start_time": "2024-09-30T08:32:36.481710Z" } }, "outputs": [], "source": [ "from mirascope.core import anthropic, prompt_template\n", "from pydantic import BaseModel, Field\n", "\n", "\n", "class SpellingAndGrammarCheck(BaseModel):\n", " has_errors: bool = Field(\n", " description=\"Whether the text has typos or grammatical errors\"\n", " )\n", "\n", "\n", "@anthropic.call(\n", " model=\"claude-3-5-sonnet-20240620\",\n", " response_model=SpellingAndGrammarCheck,\n", " json_mode=True,\n", ")\n", "@prompt_template(\n", " \"\"\"\n", " Does the following text have any typos or grammatical errors? {text}\n", " \"\"\"\n", ")\n", "def check_for_errors(text: str): ...\n", "\n", "\n", "text = \"Yestday I had a gret time!\"\n", "response = check_for_errors(text)\n", "assert response.has_errors" ] }, { "cell_type": "markdown", "id": "e8c59d429c096513", "metadata": {}, "source": [ "\n", "## Pydantic's AfterValidator\n", "\n", "We can use Pydantic's [`AfterValidator`](https://docs.pydantic.dev/latest/api/functional_validators/#pydantic.functional_validators.AfterValidator) to integrate our LLM-based validation directly into a Pydantic model:\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "4ae1574e6ac56570", "metadata": { "ExecuteTime": { "end_time": "2024-09-30T08:32:40.590085Z", "start_time": "2024-09-30T08:32:39.154588Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Validation error: 1 validation error for TextWithoutErrors\n", "text\n", " Value error, Text contains errors [type=value_error, input_value='I walkd to supermarket a... i picked up some fish?', input_type=str]\n", " For further information visit https://errors.pydantic.dev/2.8/v/value_error\n" ] } ], "source": [ "from typing import Annotated\n", "\n", "from mirascope.core import anthropic, prompt_template\n", "from pydantic import AfterValidator, BaseModel, ValidationError\n", "\n", "\n", "@anthropic.call(\n", " model=\"claude-3-5-sonnet-20240620\",\n", " response_model=SpellingAndGrammarCheck,\n", " json_mode=True,\n", ")\n", "@prompt_template(\n", " \"\"\"\n", " Does the following text have any typos or grammatical errors? {text}\n", " \"\"\"\n", ")\n", "def check_for_errors(text: str): ...\n", "\n", "\n", "class TextWithoutErrors(BaseModel):\n", " text: Annotated[\n", " str,\n", " AfterValidator(\n", " lambda t: t\n", " if not (check_for_errors(t)).has_errors\n", " else (_ for _ in ()).throw(ValueError(\"Text contains errors\"))\n", " ),\n", " ]\n", "\n", "\n", "valid = TextWithoutErrors(text=\"This is a perfectly written sentence.\")\n", "\n", "try:\n", " invalid = TextWithoutErrors(\n", " text=\"I walkd to supermarket and i picked up some fish?\"\n", " )\n", "except ValidationError as e:\n", " print(f\"Validation error: {e}\")" ] }, { "cell_type": "markdown", "id": "3a2a8c2028694bb9", "metadata": {}, "source": [ "\n", "## Reinsert Validation Errors For Improved Performance\n", "\n", "One powerful technique for enhancing LLM generations is to automatically reinsert validation errors into subsequent calls. This approach allows the LLM to learn from its previous mistakes as few-shot examples and improve it's output in real-time. We can achieve this using Mirascope's integration with Tenacity, which collects `ValidationError` messages for easy insertion into the prompt.\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "310244d961466026", "metadata": { "ExecuteTime": { "end_time": "2024-09-30T08:32:44.904901Z", "start_time": "2024-09-30T08:32:42.693052Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Corrected text: I went to the store yesterday and bought some milk.\n", "Explanation: The sentence has been corrected as follows: 'has went' was changed to 'went' (simple past tense of 'go'), and 'buyed' was changed to 'bought' (past tense of 'buy'). The subject-verb agreement was also fixed by changing 'I has' to 'I'.\n" ] } ], "source": [ "from typing import Annotated\n", "\n", "from mirascope.core import anthropic, prompt_template\n", "from mirascope.integrations.tenacity import collect_errors\n", "from pydantic import AfterValidator, BaseModel, Field, ValidationError\n", "from tenacity import retry, stop_after_attempt\n", "\n", "\n", "class SpellingAndGrammarCheck(BaseModel):\n", " has_errors: bool = Field(\n", " description=\"Whether the text has typos or grammatical errors\"\n", " )\n", "\n", "\n", "@anthropic.call(\n", " model=\"claude-3-5-sonnet-20240620\",\n", " response_model=SpellingAndGrammarCheck,\n", " json_mode=True,\n", ")\n", "@prompt_template(\n", " \"\"\"\n", " Does the following text have any typos or grammatical errors? {text}\n", " \"\"\"\n", ")\n", "def check_for_errors(text: str): ...\n", "\n", "\n", "class GrammarCheck(BaseModel):\n", " text: Annotated[\n", " str,\n", " AfterValidator(\n", " lambda t: t\n", " if not (check_for_errors(t)).has_errors\n", " else (_ for _ in ()).throw(ValueError(\"Text still contains errors\"))\n", " ),\n", " ] = Field(description=\"The corrected text with proper grammar\")\n", " explanation: str = Field(description=\"Explanation of the corrections made\")\n", "\n", "\n", "@retry(stop=stop_after_attempt(3), after=collect_errors(ValidationError))\n", "@anthropic.call(\n", " \"claude-3-5-sonnet-20240620\", response_model=GrammarCheck, json_mode=True\n", ")\n", "@prompt_template(\n", " \"\"\"\n", " {previous_errors}\n", "\n", " Correct the grammar in the following text.\n", " If no corrections are needed, return the original text.\n", " Provide an explanation of any corrections made.\n", "\n", " Text: {text}\n", " \"\"\"\n", ")\n", "def correct_grammar(\n", " text: str, *, errors: list[ValidationError] | None = None\n", ") -> anthropic.AnthropicDynamicConfig:\n", " previous_errors = f\"Previous Errors: {errors}\" if errors else \"No previous errors.\"\n", " return {\"computed_fields\": {\"previous_errors\": previous_errors}}\n", "\n", "\n", "try:\n", " text = \"I has went to the store yesterday and buyed some milk.\"\n", " result = correct_grammar(text)\n", " print(f\"Corrected text: {result.text}\")\n", " print(f\"Explanation: {result.explanation}\")\n", "except ValidationError:\n", " print(\"Failed to correct grammar after 3 attempts\")" ] }, { "cell_type": "markdown", "id": "baf8333f50b979b0", "metadata": {}, "source": [ "
\n", "

Additional Real-World Examples

\n", "\n", "
\n", "\n", "\n", "When adapting this recipe to your specific use-case, consider the following:\n", "\n", "- Tailor the prompts to provide clear instructions and relevant context for your specific validation tasks.\n", "- Balance the trade-off between validation accuracy and performance, especially when implementing retries.\n", "- Implement proper error handling and logging for production use.\n", "- Consider caching validation results for frequently validated items to improve performance.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 5 }