{ "cells": [ { "cell_type": "markdown", "id": "4c2cb6bf3e30bf22", "metadata": {}, "source": [ "# Chain of Verification: Enhancing LLM Accuracy through Self-Verification\n", "\n", "This recipe demonstrates how to implement the Chain of Verification technique using Large Language Models (LLMs) with Mirascope. Chain of Verification is a prompt engineering method that enhances an LLM's accuracy by generating and answering verification questions based on its initial response.\n", "\n", "
\n", "

Mirascope Concepts Used

\n", "\n", "
\n", "\n", "
\n", "

Background

\n", "

\n", "Chain of Verification is a prompt engineering technique where one takes a prompt and its initial LLM response then generates a checklist of questions that can be used to verify the initial answer. Each of these questions are then answered individually with separate LLM calls, and the results of each verification question are used to edit the final answer. LLMs are often more truthful when asked to verify a particular fact rather than use it in their own answer, so this technique is effective in reducing hallucinations.\n", "

\n", "
\n", "\n", "\n", "## Implementation\n", "\n", "Let's implement the Chain of Verification technique using Mirascope:\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "94d761e0986a08cb", "metadata": { "ExecuteTime": { "end_time": "2024-10-01T00:58:06.449710Z", "start_time": "2024-10-01T00:57:58.230769Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Here are five politicians who were born in New York:\n", "\n", "1. **Theodore Roosevelt** - 26th President of the United States, born in New York City, New York.\n", "2. **Al Smith** - Governor of New York and Democratic presidential candidate, born in New York City, New York.\n", "3. **Andrew Cuomo** - Former Governor of New York, born in Queens, New York City.\n", "4. **Franklin D. Roosevelt** - 32nd President of the United States, born in Hyde Park, New York.\n", "5. **Donald Trump** - 45th President of the United States, born in Queens, New York City.\n", "\n", "These individuals have all made significant contributions to American politics and governance. Note that Hillary Clinton, while a prominent politician, was actually born in Chicago, Illinois.\n" ] } ], "source": [ "import asyncio\n", "\n", "from mirascope.core import openai, prompt_template\n", "from pydantic import BaseModel, Field\n", "\n", "\n", "@openai.call(\"gpt-4o-mini\")\n", "def call(query: str) -> str:\n", " return query\n", "\n", "\n", "class VerificationQuestions(BaseModel):\n", " questions: list[str] = Field(\n", " ...,\n", " description=\"\"\"A list of questions that verifies if the response\n", " answers the original query correctly.\"\"\",\n", " )\n", "\n", "\n", "@openai.call(\"gpt-4o-mini\", response_model=VerificationQuestions)\n", "@prompt_template(\n", " \"\"\"\n", " SYSTEM:\n", " You will be given a query and a response to the query.\n", " Take the relevant statements in the response and rephrase them into questions so\n", " that they can be used to verify that they satisfy the original query.\n", " USER:\n", " Query:\n", " {query}\n", "\n", " Response:\n", " {response}\n", " \"\"\"\n", ")\n", "def get_verification_questions(query: str, response: str): ...\n", "\n", "\n", "@openai.call(\"gpt-4o-mini\")\n", "async def answer(query: str) -> str:\n", " return f\"Concisely answer the following question: {query}\"\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "@prompt_template(\n", " \"\"\"\n", " Here is the original query:\n", " {query}\n", "\n", " Here is an initial response to the query:\n", " {response}\n", "\n", " Here is some fact checking on the response:\n", " {verification_q_and_a:list}\n", "\n", " Using the knowledge you learned from verification, re-answer the original query.\n", " \"\"\"\n", ")\n", "async def cov_call(query: str) -> openai.OpenAIDynamicConfig:\n", " response = call(query).content\n", " verification_questions = get_verification_questions(query, response).questions\n", " tasks = [answer(question) for question in verification_questions]\n", " responses = await asyncio.gather(*tasks)\n", " verification_answers = [response.content for response in responses]\n", " verification_q_and_a = [\n", " [f\"Q:{q}\", f\"A:{a}\"]\n", " for q, a in zip(verification_questions, verification_answers)\n", " ]\n", " return {\n", " \"computed_fields\": {\n", " \"response\": response,\n", " \"verification_q_and_a\": verification_q_and_a,\n", " }\n", " }\n", "\n", "\n", "async def chain_of_verification(query: str):\n", " response = await cov_call(query=query)\n", " # Uncomment to see intermediate responses\n", " # print(response.user_message_param[\"content\"])\n", " return response\n", "\n", "\n", "query = \"Name 5 politicians born in New York.\"\n", "\n", "print(await chain_of_verification(query=query))" ] }, { "cell_type": "markdown", "id": "c654dafd92c7e044", "metadata": {}, "source": [ "As we can see, the Chain of Verification process has successfully identified and corrected an error in the initial response (Hillary Clinton's birthplace), demonstrating its effectiveness in improving accuracy.\n", "\n", "## Benefits and Considerations\n", "\n", "The Chain of Verification implementation offers several advantages:\n", "\n", "1. Improved accuracy by systematically verifying initial responses.\n", "2. Reduction of hallucinations and factual errors in LLM outputs.\n", "3. Transparent fact-checking process that can be easily audited.\n", "\n", "When implementing this technique, consider:\n", "\n", "- Balancing the number of verification questions with response time and computational cost.\n", "- Tailoring the verification question generation process to your specific use case.\n", "- Implementing error handling for cases where verification reveals significant discrepancies.\n", "\n", "When adapting this recipe to your specific use-case, consider:\n", "\n", "- Customizing the verification question generation process for your domain.\n", "- Implementing a feedback loop to continuously improve the verification process based on user feedback or expert review.\n", "- Combining Chain of Verification with other techniques like Chain of Thought for even more robust reasoning capabilities.\n", "- Experimenting with different LLM models for various stages of the verification process to optimize for accuracy and efficiency.\n", "\n", "By leveraging Mirascope's `call` decorator, response models, and dynamic configuration, you can easily implement and customize the Chain of Verification technique to enhance your LLM's accuracy across a wide range of applications.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 5 }