{ "cells": [ { "cell_type": "markdown", "id": "634e4707c4e77143", "metadata": {}, "source": [ "# Reverse Chain of Thought: Enhancing LLM Reasoning with Self-Reflection\n", "\n", "This recipe demonstrates how to implement the Reverse Chain of Thought technique using Large Language Models (LLMs) with Mirascope. Reverse Chain of Thought is a prompt engineering method that enhances an LLM's reasoning capabilities by encouraging it to reflect on and correct its own thought process.\n", "\n", "
\n", "

Mirascope Concepts Used

\n", "\n", "
\n", "\n", "
\n", "

Background

\n", "

\n", "Reverse chain of thought is a prompt engineering technique where a chain of thought call is made for a query, then we attempt to reconstruct the query from the attempted solution. Both the original and reconstructed query are broken down into their individual conditions, and each condition is cross-referenced with the totality of conditions for the other query to determine the existence of overlooked facts or hallucinations. The questions themselves are also compared to ensure that the two queries not only share the context but also ask the same question. This fine-grained comparison is used in a final prompt.\n", "

\n", "

\n", "In the original paper, no prompt was given for the case when mistakes do not exist, so we took the liberty of asking the model to repeat a solution without mistakes. \n", "

\n", "

\n", "Reverse chain of thought is a technique that works for any prompt which has shown signs of susceptibility to hallucinations or misinterpretations in its initial attempts to answer the question.\n", "

\n", "
\n", "\n", "## Implementation\n", "\n", "Let's implement the Reverse Chain of Thought technique using Mirascope:\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "5a9ab313645b2e80", "metadata": { "ExecuteTime": { "end_time": "2024-10-01T02:14:47.063178Z", "start_time": "2024-10-01T02:14:26.805523Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Thank you for your feedback! Based on your clarifications, here’s a revised problem prompt that corresponds to your original question while considering all the necessary conditions:\n", "\n", "---\n", "\n", "**Problem Prompt:**\n", "\n", "At the trip to the county-level scavenger hunt competition, there are 90 participants who need to be divided into groups for the event to start. Each group consists of 9 people. If \\( \\frac{3}{5} \\) of the total number of groups formed contributed by bringing back seashells, and each member of those groups brought back 2 seashells each, how many seashells were brought back in total?\n", "\n", "---\n", "\n", "This prompt encapsulates all essential components of the problem. It clarifies the number of participants, how they are grouped, the fraction of groups contributing, and the number of seashells collected by each member, leading to the final calculation. Let's go through the reasoning for clarity:\n", "\n", "1. **Total Participants**: 90 people.\n", "2. **Groups Formed**: Each group has 9 members, so:\n", " \\[\n", " \\text{Number of groups} = \\frac{90}{9} = 10\n", " \\]\n", "3. **Groups Contributing Shells**: \\( \\frac{3}{5} \\) of the total groups contributed:\n", " \\[\n", " \\text{Groups contributing} = 10 \\times \\frac{3}{5} = 6\n", " \\]\n", "4. **Total Members in Contributing Groups**: Each contributing group has 9 members:\n", " \\[\n", " \\text{Total members contributing} = 6 \\times 9 = 54\n", " \\]\n", "5. **Total Seashells Collected**: Each member brought back 2 seashells:\n", " \\[\n", " \\text{Total seashells} = 54 \\times 2 = 108\n", " \\]\n", "\n", "Thus, the total number of seashells brought back is:\n", "\n", "\\[\n", "\\boxed{108}\n", "\\] \n", "\n", "This approach correctly follows the sequence of logical deductions based on the problem's premise. Thank you for your guidance!\n" ] } ], "source": [ "import asyncio\n", "\n", "from mirascope.core import openai, prompt_template\n", "from mirascope.core.openai import OpenAICallResponse\n", "from openai.types.chat import ChatCompletionMessageParam\n", "from pydantic import BaseModel, Field\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "def zero_shot_cot(query: str) -> str:\n", " return f\"{query} Let's think step by step.\"\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "@prompt_template(\n", " \"\"\"\n", " USER:\n", " Give the concrete prompt (problem) that can generate this answer.\n", " The problem should contain all basic and necessary information and correspond to the answer.\n", " The problem can only ask for one result.\n", "\n", " {response}\n", " \"\"\"\n", ")\n", "def reconstruct_query(response: str): ...\n", "\n", "\n", "class Decomposition(BaseModel):\n", " conditions: list[str] = Field(\n", " ..., description=\"A list of conditions of the problem.\"\n", " )\n", "\n", "\n", "@openai.call(\n", " model=\"gpt-4o-mini\",\n", " response_model=Decomposition,\n", " call_params={\"tool_choice\": \"required\"},\n", ")\n", "@prompt_template(\n", " \"\"\"\n", " Please list the conditions of the problem. There may be multiple conditions.\n", " Do not list conditions not related to calculations, but list all necessary conditions.\n", " The format should be a list of conditions with one condition per item.\n", "\n", " {query}\n", " \"\"\"\n", ")\n", "async def decompose_query(query: str): ...\n", "\n", "\n", "class Comparison(BaseModel):\n", " condition: str = Field(\n", " ..., description=\"The original condition the comparison was made with, verbatim\"\n", " )\n", " deducible: bool = Field(\n", " ...,\n", " description=\"Whether the condition is deducible from the list of other conditions.\",\n", " )\n", " illustration: str = Field(\n", " ...,\n", " description=\"A quick illustration of the reason the condition is/isn't deducible from the list of other conditions.\",\n", " )\n", "\n", "\n", "@openai.call(\n", " model=\"gpt-4o-mini\",\n", " response_model=Comparison,\n", " call_params={\"tool_choice\": \"required\"},\n", ")\n", "@prompt_template(\n", " \"\"\"\n", " Given a candidate condition: '{condition}'\n", "\n", " Here is a condition list: '{condition_list}'\n", "\n", " From a mathematical point of view, can this candidate condition be deduced from the condition list?\n", " Please illustrate your reason and answer True or False.\n", " \"\"\"\n", ")\n", "async def compare_conditions(condition: str, condition_list: list[str]): ...\n", "\n", "\n", "@openai.call(\n", " model=\"gpt-4o-mini\", response_model=bool, call_params={\"tool_choice\": \"required\"}\n", ")\n", "@prompt_template(\n", " \"\"\"\n", " Q1: {original_problem}\n", " Q2: {reconstructed_problem}\n", " \n", " From a mathematical point of view, are these two problems asking the same thing at the end?\n", " \"\"\"\n", ")\n", "def compare_questions(original_problem: str, reconstructed_problem: str): ...\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "@prompt_template(\n", " \"\"\"\n", " MESSAGES: {history}\n", " USER:\n", " {mistakes_prompt}\n", " {overlooked_prompt}\n", " {hallucination_prompt}\n", " {misinterpretation_prompt}\n", " \"\"\"\n", ")\n", "async def fine_grained_comparison(\n", " history: list[ChatCompletionMessageParam], query: str, reconstructed_query: str\n", ") -> openai.OpenAIDynamicConfig:\n", " # Decompose both queries into conditions\n", " original_conditions, reconstructed_conditions = [\n", " response.conditions\n", " for response in await asyncio.gather(\n", " decompose_query(query), decompose_query(reconstructed_query)\n", " )\n", " ]\n", "\n", " # Identify overlooked/hallucinated conditions and misinterpretation of question\n", " overlooking_tasks = [\n", " compare_conditions(original_condition, reconstructed_conditions)\n", " for original_condition in original_conditions\n", " ]\n", " hallucination_tasks = [\n", " compare_conditions(reconstructed_condition, original_conditions)\n", " for reconstructed_condition in reconstructed_conditions\n", " ]\n", " full_comparison = await asyncio.gather(*(overlooking_tasks + hallucination_tasks))\n", "\n", " question_misinterpretation = compare_questions(query, reconstructed_query)\n", "\n", " overlooked_comparisons = [\n", " comparison\n", " for comparison in full_comparison[: len(original_conditions)]\n", " if not comparison.deducible\n", " ]\n", " hallucination_comparisons = [\n", " comparison\n", " for comparison in full_comparison[len(original_conditions) :]\n", " if not comparison.deducible\n", " ]\n", "\n", " # Fill out prompt depending on the comparisons\n", " if (\n", " not question_misinterpretation\n", " and not overlooked_comparisons\n", " and not hallucination_comparisons\n", " ):\n", " mistakes_prompt = \"\"\"There are no mistakes in your interpretation of the prompt.\n", " Repeat your original solution verbatim.\"\"\"\n", " overlooked_prompt = \"\"\n", " hallucination_prompt = \"\"\n", " misinterpretation_prompt = \"\"\n", " else:\n", " mistakes_prompt = (\n", " \"Here are the mistakes and reasons in your answer to the problem.\\n\"\n", " )\n", "\n", " if overlooked_comparisons:\n", " conditions = [comparison.condition for comparison in overlooked_comparisons]\n", " illustrations = [\n", " comparison.illustration for comparison in overlooked_comparisons\n", " ]\n", " overlooked_prompt = f\"\"\"\n", " Overlooked Conditions:\n", " You have ignored some real conditions:\n", " {conditions}\n", " The real problem has the conditions:\n", " {original_conditions}\n", " You should consider all real conditions in the problem.\n", " Here are the detailed reasons:\n", " {illustrations}\"\"\"\n", " else:\n", " overlooked_prompt = \"\"\n", "\n", " if hallucination_comparisons:\n", " conditions = [\n", " comparison.condition for comparison in hallucination_comparisons\n", " ]\n", " illustrations = [\n", " comparison.illustration for comparison in overlooked_comparisons\n", " ]\n", " hallucination_prompt = f\"\"\"\n", " Hallucinated Conditions\n", " You use some wrong candidate conditions:\n", " {conditions}\n", " They all can not be deduced from the true condition list.\n", " The real problem has the conditions:\n", " {original_conditions}\n", " You should consider all real conditions in the problem.\n", " Here are the detailed reasons:\n", " {illustrations}\"\"\"\n", " else:\n", " hallucination_prompt = \"\"\n", "\n", " if question_misinterpretation:\n", " misinterpretation_prompt = f\"\"\"\n", " You misunderstood the question.\n", " You think the question is: {reconstructed_query}.\n", " But the real question is: {query}\n", " They are different. You should consider the original question.\"\"\"\n", " else:\n", " misinterpretation_prompt = \"\"\n", " return {\n", " \"computed_fields\": {\n", " \"mistakes_prompt\": mistakes_prompt,\n", " \"overlooked_prompt\": overlooked_prompt,\n", " \"hallucination_prompt\": hallucination_prompt,\n", " \"misinterpretation_prompt\": misinterpretation_prompt,\n", " }\n", " }\n", "\n", "\n", "async def reverse_cot(query: str) -> OpenAICallResponse:\n", " cot_response = zero_shot_cot(query=query)\n", " reconstructed_query_response = reconstruct_query(cot_response.content)\n", " history = cot_response.messages + reconstructed_query_response.messages\n", " response = await fine_grained_comparison(\n", " history=history,\n", " query=query,\n", " reconstructed_query=reconstructed_query_response.content,\n", " )\n", " return response\n", "\n", "\n", "query = \"\"\"At the trip to the county level scavenger hunt competition 90 people \\\n", "were required to split into groups for the competition to begin. To break \\\n", "people up into smaller groups with different leaders 9-person groups were \\\n", "formed. If 3/5 of the number of groups each had members bring back 2 seashells each \\\n", "how many seashells did they bring?\"\"\"\n", "\n", "print(await reverse_cot(query=query))" ] }, { "cell_type": "markdown", "id": "e2ab5d650d9f9873", "metadata": {}, "source": [ "This implementation consists of several key functions:\n", "\n", "1. `zero_shot_cot`: Generates an initial chain of thought response.\n", "2. `reconstruct_query`: Attempts to reconstruct the original query from the chain of thought response.\n", "3. `decompose_query`: Breaks down a query into its individual conditions.\n", "4. `compare_conditions`: Compares individual conditions to determine if they are deducible from a list of other conditions.\n", "5. `compare_questions`: Checks if two questions are asking the same thing.\n", "6. `fine_grained_comparison`: Performs a detailed comparison between the original and reconstructed queries, identifying overlooked conditions, hallucinations, and misinterpretations.\n", "7. `reverse_cot`: Orchestrates the entire Reverse Chain of Thought process.\n", "\n", "## Benefits and Considerations\n", "\n", "The Reverse Chain of Thought implementation offers several advantages:\n", "\n", "1. Improved accuracy by identifying and correcting overlooked conditions, hallucinations, and misinterpretations.\n", "2. Enhanced reasoning capabilities through self-reflection and error correction.\n", "3. More robust problem-solving, especially for complex or ambiguous queries.\n", "\n", "When implementing this technique, consider:\n", "\n", "- Balancing the computational cost of multiple LLM calls with the improved accuracy.\n", "- Fine-tuning the prompts for each step to optimize the reflection and correction process.\n", "- Adapting the technique for different types of problems or domains.\n", "\n", "
\n", "

Additional Real-World Applications

\n", "\n", "
\n", "\n", "When adapting this recipe to your specific use-case, consider:\n", "\n", "- Tailoring the decomposition and comparison steps to your domain for better performance.\n", "- Implementing a feedback loop to continuously improve the quality of the Reverse Chain of Thought responses.\n", "- Combining Reverse Chain of Thought with other techniques like Self-Ask or Self-Consistency for even more powerful reasoning capabilities.\n", "\n", "By leveraging Mirascope's `call` decorator, response models, and dynamic configuration, you can easily implement and customize the Reverse Chain of Thought technique to enhance your LLM's reasoning capabilities across a wide range of applications.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 5 }