{ "cells": [ { "cell_type": "markdown", "id": "5288610e938c0d65", "metadata": {}, "source": [ "# Step-back Prompting: Enhancing LLM Reasoning with High-Level Questions\n", "\n", "This recipe demonstrates how to implement the Step-back prompting technique using Large Language Models (LLMs) with Mirascope. Step-back prompting is a method that enhances an LLM's reasoning capabilities by asking a high-level question about relevant concepts or facts before addressing the original query.\n", "\n", "
\n", "Mirascope Concepts Used\n", "\n", "
\n", "\n", "
\n", "

Background

\n", "

Step-back prompting is an alternative to the Chain of Thought technique, where one asks the LLM a high-level question about relevant concepts or facts before asking it the actual question. This technique is derived from the fact that humans often take step backs and use abstractions to arrive at an answer, and it can yield correct answers at times when Chain of Thought fails.

\n", "
\n", "\n", "## Implementation\n", "\n", "Let's implement the Step-back prompting technique using Mirascope:\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "8854c01fab9d146c", "metadata": { "ExecuteTime": { "end_time": "2024-10-01T05:39:00.834947Z", "start_time": "2024-10-01T05:38:49.351833Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "As of the 2017 NBA season, the highest-paid player was Stephen Curry. He signed a four-year, $215 million contract extension with the Golden State Warriors, which was the largest contract in NBA history at that time. This contract significantly boosted his earnings, making him the top earner in the league for that season. Other players like LeBron James and Kevin Durant were also among the highest-paid, but Curry's contract set a new benchmark in player salaries at that time.\n" ] } ], "source": [ "from mirascope.core import openai\n", "from mirascope.core.base.prompt import prompt_template\n", "\n", "few_shot_examples = [\n", " {\n", " \"original_question\": \"Which position did Knox Cunningham hold from May 1955 to Apr 1956?\",\n", " \"stepback_question\": \"Which positions have Knox Cunningham held in his career?\",\n", " },\n", " {\n", " \"original_question\": \"Who was the spouse of Anna Karina from 1968 to 1974?\",\n", " \"stepback_question\": \"Who were the spouses of Anna Karina?\",\n", " },\n", " {\n", " \"original_question\": \"Which team did Thierry Audel play for from 2007 to 2008?\",\n", " \"stepback_question\": \"Which teams did Thierry Audel play for in his career?\",\n", " },\n", " {\n", " \"original_question\": \"What was the operator of GCR Class 11E from 1913 to Dec 1922?\",\n", " \"stepback_question\": \"What were the operators of GCR Class 11E in history?\",\n", " },\n", " {\n", " \"original_question\": \"Which country did Sokolovsko belong to from 1392 to 1525?\",\n", " \"stepback_question\": \"Which countries did Sokolovsko belong to in history?\",\n", " },\n", " {\n", " \"original_question\": \"when was the last time a team from canada won the stanley cup as of 2002\",\n", " \"stepback_question\": \"which years did a team from canada won the stanley cup as of 2002\",\n", " },\n", " {\n", " \"original_question\": \"when did england last get to the semi final in a world cup as of 2019\",\n", " \"stepback_question\": \"which years did england get to the semi final in a world cup as of 2019?\",\n", " },\n", " {\n", " \"original_question\": \"what is the biggest hotel in las vegas nv as of November 28, 1993\",\n", " \"stepback_question\": \"what is the size of the hotels in las vegas nv as of November 28, 1993\",\n", " },\n", " {\n", " \"original_question\": \"who has scored most runs in t20 matches as of 2017\",\n", " \"stepback_question\": \"What are the runs of players in t20 matches as of 2017\",\n", " },\n", "]\n", "\n", "stepback_prompt = \"\"\"You are an expert at world knowledge. Your task is to step \\\n", "back and paraphrase a question to a more generic step-back question, which is \\\n", "easier to answer. Here are a few examples:\"\"\"\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "@prompt_template(\n", " \"\"\"\n", " SYSTEM: {stepback_prompt_and_examples}\n", " USER: {query}\n", " \"\"\"\n", ")\n", "def get_stepback_question(\n", " query: str, num_examples: int = 0\n", ") -> openai.OpenAIDynamicConfig:\n", " \"\"\"Gets the generic, step-back version of a query.\"\"\"\n", " if num_examples < 0 or num_examples > len(few_shot_examples):\n", " raise ValueError(\n", " \"num_examples cannot be negative or greater than number of available examples.\"\n", " )\n", " example_prompts = \"\"\n", " for i in range(num_examples):\n", " example_prompts += (\n", " f\"Original Question: {few_shot_examples[i]['original_question']}\\n\"\n", " )\n", " example_prompts += (\n", " f\"Stepback Question: {few_shot_examples[i]['stepback_question']}\\n\"\n", " )\n", " return {\n", " \"computed_fields\": {\n", " \"stepback_prompt_and_examples\": f\"{stepback_prompt}\\n{example_prompts}\"\n", " if num_examples\n", " else None\n", " }\n", " }\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "def call(query: str) -> str:\n", " \"\"\"A standard call to OpenAI.\"\"\"\n", " return query\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "@prompt_template(\n", " \"\"\"\n", " You are an expert of world knowledge. I am going to ask you a question.\n", " Your response should be comprehensive and not contradicted with the\n", " following context if they are relevant. Otherwise, ignore them if they are\n", " not relevant.\n", "\n", " {stepback_response}\n", "\n", " Original Question: {query}\n", " Answer:\n", " \"\"\"\n", ")\n", "def stepback(query: str, num_examples: int) -> openai.OpenAIDynamicConfig:\n", " \"\"\"Executes the flow of the Step-Back technique.\"\"\"\n", " stepback_question = get_stepback_question(\n", " query=query, num_examples=num_examples\n", " ).content\n", " stepback_response = call(query=stepback_question).content\n", " return {\"computed_fields\": {\"stepback_response\": stepback_response}}\n", "\n", "\n", "# Example usage\n", "query = \"\"\"Who is the highest paid player in the nba this season as of 2017\"\"\"\n", "\n", "print(stepback(query=query, num_examples=len(few_shot_examples)))" ] }, { "cell_type": "markdown", "id": "119ab54f757e1c7b", "metadata": {}, "source": [ "This implementation consists of three main functions:\n", "\n", "1. `get_stepback_question`: This function takes a query and generates a more generic, step-back version of the question.\n", "2. `call`: A standard call to OpenAI that processes the step-back question.\n", "3. `stepback`: This function orchestrates the Step-back prompting technique. It first calls `get_stepback_question` to generate a high-level question, then uses `call` to get a response to this question, and finally combines this information to answer the original query.\n", "\n", "## Benefits and Considerations\n", "\n", "The Step-back prompting implementation offers several advantages:\n", "\n", "1. Improved reasoning about complex queries by considering higher-level concepts first.\n", "2. Potential for more accurate responses in tasks that benefit from broader context.\n", "3. Ability to overcome limitations of other techniques like Chain of Thought in certain scenarios.\n", "\n", "When implementing this technique, consider:\n", "\n", "- Balancing the generality of the step-back question with its relevance to the original query.\n", "- Experimenting with different numbers of few-shot examples to optimize performance.\n", "- Adjusting the prompt for generating step-back questions based on your specific use case.\n", "\n", "
\n", "

Additional Real-World Applications

\n", "\n", "
\n", "\n", "When adapting this recipe to your specific use-case, consider:\n", "\n", "- Tailoring the few-shot examples to your domain for better performance.\n", "- Implementing a feedback loop to continuously improve the quality of step-back questions generated.\n", "- Combining Step-back prompting with other techniques like Self-consistency for even more robust reasoning capabilities.\n", "- Experimenting with different LLM models to find the best balance between performance and efficiency for your use case.\n", "\n", "By leveraging Mirascope's `call` decorator and dynamic configuration, you can easily implement and customize the Step-back prompting technique to enhance your LLM's reasoning capabilities across a wide range of applications." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 5 }