{ "cells": [ { "cell_type": "markdown", "id": "62f974e4c168f7d3", "metadata": {}, "source": [ "# Mixture of Reasoning: Enhancing LLM Performance with Multiple Techniques\n", "\n", "Mixture of Reasoning is a prompt engineering technique where you set up multiple calls, each utilizing a different prompt engineering technique. This approach is best when you want to be able to handle a wide variety of responses, or have a variety of techniques that you have found to be successful for responding to a type of prompt.\n", "\n", "
\n", "

Mirascope Concepts Used

\n", "\n", "
\n", "\n", "
\n", "

Background

\n", "

\n", "In the original paper, a trained classifier is used to determine the best answer, but since we do not have access to that, we will use an LLM to evaluate the best answer instead. To get a clean separation between the reasoning and the actual output, we'll use response_model in our final evaluation call.\n", "

\n", "
\n", "\n", "## Implementation\n", "\n", "Let's implement the Mixture of Reasoning technique using Mirascope:\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "34acd7b77e7ca225", "metadata": { "ExecuteTime": { "end_time": "2024-10-01T02:00:48.934977Z", "start_time": "2024-10-01T02:00:17.156715Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Response 1\n", "Response 1 provides a clear and systematic approach to solving the problem by setting up the equations based on the given area and perimeter. It correctly derives the quadratic equation, factors it, and finds both possible lengths and widths, concluding that the side lengths are indeed 2 and 4. Response 2 has a similar structure and is also correct but slightly less clear in its explanation. Response 3 is overly verbose and somewhat redundant in conveying the same information, which might confuse rather than clarify.\n" ] } ], "source": [ "from mirascope.core import openai, prompt_template\n", "from pydantic import BaseModel, Field\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "@prompt_template(\n", " \"\"\"\n", " Answer this question, thinking step by step.\n", " {query}\n", " \"\"\"\n", ")\n", "def cot_call(query: str): ...\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "@prompt_template(\n", " \"\"\"\n", " {query}\n", " It's very important to my career.\n", " \"\"\"\n", ")\n", "def emotion_prompting_call(query: str): ...\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\")\n", "@prompt_template(\n", " \"\"\"\n", " {query}\n", " Rephrase and expand the question, and respond.\n", " \"\"\"\n", ")\n", "def rar_call(query: str): ...\n", "\n", "\n", "class BestResponse(BaseModel):\n", " best_response: str = Field(\n", " ..., description=\"The best response of the options given, verbatim\"\n", " )\n", " reasoning: str = Field(\n", " ...,\n", " description=\"\"\"A short description of why this is the best response to\n", " the query, along with reasons why the other answers were worse.\"\"\",\n", " )\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\", response_model=BestResponse)\n", "@prompt_template(\n", " \"\"\"\n", " Here is a query: {query}\n", "\n", " Evaluate the following responses from LLMs and decide which one\n", " is the best based on correctness, fulfillment of the query, and clarity:\n", "\n", " Response 1:\n", " {cot_response}\n", "\n", " Response 2:\n", " {emotion_prompting_response}\n", "\n", " Response 3:\n", " {rar_response}\n", " \"\"\"\n", ")\n", "def mixture_of_reasoning(query: str) -> openai.OpenAIDynamicConfig:\n", " cot_response = cot_call(query=query)\n", " emotion_prompting_response = emotion_prompting_call(query=query)\n", " rar_response = rar_call(query=query)\n", "\n", " return {\n", " \"computed_fields\": {\n", " \"cot_response\": cot_response,\n", " \"emotion_prompting_response\": emotion_prompting_response,\n", " \"rar_response\": rar_response,\n", " }\n", " }\n", "\n", "\n", "prompt = \"What are the side lengths of a rectangle with area 8 and perimeter 12?\"\n", "\n", "best_response = mixture_of_reasoning(prompt)\n", "print(best_response.best_response)\n", "print(best_response.reasoning)" ] }, { "cell_type": "markdown", "id": "94a80079e7d6bd4", "metadata": {}, "source": [ "This implementation consists of several key components:\n", "\n", "1. Three different prompt engineering techniques:\n", " - `cot_call`: [Chain of Thought reasoning](../text_based/chain_of_thought.md)\n", " - `emotion_prompting_call`: [Emotion prompting](../text_based/emotion_prompting.md)\n", " - `rar_call`: [Rephrase and Respond](../text_based/rephrase_and_response.md)\n", "\n", "2. A `BestResponse` model to structure the output of the final evaluation.\n", "\n", "3. The `mixture_of_reasoning` function, which:\n", " - Calls each of the three prompt engineering techniques\n", " - Uses dynamic configuration to pass the responses to the final evaluation\n", " - Returns the best response and reasoning using the `BestResponse` model\n", "\n", "## Benefits and Considerations\n", "\n", "The Mixture of Reasoning implementation offers several advantages:\n", "\n", "1. Improved ability to handle a wide variety of queries by leveraging multiple prompt engineering techniques.\n", "2. Enhanced performance by selecting the best response from multiple approaches.\n", "3. Flexibility to add or modify prompt engineering techniques as needed.\n", "\n", "When implementing this technique, consider:\n", "\n", "- Carefully selecting the prompt engineering techniques to include based on your specific use case.\n", "- Balancing the number of techniques with computational cost and time constraints.\n", "- Fine-tuning the evaluation criteria in the final step to best suit your needs.\n", "\n", "
\n", "

Additional Real-World Applications

\n", "\n", "
\n", "\n", "When adapting this recipe to your specific use-case, consider:\n", "\n", "- Experimenting with different combinations of prompt engineering techniques.\n", "- Implementing a feedback loop to continuously improve the selection of the best response.\n", "- Tailoring the evaluation criteria to your specific domain or task requirements.\n", "- Combining Mixture of Reasoning with other techniques like Self-Consistency for even more robust reasoning capabilities.\n", "\n", "By leveraging Mirascope's `call` decorator, response models, and dynamic configuration, you can easily implement and customize the Mixture of Reasoning technique to enhance your LLM's performance across a wide range of applications." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 5 }