{ "cells": [ { "cell_type": "markdown", "id": "72942cd210501f07", "metadata": {}, "source": [ "# Search Agent with Sources\n", "\n", "This recipe shows how to use LLMs — in this case, GPT 4o mini — to answer questions using the web. Since LLMs often time hallucinate answers, it is important to fact check and verify the accuracy of the answer.\n", "\n", "
\n", "

Mirascope Concepts Used

\n", "\n", "
\n", "\n", "
\n", "

Background

\n", "

\n", "Users of Large Language Models (LLMs) often struggle to distinguish between factual content and potential hallucinations, leading to time-consuming fact-checking. By implementing source citation requirements, LLMs need to rely on verified information, thereby enhancing the accuracy of its responses and reducing the need for manual verification.\n", "

\n", "
" ] }, { "cell_type": "markdown", "id": "265d41a4", "metadata": {}, "source": [ "## Setup\n", "\n", "To set up our environment, first let's install all of the packages we will use:" ] }, { "cell_type": "code", "execution_count": null, "id": "0900d46c", "metadata": {}, "outputs": [], "source": [ "!pip install \"mirascope[openai]\" beautifulsoup4" ] }, { "cell_type": "code", "execution_count": null, "id": "12637a6d", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"OPENAI_API_KEY\"] = \"YOUR_API_KEY\"\n", "# Set the appropriate API key for the provider you're using" ] }, { "cell_type": "markdown", "id": "c094ee6d9eaa770a", "metadata": {}, "source": [ "\n", "We will need an API key for search:\n", "\n", "- [Nimble API Key](https://nimbleway.com/) or alternatively directly from Google [Custom Search](https://developers.google.com/custom-search/v1/introduction/) API.\n", "\n", "## Creating Google Search tool\n", "\n", "We use [Nimble](https://nimbleway.com/) since they provide an easy-to-use API for searching, but an alternative you can use is Google's Custom Search API. We first want to grab all the urls that are relevant to answer our question and then we take the contents of those urls, like so:" ] }, { "cell_type": "code", "execution_count": null, "id": "5046a7fabf14856d", "metadata": {}, "outputs": [], "source": [ "import requests\n", "from bs4 import BeautifulSoup\n", "\n", "NIMBLE_TOKEN = \"YOUR_NIMBLE_API_KEY\"\n", "\n", "\n", "def nimble_google_search(query: str):\n", " \"\"\"\n", " Use Nimble to get information about the query using Google Search.\n", " \"\"\"\n", " url = \"https://api.webit.live/api/v1/realtime/serp\"\n", " headers = {\n", " \"Authorization\": f\"Basic {NIMBLE_TOKEN}\",\n", " \"Content-Type\": \"application/json\",\n", " }\n", " search_data = {\n", " \"parse\": True,\n", " \"query\": query,\n", " \"search_engine\": \"google_search\",\n", " \"format\": \"json\",\n", " \"render\": True,\n", " \"country\": \"US\",\n", " \"locale\": \"en\",\n", " }\n", " response = requests.get(url, json=search_data, headers=headers)\n", " data = response.json()\n", " results = data[\"parsing\"][\"entities\"][\"OrganicResult\"]\n", " urls = [result.get(\"url\", \"\") for result in results]\n", " search_results = {}\n", " for url in urls:\n", " content = get_content(url)\n", " search_results[url] = content\n", " return search_results\n", "\n", "\n", "def get_content(url: str):\n", " data = []\n", " response = requests.get(url)\n", " content = response.content\n", " soup = BeautifulSoup(content, \"html.parser\")\n", " paragraphs = soup.find_all(\"p\")\n", " for paragraph in paragraphs:\n", " data.append(paragraph.text)\n", " return \"\\n\".join(data)" ] }, { "cell_type": "markdown", "id": "cefac0cfbe3ed78d", "metadata": {}, "source": [ "\n", "Now that we have created our tool, it’s time to create our LLM call.\n", "\n", "## Creating the first call\n", "\n", "For this call, we force the LLM to always use its tool which we will later chain.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "13a87c5e31d54f6b", "metadata": {}, "outputs": [], "source": [ "from mirascope.core import openai, prompt_template\n", "\n", "\n", "@openai.call(\n", " model=\"gpt-4o-mini\",\n", " tools=[nimble_google_search],\n", " call_params={\"tool_choice\": \"required\"},\n", ")\n", "@prompt_template(\n", " \"\"\"\n", " SYSTEM:\n", " You are a an expert at finding information on the web.\n", " Use the `nimble_google_search` function to find information on the web.\n", " Rewrite the question as needed to better find information on the web.\n", "\n", " USER:\n", " {question}\n", " \"\"\"\n", ")\n", "def search(question: str): ..." ] }, { "cell_type": "markdown", "id": "f5f017b5cc3a9433", "metadata": {}, "source": [ "\n", "We ask the LLM to rewrite the question to make it more suitable for search.\n", "\n", "Now that we have the necessary data to answer the user query and their sources, it’s time to extract all that information into a structured format using `response_model`\n", "\n", "## Extracting Search Results with Sources\n", "\n", "As mentioned earlier, it is important to fact check all answers in case of hallucination, and the first step is to ask the LLM to cite its sources:\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a4da1facf8110830", "metadata": {}, "outputs": [], "source": [ "from pydantic import BaseModel, Field\n", "\n", "\n", "class SearchResponse(BaseModel):\n", " sources: list[str] = Field(description=\"The sources of the results\")\n", " answer: str = Field(description=\"The answer to the question\")\n", "\n", "\n", "@openai.call(model=\"gpt-4o-mini\", response_model=list[SearchResponse])\n", "@prompt_template(\n", " \"\"\"\n", " SYSTEM:\n", " Extract the question, results, and sources to answer the question based on the results.\n", "\n", " Results:\n", " {results}\n", "\n", " USER:\n", " {question}\n", " \"\"\"\n", ")\n", "def extract(question: str, results: str): ..." ] }, { "cell_type": "markdown", "id": "d50ce29b7bb2f566", "metadata": {}, "source": [ "\n", "and finally we create our `run` function to execute our chain:\n" ] }, { "cell_type": "code", "execution_count": null, "id": "9547aff227a1c855", "metadata": {}, "outputs": [], "source": [ "def run(question: str):\n", " response = search(question)\n", " if tool := response.tool:\n", " output = tool.call()\n", " result = extract(question, output)\n", " return result\n", "\n", "\n", "print(run(\"What is the average price of a house in the United States?\"))" ] }, { "cell_type": "markdown", "id": "cf7fdb4d371864d7", "metadata": {}, "source": [ "
\n", "

Additional Real-World Applications

\n", "\n", "
\n", "\n", "When adapting this recipe, consider:\n", " - Adding [Tenacity](https://tenacity.readthedocs.io/en/latest/) `retry` for more a consistent extraction.\n", " - Use an LLM with web search tool to evaluate whether the answer produced is in the source.\n", " - Experiment with different model providers and version for quality and accuracy of results.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 5 }