{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6fcff7e8de11a4b7",
   "metadata": {},
   "source": [
    "# Extracting from PDF\n",
    "\n",
    "This recipe demonstrates how to leverage Large Language Models (LLMs) -- specifically the OpenAI API -- to extract pages and content from PDF files. We'll cover single PDF document as well as multiple PDF files and also use OCR to extract text from scanned documents.\n",
    "\n",
    "<div class=\"admonition tip\">\n",
    "<p class=\"admonition-title\">Mirascope Concepts Used</p>\n",
    "<ul>\n",
    "<li><a href=\"../../../learn/prompts/\">Prompts</a></li>\n",
    "<li><a href=\"../../../learn/calls/\">Calls</a></li>\n",
    "<li><a href=\"../../../learn/response_models/\">Response Models</a></li>\n",
    "</ul>\n",
    "</div>\n",
    "\n",
    "<div class=\"admonition note\">\n",
    "<p class=\"admonition-title\">Background</p>\n",
    "<p>\n",
    "Prior to LLMs, extracting pages from pdf files has been a time-consuming and expensive task. Natural Language Processing (NLP) would be used to identify and categorize information in text specific to the PDF document, or worse yet manually by humans. This would need to happen on every new PDF with new categories, which is not scalable. LLMs possess the ability to understand context, and the versatility to handle diverse data beyond PDFs such as word documents, powerpoints, email, and more.\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "766aa672",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "To set up our environment, first let's install all of the packages we will use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d2893e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \"mirascope[openai]\" PyMuPDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93a9fff1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"YOUR_API_KEY\"\n",
    "# Set the appropriate API key for the provider you're using"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "308d65255c9bb856",
   "metadata": {},
   "source": [
    "\n",
    "## Text-Based Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d630882b75a756",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-09-30T12:02:48.661805Z",
     "start_time": "2024-09-30T12:02:47.454951Z"
    }
   },
   "outputs": [],
   "source": [
    "import asyncio\n",
    "\n",
    "from mirascope.core import openai, prompt_template\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "\n",
    "class ResumeInfo(BaseModel):\n",
    "    name: str = Field(..., description=\"The name of the person\")\n",
    "    email: str = Field(..., description=\"The email of the person\")\n",
    "    phone: str = Field(..., description=\"The phone number of the person\")\n",
    "    skills: list[str] = Field(..., description=\"The skills of the person\")\n",
    "    experience: list[str] = Field(..., description=\"The experience of the person\")\n",
    "    education: list[str] = Field(..., description=\"The education of the person\")\n",
    "\n",
    "\n",
    "@openai.call(model=\"gpt-4o\", response_model=ResumeInfo)\n",
    "@prompt_template(\n",
    "    \"\"\"\n",
    "    Extract the resume information from the pdf file.\n",
    "    {pdf_text}\n",
    "    \"\"\"\n",
    ")\n",
    "async def extract_resume_info(pdf_text: str): ...\n",
    "\n",
    "\n",
    "async def convert_pdf_to_text(pdf_file_path: str) -> str:\n",
    "    doc = fitz.open(pdf_file_path, filetype=\"pdf\")\n",
    "    text = []\n",
    "    for page in doc:\n",
    "        text.append(page.get_text())  # type: ignore\n",
    "    return \"\\n\".join(text)\n",
    "\n",
    "\n",
    "async def process_pdf(pdf_file_path: str) -> ResumeInfo:\n",
    "    pdf_text = await convert_pdf_to_text(pdf_file_path)\n",
    "    return await extract_resume_info(pdf_text)\n",
    "\n",
    "\n",
    "async def run(pdf_file_paths: list[str]):\n",
    "    tasks = [process_pdf(pdf_file_path) for pdf_file_path in pdf_file_paths]\n",
    "    await asyncio.gather(*tasks)\n",
    "\n",
    "\n",
    "# In jupyter notebook to run an async function\n",
    "await run([\"resume.pdf\"])\n",
    "\n",
    "# In Python scripts to run an async function\n",
    "# asyncio.run(run([\"resume.pdf\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a868a8bc95e8041c",
   "metadata": {},
   "source": [
    "\n",
    "This script takes in multiple text PDF documents and extracts them into a `ResumeInfo` Pydantic model. However, PDF documents contain not only text but images, tables, and more. In order to handle these more complex documents, we use Optical Character Recognition (OCR).\n",
    "\n",
    "## Vision-Based Extraction\n",
    "\n",
    "For PDFs with complex layouts or embedded images, vision-based extraction can be more effective. We first write a helper function that uses `PyMuPDF` to convert the PDF document into a PNG image:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50c2e63090c31fae",
   "metadata": {},
   "outputs": [],
   "source": [
    "import fitz\n",
    "\n",
    "\n",
    "async def convert_pdf_to_png(pdf_file_path: str) -> list[bytes]:\n",
    "    doc = fitz.open(pdf_file_path, filetype=\"pdf\")\n",
    "    images = []\n",
    "    for page_num in range(len(doc)):\n",
    "        page = doc.load_page(page_num)\n",
    "        pix = page.get_pixmap()\n",
    "        images.append(pix.tobytes(output=\"png\"))\n",
    "    return images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7fddaaf4ec8997c",
   "metadata": {},
   "source": [
    "Afterwards, we update our previously defined call function to take in image bytes and run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "761a2aae6990375f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mirascope.core import openai, prompt_template\n",
    "\n",
    "\n",
    "@openai.call(model=\"gpt-4o-mini\", response_model=ResumeInfo)\n",
    "@prompt_template(\n",
    "    \"\"\"\n",
    "    Extract the resume information from the pdf file.\n",
    "    {pdf_imgs::images}\n",
    "    \"\"\"\n",
    ")\n",
    "async def extract_resume_info_as_image(pdf_imgs: list[bytes]): ...\n",
    "\n",
    "\n",
    "async def process_pdf_as_image(pdf_file_path: str) -> ResumeInfo:\n",
    "    pdf_pngs = await convert_pdf_to_png(pdf_file_path)\n",
    "    return await extract_resume_info_as_image(pdf_pngs)\n",
    "\n",
    "\n",
    "# In jupyter notebook to run an async function\n",
    "await process_pdf_as_image(\"resume.pdf\")\n",
    "\n",
    "# In Python scripts to run an async function\n",
    "# asyncio.run(process_pdf_as_image(\"resume.pdf\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b879c67813c227c9",
   "metadata": {},
   "source": [
    "\n",
    "This approach converts the PDF to images, allowing the AI model to analyze the visual layout and content. It's particularly useful for PDFs with non-standard layouts or those containing charts and diagrams.\n",
    "\n",
    "Not all providers support vision, so be sure to check provider documentation.\n",
    "\n",
    "<div class=\"admonition tip\">\n",
    "<p class=\"admonition-title\">Additional Real-World Examples</p>\n",
    "<ul>\n",
    "<li><b>Automating document processing</b>: Take invoices, statements, tax-forms, etc and other manual data entry tasks and automate them.</li>\n",
    "<li><b>Medical Document Integrations</b>: Extract patient records, often a PDF document into Electronic Health Record (EHR) systems.</li>\n",
    "<li><b>Resume Parsing Advanced</b>: Update the <code>ResumeInfo</code> schema to a format for a Customer Relationship Management (CRM) tool.</li>\n",
    "</ul>\n",
    "</div>\n",
    "\n",
    "When adapting this recipe to your specific use-case, consider the following:\n",
    "\n",
    "    - OCR accuracy: For better OCR results when using Vision-based extraction, use high-quality scans.\n",
    "    - Multiple providers: Use multiple LLM providers to verify whether the extracted information is accurate and not hallucination.\n",
    "    - Implement Pydantic `ValidationError` and Tenacity `retry` to improve reliability and accuracy.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}