{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Text Classification\n", "\n", "In this recipe we’ll explore using Mirascope to implement binary classification, multi-class classification, and various other extensions of these classification techniques — specifically using Python the OpenAI API. We will also compare these solutions with more traditional machine learning and Natural Language Processing (NLP) techniques.\n", "\n", "\n", "
\n", "

Mirascope Concepts Used

\n", "\n", "
\n", "\n", "
\n", "

Background

\n", "

\n", " Text classification is a fundamental classification problem and NLP task that involves categorizing text documents into predefined classes or categories. Historically this has required training text classifiers through more traditional machine learning methods. Large Language Models (LLMs) have revolutionized this field, making sophisticated classification tasks accessible through simple API calls and thoughtful prompt engineering.\n", "

\n", "
\n", "\n", "## Setup\n", "\n", "Let's start by installing Mirascope and its dependencies:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install \"mirascope[openai]\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"OPENAI_API_KEY\"] = \"YOUR_API_KEY\"\n", "# Set the appropriate API key for the provider you're using" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Binary Classification: Spam Detection\n", "\n", "Binary classification involves categorizing text into one of two classes. We'll demonstrate this by creating a spam detector that classifies text as either spam or not spam.\n", "\n", "For binary classification, we can extract a boolean value by setting `response_model=bool` and prompting the model to classify the text:\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2024-09-30T02:18:45.618737Z", "start_time": "2024-09-30T02:18:43.779663Z" } }, "outputs": [], "source": [ "from mirascope.core import openai, prompt_template\n", "\n", "\n", "@openai.call(\"gpt-4o-mini\", response_model=bool)\n", "def classify_spam(text: str) -> str:\n", " return f\"Classify the following text as spam or not spam: {text}\"\n", "\n", "\n", "text = \"Would you like to buy some cheap viagra?\"\n", "label = classify_spam(text)\n", "assert label is True # This text is classified as spam\n", "\n", "text = \"Hi! It was great meeting you today. Let's stay in touch!\"\n", "label = classify_spam(text)\n", "assert label is False # This text is classified as not spam" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Multi-Class Classification: Sentiment Analysis\n", "\n", "Multi-class classification extends the concept to scenarios where we need to categorize text into one of several classes. We'll demonstrate this with a sentiment analysis task.\n", "\n", "First, we define an `Enum` to represent our sentiment labels:\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2024-09-30T02:18:47.329859Z", "start_time": "2024-09-30T02:18:47.327776Z" } }, "outputs": [], "source": [ "from enum import Enum\n", "\n", "\n", "class Sentiment(Enum):\n", " NEGATIVE = \"negative\"\n", " NEUTRAL = \"neutral\"\n", " POSITIVE = \"positive\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we set `response_model=Sentiment` to analyze the sentiment of the given text:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2024-09-30T02:18:51.710915Z", "start_time": "2024-09-30T02:18:49.880241Z" } }, "outputs": [], "source": [ "from mirascope.core import openai\n", "\n", "\n", "@openai.call(\"gpt-4o-mini\", response_model=Sentiment)\n", "def classify_sentiment(text: str) -> str:\n", " return f\"Classify the sentiment of the following text: {text}\"\n", "\n", "\n", "text = \"I hate this product. It's terrible.\"\n", "label = classify_sentiment(text)\n", "assert label == Sentiment.NEGATIVE\n", "\n", "text = \"I don't feel strongly about this product.\"\n", "label = classify_sentiment(text)\n", "assert label == Sentiment.NEUTRAL\n", "\n", "text = \"I love this product. It's amazing!\"\n", "label = classify_sentiment(text)\n", "assert label == Sentiment.POSITIVE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classification with Reasoning\n", "\n", "So far we've demonstrated using simple types like `bool` and `Enum` for classification, but we can extend this approach using Pydantic's `BaseModel` class to extract additional information beyond just the classification label.\n", "\n", "For example, we can gain insight to the LLMs reasoning for the classified label simply by including a reasoning field in our response model and updating the prompt:\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2024-09-30T02:18:55.076718Z", "start_time": "2024-09-30T02:18:53.356795Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sentiment: Sentiment.NEUTRAL\n", "Reasoning: The text expresses a positive sentiment towards the product because the speaker is willing to recommend it. However, the mention of 'if it were cheaper' introduces a condition that makes the overall sentiment appear somewhat negative, as it suggests dissatisfaction with the current price. Therefore, the sentiment can be classified as neutral, as it acknowledges both a positive recommendation but also a negative aspect regarding pricing.\n" ] } ], "source": [ "from enum import Enum\n", "\n", "from mirascope.core import openai\n", "from pydantic import BaseModel\n", "\n", "\n", "class Sentiment(Enum):\n", " NEGATIVE = \"negative\"\n", " NEUTRAL = \"neutral\"\n", " POSITIVE = \"positive\"\n", "\n", "\n", "class SentimentWithReasoning(BaseModel):\n", " reasoning: str\n", " sentiment: Sentiment\n", "\n", "\n", "@openai.call(\"gpt-4o-mini\", response_model=SentimentWithReasoning)\n", "@prompt_template(\n", " \"\"\"\n", " Classify the sentiment of the following text: {text}.\n", " Explain your reasoning for the classified sentiment.\n", " \"\"\"\n", ")\n", "def classify_sentiment_with_reasoning(text: str): ...\n", "\n", "\n", "text = \"I would recommend this product if it were cheaper...\"\n", "response = classify_sentiment_with_reasoning(text)\n", "print(f\"Sentiment: {response.sentiment}\")\n", "print(f\"Reasoning: {response.reasoning}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Handling Uncertainty\n", "\n", "When dealing with LLMs for classification tasks, it's important to account for cases where the model might be uncertain about its prediction. We can modify our approach to include a certainty score and handle cases where the model's confidence is below a certain threshold.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2024-09-30T02:19:01.790560Z", "start_time": "2024-09-30T02:18:59.676426Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The model is not certain enough about the classification.\n" ] } ], "source": [ "from pydantic import BaseModel, Field\n", "\n", "\n", "class SentimentAnalysisWithCertainty(BaseModel):\n", " sentiment: Sentiment\n", " certainty: float = Field(..., ge=0, le=1)\n", " reasoning: str\n", "\n", "\n", "class SentimentWithCertainty(BaseModel):\n", " sentiment: Sentiment\n", " reasoning: str\n", " certainty: float\n", "\n", "\n", "@openai.call(\"gpt-4o-mini\", response_model=SentimentWithCertainty)\n", "@prompt_template(\n", " \"\"\"\n", " Classify the sentiment of the following text: {text}\n", " Explain your reasoning for the classified sentiment.\n", " Also provide a certainty score between 0 and 1, where 1 is absolute certainty.\n", " \"\"\"\n", ")\n", "def classify_sentiment_with_certainty(text: str): ...\n", "\n", "\n", "text = \"This is the best product ever. And the worst.\"\n", "response = classify_sentiment_with_certainty(text)\n", "if response.certainty > 0.8:\n", " print(f\"Sentiment: {response.sentiment}\")\n", " print(f\"Reasoning: {response.reasoning}\")\n", " print(f\"Certainty: {response.certainty}\")\n", "else:\n", " print(\"The model is not certain enough about the classification.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "

Additional Real-World Applications

\n", "\n", "
\n", "\n", "When adapting this recipe to your specific use-case, consider the following:\n", "\n", "- Refine your prompts to provide clear instructions and relevant context for your specific classification task.\n", "- Experiment with different model providers and version to balance accuracy and speed.\n", "- Implement error handling and fallback mechanisms for cases where the model's classification is uncertain.\n", "- Consider using a combination of classifiers for more complex categorization tasks.\n", "\n", "## Comparison with Traditional Machine Learning Models\n", "\n", "Training text classification models requires a much more involved workflow:\n", "\n", "- Preprocessing:\n", " - Read in data, clean and standardize it, and split it into training, validation, and test datasets\n", "- Feature Extraction:\n", " - Basic: bag of words, TF-IDF\n", " - Advanced: word embeddings, contextual embeddings\n", "- Classification Algorithm / Machine Learning Algorithm:\n", " - Basic: Naive Bayes, logistic regression, linear classifiers\n", " - Advanced: Neural networks, transformers (e.g. BERT)\n", "- Model Training:\n", " - Train on training data and validate on validation data, adjusting batch size and epochs.\n", " - Things like activation layers and optimizers can greatly impact the quality of the final trained model\n", "- Model Evaluation:\n", " - Evaluate model quality on the test dataset using metrics such as F1-score, recall, precision, accuracy — whichever metric best suits your use-case\n", " \n", "Many frameworks such as TensorFlow and PyTorch make implementing such workflows easier, but it is still far more involved that the approach we showed in the beginning using Mirascope.\n", "\n", "If you’re interested in taking a deeper dive into this more traditional approach, the [TensorFlow IMDB Text Classification](https://www.tensorflow.org/tutorials/keras/text_classification_with_hub) tutorial is a great place to start.\n", "\n", "\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 4 }