Web Search Agent¶
In this recipe, we'll explore using Mirascope to enhance our Large Language Model (LLM) — specifically OpenAI's GPT-4o mini — by providing it with access to the web and its contents. We will be using DuckDuckGo's API as a tool for our Agentic workflow.
Background
In the past, users had to rely on search engines and manually browse through multiple web pages to find answers to their questions. Large Language Models (LLMs) have revolutionized this process. They can efficiently utilize search engine results pages (SERPs) and extract relevant content from websites. By leveraging this information, LLMs can quickly provide accurate answers to user queries, eliminating the need for active searching. Users can simply pose their questions and let the LLM work in the background, significantly streamlining the information retrieval process.
Setup¶
To set up our environment, first let's install all of the packages we will use:
!pip install "mirascope[openai]" beautifulsoup4 duckduckgo-search requests
import os
os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"
# Set the appropriate API key for the provider you're using
Make sure to also set your OPENAI_API_KEY
if you haven't already. We are using duckduckgo-search
since it does not require an API key, but feel free to use Google Search API or other search engine APIs.
Add DuckDuckGo Tool¶
The first step is to create a WebAssistant
that first conducts a web search based on the user's query. Let’s go ahead and create our WebAssistant
and add our web search tool:
from duckduckgo_search import DDGS
from mirascope.core import BaseMessageParam, openai, prompt_template
from pydantic import BaseModel
class WebAssistantBase(BaseModel):
messages: list[BaseMessageParam | openai.OpenAIMessageParam] = []
search_history: list[str] = []
max_results_per_query: int = 2
def _web_search(self, queries: list[str]) -> str:
"""Performs web searches for given queries and returns URLs.
Args:
queries: List of search queries.
Returns:
str: Newline-separated URLs from search results or error messages.
Raises:
Exception: If web search fails entirely.
"""
try:
urls = []
for query in queries:
results = DDGS(proxies=None).text(
query, max_results=self.max_results_per_query
)
for result in results:
link = result["href"]
try:
urls.append(link)
except Exception as e:
urls.append(
f"{type(e)}: Failed to parse content from URL {link}"
)
self.search_history.append(query)
return "\n\n".join(urls)
except Exception as e:
return f"{type(e)}: Failed to search the web for text"
We are grabbing the first 2 results that best match each of our user queries and retrieving their URLs. We save our search results into search_history
to provide as context for future searches.
We also want to setup our extract_content
tool which will take in a url and grab the HTML content.
import re
import requests
from bs4 import BeautifulSoup
def extract_content(url: str) -> str:
"""Extract the main content from a webpage.
Args:
url: The URL of the webpage to extract the content from.
Returns:
The extracted content as a string.
"""
try:
response = requests.get(url, timeout=5)
soup = BeautifulSoup(response.content, "html.parser")
unwanted_tags = ["script", "style", "nav", "header", "footer", "aside"]
for tag in unwanted_tags:
for element in soup.find_all(tag):
element.decompose()
main_content = (
soup.find("main")
or soup.find("article")
or soup.find("div", class_=re.compile("content|main"))
)
if main_content:
text = main_content.get_text(separator="\n", strip=True)
else:
text = soup.get_text(separator="\n", strip=True)
lines = (line.strip() for line in text.splitlines())
return "\n".join(line for line in lines if line)
except Exception as e:
return f"{type(e)}: Failed to extract content from URL {url}"
Notice that we did not define extract_content
as a method of WebAssistant
, since extract_content
does not need to update state.
Now that our tools are setup, we can proceed to implement the Q&A functionality of our WebAssistant
.
Add Q&A Functionality¶
Now that we have our tools we can now create our prompt_template
and _stream
function. We engineer the prompt to first use our _web_search
tool, then extract_content
from the tool before answering the user question based on the retrieved content:
from datetime import datetime
class WebAssistantBaseWithStream(WebAssistantBase):
@openai.call(model="gpt-4o-mini", stream=True)
@prompt_template(
"""
SYSTEM:
You are an expert web searcher. Your task is to answer the user's question using the provided tools.
The current date is {current_date}.
You have access to the following tools:
- `_web_search`: Search the web when the user asks a question. Follow these steps for EVERY web search query:
1. There is a previous search context: {self.search_history}
2. There is the current user query: {question}
3. Given the previous search context, generate multiple search queries that explores whether the new query might be related to or connected with the context of the current user query.
Even if the connection isn't immediately clear, consider how they might be related.
- `extract_content`: Parse the content of a webpage.
When calling the `_web_search` tool, the `body` is simply the body of the search
result. You MUST then call the `extract_content` tool to get the actual content
of the webpage. It is up to you to determine which search results to parse.
Once you have gathered all of the information you need, generate a writeup that
strikes the right balance between brevity and completeness based on the context of the user's query.
MESSAGES: {self.messages}
USER: {question}
"""
)
async def _stream(self, question: str) -> openai.OpenAIDynamicConfig:
return {
"tools": [self._web_search, extract_content],
"computed_fields": {
"current_date": datetime.now().strftime("%Y-%m-%d %H:%M:%S")
},
}
async def _step(self, question: str):
print(self.messages)
response = await self._stream(question)
tools_and_outputs = []
async for chunk, tool in response:
if tool:
print(f"using {tool._name()} tool with args: {tool.args}")
tools_and_outputs.append((tool, tool.call()))
else:
print(chunk.content, end="", flush=True)
if response.user_message_param:
self.messages.append(response.user_message_param)
self.messages.append(response.message_param)
if tools_and_outputs:
self.messages += response.tool_message_params(tools_and_outputs)
await self._step("")
async def run(self):
while True:
question = input("(User): ")
if question == "exit":
break
print(f"(User): {question}", flush=True)
print("(Assistant): ", end="", flush=True)
await self._step(question)
print()
There are a few things to note:
- We set our
@openai.call()
tostream=True
to provide a more responsive user experience. - We give the LLM the current date so the user does not need to provide that.
- We instruct our LLM on how to use the tools.
- User queries can often times be ambiguous so giving as much context to the LLM when it generates the search query is crucial.
- Multiple search queries are generated for user queries that might rely on previous context.
Example search queries¶
await WebAssistantBaseWithStream().run()
(Assistant): [] using _web_search tool with args: {'queries': ['LLM development tool library', 'best libraries for LLM development', 'open source LLM libraries', 'popular LLM frameworks for developers']} [{'role': 'user', 'content': 'I am a SWE looking for a LLM dev tool library'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"queries":["LLM development tool library","best libraries for LLM development","open source LLM libraries","popular LLM frameworks for developers"]}', 'name': '_web_search'}, 'id': 'call_eVVmR6gCJnuOMDNDyKHxVPd4'}]}, {'role': 'tool', 'content': 'https://github.com/tensorchord/awesome-llmops\n\nhttps://github.com/M1n9X/llm_agents_devtools\n\nhttps://github.com/jihoo-kim/awesome-production-llm\n\nhttps://github.com/tensorchord/awesome-llmops\n\nhttps://www.datacamp.com/blog/top-open-source-llms\n\nhttps://github.com/tensorchord/awesome-llmops\n\nhttps://llmmodels.org/blog/top-10-open-source-llm-frameworks-2024/\n\nhttps://github.com/tensorchord/awesome-llmops', 'tool_call_id': 'call_eVVmR6gCJnuOMDNDyKHxVPd4', 'name': '_web_search'}] using extract_content tool with args: {'url': 'https://github.com/tensorchord/awesome-llmops'} using extract_content tool with args: {'url': 'https://github.com/M1n9X/llm_agents_devtools'} using extract_content tool with args: {'url': 'https://github.com/jihoo-kim/awesome-production-llm'} using extract_content tool with args: {'url': 'https://llmmodels.org/blog/top-10-open-source-llm-frameworks-2024/'} [{'role': 'user', 'content': 'I am a SWE looking for a LLM dev tool library'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"queries":["LLM development tool library","best libraries for LLM development","open source LLM libraries","popular LLM frameworks for developers"]}', 'name': '_web_search'}, 'id': 'call_eVVmR6gCJnuOMDNDyKHxVPd4'}]}, {'role': 'tool', 'content': 'https://github.com/tensorchord/awesome-llmops\n\nhttps://github.com/M1n9X/llm_agents_devtools\n\nhttps://github.com/jihoo-kim/awesome-production-llm\n\nhttps://github.com/tensorchord/awesome-llmops\n\nhttps://www.datacamp.com/blog/top-open-source-llms\n\nhttps://github.com/tensorchord/awesome-llmops\n\nhttps://llmmodels.org/blog/top-10-open-source-llm-frameworks-2024/\n\nhttps://github.com/tensorchord/awesome-llmops', 'tool_call_id': 'call_eVVmR6gCJnuOMDNDyKHxVPd4', 'name': '_web_search'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"url": "https://github.com/tensorchord/awesome-llmops"}', 'name': 'extract_content'}, 'id': 'call_zIolfSqkgQdsqi3BsRkXkWXl'}, {'type': 'function', 'function': {'arguments': '{"url": "https://github.com/M1n9X/llm_agents_devtools"}', 'name': 'extract_content'}, 'id': 'call_Vh9DH7mDWzXzsQoHKaWakHtS'}, {'type': 'function', 'function': {'arguments': '{"url": "https://github.com/jihoo-kim/awesome-production-llm"}', 'name': 'extract_content'}, 'id': 'call_VXxtxhySFYvaXlxhDqgLVzdu'}, {'type': 'function', 'function': {'arguments': '{"url": "https://llmmodels.org/blog/top-10-open-source-llm-frameworks-2024/"}', 'name': 'extract_content'}, 'id': 'call_NUXKesTLiBqnO48zOZYVgfF7'}]}, {'role': 'tool', 'content': 'tensorchord\n/\nAwesome-LLMOps\nPublic\nNotifications\nYou must be signed in to change notification settings\nFork\n362\nStar\n3.8k\nAn awesome & curated list of best LLMOps tools for developers\nLicense\nCC0-1.0 license\n3.8k\nstars\n362\nforks\nBranches\nTags\nActivity\nStar\nNotifications\nYou must be signed in to change notification settings\ntensorchord/Awesome-LLMOps\nThis commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.\nmain\nBranches\nTags\nGo to file\nCode\nFolders and files\nName\nName\nLast commit message\nLast commit date\nLatest commit\nHistory\n155 Commits\nscripts\nscripts\n.gitignore\n.gitignore\nLICENSE\nLICENSE\nREADME.md\nREADME.md\ncontributing.md\ncontributing.md\nView all files\nRepository files navigation\nAwesome LLMOps\nAn awesome & curated list of the best LLMOps tools for developers.\nContribute\nContributions are most welcome, please adhere to the\ncontribution guidelines\n.\nTable of Contents\nTable of Contents\nModel\nLarge Language Model\nCV Foundation Model\nAudio Foundation Model\nServing\nLarge Model Serving\nFrameworks/Servers for Serving\nObservability\nSecurity\nLLMOps\nSearch\nVector search\nCode AI\nTraining\nIDEs and Workspaces\nFoundation Model Fine Tuning\nFrameworks for Training\nExperiment Tracking\nVisualization\nModel Editing\nData\nData Management\nData Storage\nData Tracking\nFeature Engineering\nData/Feature enrichment\nLarge Scale Deployment\nML Platforms\nWorkflow\nScheduling\nModel Management\nPerformance\nML Compiler\nProfiling\nAutoML\nOptimizations\nFederated ML\nAwesome Lists\nModel\nLarge Language Model\nProject\nDetails\nRepository\nAlpaca\nCode and documentation to train Stanford\'s Alpaca models, and generate the data.\nBELLE\nA 7B Large Language Model fine-tune by 34B Chinese Character Corpus, based on LLaMA and Alpaca.\nBloom\nBigScience Large Open-science Open-access Multilingual Language Model\ndolly\nDatabricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform\nFalcon 40B\nFalcon-40B-Instruct is a 40B parameters causal decoder-only model built by TII based on Falcon-40B and finetuned on a mixture of Baize. It is made available under the Apache 2.0 license.\nFastChat (Vicuna)\nAn open platform for training, serving, and evaluating large language models. Release repo for Vicuna and FastChat-T5.\nGemma\nGemma is a family of lightweight, open models built from the research and technology that Google used to create the Gemini models.\nGLM-6B (ChatGLM)\nAn Open Bilingual Pre-Trained Model, quantization of ChatGLM-130B, can run on consumer-level GPUs.\nChatGLM2-6B\nChatGLM2-6B is the second-generation version of the open-source bilingual (Chinese-English) chat model\nChatGLM-6B\n.\nGLM-130B (ChatGLM)\nAn Open Bilingual Pre-Trained Model (ICLR 2023)\nGPT-NeoX\nAn implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.\nLuotuo\nA Chinese LLM, Based on LLaMA and fine tune by Stanford Alpaca, Alpaca LoRA, Japanese-Alpaca-LoRA.\nMixtral-8x7B-v0.1\nThe Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts.\nStableLM\nStableLM: Stability AI Language Models\n⬆ back to ToC\nCV Foundation Model\nProject\nDetails\nRepository\ndisco-diffusion\nA frankensteinian amalgamation of notebooks, models and techniques for the generation of AI Art and Animations.\nmidjourney\nMidjourney is an independent research lab exploring new mediums of thought and expanding the imaginative powers of the human species.\nsegment-anything (SAM)\nproduces high quality object masks from input prompts such as points or boxes, and it can be used to generate masks for all objects in an image.\nstable-diffusion\nA latent text-to-image diffusion model\nstable-diffusion v2\nHigh-Resolution Image Synthesis with Latent Diffusion Models\n⬆ back to ToC\nAudio Foundation Model\nProject\nDetails\nRepository\nbark\nBark is a transformer-based text-to-audio model created by Suno. Bark can generate highly realistic, multilingual speech as well as other audio - including music, background noise and simple sound effects.\nwhisper\nRobust Speech Recognition via Large-Scale Weak Supervision\nServing\nLarge Model Serving\nProject\nDetails\nRepository\nAlpaca-LoRA-Serve\nAlpaca-LoRA as Chatbot service\nCTranslate2\nfast inference engine for Transformer models in C++\nClip-as-a-service\nserving the OpenAI CLIP model\nDeepSpeed-MII\nMII makes low-latency and high-throughput inference possible, powered by DeepSpeed.\nFaster Whisper\nfast inference engine for whisper in C++ using CTranslate2.\nFlexGen\nRunning large language models on a single GPU for throughput-oriented scenarios.\nFlowise\nDrag & drop UI to build your customized LLM flow using LangchainJS.\nllama.cpp\nPort of Facebook\'s LLaMA model in C/C++\nInfinity\nRest API server for serving text-embeddings\nModelz-LLM\nOpenAI compatible API for LLMs and embeddings (LLaMA, Vicuna, ChatGLM and many others)\nOllama\nServe Llama 2 and other large language models locally from command line or through a browser interface.\nTensorRT-LLM\nInference engine for TensorRT on Nvidia GPUs\ntext-generation-inference\nLarge Language Model Text Generation Inference\ntext-embeddings-inference\nInference for text-embedding models\ntokenizers\n💥 Fast State-of-the-Art Tokenizers optimized for Research and Production\nvllm\nA high-throughput and memory-efficient inference and serving engine for LLMs.\nwhisper.cpp\nPort of OpenAI\'s Whisper model in C/C++\nx-stable-diffusion\nReal-time inference for Stable Diffusion - 0.88s latency. Covers AITemplate, nvFuser, TensorRT, FlashAttention.\n⬆ back to ToC\nFrameworks/Servers for Serving\nProject\nDetails\nRepository\nBentoML\nThe Unified Model Serving Framework\nJina\nBuild multimodal AI services via cloud native technologies · Model Serving · Generative AI · Neural Search · Cloud Native\nMosec\nA machine learning model serving framework with dynamic batching and pipelined stages, provides an easy-to-use Python interface.\nTFServing\nA flexible, high-performance serving system for machine learning models.\nTorchserve\nServe, optimize and scale PyTorch models in production\nTriton Server (TRTIS)\nThe Triton Inference Server provides an optimized cloud and edge inferencing solution.\nlangchain-serve\nServerless LLM apps on Production with Jina AI Cloud\nlanarky\nFastAPI framework to build production-grade LLM applications\nray-llm\nLLMs on Ray - RayLLM\nXinference\nReplace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you\'re empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.\n⬆ back to ToC\nSecurity\nFrameworks for LLM security\nProject\nDetails\nRepository\nPlexiglass\nA Python Machine Learning Pentesting Toolbox for Adversarial Attacks. Works with LLMs, DNNs, and other machine learning algorithms.\n⬆ back to ToC\nObservability\nProject\nDetails\nRepository\nAzure OpenAI Logger\n"Batteries included" logging solution for your Azure OpenAI instance.\nDeepchecks\nTests for Continuous Validation of ML Models & Data. Deepchecks is a Python package for comprehensively validating your machine learning models and data with minimal effort.\nEvidently\nAn open-source framework to evaluate, test and monitor ML and LLM-powered systems.\nFiddler AI\nEvaluate, monitor, analyze, and improve machine learning and generative models from pre-production to production. Ship more ML and LLMs into production, and monitor ML and LLM metrics like hallucination, PII, and toxicity.\nGiskard\nTesting framework dedicated to ML models, from tabular to LLMs. Detect risks of biases, performance issues and errors in 4 lines of code.\nGreat Expectations\nAlways know what to expect from your data.\nwhylogs\nThe open standard for data logging\n⬆ back to ToC\nLLMOps\nProject\nDetails\nRepository\nagenta\nThe LLMOps platform to build robust LLM apps. Easily experiment and evaluate different prompts, models, and workflows to build robust apps.\nAI studio\nA Reliable Open Source AI studio to build core infrastructure stack for your LLM Applications. It allows you to gain visibility, make your application reliable, and prepare it for production with features such as caching, rate limiting, exponential retry, model fallback, and more.\nArize-Phoenix\nML observability for LLMs, vision, language, and tabular models.\nBudgetML\nDeploy a ML inference service on a budget in less than 10 lines of code.\ndeeplake\nStream large multimodal datasets to achieve near 100% GPU utilization. Query, visualize, & version control data. Access data w/o the need to recompute the embeddings for the model finetuning.\nDify\nOpen-source framework aims to enable developers (and even non-developers) to quickly build useful applications based on large language models, ensuring they are visual, operable, and improvable.\nDstack\nCost-effective LLM development in any cloud (AWS, GCP, Azure, Lambda, etc).\nEmbedchain\nFramework to create ChatGPT like bots over your dataset.\nEvidently\nAn open-source framework to evaluate, test and monitor ML and LLM-powered systems.\nFiddler AI\nEvaluate, monitor, analyze, and improve MLOps and LLMOps from pre-production to production.\nGlide\nCloud-Native LLM Routing Engine. Improve LLM app resilience and speed.\ngotoHuman\nBring a\nhuman into the loop\nin your LLM-based and agentic workflows. Prompt users to approve actions, select next steps, or review and validate generated results.\nGPTCache\nCreating semantic cache to store responses from LLM queries.\nGPUStack\nAn open-source GPU cluster manager for running and managing LLMs\nHaystack\nQuickly compose applications with LLM Agents, semantic search, question-answering and more.\nHelicone\nOpen-source LLM observability platform for logging, monitoring, and debugging AI applications. Simple 1-line integration to get started.\nIzlo\nPrompt management tools for teams. Store, improve, test, and deploy your prompts in one unified workspace.\nKeywords AI\nA unified DevOps platform for AI software. Keywords AI makes it easy for developers to build LLM applications.\nlangchain\nBuilding applications with LLMs through composability\nLangFlow\nAn effortless way to experiment and prototype LangChain flows with drag-and-drop components and a chat interface.\nLangfuse\nOpen Source LLM Engineering Platform: Traces, evals, prompt management and metrics to debug and improve your LLM application.\nLangKit\nOut-of-the-box LLM telemetry collection library that extracts features and profiles prompts, responses and metadata about how your LLM is performing over time to find problems at scale.\nLiteLLM 🚅\nA simple & light 100 line package to\nstandardize LLM API calls\nacross OpenAI, Azure, Cohere, Anthropic, Replicate API Endpoints\nLiteral AI\nMulti-modal LLM observability and evaluation platform. Create prompt templates, deploy prompts versions, debug LLM runs, create datasets, run evaluations, monitor LLM metrics and collect human feedback.\nLlamaIndex\nProvides a central interface to connect your LLMs with external data.\nLLMApp\nLLM App is a Python library that helps you build real-time LLM-enabled data pipelines with few lines of code.\nLLMFlows\nLLMFlows is a framework for building simple, explicit, and transparent LLM applications such as chatbots, question-answering systems, and agents.\nLLMonitor\nObservability and monitoring for AI apps and agents. Debug agents with powerful tracing and logging. Usage analytics and dive deep into the history of your requests. Developer friendly modules with plug-and-play integration into LangChain.\nmagentic\nSeamlessly integrate LLMs as Python functions. Use type annotations to specify structured output. Mix LLM queries and function calling with regular Python code to create complex LLM-powered functionality.\nManag.ai\nYour all-in-one prompt management and observability platform. Craft, track, and perfect your LLM prompts with ease.\nMirascope\nIntuitive convenience tooling for lightning-fast, efficient development and ensuring quality in LLM-based applications\nOpenLIT\nOpenLIT is an OpenTelemetry-native GenAI and LLM Application Observability tool and provides OpenTelmetry Auto-instrumentation for monitoring LLMs, VectorDBs and Frameworks. It provides valuable insights into token & cost usage, user interaction, and performance related metrics.\nOpik\nConfidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.\nParea AI\nPlatform and SDK for AI Engineers providing tools for LLM evaluation, observability, and a version-controlled enhanced prompt playground.\nPezzo 🕹️\nPezzo is the open-source LLMOps platform built for developers and teams. In just two lines of code, you can seamlessly troubleshoot your AI operations, collaborate and manage your prompts in one place, and instantly deploy changes to any environment.\nPromptHub\nFull stack prompt management tool designed to be usable by technical and non-technical team members. Test, version, collaborate, deploy, and monitor, all from one place.\npromptfoo\nOpen-source tool for testing & evaluating prompt quality. Create test cases, automatically check output quality and catch regressions, and reduce evaluation cost.\nPromptFoundry\nThe simple prompt engineering and evaluation tool designed for developers building AI applications.\nPromptLayer 🍰\nPrompt Engineering platform. Collaborate, test, evaluate, and monitor your LLM applications\nPromptMage\nOpen-source tool to simplify the process of creating and managing LLM workflows and prompts as a self-hosted solution.\nPrompteams\nPrompt management system. Version, test, collaborate, and retrieve prompts through real-time APIs. Have GitHub style with repos, branches, and commits (and commit history).\nprompttools\nOpen-source tools for testing and experimenting with prompts. The core idea is to enable developers to evaluate prompts using familiar interfaces like code and notebooks. In just a few lines of codes, you can test your prompts and parameters across different models (whether you are using OpenAI, Anthropic, or LLaMA models). You can even evaluate the retrieval accuracy of vector databases.\nTreeScale\nAll In One Dev Platform For LLM Apps. Deploy LLM-enhanced APIs seamlessly using tools for prompt optimization, semantic querying, version management, statistical evaluation, and performance tracking. As a part of the developer friendly API implementation TreeScale offers Elastic LLM product, which makes a unified API Endpoint for all major LLM providers and open source models.\nTrueFoundry\nDeploy LLMOps tools like Vector DBs, Embedding server etc on your own Kubernetes (EKS,AKS,GKE,On-prem) Infra including deploying, Fine-tuning, tracking Prompts and serving Open Source LLM Models with full Data Security and Optimal GPU Management. Train and Launch your LLM Application at Production scale with best Software Engineering practices.\nReliableGPT 💪\nHandle OpenAI Errors (overloaded OpenAI servers, rotated keys, or context window errors) for your production LLM Applications.\nPortkey\nControl Panel with an observability suite & an AI gateway — to ship fast, reliable, and cost-efficient apps.\nVellum\nAn AI product development platform to experiment with, evaluate, and deploy advanced LLM apps.\nWeights & Biases (Prompts)\nA suite of LLMOps tools within the developer-first W&B MLOps platform. Utilize W&B Prompts for visualizing and inspecting LLM execution flow, tracking inputs and outputs, viewing intermediate results, securely managing prompts and LLM chain configurations.\nWordware\nA web-hosted IDE where non-technical domain experts work with AI Engineers to build task-specific AI agents. It approaches prompting as a new programming language rather than low/no-code blocks.\nxTuring\nBuild and control your personal LLMs with fast and efficient fine-tuning.\nZenML\nOpen-source framework for orchestrating, experimenting and deploying production-grade ML solutions, with built-in\nlangchain\n&\nllama_index\nintegrations.\n⬆ back to ToC\nSearch\nVector search\nProject\nDetails\nRepository\nAquilaDB\nAn easy to use Neural Search Engine. Index latent vectors along with JSON metadata and do efficient k-NN search.\nAwadb\nAI Native database for embedding vectors\nChroma\nthe open source embedding database\nInfinity\nThe AI-native database built for LLM applications, providing incredibly fast vector and full-text search\nLancedb\nDeveloper-friendly, serverless vector database for AI applications. Easily add long-term memory to your LLM apps!\nMarqo\nTensor search for humans.\nMilvus\nVector database for scalable similarity search and AI applications.\nPinecone\nThe Pinecone vector database makes it easy to build high-performance vector search applications. Developer-friendly, fully managed, and easily scalable without infrastructure hassles.\npgvector\nOpen-source vector similarity search for Postgres.\npgvecto.rs\nVector database plugin for Postgres, written in Rust, specifically designed for LLM.\nQdrant\nVector Search Engine and Database for the next generation of AI applications. Also available in the cloud\ntxtai\nBuild AI-powered semantic search applications\nVald\nA Highly Scalable Distributed Vector Search Engine\nVearch\nA distributed system for embedding-based vector retrieval\nVectorDB\nA Python vector database you just need - no more, no less.\nVellum\nA managed service for ingesting documents and performing hybrid semantic/keyword search across them. Comes with out-of-box support for OCR, text chunking, embedding model experimentation, metadata filtering, and production-grade APIs.\nWeaviate\nWeaviate is an open source vector search engine that stores both objects and vectors, allowing for combining vector search with structured filtering with the fault-tolerance and scalability of a cloud-native database, all accessible through GraphQL, REST, and various language clients.\n⬆ back to ToC\nCode AI\nProject\nDetails\nRepository\nCodeGeeX\nCodeGeeX: An Open Multilingual Code Generation Model (KDD 2023)\nCodeGen\nCodeGen is an open-source model for program synthesis. Trained on TPU-v4. Competitive with OpenAI Codex.\nCodeT5\nOpen Code LLMs for Code Understanding and Generation.\nContinue\n⏩ the open-source autopilot for software development—bring the power of ChatGPT to VS Code\nfauxpilot\nAn open-source alternative to GitHub Copilot server\ntabby\nSelf-hosted AI coding assistant. An opensource / on-prem alternative to GitHub Copilot.\nTraining\nIDEs and Workspaces\nProject\nDetails\nRepository\ncode server\nRun VS Code on any machine anywhere and access it in the browser.\nconda\nOS-agnostic, system-level binary package manager and ecosystem.\nDocker\nMoby is an open-source project created by Docker to enable and accelerate software containerization.\nenvd\n🏕️ Reproducible development environment for AI/ML.\nJupyter Notebooks\nThe Jupyter notebook is a web-based notebook environment for interactive computing.\nKurtosis\nA build, packaging, and run system for ephemeral multi-container environments.\nWordware\nA web-hosted IDE where non-technical domain experts work with AI Engineers to build task-specific AI agents. It approaches prompting as a new programming language rather than low/no-code blocks.\n⬆ back to ToC\nFoundation Model Fine Tuning\nProject\nDetails\nRepository\nalpaca-lora\nInstruct-tune LLaMA on consumer hardware\nfinetuning-scheduler\nA PyTorch Lightning extension that accelerates and enhances foundation model experimentation with flexible fine-tuning schedules.\nFlyflow\nOpen source, high performance fine tuning as a service for GPT4 quality models with 5x lower latency and 3x lower cost\nLMFlow\nAn Extensible Toolkit for Finetuning and Inference of Large Foundation Models\nLora\nUsing Low-rank adaptation to quickly fine-tune diffusion models.\npeft\nState-of-the-art Parameter-Efficient Fine-Tuning.\np-tuning-v2\nAn optimized prompt tuning strategy achieving comparable performance to fine-tuning on small/medium-sized models and sequence tagging challenges.\n(ACL 2022)\nQLoRA\nEfficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.\nTRL\nTrain transformer language models with reinforcement learning.\n⬆ back to ToC\nFrameworks for Training\nProject\nDetails\nRepository\nAccelerate\n🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision.\nApache MXNet\nLightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler.\naxolotl\nA tool designed to streamline the fine-tuning of various AI models, offering support for multiple configurations and architectures.\nCaffe\nA fast open framework for deep learning.\nCandle\nMinimalist ML framework for Rust .\nColossalAI\nAn integrated large-scale model training system with efficient parallelization techniques.\nDeepSpeed\nDeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.\nHorovod\nDistributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.\nJax\nAutograd and XLA for high-performance machine learning research.\nKedro\nKedro is an open-source Python framework for creating reproducible, maintainable and modular data science code.\nKeras\nKeras is a deep learning API written in Python, running on top of the machine learning platform TensorFlow.\nLightGBM\nA fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.\nMegEngine\nMegEngine is a fast, scalable and easy-to-use deep learning framework, with auto-differentiation.\nmetric-learn\nMetric Learning Algorithms in Python.\nMindSpore\nMindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios.\nOneflow\nOneFlow is a performance-centered and open-source deep learning framework.\nPaddlePaddle\nMachine Learning Framework from Industrial Practice.\nPyTorch\nTensors and Dynamic neural networks in Python with strong GPU acceleration.\nPyTorch Lightning\nDeep learning framework to train, deploy, and ship AI products Lightning fast.\nXGBoost\nScalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library.\nscikit-learn\nMachine Learning in Python.\nTensorFlow\nAn Open Source Machine Learning Framework for Everyone.\nVectorFlow\nA minimalist neural network library optimized for sparse data and single machine environments.\n⬆ back to ToC\nExperiment Tracking\nProject\nDetails\nRepository\nAim\nan easy-to-use and performant open-source experiment tracker.\nClearML\nAuto-Magical CI/CD to streamline your ML workflow. Experiment Manager, MLOps and Data-Management\nComet\nComet is an MLOps platform that offers experiment tracking, model production management, a model registry, and full data lineage from training straight through to production. Comet plays nicely with all your favorite tools, so you don\'t have to change your existing workflow. Comet Opik to confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle!\nGuild AI\nExperiment tracking, ML developer tools.\nMLRun\nMachine Learning automation and tracking.\nKedro-Viz\nKedro-Viz is an interactive development tool for building data science pipelines with Kedro. Kedro-Viz also allows users to view and compare different runs in the Kedro project.\nLabNotebook\nLabNotebook is a tool that allows you to flexibly monitor, record, save, and query all your machine learning experiments.\nSacred\nSacred is a tool to help you configure, organize, log and reproduce experiments.\nWeights & Biases\nA developer first, lightweight, user-friendly experiment tracking and visualization tool for machine learning projects, streamlining collaboration and simplifying MLOps. W&B excels at tracking LLM-powered applications, featuring W&B Prompts for LLM execution flow visualization, input and output monitoring, and secure management of prompts and LLM chain configurations.\n⬆ back to ToC\nVisualization\nProject\nDetails\nRepository\nFiddler AI\nRich dashboards, reports, and UMAP to perform root cause analysis, pinpoint problem areas, like correctness, safety, and privacy issues, and improve LLM outcomes.\nManiford\nA model-agnostic visual debugging tool for machine learning.\nnetron\nVisualizer for neural network, deep learning, and machine learning models.\nOpenOps\nBring multiple data streams into one dashboard.\nTensorBoard\nTensorFlow\'s Visualization Toolkit.\nTensorSpace\nNeural network 3D visualization framework, build interactive and intuitive model in browsers, support pre-trained deep learning models from TensorFlow, Keras, TensorFlow.js.\ndtreeviz\nA python library for decision tree visualization and model interpretation.\nZetane Viewer\nML models and internal tensors 3D visualizer.\nZeno\nAI evaluation platform for interactively exploring data and model outputs.\nModel Editing\nProject\nDetails\nRepository\nFastEdit\nFastEdit aims to assist developers with injecting fresh and customized knowledge into large language models efficiently using one single command.\n⬆ back to ToC\nData\nData Management\nProject\nDetails\nRepository\nArtiVC\nA version control system to manage large files. Lake is a dataset format with a simple API for creating, storing, and collaborating on AI datasets of any size.\nDolt\nGit for Data.\nDVC\nData Version Control - Git for Data & Models - ML Experiments Management.\nDelta-Lake\nStorage layer that brings scalable, ACID transactions to Apache Spark and other engines.\nPachyderm\nPachyderm is a version control system for data.\nQuilt\nA self-organizing data hub for S3.\n⬆ back to ToC\nData Storage\nProject\nDetails\nRepository\nJuiceFS\nA distributed POSIX file system built on top of Redis and S3.\nLakeFS\nGit-like capabilities for your object storage.\nLance\nModern columnar data format for ML implemented in Rust.\n⬆ back to ToC\nData Tracking\nProject\nDetails\nRepository\nPiperider\nA CLI tool that allows you to build data profiles and write assertion tests for easily evaluating and tracking your data\'s reliability over time.\nLUX\nA Python library that facilitates fast and easy data exploration by automating the visualization and data analysis process.\n⬆ back to ToC\nFeature Engineering\nProject\nDetails\nRepository\nFeatureform\nThe Virtual Feature Store. Turn your existing data infrastructure into a feature store.\nFeatureTools\nAn open source python framework for automated feature engineering\n⬆ back to ToC\nData/Feature enrichment\nProject\nDetails\nRepository\nUpgini\nFree automated data & feature enrichment library for machine learning: automatically searches through thousands of ready-to-use features from public and community shared data sources and enriches your training dataset with only the accuracy improving features\nFeast\nAn open source feature store for machine learning.\ndistilabel\n⚗️ distilabel is a framework for synthetic data and AI feedback for AI engineers that require high-quality outputs, full data ownership, and overall efficiency.\n⬆ back to ToC\nLarge Scale Deployment\nML Platforms\nProject\nDetails\nRepository\nComet\nComet is an MLOps platform that offers experiment tracking, model production management, a model registry, and full data lineage from training straight through to production. Comet plays nicely with all your favorite tools, so you don\'t have to change your existing workflow. Comet Opik to confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle!\nClearML\nAuto-Magical CI/CD to streamline your ML workflow. Experiment Manager, MLOps and Data-Management.\nHopsworks\nHopsworks is a MLOps platform for training and operating large and small ML systems, including fine-tuning and serving LLMs. Hopsworks includes both a feature store and vector database for RAG.\nOpenLLM\nAn open platform for operating large language models (LLMs) in production. Fine-tune, serve, deploy, and monitor any LLMs with ease.\nMLflow\nOpen source platform for the machine learning lifecycle.\nMLRun\nAn open MLOps platform for quickly building and managing continuous ML applications across their lifecycle.\nModelFox\nModelFox is a platform for managing and deploying machine learning models.\nKserve\nStandardized Serverless ML Inference Platform on Kubernetes\nKubeflow\nMachine Learning Toolkit for Kubernetes.\nPAI\nResource scheduling and cluster management for AI.\nPolyaxon\nMachine Learning Management & Orchestration Platform.\nPrimehub\nAn effortless infrastructure for machine learning built on the top of Kubernetes.\nOpenModelZ\nOne-click machine learning deployment (LLM, text-to-image and so on) at scale on any cluster (GCP, AWS, Lambda labs, your home lab, or even a single machine).\nSeldon-core\nAn MLOps framework to package, deploy, monitor and manage thousands of production machine learning models\nStarwhale\nAn MLOps/LLMOps platform for model building, evaluation, and fine-tuning.\nTrueFoundry\nA PaaS to deploy, Fine-tune and serve LLM Models on a company’s own Infrastructure with Data Security and Optimal GPU and Cost Management. Launch your LLM Application at Production scale with best DevSecOps practices.\nWeights & Biases\nA lightweight and flexible platform for machine learning experiment tracking, dataset versioning, and model management, enhancing collaboration and streamlining MLOps workflows. W&B excels at tracking LLM-powered applications, featuring W&B Prompts for LLM execution flow visualization, input and output monitoring, and secure management of prompts and LLM chain configurations.\n⬆ back to ToC\nWorkflow\nProject\nDetails\nRepository\nAirflow\nA platform to programmatically author, schedule and monitor workflows.\naqueduct\nAn Open-Source Platform for Production Data Science\nArgo Workflows\nWorkflow engine for Kubernetes.\nFlyte\nKubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale.\nHamilton\nA lightweight framework to represent ML/language model pipelines as a series of python functions.\nKubeflow Pipelines\nMachine Learning Pipelines for Kubeflow.\nLangFlow\nAn effortless way to experiment and prototype LangChain flows with drag-and-drop components and a chat interface.\nMetaflow\nBuild and manage real-life data science projects with ease!\nPloomber\nThe fastest way to build data pipelines. Develop iteratively, deploy anywhere.\nPrefect\nThe easiest way to automate your data.\nVDP\nAn open-source unstructured data ETL tool to streamline the end-to-end unstructured data processing pipeline.\nZenML\nMLOps framework to create reproducible pipelines.\n⬆ back to ToC\nScheduling\nProject\nDetails\nRepository\nKueue\nKubernetes-native Job Queueing.\nPAI\nResource scheduling and cluster management for AI (Open-sourced by Microsoft).\nSlurm\nA Highly Scalable Workload Manager.\nVolcano\nA Cloud Native Batch System (Project under CNCF).\nYunikorn\nLight-weight, universal resource scheduler for container orchestrator systems.\n⬆ back to ToC\nModel Management\nProject\nDetails\nRepository\nComet\nComet is an MLOps platform that offers Model Production Management, a Model Registry, and full model lineage from training straight through to production. Use Comet for model reproducibility, model debugging, model versioning, model visibility, model auditing, model governance, and model monitoring.\ndvc\nML Experiments Management - Data Version Control - Git for Data & Models\nModelDB\nOpen Source ML Model Versioning, Metadata, and Experiment Management\nMLEM\nA tool to package, serve, and deploy any ML model on any platform.\normb\nDocker for Your ML/DL Models Based on OCI Artifacts\n⬆ back to ToC\nPerformance\nML Compiler\nProject\nDetails\nRepository\nONNX-MLIR\nCompiler technology to transform a valid Open Neural Network Exchange (ONNX) graph into code that implements the graph with minimum runtime support.\nbitsandbytes\nAccessible large language models via k-bit quantization for PyTorch.\nTVM\nOpen deep learning compiler stack for cpu, gpu and specialized accelerators\n⬆ back to ToC\nProfiling\nProject\nDetails\nRepository\noctoml-profile\noctoml-profile is a python library and cloud service designed to provide the simplest experience for assessing and optimizing the performance of PyTorch models on cloud hardware with state-of-the-art ML acceleration technology.\nscalene\na high-performance, high-precision CPU, GPU, and memory profiler for Python\n⬆ back to ToC\nAutoML\nProject\nDetails\nRepository\nArchai\na platform for Neural Network Search (NAS) that allows you to generate efficient deep networks for your applications.\nautoai\nA framework to find the best performing AI/ML model for any AI problem.\nAutoGL\nAn autoML framework & toolkit for machine learning on graphs\nAutoGluon\nAutoML for Image, Text, and Tabular Data.\nautoml-gs\nProvide an input CSV and a target field to predict, generate a model + code to run it.\nautokeras\nAutoML library for deep learning.\nAuto-PyTorch\nAutomatic architecture search and hyperparameter optimization for PyTorch.\nauto-sklearn\nan automated machine learning toolkit and a drop-in replacement for a scikit-learn estimator.\nDragonfly\nAn open source python library for scalable Bayesian optimisation.\nDetermined\nscalable deep learning training platform with integrated hyperparameter tuning support; includes Hyperband, PBT, and other search methods.\nDEvol (DeepEvolution)\na basic proof of concept for genetic architecture search in Keras.\nEvalML\nAn open source python library for AutoML.\nFEDOT\nAutoML framework for the design of composite pipelines.\nFLAML\nFast and lightweight AutoML (\npaper\n).\nGoptuna\nA hyperparameter optimization framework, inspired by Optuna.\nHpBandSter\na framework for distributed hyperparameter optimization.\nHPOlib2\na library for hyperparameter optimization and black box optimization benchmarks.\nHyperband\nopen source code for tuning hyperparams with Hyperband.\nHypernets\nA General Automated Machine Learning Framework.\nHyperopt\nDistributed Asynchronous Hyperparameter Optimization in Python.\nhyperunity\nA toolset for black-box hyperparameter optimisation.\nIntelli\nA framework to connect a flow of ML models by applying graph theory.\nKatib\nKatib is a Kubernetes-native project for automated machine learning (AutoML).\nKeras Tuner\nHyperparameter tuning for humans.\nlearn2learn\nPyTorch Meta-learning Framework for Researchers.\nLudwig\na toolbox built on top of TensorFlow that allows to train and test deep learning models without the need to write code.\nMOE\na global, black box optimization engine for real world metric optimization by Yelp.\nModel Search\na framework that implements AutoML algorithms for model architecture search at scale.\nNASGym\na proof-of-concept OpenAI Gym environment for Neural Architecture Search (NAS).\nNNI\nAn open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.\nOptuna\nA hyperparameter optimization framework.\nPycaret\nAn open-source, low-code machine learning library in Python that automates machine learning workflows.\nRay Tune\nScalable Hyperparameter Tuning.\nREMBO\nBayesian optimization in high-dimensions via random embedding.\nRoBO\na Robust Bayesian Optimization framework.\nscikit-optimize(skopt)\nSequential model-based optimization with a\nscipy.optimize\ninterface.\nSpearmint\na software package to perform Bayesian optimization.\nTPOT\none of the very first AutoML methods and open-source software packages.\nTorchmeta\nA Meta-Learning library for PyTorch.\nVegas\nan AutoML algorithm tool chain by Huawei Noah\'s Arb Lab.\n⬆ back to ToC\nOptimizations\nProject\nDetails\nRepository\nFeatherCNN\nFeatherCNN is a high performance inference engine for convolutional neural networks.\nForward\nA library for high performance deep learning inference on NVIDIA GPUs.\nNCNN\nncnn is a high-performance neural network inference framework optimized for the mobile platform.\nPocketFlow\nuse AutoML to do model compression.\nTensorFlow Model Optimization\nA suite of tools that users, both novice and advanced, can use to optimize machine learning models for deployment and execution.\nTNN\nA uniform deep learning inference framework for mobile, desktop and server.\noptimum-tpu\nGoogle TPU optimizations for transformers models\n⬆ back to ToC\nFederated ML\nProject\nDetails\nRepository\nEasyFL\nAn Easy-to-use Federated Learning Platform\nFATE\nAn Industrial Grade Federated Learning Framework\nFedML\nThe federated learning and analytics library enabling secure and collaborative machine learning on decentralized data anywhere at any scale. Supporting large-scale cross-silo federated learning, cross-device federated learning on smartphones/IoTs, and research simulation.\nFlower\nA Friendly Federated Learning Framework\nHarmonia\nHarmonia is an open-source project aiming at developing systems/infrastructures and libraries to ease the adoption of federated learning (abbreviated to FL) for researches and production usage.\nTensorFlow Federated\nA framework for implementing federated learning\n⬆ back to ToC\nAwesome Lists\nProject\nDetails\nRepository\nAwesome Argo\nA curated list of awesome projects and resources related to Argo\nAwesome AutoDL\nAutomated Deep Learning: Neural Architecture Search Is Not the End (a curated list of AutoDL resources and an in-depth analysis)\nAwesome AutoML\nCurating a list of AutoML-related research, tools, projects and other resources\nAwesome AutoML Papers\nA curated list of automated machine learning papers, articles, tutorials, slides and projects\nAwesome-Code-LLM\n👨\u200d💻 An awesome and curated list of best code-LLM for research.\nAwesome Federated Learning Systems\nA curated list of Federated Learning Systems related academic papers, articles, tutorials, slides and projects.\nAwesome Federated Learning\nA curated list of federated learning publications, re-organized from Arxiv (mostly)\nawesome-federated-learning\nacc\nAll materials you need for Federated Learning: blogs, videos, papers, and softwares, etc.\nAwesome Open MLOps\nThis is the Fuzzy Labs guide to the universe of free and open source MLOps tools.\nAwesome Production Machine Learning\nA curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning\nAwesome Tensor Compilers\nA list of awesome compiler projects and papers for tensor computation and deep learning.\nkelvins/awesome-mlops\nA curated list of awesome MLOps tools.\nvisenger/awesome-mlops\nMachine Learning Operations - An awesome list of references for MLOps\ncurrentslab/awesome-vector-search\nA curated list of awesome vector search framework/engine, library, cloud service and research papers to vector similarity search.\npleisto/flappy\nProduction-Ready LLM Agent SDK for Every Developer\n⬆ back to ToC\nAbout\nAn awesome & curated list of best LLMOps tools for developers\nTopics\nawesome-list\nmlops\nai-development-tools\nllmops\nResources\nReadme\nLicense\nCC0-1.0 license\nActivity\nCustom properties\nStars\n3.8k\nstars\nWatchers\n65\nwatching\nForks\n362\nforks\nReport repository\nReleases\nNo releases published\nPackages\n0\nNo packages published\nContributors\n84\n+ 70 contributors\nLanguages\nShell\n86.2%\nPython\n13.8%', 'tool_call_id': 'call_zIolfSqkgQdsqi3BsRkXkWXl', 'name': 'extract_content'}, {'role': 'tool', 'content': 'M1n9X\n/\nllm_agents_devtools\nPublic\nNotifications\nYou must be signed in to change notification settings\nFork\n3\nStar\n30\nA curated list of autonomous agents and developer tools powered by LLM.\n30\nstars\n3\nforks\nBranches\nTags\nActivity\nStar\nNotifications\nYou must be signed in to change notification settings\nM1n9X/llm_agents_devtools\nThis commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.\nmain\nBranches\nTags\nGo to file\nCode\nFolders and files\nName\nName\nLast commit message\nLast commit date\nLatest commit\nHistory\n62 Commits\nREADME.md\nREADME.md\nView all files\nRepository files navigation\n⚡️ LLM Agents and DevTools\nThis is a curated list of autonomous agents and developer tools powered by LLM.\nAgents\nAgent Platforms\nApp Generators\nCoding Assistants\nCognitive Architectures\nDigital Humans\nDocumentation\nEmbodied AI\nEvaluation\nFoundation Models\nIDEs\nKnowledge Bases\nLLMOps\nOpenAI Plugins\nPR Agents\nPrompts Tuning\nReasoning and Planning\nResearch\nSearch\nSecurity\nShell Assistants\nTraining & Deployment\nAgents\nAdala\n- Autonomous DAta (Labeling) Agent framework.\nAgentGPT\n- 🤖 Assemble, configure, and deploy autonomous AI Agents in your browser.\nAutoGPT\n— An experimental open-source attempt to make GPT-4 fully autonomous.\nBabyAGI\n— An AI-powered task management system.\nBambooAI\n- A lightweight library that leverages Language Models (LLMs) to enable natural language interactions, allowing you to source and converse with data.\nBlockAGI\n- Your Self-Hosted, Hackable Research Agent Inspired by AutoGPT.\nCAMEL\n- Communicative Agents for “Mind” Exploration of Large Scale Language Model Society.\nChat-Haruhi-Suzumiya\n- Reviving Anime Character in Reality via Large Language Model.\nDB-GPT\n- by TsinghuaDatabaseGroup. LLM As Database Administrator.\nDB-GPT\n- by eosphoros-ai. Revolutionizing Database Interactions with Private LLM Technology.\nData-Copilot\n- Bridging Billions of Data and Humans with Autonomous Workflow.\nDemoGPT\n— Auto Gen-AI App Generator with the Power of Llama 2\nDevOpsGPT\n- DevOpsGPT: AI-Driven Software Development Automation Solution\nFactory\n— Agents for code generation. Waitlisted.\nGPT Engineer\n—\xa0CLI agent that generates a repository from a prompt, and asks clarifying questions.\nGPT Migrate\n—\xa0CLI agent that converts a full-stack application from one language or framework to another. Uses GPT-4 32k context.\nGeneralAgent\n- A simple, general, customizable Agent Framework.\nGitWit\n—\xa0Web-based agent for adding features to full-stack apps in Git repositories.\nGorilla\n- Large Language Model Connected with Massive APIs.\nGrit\n- GitHub-integrated agent for automating maintenance tasks and other development work.\nHyperWrite\n- Personal Assistant — AI Agent\nMuzic\n- Music Understanding and Generation with Artificial Intelligence.\nSalesGPT\n- Context-aware AI Sales Agent to automate sales outreach.\nSecond.dev\n— A platform for adding features to full-stack apps.\nShortGPT\n- Experimental AI framework for automated short/video content creation.\nSmol Developer\n— CLI agent that generates a repository from a prompt. Uses OpenAI and Anthropic.\nTeenage-AGI\n- Use OpenAI and Pinecone to Give memory to an AI agent and also allows it to "think" before making an action (outputting text)\nTransformers Agent\n- HuggingFace Transformers Agent offer a natural language API built on transformers and a curated set of tools. An agent interprets user natural language input & use tools to fulfil the user request.\nWebArena\n- A Realistic Web Environment for Building Autonomous Agents.\nautomata\n- Automata\'s objective is to evolve into a fully autonomous, self-programming Artificial Intelligence system.\nbeebot\n- An Autonomous AI Agent that works.\ngpt-researcher\n- GPT based autonomous agent that does online comprehensive research on any given topic.\ngpt_academic\n- 为 ChatGPT/GLM 提供图形交互界面,特别优化论文阅读/润色/写作体验,模块化设计,支持自定义快捷按钮&函数插件,支持 Python 和 C++等项目剖析&自译解功能,PDF/LaTex 论文翻译&总结功能,支持并行问询多种 LLM 模型,支持 chatglm2 等本地模型。兼容文心一言, moss, llama2, rwkv, claude2, 通义千问, 书生, 讯飞星火等。\nlemon-agent\n- Plan-Validate-Solve (PVS) Agent for accurate, reliable and reproducable workflow automation.\nloopgpt\n- A re-implementation of the popular Auto-GPT project as a proper python package, written with modularity and extensibility in mind.\nmini-agi\n- A minimal general-purpose autonomous agent based on GPT-3.5 / GPT-4. Can analyze stock prices, perform network security tests, create art, and order pizza.\nrci-agent\n- RCI Agent for MiniWoB++\ntalk\n- Talk with ChatGPT using your VOICE\nAgent Platforms\nAgentBench\n- A Comprehensive Benchmark to Evaluate LLMs as Agents.\nAutoAgents\n- Generate different roles for GPTs to form a collaborative entity for complex tasks.\nAutoGen\n- Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework.\nBOLAA\n- Benchmarking and Orchestrating LLM-augmented Autonomous Agents.\nE2B\n—\xa0Open source cloud platform for hosting LLM-based agents. Supports\nSmol Developer\n.\nGeniA\n- Your Engineering Gen AI Team member 🧬🤖💻\nGentopia\n- Build AGI through Interaction of Specialized Agents.\nJARVIS\n- A system to connect LLMs with ML community.\nLLM-ToolMaker\n- Large Language Models as Tool Makers.\nOpenAGI\n- An open-source AGI research platform, specifically designed to offer complex, multi-step tasks and accompanied by task-specific datasets, evaluation metrics, and a diverse range of extensible models.\nOpenAgents\n- An Open Platform for Language Agents in the Wild\nPromethAI-Backend\n- Open-source framework that gives you AI Agents that help you navigate decision-making, get personalized goals and execute them.\nSuperAGI\nOpen source platform for hosting LLM-based agents including\nSuperCoder\n.\nSuperagent\n- Build, deploy, and manage LLM-powered agents.\nToolBench\n- An open platform for training, serving, and evaluating large language model for tool learning.\nXAgent\n- An Autonomous LLM Agent for Complex Task Solving.\nagents\n- An Open-source Framework for Autonomous Language Agents.\nai-legion\n- An LLM-powered autonomous agent platform.\nchidori\n- A reactive runtime for building durable AI agents.\ndataberry\n- The no-code platform for building custom LLM Agents.\nix\n- Autonomous GPT-4 agent platform.\nlagent\n- A lightweight framework for building LLM-based agents.\nopenagent\n- Microservices approach to AGI. Modular components for AI apps or AGI agents. (... and solving some wicked LLM problems like ⚡ 2X faster LLaMA 2)\nrivet\n- AI agent and prompt chaining IDE and library.\nsemantic-kernel\n- Integrate cutting-edge LLM technology quickly and easily into your apps.\nsmart_agent\n- An open-source project offering a comprehensive library for deconstructing complex tasks and dispatching functions within a toolkit.\ntoolformer-pytorch\n- Language Models That Can Use Tools, by MetaAI.\nworkgpt\n- A GPT agent framework for invoking APIs.\nApp Generators\nGPT Pilot\n- Code out the entire app as you oversee the code being written.\nLiterally anything\n- HTML and JavaScript web app generator.\nOpenCopilot\n- AI Copilot for your own SaaS product. Open source AI sidekick for everyone.\nPico\n- End-to-end micro app generator with instant deployment.\ngpt-pilot\n- PoC for a scalable dev tool that writes entire apps from scratch while the developer oversees the implementation.\nreact-agent\n- The open-source React.js Autonomous LLM Agent.\ntextbase\n- A simple framework for building AI chatbots.\nwasp\n- The fastest way to develop full-stack web apps with React & Node.js.\nCoding Assistants\nAI Code Convert\n—\xa0A web tool for translating code between programming languages.\nAI Code Playground\n— CodeChain is a web tool for refactoring and improving code.\nAPI Copilot\n— Assistant for backend API development.\nAdrenaline\n- Web-based chatbot using AI and ASTs to answer questions about your codebase.\nAmazon CodeWhisperer\n- An AI coding companion that generates whole line and full function code suggestions in your IDE to help you get more done faster.\nBito AI\n— Bito is powered by OpenAI\'s ChatGPT and GPT-4, which revolutionizes the way developers write code. Generate high-quality AI-powered code, AI-assisted code completion, optimize code performance, explain complex code snippets, generate unit tests, and more – all effortlessly.\nBlackbox\n— VS Code extension with autocomplete and chat including links to online coding references.\nCodeApex\n- A bilingual programming evaluation benchmark for Large Language Models.\nCodeGeeX\nOpen source assistant based on the CodeGeeX LLM with chat, completion, and refactoring. Extensions for 9 editors including VS Code, and PyCharm.\nCodeMate\n—\xa0VS Code extension for debugging and optimizing code.\nCodePal\n— A web tool for quickly generating or refactoring code.\nCodeSquire\n— Chrome extension that adds autocomplete to Google Colab, BigQuery, and JupyterLab.\nCodeium\n— Assistant with autocomplete, natural language search and chat. Extensions for 21 editors including VS Code, JetBrains, Neovim, Vim, Emacs, Eclipse, PyCharm, and Xcode. Enterprise version includes codebase-specific fine-tuning.\nContinue\n— VS Code extension with chat, refactor, and code generation. Edits multiple files and runs commands on your behalf.\nGenie AI\n— A Visual Studio Code - ChatGPT Integration. Supports GPT-4, GPT3.5, GPT3 and Codex models. Create new files, view diffs with one click; your copilot to learn code, add tests, find bugs and more.\nGitHub Copilot X\n— A VS Code extension with chat, pull request text generation, and unit test generation.\nIncognito Pilot\n—\xa0Open source assistant with built-in Python editor and interpreter.\nMagnet\n— Web-based chatbot with repositories and issues as context.\nQuack AI\n—\xa0VS Code extension for adhering to project coding guidelines. Waitlist.\nRefact\n—\xa0Open-Source Coding Assistant with Fine-Tuning on codebase, autocompletion, code refactoring, code analysis, integrated chat and more!\nReplit Ghostwriter Chat\n— Assistant built into\nReplit\nwith chat, proactive debugging, and autocomplete. Uses OpenAI for chat and\nreplit-code-v1-3b\n(OS) for autocomplete.\nRift\n- An AI-native language server for your personal AI software engineer.\nSource Graph Cody\n- Assistant with chat, refactoring, and unit test generation. Extensions for VS Code and IntelliJ. Also available as a web app.\nTabby\n—\xa0Open source, self-hosted code completion assistant. Extensions for VS Code and Vim.\nTabnine\n(Source)\n— Open source, self-hosted code completion assistant. Extensions for 15 editors including VS Code, IntelliJ, Neovim, Eclipse, and PyCharm.\nai-jsx\n- The AI Application Framework for Javascript.\nax\n- A comprehensive AI framework for TypeScript.\nclippinator\n- AI programming assistant.\ncodeinterpreter-api\n- Open source implementation of the ChatGPT Code Interpreter 👾\ncody\n- AI that knows your entire codebase.\nevalgpt\n- An code interpreter framework that utilizes large language models to automate the process of code-writing and execution, delivering precise results for user-defined tasks.\ngpt-migrate\n- Easily migrate your codebase from one framework or language to another.\nopen-interpreter\n- OpenAI\'s Code Interpreter in your terminal, running locally.\ntalk-codebase\n- CLI chatbot with repository as context. Supports OpenAI as well as locally running LLMs via GPT4All.\nCognitive Architectures\nACT-R\n- Adaptive Control of Thought—Rational.\nCoALA\n- Cognitive Architectures for Language Agents.\nMemGPT\n- Teaching LLMs memory management for unbounded context 📚🦙\nSoar\n- A general cognitive architecture for systems that exhibit intelligent behavior.\nDigital Humans\nGPT-vup\n- Live digital host for BIliBili/Tiktok.\nFaceChain\n- A deep-learning toolchain for generating your Digital-Twin.\nDocumentation\nDocify\n—\xa0A VS Code extension to generate docstrings.\nMintlify Writer\n—\xa0A VS Code extension to generate docstrings.\nTrelent\n— A VS Code extension to generate docstrings. Uses proprietary models.\ngpt-author\n- Utilize a chain of GPT-4, Stable Diffusion, and Anthropic API calls to generate an original fantasy novel.\nEmbodied AI\nAgentSims\n- An easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in.\nAgentVerse\n- A flexible framework that simplifies the process of building custom multi-agent environments for large language models (LLMs).\nChatArena\n- A Multi-Agent Language Game Environments for LLMs. The goal is to develop communication and collaboration capabilities of AIs.\nChatDev\n- Create Customized Software using Natural Language Idea (through Multi-Agent Collaboration)\nGPTeam\n- An open-source multi-agent simulation.\nGenerative Agents\n- Interactive Simulacra of Human Behavior.\nGeniA\n- Your Engineering Gen AI Team member 🧬🤖💻\nHumanoidAgents\n- Humanoid Agents: Platform for Simulating Human-like Generative Agents.\nLASER\n- LLM Agent with State-Space Exploration for Web Navigation.\nMetaGPT\n— The Multi-Agent Framework: Given one line Requirement, return PRD, Design, Tasks, Repo.\nMind2Web\n- Towards a Generalist Agent for the Web.\nMindAgent\n- Emergent Gaming Interaction.\nRT-2\n- Robotic Transformer 2 (RT-2): The Vision-Language-Action Model.\nRoboAgent\n- Towards Sample Efficient Robot Manipulation with Semantic Augmentations and Action Chunking.\nVoxPoser\n- Composable 3D Value Maps for Robotic Manipulation with Language Models\nVoyager\n- An Open-Ended Embodied Agent with Large Language Models.\nai-town\n- A deployable starter kit for building and customizing your own version of AI town - a virtual town where AI characters live, chat and socialize.\naloha\n- A Low-cost Open-source Hardware System for Bimanual Teleoperation.\ndev-gpt\n- Your Virtual Development Team.\nEvaluation\nGPT-Fathom\n- Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond.\nInstructEval\n- Evaluation suite for the systematic evaluation of instruction selection methods.\nSmartPlay\n- A benchmark for Large Language Models (LLMs). It is designed to be easy to use, and to provide a wide variety of games to test agents on.\ninstruct-eval\n- Quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.\nFoundation Models\nCodeGeeX2\n- A More Powerful Multilingual Code Generation Model.\nCodeGen\n- A family of open-source model for program synthesis. Trained on TPU-v4. Competitive with OpenAI Codex.\nCodeLlama\n- A family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks.\nDeepSeek-Coder\n- Let the Code Write Itself.\nLM Studio\n- Discover, download, and run local LLMs\nMAmmoTH\n- Building Math Generalist Models through Hybrid Instruction Tuning.\nMFTCoder\n- An open-source project of CodeFuse for multitasking Code-LLMs(large language model for code tasks), which includes models, datasets, training codebases and inference guides.\nMagic\n— Company promising two products, an assistant and\nLTM-1\n, an underlying foundation model trained on code. Waitlist.\nMathGLM\n- GPT Can Solve Mathematical Problems Without a Calculator.\nMistral-7B-v0.1\nNExT-GPT\n- Any-to-Any Multimodal Large Language Model.\nOpenChat\n- Advancing Open-source Language Models with Imperfect Data\nPhind-CodeLlama-34B-v1\n- Beating GPT-4 on HumanEval with a Fine-Tuned CodeLlama-34B.\nStarCoder\n- A helpful coding assistant.\nTinyLlama\n- An open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.\nWizardCoder\n- Empowering Code Large Language Models with Evol-Instruct.\ngpt4all\n- open-source LLM chatbots that you can run anywhere.\ngpt4free\n- The official gpt4free repository | various collection of powerful language models.\nincoder\n- A Generative Model for Code Infilling and Synthesis.\nlitellm\n- Call all LLM APIs using the OpenAI format. Use Azure, OpenAI, Cohere, Anthropic, Ollama, VLLM, Sagemaker, HuggingFace, Replicate (100+ LLMs)\nprivateGPT\n- Interact privately with your documents using the power of GPT, 100% privately, no data leaks.\nreplit-code-v1-3b\n- A 2.7B Causal Language Model focused on Code Completion.\nstreaming-llm\n- Efficient Streaming Language Models with Attention Sinks.\nviper\n- ViperGPT: Visual Inference via Python Execution for Reasoning\nIDEs\nCodeStory\n—\xa0An IDE with chat, code explanations, auto-generated commits and PR summaries. Forked from VSCodium.\nCursor\n— An IDE with chat, edit, generate and debug features. Forked from VSCodium, so the interface is similar to VS Code. Uses OpenAI.\nMutable\n— Web-based IDE, integrated with a chatbot and GitHub.\nKnowledge Bases\nALCE\n- Enabling Large Language Models to Generate Text with Citations.\nLangchain-Chatchat\n- Local knowledge based LLM (like ChatGLM) QA app with langchain.\nPDFTriage\n- Question Answering over Long, Structured Documents.\nRobby-chatbot\n- AI chatbot 🤖 for chat with CSV, PDF, TXT files 📄 and YTB videos 🎥 | using Langchain🦜 | OpenAI | Streamlit ⚡\nVerba\n- Retrieval Augmented Generation (RAG) chatbot powered by Weaviate.\nembedchain\n- Framework to easily create LLM powered bots over any dataset.\ngpt-runner\n- Conversations with your files! Manage and run your AI presets!\nkhoj\n- An AI personal assistant for your digital brain. Khoj provides you with an easy way to setup the infrastructure to search and chat with your personal knowledge base, be it markdown, org, image, or PDF files.\ntxtai\n- All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows.\nLLMOps\nAGiXT\n- A dynamic AI Automation Platform that seamlessly orchestrates instruction management and complex task execution across diverse AI providers.\nBrowserGPT\n- Command your browser with GPT.\nCarbonate\n— End-to-end testing using natural language. Integrates into your existing test suite (currently Jest, PHPUnit and Python\'s unittest).\nDiffBlue\n- Automatically generated unit tests for Java.\nLLMStack\n- No-code platform to build generative AI apps, chatbots and agents with your data.\nLocalAI\n- 🤖 Self-hosted, community-driven, local OpenAI compatible API. Drop-in replacement for OpenAI running LLMs on consumer-grade hardware. Free Open Source OpenAI alternative. No GPU required. Runs ggml, gguf, GPTQ, onnx, TF compatible models: llama, llama2, gpt4all, rwkv, whisper, vicuna, koala, cerebras, falcon, dolly, starcoder, and many others.\nMeticulous.ai\n- Automatically generated, automatically maintained end-to-end tests: as your app evolves so does your test suite.\nOctoMind\n- Auto-maintenance and generated browser-based end-to-end-tests integrated into Github Actions, Azure DevOps and more.\nTaxyAI browser-extension\n- Automate your browser with GPT-4.\nTraceloop\n- Uses OpenTelemetry tracing data with generative AI to improve system reliability.\nhaystack\n- 🔍 LLM orchestration framework to build customizable, production-ready LLM applications.\nhelicone\n- An open-source observability platform for Language Learning Models (LLMs).\nlangfuse\n- Open source observability and analytics for LLM applications.\npezzo\n- 🕹️ Open-source, developer-first LLMOps platform designed to streamline prompt design, version management, instant delivery, collaboration, troubleshooting, observability and more.\npromptflow\n- Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.\nzeno\n- AI Data Management & Evaluation Platform.\nOpenAI Plugins\nChatWithGit\n— Enables ChatGPT to search GitHub and return links to relevant repositories.\nCode ChatGPT Plugin\n— Open source example of a ChatGPT plugin that pulls context from a directory of files.\nPR Agents\nAutoPR\n- Fix issues with AI-generated pull requests, powered by ChatGPT.\nBitBuilder\n—\xa0A GitHub integration to generate pull requests from issues.\nCode Review GPT\n—\xa0An open source tool for reviewing PRs. Works as GitHub action, Gitlab CLI or locally.\nCodeRabbit\n—\xa0Customizable CI to add summaries and code suggestions to PRs.\nCodeium PR Agent\n—\xa0Open source tool for automated code reviews.\nNova\n- CI bot to add actions such as summaries and tests to new PRs.\nSweep\n—\xa0AI junior dev: GitHub integration to generate, test, and self-review pull requests from issues.\nPrompts Tuning\nAnalogical Prompting\n- A new prompting approach designed to automatically guide the reasoning process of large language models.\nEvoPrompt\n- Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers.\nFooocus\n- Focus on prompting and generating.\nLangGPT\n- LangGPT: Empowering everyone to become a prompt expert!🚀 Structured Prompt,结构化提示词。\nPrompt-Engineering-Guide\n- 🐙 Guides, papers, lecture, notebooks and resources for prompt engineering.\nYiVal\n- 🚀 From Demo to Production.🚀 YiVal is an open source GenAI-Ops framework that allows you to iteratively tune and evaluate your AIGC prompts, model metadata, model params, and retrieval configs all at once with your preferred choices of test dataset generation, evaluation algorithms and improvement strategies.\nguidance\n- A guidance language for controlling large language models.\noutlines\n- Generative Model Programming.\nprompttools\n- Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).\nReasoning and Planning\nAoT\n- Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models.\nCoD\n- From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting.\nCoT\n- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.\nGoT\n- Graph of Thoughts: Solving Elaborate Problems with Large Language Models.\nReAct\n- Synergizing Reasoning and Acting in Language Models.\nReWOO\n- Decoupling Reasoning from Observations for Efficient Augmented Language Models.\nSCoT\n- Structured Chain-of-Thought Prompting for Code Generation.\nScrews\n-A Modular Framework for Reasoning with Revisions.\nSwiftSage\n- A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks.\nToT\n- Deliberate Problem Solving with Large Language Models that Elevates Model Reasoning by atleast 70%.\nllm-reasoners\n- reasoning as planning (RAP), a library for advanced large language model reasoning.\nreflexion\n- Reflexion: Language Agents with Verbal Reinforcement Learning.\nsaycan\n- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.\nself-ask\n- Measuring and Narrowing the Compositionality Gap in Language Models.\nResearch\nArxivGPT\n- A Google Chrome plug-in that helps you quickly understand the content of arXiv papers.\nChatPDF\n- Chat with any PDF using the new ChatGPT API.\nChatPaper\n- Use ChatGPT to summarize the arXiv papers. 全流程加速科研,利用 chatgpt 进行论文全文总结+专业翻译+润色+审稿+审稿回复.\nChemCrow\n- An open source package for the accurate solution of reasoning-intensive chemical tasks.\nOpenBBTerminal\n- OpenBB is committed to build the future of investment research by focusing on an open source infrastructure accessible to everyone, everywhere.\ngpt-researcher\n- GPT based autonomous agent that does online comprehensive research on any given topic.\nqlib\n- An AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions.\nSecurity\nNeMo-Guardrails\n- An open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.\nguardrails\n- An open-source Python package for specifying structure and type, validating and correcting the outputs of large language models (LLMs).\nShell Assistants\nAider\n— CLI assistant and agent that generates changes and commits to repositories. Uses OpenAI.\nAskCommand\n- Web based tool to generate Unix commands from text automatically using AI.\nButterfish\n- CLI tool that embeds ChatGPT in your shell for easy access. Includes simple agentic capabilities.\nMentat\n— CLI assistant and agent that makes changes to repositories.\nShell-AI\n- LangChain powered shell command generate and run CLI.\nSearch\nBloop\n— Natural language search for repositories.\nBuildt\n— Natural language search for repositories.\nMetaphor\n- A search engine designed from scratch using AI.\nPandasAI\n- A Python library that integrates generative artificial intelligence capabilities into pandas, making dataframes conversational.\nPerplexity\n- Unlock the power of knowledge with information discovery and sharing.\nTraining & Deployment\nHCP-Diffusion\n- A universal Stable-Diffusion toolbox.\nLMDeploy\n- A toolkit for compressing, deploying, and serving LLMs.\nMedusa\n- Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads.\nlangchain-production-starter\n- Deploy LangChain Agents and connect them to Telegram.\nllm-applications\n- A comprehensive guide to building RAG-based LLM applications for production.\nludwig\n- Low-code framework for building custom LLMs, neural networks, and other AI models.\nonprem\n- A tool for running on-premises large language models with non-public data.\nprompt2model\n- Generate Deployable Models from Natural Language Instructions.\nvllm\n- A high-throughput and memory-efficient inference and serving engine for LLMs.\nAbout\nA curated list of autonomous agents and developer tools powered by LLM.\nResources\nReadme\nActivity\nStars\n30\nstars\nWatchers\n2\nwatching\nForks\n3\nforks\nReport repository\nReleases\nNo releases published\nPackages\n0\nNo packages published', 'tool_call_id': 'call_Vh9DH7mDWzXzsQoHKaWakHtS', 'name': 'extract_content'}, {'role': 'tool', 'content': "jihoo-kim\n/\nawesome-production-llm\nPublic\nNotifications\nYou must be signed in to change notification settings\nFork\n25\nStar\n322\nA curated list of awesome open-source libraries for production LLM\nLicense\nMIT license\n322\nstars\n25\nforks\nBranches\nTags\nActivity\nStar\nNotifications\nYou must be signed in to change notification settings\njihoo-kim/awesome-production-llm\nThis commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.\nmain\nBranches\nTags\nGo to file\nCode\nFolders and files\nName\nName\nLast commit message\nLast commit date\nLatest commit\nHistory\n4 Commits\nLICENSE\nLICENSE\nREADME.md\nREADME.md\nView all files\nRepository files navigation\nAwesome-Production-LLM\nThis repository contains a curated list of awesome open-source libraries for production large language models.\nNewly updated\n[2024.09.03] A new category\n🎓LLM Courses / Education\nhas been added.\n[2024.08.01] A new category\n🍳LLM Cookbook / Examples\nhas been added.\nQuick links\n📚LLM Data Preprocessing\n🤖LLM Training / Finetuning\n📊LLM Evaluation / Benchmark\n🚀LLM Serving / Inference\n🛠️LLM Application / RAG\n🧐LLM Testing / Monitoring\n🛡️LLM Guardrails / Security\n🍳LLM Cookbook / Examples\n🎓LLM Courses / Education\nLLM Data Preprocessing\ndata-juicer\n(\nModelScope\n)\nA one-stop data processing system to make data higher-quality, juicier, and more digestible for (multimodal) LLMs!\ndatatrove\n(\nHuggingFace\n)\nFreeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.\ndolma\n(\nAllenAI\n)\nData and tools for generating and inspecting OLMo pre-training data.\ndataverse\n(\nUpstage\n)\nThe Universe of Data. All about data, data science, and data engineering\nNeMo-Curator\n(\nNVIDIA\n)\nScalable toolkit for data curation\ndps\n(\nEleutherAI\n)\nData processing system for polyglot\nLLM Training / Finetuning\nnanoGPT\n(\nkarpathy\n)\nThe simplest, fastest repository for training/finetuning medium-sized GPTs.\nLLaMA-Factory\nA WebUI for Efficient Fine-Tuning of 100+ LLMs (ACL 2024)\npeft\n(\nHuggingFace\n)\nPEFT: State-of-the-art Parameter-Efficient Fine-Tuning.\nllama-recipes\n(\nMeta\n)\nScripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs.\nMegatron-LM\n(\nNVIDIA\n)\nOngoing research training transformer models at scale\nlitgpt\n(\nLightningAI\n)\n20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.\ntrl\n(\nHuggingFace\n)\nTrain transformer language models with reinforcement learning.\nLMFlow\n(\nOptimalScale\n)\nAn Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.\ngpt-neox\n(\nEleutherAI\n)\nAn implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries\ntorchtune\n(\nPyTorch\n)\nA Native-PyTorch Library for LLM Fine-tuning\nxtuner\n(\nInternLM\n)\nAn efficient, flexible and full-featured toolkit for fine-tuning LLM (InternLM2, Llama3, Phi3, Qwen, Mistral, ...)\nnanotron\n(\nHuggingFace\n)\nMinimalistic large language model 3D-parallelism training\nLLM Evaluation / Benchmark\nevals\n(\nOpenAI\n)\nEvals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.\nlm-evaluation-harness\n(\nEleutherAI\n)\nA framework for few-shot evaluation of language models.\nopencompass\n(\nOpenCompass\n)\n- OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.\ndeepeval\n(\nConfidentAI\n)\nThe LLM Evaluation Framework\nlighteval\n(\nHuggingFace\n)\nLightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.\nevalverse\n(\nUpstage\n)\nThe Universe of Evaluation. All about the evaluation for LLMs.\nLLM Serving / Inference\nollama\n(\nOllama\n)\nGet up and running with Llama 3.1, Mistral, Gemma 2, and other large language models.\ngpt4all\n(\nNomicAI\n)\nGPT4All: Chat with Local LLMs on Any Device\nllama.cpp\nLLM inference in C/C++\nFastChat\n(\nLMSYS\n)\nAn open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.\nvllm\nA high-throughput and memory-efficient inference and serving engine for LLMs\nguidance\n(\nguidance-ai\n)\nA guidance language for controlling large language models.\nLiteLLM\n(\nBerriAI\n)\nCall all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate, Groq (100+ LLMs)\nOpenLLM\n(\nBentoML\n)\nRun any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.\ntext-generation-inference\n(\nHuggingFace\n)\nLarge Language Model Text Generation Inference\nTensorRT-LLM\n(\nNVIDIA\n)\nTensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.\nLMDeploy\n(\nInternLM\n)\nLMDeploy is a toolkit for compressing, deploying, and serving LLMs.\nRouteLLM\n(\nLMSYS\n)\nA framework for serving and evaluating LLM routers - save LLM costs without compromising quality!\nLLM Application / RAG\nAutoGPT\nAutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.\nlangchain\n(\nLangChain\n)\nBuild context-aware reasoning applications\nMetaGPT\nThe Multi-Agent Framework: First AI Software Company, Towards Natural Language Programming\ndify\n(\nLangGenius\n)\nDify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.\nllama_index\n(\nLlamaIndex\n)\nLlamaIndex is a data framework for your LLM applications\nFlowise\n(\nFlowiseAI\n)\nDrag & drop UI to build your customized LLM flow\nmem0\n(\nMem0\n)\nThe memory layer for Personalized AI\nhaystack\n(\nDeepset\n)\nLLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data.\nGraphRAG\n(\nMicrosoft\n)\nA modular graph-based Retrieval-Augmented Generation (RAG) system\nRAGFlow\n(\nInfiniFlow\n)\nRAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.\nllmware\n(\nLLMware.ai\n)\nUnified framework for building enterprise RAG pipelines with small, specialized models\nllama-agentic-system\n(\nMeta\n)\nAgentic components of the Llama Stack APIs\nLLM Testing / Monitoring\npromptflow\n(\nMicrosoft\n)\nBuild high-quality LLM apps - from prototyping, testing to production deployment and monitoring.\nlangfuse\n(\nLangfuse\n)\nOpen source LLM engineering platform: Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more.\nevidently\n(\nEvidentlyAI\n)\nEvidently is \u200b\u200ban open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.\ngiskard\n(\nGiskard\n)\nOpen-Source Evaluation & Testing for LLMs and ML models\npromptfoo\n(\npromptfoo\n)\nTest your prompts, agents, and RAGs. Redteaming, pentesting, vulnerability scanning for LLMs. Improve your app's quality and catch problems. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.\nphoenix\n(\nArizeAI\n)\nAI Observability & Evaluation\nagenta\n(\nAgenta.ai\n)\nThe all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.\nLLM Guardrails / Security\nNeMo-Guardrails\n(\nNVIDIA\n)\nNeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.\nguardrails\n(\nGuardrailsAI\n)\nAdding guardrails to large language models.\nPurpleLlama\n(\nMeta\n)\nSet of tools to assess and improve LLM security.\nllm-guard\n(\nProtectAI\n)\nThe Security Toolkit for LLM Interactions\nLLM Cookbook / Examples\nopenai-cookbook\n(\nOpenAI\n)\nExamples and guides for using the OpenAI API\ngemini-cookbook\n(\nGoogle\n)\nExamples and guides for using the Gemini API.\nanthropic-cookbook\n(\nAnthropic\n)\nA collection of notebooks/recipes showcasing some fun and effective ways of using Claude.\namazon-bedrock-workshop\n(\nAWS\n)\nThis is a workshop designed for Amazon Bedrock a foundational model service.\nPhi-3CookBook\n(\nMicrosoft\n)\nThis is a Phi-3 book for getting started with Phi-3. Phi-3, a family of open AI models developed by Microsoft.\nmistral-cookbook\n(\nMistral\n)\nThe Mistral Cookbook features examples contributed by Mistralers and our community, as well as our partners.\namazon-bedrock-samples\n(\nAWS\n)\nThis repository contains examples for customers to get started using the Amazon Bedrock Service. This contains examples for all available foundational models\ncohere-notebooks\n(\nCohere\n)\nCode examples and jupyter notebooks for the Cohere Platform\ngemma-cookbook\n(\nGoogle\n)\nA collection of guides and examples for the Gemma open models from Google.\nupstage-cookbook\n(\nUpstage\n)\nUpstage api examples and guides\nLLM Courses / Education\ngenerative-ai-for-beginners\n(\nMicrosoft\n)\n18 Lessons, Get Started Building with Generative AI\nllm-course\nCourse to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.\nLLMs-from-scratch\nImplementing a ChatGPT-like LLM in PyTorch from scratch, step by step\nhands-on-llms\nLearn about LLMs, LLMOps, and vector DBs for free by designing, training, and deploying a real-time financial advisor LLM system ~ source code + video & reading materials\nllm-zoomcamp\n(\nDataTalksClub\n)\nLLM Zoomcamp - a free online course about building a Q&A system\nllm-twin-course\n(\nDecodingML\n)\nLearn for free how to build an end-to-end production-ready LLM & RAG system using LLMOps best practices: ~ source code + 12 hands-on lessons\nAcknowledgements\nThis project is inspired by\nAwesome Production Machine Learning\n.\nAbout\nA curated list of awesome open-source libraries for production LLM\nResources\nReadme\nLicense\nMIT license\nActivity\nStars\n322\nstars\nWatchers\n4\nwatching\nForks\n25\nforks\nReport repository\nReleases\nNo releases published\nPackages\n0\nNo packages published", 'tool_call_id': 'call_VXxtxhySFYvaXlxhDqgLVzdu', 'name': 'extract_content'}, {'role': 'tool', 'content': 'Top 10 Open-Source LLM Frameworks 2024\npublished on 09 May 2024\nLarge Language Models\n(LLMs) have revolutionized how machines understand and generate human-like text. Open-source LLMs make AI technology accessible, enabling developers and researchers to innovate. Here are the top 10 open-source LLM frameworks available in 2024:\nLLaMA 2\n- Powerful LLM from\nMeta\nwith up to 70B parameters, multilingual support, customizability, and an active community.\nGPT-NeoX-20B\n- 20B parameter autoregressive LLM from\nEleutherAI\n, open-source with strong performance.\nBLOOM\n- 176B parameter LLM from\nBigScience\n, supports 46 languages, open-source.\nOPT-175B\n- 175B parameter LLM comparable to GPT-3 but with a smaller carbon footprint.\nCodeGen\n- LLM from\nSalesforce\nfor streamlining software development.\nBERT\n- Bidirectional Encoder from Google for understanding context and generating language.\nT5\n- Text-to-text transformer from Google for various NLP tasks.\nFalcon-40B\n- 40B parameter LLM from TII, supports multiple languages, open-source.\nVicuna 33B\n- 33B parameter open-source chatbot with competitive performance.\nGPT-J\n- Massive 6T parameter LLM from EleutherAI for high-quality text generation.\nOpen-source LLMs offer cost savings, flexibility, community support, transparency, and foster innovation. However, they face challenges like resource requirements, potential biases, framework selection, security, and integration.\nQuick Comparison\nFramework\nParameters\nMultilingual\nOpen Source\nKey Features\nLLaMA 2\nUp to 70B\nYes\nYes\nSpeed, customizability, community\nGPT-NeoX-20B\n20B\nNo\nYes\nStrong few-shot reasoner\nBLOOM\n176B\nYes (46 languages)\nYes\nSupports programming languages\nOPT-175B\n175B\nNo\nYes\nComparable to GPT-3, lower carbon footprint\nCodeGen\n-\nNo\nYes\nStreamlines software development\nBERT\nFlexible\nNo\nYes\nUnderstands context, generates language\nT5\nFlexible\nNo\nYes\nUnified framework for NLP tasks\nFalcon-40B\nUp to 40B\nYes\nYes\nEfficient inference, scalable\nVicuna 33B\n33B\nNo\nYes\nCompetitive chatbot performance\nGPT-J\n6T\nNo\nYes\nMassive model for high-quality text\nKey Features of Top LLM Frameworks\nWhen choosing an open-source LLM framework, several key features set the top models apart. These features include:\nModel Size and Parameters\nFeature\nDescription\nModel Size\nThe number of parameters in an LLM framework affects its performance and capabilities. Larger models can process and generate more complex text, but require more resources and memory.\nMultilingual Support\nFeature\nDescription\nMultilingual Support\nMany LLM frameworks support multiple languages, making them suitable for a broader range of use cases, such as language translation, sentiment analysis, or text summarization.\nCustomizability\nFeature\nDescription\nCustomizability\nTop LLM frameworks allow developers to fine-tune models for specific tasks or domains, improving performance and accuracy.\nCommunity Involvement\nFeature\nDescription\nCommunity Involvement\nAn active community contributes to the framework\'s development, provides support, and shares knowledge, making it easier for new users to adopt and integrate the framework.\nPerformance Efficiency\nFeature\nDescription\nPerformance Efficiency\nTop frameworks optimize their models for performance, ensuring they can handle large datasets and generate text quickly.\nWhen evaluating open-source LLM frameworks, consider these key features to choose the best model for your specific use case and requirements.\n1. LLaMA 2\nLLaMA 2 is a powerful open-source\nlarge language model\n(LLM) developed by Meta. It has gained significant attention in the AI research community for its speed and capabilities.\nModel Size and Parameters\nLLaMA 2 offers three size versions with 7 billion, 13 billion, and 70 billion parameters. The larger models support direct dialogue applications and demonstrate superior capabilities in various dimensions.\nModel Size\nParameters\nSmall\n7 billion\nMedium\n13 billion\nLarge\n70 billion\nMultilingual Support\nLLaMA 2 supports multiple languages, making it suitable for various use cases such as language translation, sentiment analysis, or text summarization.\nCustomizability\nLLaMA 2 allows developers to fine-tune models for specific tasks or domains, improving performance and accuracy.\nCommunity and Support\nLLaMA 2 has an active community that contributes to its ongoing development and optimization. The model is available on\nGitHub\nas free, open-source Python code, and Meta provides official support and resources for developers.\nPerformance and Efficiency\nLLaMA 2 optimizes its models for performance, ensuring they can handle large datasets and generate text quickly.\nOverall, LLaMA 2 is a powerful and flexible open-source LLM framework that offers a range of benefits for developers and researchers. Its large model size, multilingual support, customizability, and active community make it an attractive choice for various NLP applications.\n2. GPT-NeoX-20B\nGPT-NeoX-20B is a powerful\nopen-source large language model\n(LLM) developed by EleutherAI. It is a 20 billion parameter autoregressive language model trained on the Pile, with freely and openly available weights through a permissive license.\nModel Size and Parameters\nGPT-NeoX-20B has a large model size, with 20 billion parameters. This enables it to perform well on various language-understanding, mathematics, and knowledge-based tasks.\nModel Size\nParameters\nGPT-NeoX-20B\n20 billion\nPerformance and Efficiency\nGPT-NeoX-20B has been evaluated on various natural language tasks, including zero-shot performance. The results show that it is a strong few-shot reasoner and gains significant performance when evaluated five-shot.\nCommunity and Support\nGPT-NeoX-20B is open-sourced, with its training and evaluation code, as well as the model weights, available on GitHub. This allows developers to fine-tune the model for specific tasks or domains, improving performance and accuracy. The active community and open-source nature of the model ensure ongoing development and optimization.\nOverall, GPT-NeoX-20B is a powerful and flexible open-source LLM framework that offers a range of benefits for developers and researchers. Its large model size, performance, and open-source nature make it an attractive choice for various NLP applications.\n3. BLOOM\nBLOOM is a large open-source language model developed by BigScience, a global collaboration of over 1,000 AI researchers. It has a decoder-only architecture derived from\nMegatron-LM GPT2\n.\nModel Size and Parameters\nBLOOM has a total of\n176 billion parameters\n,\n70 layers\n, and\n112 attention heads\n.\nMultilingual Support\nBLOOM can generate coherent text in\n46 natural languages\nand\n13 programming languages\n, making it suitable for applications that require multilingual support.\nCommunity and Support\nBLOOM is open-sourced, with its training and evaluation code, as well as the model weights, available on GitHub. This allows developers to fine-tune the model for specific tasks or domains, improving performance and accuracy.\nPerformance and Efficiency\nBLOOM has been evaluated on various natural language tasks, including zero-shot performance. The results show that it is a strong few-shot reasoner and gains significant performance when evaluated five-shot.\nOverall, BLOOM is a powerful and flexible open-source LLM framework that offers a range of benefits for developers and researchers. Its large model size, multilingual support, and open-source nature make it an attractive choice for various NLP applications.\n4. OPT-175B\nOPT-175B is a large language model with\n175 billion parameters\n, making it a powerful tool for various natural language processing tasks.\nModel Size and Parameters\nModel Size\nParameters\nOPT-175B\n175 billion\nCommunity and Support\nOPT-175B is open-sourced, with its code and trained model weights available on GitHub. This allows developers to fine-tune the model for specific tasks or domains, improving performance and accuracy. The model is released under a non-commercial license and is intended for use by researchers "affiliated with organizations in government, civil society, and academia" as well as industry researchers.\nPerformance and Efficiency\nOPT-175B\'s performance is comparable to GPT-3, while requiring only 1/7th of GPT-3\'s training carbon footprint. The model has been evaluated on various natural language tasks, including:\nQuestion answering\nWriting articles\nSolving math problems\nIn some tasks, such as the WIC task, OPT-175B outperformed GPT models. However, it underperformed in tasks like ARC Challenge and MultiRC. Overall, OPT-175B is a strong open-source LLM framework that offers a range of benefits for developers and researchers.\n5. CodeGen\nCodeGen is an open-source large language model developed by Salesforce AI Research. It has gained attention in the developer community for its potential to streamline software development processes and boost productivity.\nModel Size and Parameters\nCodeGen\'s model size and parameters are not explicitly stated. However, its capabilities and performance are noteworthy.\nCommunity and Support\nCodeGen is open-sourced, with its code and trained model weights available on GitHub. This allows developers to fine-tune the model for specific tasks or domains, improving performance and accuracy. The open-source nature enables a community-driven approach to development and support.\nPerformance and Efficiency\nCodeGen\'s performance is promising, with the potential to save developers time and focus on more complex tasks. While specific performance metrics are not available, CodeGen\'s capabilities and community support make it a strong open-source LLM framework.\n6. BERT\nBERT (Bidirectional Encoder Representations from Transformers) is a powerful open-source language model developed by Google in 2018. It has significantly impacted the field of natural language processing (NLP) with its ability to understand context and generate human-like language.\nModel Size and Parameters\nBERT\'s model size is flexible and can be fine-tuned for specific tasks and domains. The original BERT model was trained on a large plain text corpus, using a bidirectional method to analyze language and understand context.\nModel Size\nDescription\nFlexible\nCan be fine-tuned for specific tasks and domains\nCommunity and Support\nBERT is open-sourced, with its code and trained model weights available on GitHub. This has led to a large community of developers and researchers contributing to its development and fine-tuning.\nPerformance and Efficiency\nBERT\'s performance is notable, with its ability to generate language based on context. It has been shown to be accurate in detecting sentiment and classifying language based on the sentiment expressed.\nOverall, BERT is a powerful and widely adopted open-source LLM framework that has significantly impacted the field of NLP. Its ability to understand context and generate human-like language makes it a valuable tool for a wide range of applications.\nsbb-itb-f3e41df\n7. T5\nT5 is a text-to-text transformer model developed by Google AI. It uses a unified framework to tackle various natural language processing (NLP) tasks.\nModel Size and Parameters\nT5\'s model size is flexible, and it can be fine-tuned for specific tasks and domains. Its architecture is based on the transformer architecture, blending BERT\'s and GPT\'s pre-training approaches.\nModel Size\nDescription\nFlexible\nCan be fine-tuned for specific tasks and domains\nCommunity and Support\nT5 is an open-source model, with its code and trained model weights available on GitHub. This has led to a large community of developers and researchers contributing to its development and fine-tuning.\nPerformance and Efficiency\nT5\'s performance is notable, with its ability to generate language based on context. It has been shown to be accurate in various NLP tasks, including:\nMachine translation\nAutomated summarization\nCode-related tasks\nT5\'s efficiency is also impressive, with its ability to optimize computation resources.\nOverall, T5 is a powerful and versatile open-source LLM framework that has significantly impacted the field of NLP. Its ability to understand context and generate human-like language makes it a valuable tool for a wide range of applications.\n8. Falcon-40B\nFalcon-40B is a large open-source language model developed by the Technology Innovation Institute (TII) in Abu Dhabi. With 40 billion parameters, it is one of the largest language models ever created, making it a powerful tool for various natural language processing (NLP) tasks.\nModel Size and Parameters\nFalcon-40B has three variants: 1B, 7B, and 40B parameters. Its extensive training on a large dataset of text and code enables it to possess a wide range of knowledge and capabilities.\nModel Size\nParameters\nSmall\n1 billion\nMedium\n7 billion\nLarge\n40 billion\nMultilingual Support\nFalcon-40B supports multiple languages, including English, German, Spanish, French, Italian, Portuguese, Polish, Dutch, Romanian, Czech, and Swedish. This makes it a versatile foundation model that can be used for applications such as translation, question answering, and summarizing information.\nCommunity and Support\nFalcon-40B is an open-source model, which means it is freely available to anyone who wants to use it. Its open-source nature has led to a growing community of users who contribute to its development and fine-tuning.\nPerformance and Efficiency\nFalcon-40B has showcased its exceptional performance on various benchmarks, including the Hugging Face OpenLLM Leaderboard. Its performance is notable, with its ability to generate human-like language and understand context. It has been shown to be accurate in various NLP tasks, including machine translation, automated summarization, and text generation. Additionally, Falcon-40B\'s architecture is optimized for efficient inference, resulting in higher inference speed and scalability.\n9. Vicuna 33B\nVicuna 33B is an open-source chatbot that has demonstrated competitive performance compared to other open-source models like Stanford Alpaca. It is an enhanced version of the Vicuna-13B model, with a larger parameter size of 33 billion.\nModel Size and Parameters\nModel Size\nParameters\nVicuna 33B\n33 billion\nCommunity and Support\nVicuna 33B is an open-source model, which means it is freely available to anyone who wants to use it. Its open-source nature has led to a growing community of users who contribute to its development and fine-tuning.\nPerformance and Efficiency\nVicuna 33B has showcased its exceptional performance on various benchmarks. Its architecture is optimized for efficient inference, resulting in higher inference speed and scalability. The model\'s performance has been evaluated by creating a set of 80 diverse questions and utilizing GPT-4 to judge the model outputs. The results have shown that Vicuna 33B provides high-quality responses, making it a powerful tool for various natural language processing (NLP) tasks.\n10.\nEleutherAI\n\'s GPT-J\nEleutherAI\'s GPT-J is a massive language model with 6 trillion parameters, making it one of the largest publicly available language models. This size enables it to generate high-quality text that is often indistinguishable from human writing.\nModel Size and Parameters\nModel Size\nParameters\nGPT-J\n6 trillion\nCommunity and Support\nGPT-J is an open-source model, which means it is freely available to anyone who wants to use it. The model\'s GitHub repository provides access to its code, pre-trained weight files, and a demo website. This open-source nature has led to a growing community of users who contribute to its development and fine-tuning.\nPerformance and Efficiency\nGPT-J has demonstrated impressive performance on various benchmarks, showcasing its ability to generate coherent and contextually appropriate text. Its architecture is optimized for efficient inference, resulting in higher inference speed and scalability. The model\'s performance has been evaluated on various down-streaming tasks, and it has achieved state-of-the-art results in many cases.\nWhile GPT-J\'s massive size and parameter count require significant computational resources, making it less accessible to individual developers, its potential for advanced applications in language generation and understanding cannot be underestimated.\nBenefits of Open-Source LLMs\nOpen-source LLM frameworks offer several advantages that make them an attractive choice for developers, researchers, and organizations.\nCost Savings\nOne of the primary benefits is the elimination of licensing fees, reducing the financial burden associated with proprietary models. This cost savings enables organizations to allocate resources more efficiently, promoting innovation and growth.\nFlexibility and Customization\nOpen-source LLMs provide the freedom to customize and tailor models to specific needs, allowing for greater control over the development process. This flexibility is particularly valuable for organizations with unique requirements or those operating in niche domains.\nCommunity Support and Collaboration\nThe collaborative nature of open-source projects fosters a community-driven approach, where developers can share knowledge, expertise, and resources. This collective effort leads to faster development, improved model performance, and reduced errors.\nTransparency and Accountability\nOpen-source LLMs offer transparency and accountability, as developers can inspect, audit, and validate the models, ensuring that they are fair, unbiased, and secure. This transparency is essential for building trust in AI systems, particularly in high-stakes applications.\nInnovation and Advancement\nBy providing access to the source code and model architecture, open-source LLMs enable developers to experiment, modify, and improve upon existing models. This leads to the creation of new models, techniques, and applications, driving the advancement of AI technology.\nIn summary, open-source LLMs offer a range of benefits that make them an attractive choice for developers, researchers, and organizations. By leveraging these frameworks, developers can reduce costs, increase flexibility, and promote innovation and collaboration.\nChallenges and Considerations\nWhen working with open-source LLM frameworks, developers and organizations may encounter several challenges and considerations.\nComputational Resource Requirements\nResource Intensive\n: Training and deploying LLMs require significant computational power, memory, and storage. This can be costly and resource-intensive.\nPotential Biases in Models\nFairness and Transparency\n: LLMs can perpetuate biases present in the training data, leading to unfair or discriminatory outcomes. Developers must ensure their models are trained on diverse, representative data and implement measures to detect and mitigate biases.\nSelecting the Right Framework\nChoosing the Best Fit\n: With numerous open-source LLM frameworks available, selecting the right framework for a specific use case can be challenging. Developers must evaluate the strengths and weaknesses of each framework, considering factors such as model performance, customization requirements, and community support.\nSecurity and Privacy Concerns\nProtecting Sensitive Data\n: LLMs can pose security and privacy risks if not implemented correctly. Developers must ensure their models are secure, and sensitive data is protected from unauthorized access or breaches.\nCustomization and Integration\nSeamless Integration\n: Customizing and integrating LLMs with existing systems and infrastructure can be a complex task. Developers must consider the compatibility of their chosen framework with their existing technology stack and ensure seamless integration to achieve desired outcomes.\nBy understanding these challenges and considerations, developers and organizations can better navigate the complexities of working with open-source LLM frameworks and unlock the full potential of these powerful technologies.\nConclusion\nThe top 10 open-source LLM frameworks discussed in this article have the potential to transform the field of natural language processing. By using these frameworks, developers and organizations can create advanced NLP applications that were previously inaccessible due to proprietary constraints.\nKey Takeaways\nOpen-source LLM frameworks foster collaboration, innovation, and transparency.\nThey drive AI innovation forward by empowering developers with the tools they need to create cutting-edge NLP applications.\nAs we move forward in this exciting era of AI development, it\'s essential to recognize the importance of open-source LLM frameworks in unlocking new possibilities and pushing the boundaries of what\'s possible with language models.\nRelated posts\nUnsupervised Pre-training vs. Supervised Fine-tuning for LLMs\n5 Types of Autoencoding Language Models\n10 Hyperparameter Tuning Tips for LLM Fine-Tuning\nGerman BERT: Top Language Model for German NLP\nRead more\nThe Future of Search: How AI Search Engines are Changing the Internet\nThe Future of Voice to Text: How AI is Revolutionizing Communication\nExploring ChatGPT Alternatives for Enhanced AI Writing and Chatbots', 'tool_call_id': 'call_NUXKesTLiBqnO48zOZYVgfF7', 'name': 'extract_content'}] As a software engineer (SWE) looking for libraries to aid in the development of large language models (LLMs), there are several noteworthy tools and libraries you can explore. Here’s a curated overview of some popular options: ### 1. **Awesome LLMOps** - **GitHub Repository:** [Awesome LLMOps](https://github.com/tensorchord/awesome-llmops) - This repository features a curated list of tools specifically for LLM operations (LLMOps), including: - **Training & Fine-Tuning Frameworks:** Tools for efficient model training. - **Data Management:** Libraries focused on data curation and preprocessing. - **Model Serving:** Options for deploying models into production. - **Observability Tools:** Solutions for monitoring and debugging LLM performance. ### 2. **LLM Agents DevTools** - **GitHub Repository:** [LLM Agents DevTools](https://github.com/M1n9X/llm_agents_devtools) - This collection includes a range of developer tools and autonomous agents that leverage LLMs. It covers: - **Coding Assistants:** AI tools that support software development. - **Agent Frameworks:** Platforms for building and managing autonomous agents. ### 3. **Awesome Production LLM** - **GitHub Repository:** [Awesome Production LLM](https://github.com/jihoo-kim/awesome-production-llm) - A resource for developers looking for libraries that facilitate the deployment of LLMs in production. It includes: - **Preprocessing Tools:** Libraries for preparing data for LLMs. - **Evaluation Protocols:** Tools to benchmark model performance. - **Serving & Inference Frameworks:** Libraries to enable efficient model inference and serving. ### 4. **Top Open-Source LLM Frameworks for 2024** - **Source:** [LLM Frameworks Article](https://llmmodels.org/blog/top-10-open-source-llm-frameworks-2024/) - This article summarizes the top open-source LLM frameworks, highlighting key features such as model size, customizability, and community support. Some notable frameworks include: - **LLaMA 2** - **GPT-NeoX** - **BLOOM** - **OPT-175B** - **CodeGen** - **T5** - Each framework has its strengths, making it important to choose one that fits your specific needs. ### Considerations for Choosing a Library When selecting a library or framework for LLM development, consider factors such as: - **Community Support:** A strong community can provide help and shared resources. - **Documentation:** Clear documentation can ease the learning curve. - **Customizability:** The ability to adapt a library to fit specific project requirements is crucial. - **Performance Metrics:** Frameworks with benchmarks can help assess their suitability for your use cases. These resources should provide a solid foundation for your LLM development projects, enabling you to leverage state-of-the-art capabilities in natural language processing and generation.
Search Queries:
- LLM development tool libraries
- best libraries for LLM development
- software engineering tools for LLM
- open source LLM libraries for developers
- programming libraries for large language models
By prompting the LLM to generate multiple queries, the LLM has access to a wide range of relevant information, including both open-source and commercial products, which it would have a significantly lower chance of doing with a single query.
await WebAssistantBaseWithStream().run()
(Assistant): [] using _web_search tool with args: {'queries': ['Mirascope library', 'Mirascope Python library', 'Mirascope programming library', 'what is Mirascope library']} [{'role': 'user', 'content': 'What is Mirascope library?'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"queries":["Mirascope library","Mirascope Python library","Mirascope programming library","what is Mirascope library"]}', 'name': '_web_search'}, 'id': 'call_2olv5hg3ZzMFXMwwfIKvrFno'}]}, {'role': 'tool', 'content': 'https://github.com/mirascope/mirascope\n\nhttps://www.mirascope.io/\n\nhttps://pypi.org/project/mirascope/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/learn/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/WHY/\n\nhttps://github.com/mirascope/mirascope', 'tool_call_id': 'call_2olv5hg3ZzMFXMwwfIKvrFno', 'name': '_web_search'}] using extract_content tool with args: {'url': 'https://www.mirascope.io/'} using extract_content tool with args: {'url': 'https://docs.mirascope.io/latest/learn/'} [{'role': 'user', 'content': 'What is Mirascope library?'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"queries":["Mirascope library","Mirascope Python library","Mirascope programming library","what is Mirascope library"]}', 'name': '_web_search'}, 'id': 'call_2olv5hg3ZzMFXMwwfIKvrFno'}]}, {'role': 'tool', 'content': 'https://github.com/mirascope/mirascope\n\nhttps://www.mirascope.io/\n\nhttps://pypi.org/project/mirascope/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/learn/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/WHY/\n\nhttps://github.com/mirascope/mirascope', 'tool_call_id': 'call_2olv5hg3ZzMFXMwwfIKvrFno', 'name': '_web_search'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"url": "https://www.mirascope.io/"}', 'name': 'extract_content'}, 'id': 'call_SugLNjl2qFQu6nDhdZ1eNzka'}, {'type': 'function', 'function': {'arguments': '{"url": "https://docs.mirascope.io/latest/learn/"}', 'name': 'extract_content'}, 'id': 'call_hDmkCB8DRPyvsZI1X82Wsyhh'}]}, {'role': 'tool', 'content': 'Join our beta list!\nGet updates and early access to try out new features as a beta tester.\nThank you! Your submission has been received!\nOops! Something went wrong while submitting the form.\nCore Components\nCalls\nStreams\nTools\nStreaming Tools\nStructured Outputs\nStructured Streaming\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\[email protected](\n"gpt-4o-mini"\n)\n4\n@prompt_template(\n"Recommend a {genre} book"\n)\n5\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n6\n7\nresponse = recommend_book(\n"fantasy"\n)\n8\nprint\n(response)\n9\n# > Sure! I would recommend The Name of the Wind by...\n1\[email protected](\n"gpt-4o-mini"\n, stream=\nTrue\n)\n2\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n3\n"""Recommend a {genre} book."""\n4\n5\nstream = recommend_book(\n"fantasy"\n)\n6\nfor\nchunk, _\nin\nstream:\n7\nprint\n(chunk, end=\n""\n, flush=\nTrue\n)\n8\n# > Sure! I would recommend...\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\ndef\nformat_book\n(\ntitle:\nstr\n, author:\nstr\n):\n4\nreturn\nf"\n{title}\nby\n{author}\n"\n5\n6\[email protected](\n7\n"gpt-4o-mini"\n,\n8\ntools=[format_book],\n9\ntool_choice=\n"required"\n,\n10\n)\n11\n@prompt_template(\n"Recommend a {genre} book"\n)\n12\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n13\n14\nresponse = recommend_book(\n"fantasy"\n)\n15\nif\ntool := response.tool\n16\nprint\n(tool.call())\n17\n# > The Name of the Wind by Patrick Rothfuss\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\[email protected](\n4\n"gpt-4o-mini"\n,\n5\nstream=\nTrue\n,\n6\ntools=[format_book],\n7\ntool_choice=\n"required"\n8\n)\n9\n@prompt_template(\n"Recommend two (2) {genre} books"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n11\n12\nstream = recommend_book(\n"fantasy"\n)\n13\nfor\nchunk, tool\nin\nstream:\n14\nif\ntool:\n15\nprint\n(tool.call())\n16\nelse\n:\n17\nprint\n(chunk, end=\n""\n, flush=\nTrue\n)\n18\n# > The Name of the Wind by Patrick Rothfuss\n19\n# > Mistborn: The Final Empire by Brandon Sanderson\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\nfrom\npydantic\nimport\nBaseModel\n3\n4\nclass\nBook\n(\nBaseModel\n):\n5\ntitle:\nstr\n6\nauthor:\nstr\n7\n8\[email protected](\n"gpt-4o-mini"\n, response_model=Book\n)\n9\n@prompt_template(\n"Recommend a {genre} book"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n11\n12\nbook = recommend_book(\n"fantasy"\n)\n13\nassert\nisinstance\n(book, Book)\n14\nprint\n(book)\n15\n# > title=\'The Name of the Wind\' author=\'Patrick Rothfuss\'\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\nfrom\npydantic\nimport\nBaseModel\n3\n4\nclass\nBook\n(\nBaseModel\n):\n5\ntitle:\nstr\n6\nauthor:\nstr\n7\n8\[email protected](\n"gpt-4o-mini"\n, stream=\nTrue\n, response_model=Book\n)\n9\n@prompt_template(\n"Recommend a {genre} book"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n11\n12\nbook_stream = recommend_book(\n"fantasy"\n)\n13\nfor\npartial_book\nin\nbook_stream:\n14\nprint\n(partial_book)\n15\n# > title=None author=None\n16\n# > title=\'The Name\' author=None\n17\n# > title=\'The Name of the Wind\' author=None\n18\n# > title=\'The Name of the Wind\' author=\'Patrick\'\n19\n# > title=\'The Name of the Wind\' author=\'Patrick Rothfuss\'\n...and more!\nExamples\nYou\'re in control\nSimple and transparent abstractions ensure you\'re in the driver\'s seat. Abstractions must be useful without getting in your way and blinding your view.\nType hints you\'ll love\nWe’ve taken care of all of the annoying Python typing so that you can have proper type hints in as simple an interface as possible.\nBuild\nyour\nway\nSimple, clean building blocks enable fast and reliable development without forcing you to do things our way. While we have recommendations and best practices, you\'re still always able to do things\nyour\nway to best suit\nyour\nneeds.\nTrusted by engineers building the next generation of AI-native applications\nThe Pydantic inspired LLM framework the space has been missing. Simple, modular, extensible...helps where you need it, stays out of your way when you don\'t.\nVince Trost\nCo-Founder / Plastic Labs\nMirascope\'s simplicity made it the natural next step from a provider\'s API (OpenAI) — all without fighting unnecessary complexity of other tools like LangChain. We have all the bells and whistles necessary for production while maintaining ease of use and easy onboarding of new team members.\nJake Duth\nCo-Founder &\xa0CTO / Reddy\nAs the author of Mirascope, I\'m certainly biased, but there\'s a reason I\'ve worked day and night on this package — it solves the problems I have with existing developer tools. Mirascope is simple and easy to use, and honestly building with Mirascope is the most fun I\'ve had coding in my life.\nWilliam Bakst\nFounder / Mirascope\nStart Building With Mirascope\nMirascope can help improve your application development experience no matter how simple or complex.\nIt will feel like writing the Python you already know.\nRead the Docs\nGitHub | ⭐ ${starCount}\nJoin the Community', 'tool_call_id': 'call_SugLNjl2qFQu6nDhdZ1eNzka', 'name': 'extract_content'}, {'role': 'tool', 'content': "Learn Mirascope\n¶\nThis section is designed to help you master Mirascope, a toolkit for building AI-powered applications with Large Language Models (LLMs).\nMirascope is a powerful, flexible, and user-friendly library that simplifies the process of working with LLMs. Whether you're building chatbots, content generators, or complex AI-driven agent systems, Mirascope provides the tools you need to streamline your development process and create powerful, robust applications.\nOur documentation is tailored for developers who have at least some experience with Python and LLMs. Whether you're coming from other development tool libraries or have worked directly with provider SDKs and APIs, Mirascope offers a familiar but enhanced experience.\nKey Features and Benefits\n¶\nType Safety and Superior Editor Support\n: We've prioritized proper type hints and type safety, ensuring you get world-class editor support. This means fewer errors, better autocomplete, and a smoother development experience in as simple and seamless an interface as possible.\nProvider-Agnostic Design\n: Mirascope works seamlessly with multiple LLM providers, allowing you to switch between them effortlessly or use multiple providers in the same project.\nSimplicity and Ease of Use\n: We've designed Mirascope with simplicity in mind. You get powerful features without unnecessary complexity, making it easy to get started and scale your projects.\nComprehensive Tooling\n: From prompt engineering to response parsing, Mirascope offers a complete suite of tools for every aspect of LLM application development.\nCore Components\n¶\nMirascope is built around the following core components, each designed to handle specific aspects of LLM interaction and application development. Here's a quick overview with links to detailed documentation:\nPrompts\n: Learn how to create and manage prompts effectively.\nCalls\n: Understand how to make calls to LLMs using Mirascope.\nStreams\n: Explore streaming responses for real-time applications.\nTools\n: Discover how to extend LLM capabilities with custom tools.\nDynamic Configuration\n: Learn to adjust LLM behavior at runtime.\nChaining\n: Understand the art of chaining multiple LLM calls for complex tasks.\nJSON Mode\n: Work with structured data responses from LLMs.\nResponse Models\n: Define and use structured output models with automatic validation.\nOutput Parsers\n: Process and transform custom LLM output structures effectively.\nAsync\n: Maximize efficiecy with asynchronous programming.\nEvals\n: Apply core components to build evaluation strategies for your LLM applications.\nAgents\n: Put everything together to build advanced AI agents using Mirascope.\nNext Steps\n¶\nWe encourage you to dive into each component's documentation to gain a deeper understanding of Mirascope's capabilities. Start with the topics that align most closely with your immediate needs, but don't hesitate to explore all areas – you might discover new ways to enhance your LLM applications!\nAs you progress, you'll find advanced topics and best practices throughout the documentation. These will help you optimize your use of Mirascope and build increasingly sophisticated AI-powered applications.\nHappy learning, and welcome to the world of development with Mirascope!\nBack to top", 'tool_call_id': 'call_hDmkCB8DRPyvsZI1X82Wsyhh', 'name': 'extract_content'}] The **Mirascope library** is a powerful toolkit designed for developers working with **Large Language Models (LLMs)**. It streamlines the process of building AI-powered applications, such as chatbots, content generators, and complex AI-driven systems. Here are some key aspects of Mirascope: ### Key Features and Benefits: 1. **User-Friendly Design**: Mirascope is engineered for simplicity, providing necessary functionalities without unnecessary complexity. It aims to create a seamless development experience that feels familiar to Python developers. 2. **Type Safety**: The library emphasizes type hints and type safety, improving editor support and reducing potential errors during development. 3. **Provider-Agnostic**: Mirascope can interact with multiple LLM providers, enabling developers to switch between them easily or even use several providers within the same project. 4. **Comprehensive Tooling**: It offers a complete suite of tools to handle all aspects of LLM application development—from prompt engineering to response parsing. 5. **Modular and Extensible**: Mirascope is built with core components that allow developers to manage prompts, make API calls, handle streaming responses, and much more, all while maintaining the ability to customize and extend functionalities. ### Core Components: - **Prompts**: Facilitate efficient creation and management. - **Calls**: Handle interactions with LLMs. - **Streams**: Support real-time applications with streaming responses. - **Tools**: Allow for extending LLM capabilities. - **Chaining**: Enable coordinating multiple LLM calls for complex tasks. - **Output Parsing**: Simplify the processing and transformation of responses. ### Application Scenarios: Mirascope can be utilized for a range of applications like chatbots, content generation, and AI-driven agents, helping developers integrate sophisticated AI functionalities into their products. ### Conclusion: Mirascope serves as an essential tool for modern developers looking to harness AI capabilities efficiently and effectively, encouraging advanced application development while ensuring ease of use and flexibility. For more detailed information, you can refer to the [official Mirascope documentation](https://docs.mirascope.io/latest/learn/) or visit its [homepage](https://www.mirascope.io/). (Assistant): [{'role': 'user', 'content': 'What is Mirascope library?'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"queries":["Mirascope library","Mirascope Python library","Mirascope programming library","what is Mirascope library"]}', 'name': '_web_search'}, 'id': 'call_2olv5hg3ZzMFXMwwfIKvrFno'}]}, {'role': 'tool', 'content': 'https://github.com/mirascope/mirascope\n\nhttps://www.mirascope.io/\n\nhttps://pypi.org/project/mirascope/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/learn/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/WHY/\n\nhttps://github.com/mirascope/mirascope', 'tool_call_id': 'call_2olv5hg3ZzMFXMwwfIKvrFno', 'name': '_web_search'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"url": "https://www.mirascope.io/"}', 'name': 'extract_content'}, 'id': 'call_SugLNjl2qFQu6nDhdZ1eNzka'}, {'type': 'function', 'function': {'arguments': '{"url": "https://docs.mirascope.io/latest/learn/"}', 'name': 'extract_content'}, 'id': 'call_hDmkCB8DRPyvsZI1X82Wsyhh'}]}, {'role': 'tool', 'content': 'Join our beta list!\nGet updates and early access to try out new features as a beta tester.\nThank you! Your submission has been received!\nOops! Something went wrong while submitting the form.\nCore Components\nCalls\nStreams\nTools\nStreaming Tools\nStructured Outputs\nStructured Streaming\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\[email protected](\n"gpt-4o-mini"\n)\n4\n@prompt_template(\n"Recommend a {genre} book"\n)\n5\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n6\n7\nresponse = recommend_book(\n"fantasy"\n)\n8\nprint\n(response)\n9\n# > Sure! I would recommend The Name of the Wind by...\n1\[email protected](\n"gpt-4o-mini"\n, stream=\nTrue\n)\n2\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n3\n"""Recommend a {genre} book."""\n4\n5\nstream = recommend_book(\n"fantasy"\n)\n6\nfor\nchunk, _\nin\nstream:\n7\nprint\n(chunk, end=\n""\n, flush=\nTrue\n)\n8\n# > Sure! I would recommend...\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\ndef\nformat_book\n(\ntitle:\nstr\n, author:\nstr\n):\n4\nreturn\nf"\n{title}\nby\n{author}\n"\n5\n6\[email protected](\n7\n"gpt-4o-mini"\n,\n8\ntools=[format_book],\n9\ntool_choice=\n"required"\n,\n10\n)\n11\n@prompt_template(\n"Recommend a {genre} book"\n)\n12\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n13\n14\nresponse = recommend_book(\n"fantasy"\n)\n15\nif\ntool := response.tool\n16\nprint\n(tool.call())\n17\n# > The Name of the Wind by Patrick Rothfuss\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\[email protected](\n4\n"gpt-4o-mini"\n,\n5\nstream=\nTrue\n,\n6\ntools=[format_book],\n7\ntool_choice=\n"required"\n8\n)\n9\n@prompt_template(\n"Recommend two (2) {genre} books"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n11\n12\nstream = recommend_book(\n"fantasy"\n)\n13\nfor\nchunk, tool\nin\nstream:\n14\nif\ntool:\n15\nprint\n(tool.call())\n16\nelse\n:\n17\nprint\n(chunk, end=\n""\n, flush=\nTrue\n)\n18\n# > The Name of the Wind by Patrick Rothfuss\n19\n# > Mistborn: The Final Empire by Brandon Sanderson\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\nfrom\npydantic\nimport\nBaseModel\n3\n4\nclass\nBook\n(\nBaseModel\n):\n5\ntitle:\nstr\n6\nauthor:\nstr\n7\n8\[email protected](\n"gpt-4o-mini"\n, response_model=Book\n)\n9\n@prompt_template(\n"Recommend a {genre} book"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n11\n12\nbook = recommend_book(\n"fantasy"\n)\n13\nassert\nisinstance\n(book, Book)\n14\nprint\n(book)\n15\n# > title=\'The Name of the Wind\' author=\'Patrick Rothfuss\'\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\nfrom\npydantic\nimport\nBaseModel\n3\n4\nclass\nBook\n(\nBaseModel\n):\n5\ntitle:\nstr\n6\nauthor:\nstr\n7\n8\[email protected](\n"gpt-4o-mini"\n, stream=\nTrue\n, response_model=Book\n)\n9\n@prompt_template(\n"Recommend a {genre} book"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n11\n12\nbook_stream = recommend_book(\n"fantasy"\n)\n13\nfor\npartial_book\nin\nbook_stream:\n14\nprint\n(partial_book)\n15\n# > title=None author=None\n16\n# > title=\'The Name\' author=None\n17\n# > title=\'The Name of the Wind\' author=None\n18\n# > title=\'The Name of the Wind\' author=\'Patrick\'\n19\n# > title=\'The Name of the Wind\' author=\'Patrick Rothfuss\'\n...and more!\nExamples\nYou\'re in control\nSimple and transparent abstractions ensure you\'re in the driver\'s seat. Abstractions must be useful without getting in your way and blinding your view.\nType hints you\'ll love\nWe’ve taken care of all of the annoying Python typing so that you can have proper type hints in as simple an interface as possible.\nBuild\nyour\nway\nSimple, clean building blocks enable fast and reliable development without forcing you to do things our way. While we have recommendations and best practices, you\'re still always able to do things\nyour\nway to best suit\nyour\nneeds.\nTrusted by engineers building the next generation of AI-native applications\nThe Pydantic inspired LLM framework the space has been missing. Simple, modular, extensible...helps where you need it, stays out of your way when you don\'t.\nVince Trost\nCo-Founder / Plastic Labs\nMirascope\'s simplicity made it the natural next step from a provider\'s API (OpenAI) — all without fighting unnecessary complexity of other tools like LangChain. We have all the bells and whistles necessary for production while maintaining ease of use and easy onboarding of new team members.\nJake Duth\nCo-Founder &\xa0CTO / Reddy\nAs the author of Mirascope, I\'m certainly biased, but there\'s a reason I\'ve worked day and night on this package — it solves the problems I have with existing developer tools. Mirascope is simple and easy to use, and honestly building with Mirascope is the most fun I\'ve had coding in my life.\nWilliam Bakst\nFounder / Mirascope\nStart Building With Mirascope\nMirascope can help improve your application development experience no matter how simple or complex.\nIt will feel like writing the Python you already know.\nRead the Docs\nGitHub | ⭐ ${starCount}\nJoin the Community', 'tool_call_id': 'call_SugLNjl2qFQu6nDhdZ1eNzka', 'name': 'extract_content'}, {'role': 'tool', 'content': "Learn Mirascope\n¶\nThis section is designed to help you master Mirascope, a toolkit for building AI-powered applications with Large Language Models (LLMs).\nMirascope is a powerful, flexible, and user-friendly library that simplifies the process of working with LLMs. Whether you're building chatbots, content generators, or complex AI-driven agent systems, Mirascope provides the tools you need to streamline your development process and create powerful, robust applications.\nOur documentation is tailored for developers who have at least some experience with Python and LLMs. Whether you're coming from other development tool libraries or have worked directly with provider SDKs and APIs, Mirascope offers a familiar but enhanced experience.\nKey Features and Benefits\n¶\nType Safety and Superior Editor Support\n: We've prioritized proper type hints and type safety, ensuring you get world-class editor support. This means fewer errors, better autocomplete, and a smoother development experience in as simple and seamless an interface as possible.\nProvider-Agnostic Design\n: Mirascope works seamlessly with multiple LLM providers, allowing you to switch between them effortlessly or use multiple providers in the same project.\nSimplicity and Ease of Use\n: We've designed Mirascope with simplicity in mind. You get powerful features without unnecessary complexity, making it easy to get started and scale your projects.\nComprehensive Tooling\n: From prompt engineering to response parsing, Mirascope offers a complete suite of tools for every aspect of LLM application development.\nCore Components\n¶\nMirascope is built around the following core components, each designed to handle specific aspects of LLM interaction and application development. Here's a quick overview with links to detailed documentation:\nPrompts\n: Learn how to create and manage prompts effectively.\nCalls\n: Understand how to make calls to LLMs using Mirascope.\nStreams\n: Explore streaming responses for real-time applications.\nTools\n: Discover how to extend LLM capabilities with custom tools.\nDynamic Configuration\n: Learn to adjust LLM behavior at runtime.\nChaining\n: Understand the art of chaining multiple LLM calls for complex tasks.\nJSON Mode\n: Work with structured data responses from LLMs.\nResponse Models\n: Define and use structured output models with automatic validation.\nOutput Parsers\n: Process and transform custom LLM output structures effectively.\nAsync\n: Maximize efficiecy with asynchronous programming.\nEvals\n: Apply core components to build evaluation strategies for your LLM applications.\nAgents\n: Put everything together to build advanced AI agents using Mirascope.\nNext Steps\n¶\nWe encourage you to dive into each component's documentation to gain a deeper understanding of Mirascope's capabilities. Start with the topics that align most closely with your immediate needs, but don't hesitate to explore all areas – you might discover new ways to enhance your LLM applications!\nAs you progress, you'll find advanced topics and best practices throughout the documentation. These will help you optimize your use of Mirascope and build increasingly sophisticated AI-powered applications.\nHappy learning, and welcome to the world of development with Mirascope!\nBack to top", 'tool_call_id': 'call_hDmkCB8DRPyvsZI1X82Wsyhh', 'name': 'extract_content'}, {'role': 'assistant', 'content': 'The **Mirascope library** is a powerful toolkit designed for developers working with **Large Language Models (LLMs)**. It streamlines the process of building AI-powered applications, such as chatbots, content generators, and complex AI-driven systems. Here are some key aspects of Mirascope:\n\n### Key Features and Benefits:\n1. **User-Friendly Design**: Mirascope is engineered for simplicity, providing necessary functionalities without unnecessary complexity. It aims to create a seamless development experience that feels familiar to Python developers.\n \n2. **Type Safety**: The library emphasizes type hints and type safety, improving editor support and reducing potential errors during development.\n\n3. **Provider-Agnostic**: Mirascope can interact with multiple LLM providers, enabling developers to switch between them easily or even use several providers within the same project.\n\n4. **Comprehensive Tooling**: It offers a complete suite of tools to handle all aspects of LLM application development—from prompt engineering to response parsing.\n\n5. **Modular and Extensible**: Mirascope is built with core components that allow developers to manage prompts, make API calls, handle streaming responses, and much more, all while maintaining the ability to customize and extend functionalities.\n\n### Core Components:\n- **Prompts**: Facilitate efficient creation and management.\n- **Calls**: Handle interactions with LLMs.\n- **Streams**: Support real-time applications with streaming responses.\n- **Tools**: Allow for extending LLM capabilities.\n- **Chaining**: Enable coordinating multiple LLM calls for complex tasks.\n- **Output Parsing**: Simplify the processing and transformation of responses.\n\n### Application Scenarios:\nMirascope can be utilized for a range of applications like chatbots, content generation, and AI-driven agents, helping developers integrate sophisticated AI functionalities into their products.\n\n### Conclusion:\nMirascope serves as an essential tool for modern developers looking to harness AI capabilities efficiently and effectively, encouraging advanced application development while ensuring ease of use and flexibility.\n\nFor more detailed information, you can refer to the [official Mirascope documentation](https://docs.mirascope.io/latest/learn/) or visit its [homepage](https://www.mirascope.io/).'}] using _web_search tool with args: {'queries': ['exot programming', 'exot library', 'exot software', 'exot application', 'exot API', 'Mirascope exot connection']} [{'role': 'user', 'content': 'What is Mirascope library?'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"queries":["Mirascope library","Mirascope Python library","Mirascope programming library","what is Mirascope library"]}', 'name': '_web_search'}, 'id': 'call_2olv5hg3ZzMFXMwwfIKvrFno'}]}, {'role': 'tool', 'content': 'https://github.com/mirascope/mirascope\n\nhttps://www.mirascope.io/\n\nhttps://pypi.org/project/mirascope/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/learn/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/WHY/\n\nhttps://github.com/mirascope/mirascope', 'tool_call_id': 'call_2olv5hg3ZzMFXMwwfIKvrFno', 'name': '_web_search'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"url": "https://www.mirascope.io/"}', 'name': 'extract_content'}, 'id': 'call_SugLNjl2qFQu6nDhdZ1eNzka'}, {'type': 'function', 'function': {'arguments': '{"url": "https://docs.mirascope.io/latest/learn/"}', 'name': 'extract_content'}, 'id': 'call_hDmkCB8DRPyvsZI1X82Wsyhh'}]}, {'role': 'tool', 'content': 'Join our beta list!\nGet updates and early access to try out new features as a beta tester.\nThank you! Your submission has been received!\nOops! Something went wrong while submitting the form.\nCore Components\nCalls\nStreams\nTools\nStreaming Tools\nStructured Outputs\nStructured Streaming\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\[email protected](\n"gpt-4o-mini"\n)\n4\n@prompt_template(\n"Recommend a {genre} book"\n)\n5\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n6\n7\nresponse = recommend_book(\n"fantasy"\n)\n8\nprint\n(response)\n9\n# > Sure! I would recommend The Name of the Wind by...\n1\[email protected](\n"gpt-4o-mini"\n, stream=\nTrue\n)\n2\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n3\n"""Recommend a {genre} book."""\n4\n5\nstream = recommend_book(\n"fantasy"\n)\n6\nfor\nchunk, _\nin\nstream:\n7\nprint\n(chunk, end=\n""\n, flush=\nTrue\n)\n8\n# > Sure! I would recommend...\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\ndef\nformat_book\n(\ntitle:\nstr\n, author:\nstr\n):\n4\nreturn\nf"\n{title}\nby\n{author}\n"\n5\n6\[email protected](\n7\n"gpt-4o-mini"\n,\n8\ntools=[format_book],\n9\ntool_choice=\n"required"\n,\n10\n)\n11\n@prompt_template(\n"Recommend a {genre} book"\n)\n12\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n13\n14\nresponse = recommend_book(\n"fantasy"\n)\n15\nif\ntool := response.tool\n16\nprint\n(tool.call())\n17\n# > The Name of the Wind by Patrick Rothfuss\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\[email protected](\n4\n"gpt-4o-mini"\n,\n5\nstream=\nTrue\n,\n6\ntools=[format_book],\n7\ntool_choice=\n"required"\n8\n)\n9\n@prompt_template(\n"Recommend two (2) {genre} books"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n11\n12\nstream = recommend_book(\n"fantasy"\n)\n13\nfor\nchunk, tool\nin\nstream:\n14\nif\ntool:\n15\nprint\n(tool.call())\n16\nelse\n:\n17\nprint\n(chunk, end=\n""\n, flush=\nTrue\n)\n18\n# > The Name of the Wind by Patrick Rothfuss\n19\n# > Mistborn: The Final Empire by Brandon Sanderson\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\nfrom\npydantic\nimport\nBaseModel\n3\n4\nclass\nBook\n(\nBaseModel\n):\n5\ntitle:\nstr\n6\nauthor:\nstr\n7\n8\[email protected](\n"gpt-4o-mini"\n, response_model=Book\n)\n9\n@prompt_template(\n"Recommend a {genre} book"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n11\n12\nbook = recommend_book(\n"fantasy"\n)\n13\nassert\nisinstance\n(book, Book)\n14\nprint\n(book)\n15\n# > title=\'The Name of the Wind\' author=\'Patrick Rothfuss\'\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\nfrom\npydantic\nimport\nBaseModel\n3\n4\nclass\nBook\n(\nBaseModel\n):\n5\ntitle:\nstr\n6\nauthor:\nstr\n7\n8\[email protected](\n"gpt-4o-mini"\n, stream=\nTrue\n, response_model=Book\n)\n9\n@prompt_template(\n"Recommend a {genre} book"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n11\n12\nbook_stream = recommend_book(\n"fantasy"\n)\n13\nfor\npartial_book\nin\nbook_stream:\n14\nprint\n(partial_book)\n15\n# > title=None author=None\n16\n# > title=\'The Name\' author=None\n17\n# > title=\'The Name of the Wind\' author=None\n18\n# > title=\'The Name of the Wind\' author=\'Patrick\'\n19\n# > title=\'The Name of the Wind\' author=\'Patrick Rothfuss\'\n...and more!\nExamples\nYou\'re in control\nSimple and transparent abstractions ensure you\'re in the driver\'s seat. Abstractions must be useful without getting in your way and blinding your view.\nType hints you\'ll love\nWe’ve taken care of all of the annoying Python typing so that you can have proper type hints in as simple an interface as possible.\nBuild\nyour\nway\nSimple, clean building blocks enable fast and reliable development without forcing you to do things our way. While we have recommendations and best practices, you\'re still always able to do things\nyour\nway to best suit\nyour\nneeds.\nTrusted by engineers building the next generation of AI-native applications\nThe Pydantic inspired LLM framework the space has been missing. Simple, modular, extensible...helps where you need it, stays out of your way when you don\'t.\nVince Trost\nCo-Founder / Plastic Labs\nMirascope\'s simplicity made it the natural next step from a provider\'s API (OpenAI) — all without fighting unnecessary complexity of other tools like LangChain. We have all the bells and whistles necessary for production while maintaining ease of use and easy onboarding of new team members.\nJake Duth\nCo-Founder &\xa0CTO / Reddy\nAs the author of Mirascope, I\'m certainly biased, but there\'s a reason I\'ve worked day and night on this package — it solves the problems I have with existing developer tools. Mirascope is simple and easy to use, and honestly building with Mirascope is the most fun I\'ve had coding in my life.\nWilliam Bakst\nFounder / Mirascope\nStart Building With Mirascope\nMirascope can help improve your application development experience no matter how simple or complex.\nIt will feel like writing the Python you already know.\nRead the Docs\nGitHub | ⭐ ${starCount}\nJoin the Community', 'tool_call_id': 'call_SugLNjl2qFQu6nDhdZ1eNzka', 'name': 'extract_content'}, {'role': 'tool', 'content': "Learn Mirascope\n¶\nThis section is designed to help you master Mirascope, a toolkit for building AI-powered applications with Large Language Models (LLMs).\nMirascope is a powerful, flexible, and user-friendly library that simplifies the process of working with LLMs. Whether you're building chatbots, content generators, or complex AI-driven agent systems, Mirascope provides the tools you need to streamline your development process and create powerful, robust applications.\nOur documentation is tailored for developers who have at least some experience with Python and LLMs. Whether you're coming from other development tool libraries or have worked directly with provider SDKs and APIs, Mirascope offers a familiar but enhanced experience.\nKey Features and Benefits\n¶\nType Safety and Superior Editor Support\n: We've prioritized proper type hints and type safety, ensuring you get world-class editor support. This means fewer errors, better autocomplete, and a smoother development experience in as simple and seamless an interface as possible.\nProvider-Agnostic Design\n: Mirascope works seamlessly with multiple LLM providers, allowing you to switch between them effortlessly or use multiple providers in the same project.\nSimplicity and Ease of Use\n: We've designed Mirascope with simplicity in mind. You get powerful features without unnecessary complexity, making it easy to get started and scale your projects.\nComprehensive Tooling\n: From prompt engineering to response parsing, Mirascope offers a complete suite of tools for every aspect of LLM application development.\nCore Components\n¶\nMirascope is built around the following core components, each designed to handle specific aspects of LLM interaction and application development. Here's a quick overview with links to detailed documentation:\nPrompts\n: Learn how to create and manage prompts effectively.\nCalls\n: Understand how to make calls to LLMs using Mirascope.\nStreams\n: Explore streaming responses for real-time applications.\nTools\n: Discover how to extend LLM capabilities with custom tools.\nDynamic Configuration\n: Learn to adjust LLM behavior at runtime.\nChaining\n: Understand the art of chaining multiple LLM calls for complex tasks.\nJSON Mode\n: Work with structured data responses from LLMs.\nResponse Models\n: Define and use structured output models with automatic validation.\nOutput Parsers\n: Process and transform custom LLM output structures effectively.\nAsync\n: Maximize efficiecy with asynchronous programming.\nEvals\n: Apply core components to build evaluation strategies for your LLM applications.\nAgents\n: Put everything together to build advanced AI agents using Mirascope.\nNext Steps\n¶\nWe encourage you to dive into each component's documentation to gain a deeper understanding of Mirascope's capabilities. Start with the topics that align most closely with your immediate needs, but don't hesitate to explore all areas – you might discover new ways to enhance your LLM applications!\nAs you progress, you'll find advanced topics and best practices throughout the documentation. These will help you optimize your use of Mirascope and build increasingly sophisticated AI-powered applications.\nHappy learning, and welcome to the world of development with Mirascope!\nBack to top", 'tool_call_id': 'call_hDmkCB8DRPyvsZI1X82Wsyhh', 'name': 'extract_content'}, {'role': 'assistant', 'content': 'The **Mirascope library** is a powerful toolkit designed for developers working with **Large Language Models (LLMs)**. It streamlines the process of building AI-powered applications, such as chatbots, content generators, and complex AI-driven systems. Here are some key aspects of Mirascope:\n\n### Key Features and Benefits:\n1. **User-Friendly Design**: Mirascope is engineered for simplicity, providing necessary functionalities without unnecessary complexity. It aims to create a seamless development experience that feels familiar to Python developers.\n \n2. **Type Safety**: The library emphasizes type hints and type safety, improving editor support and reducing potential errors during development.\n\n3. **Provider-Agnostic**: Mirascope can interact with multiple LLM providers, enabling developers to switch between them easily or even use several providers within the same project.\n\n4. **Comprehensive Tooling**: It offers a complete suite of tools to handle all aspects of LLM application development—from prompt engineering to response parsing.\n\n5. **Modular and Extensible**: Mirascope is built with core components that allow developers to manage prompts, make API calls, handle streaming responses, and much more, all while maintaining the ability to customize and extend functionalities.\n\n### Core Components:\n- **Prompts**: Facilitate efficient creation and management.\n- **Calls**: Handle interactions with LLMs.\n- **Streams**: Support real-time applications with streaming responses.\n- **Tools**: Allow for extending LLM capabilities.\n- **Chaining**: Enable coordinating multiple LLM calls for complex tasks.\n- **Output Parsing**: Simplify the processing and transformation of responses.\n\n### Application Scenarios:\nMirascope can be utilized for a range of applications like chatbots, content generation, and AI-driven agents, helping developers integrate sophisticated AI functionalities into their products.\n\n### Conclusion:\nMirascope serves as an essential tool for modern developers looking to harness AI capabilities efficiently and effectively, encouraging advanced application development while ensuring ease of use and flexibility.\n\nFor more detailed information, you can refer to the [official Mirascope documentation](https://docs.mirascope.io/latest/learn/) or visit its [homepage](https://www.mirascope.io/).'}, {'role': 'user', 'content': 'exot'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"queries":["exot programming","exot library","exot software","exot application","exot API","Mirascope exot connection"]}', 'name': '_web_search'}, 'id': 'call_xmAYDMXpqwCBNqZnAPq5edtg'}]}, {'role': 'tool', 'content': 'https://stackoverflow.com/questions/73663/how-do-i-terminate-a-script\n\nhttps://www.lifewire.com/how-to-force-quit-a-program-in-windows-2625781\n\nhttps://www.neb.com/en/products/m0265-exonuclease-t\n\nhttps://www.fsis.usda.gov/inspection/import-export/import-export-library\n\nhttps://exot.dev/\n\nhttps://exoft.net/\n\nhttps://learn.microsoft.com/en-us/dotnet/api/system.windows.forms.application.exit?view=windowsdesktop-8.0\n\nhttps://stackoverflow.com/questions/12977924/how-do-i-properly-exit-a-c-sharp-application\n\nhttps://developer.exotel.com/\n\nhttps://developer.exotel.com/api\n\nhttps://docs.mirascope.io/latest/\n\nhttps://docs.mirascope.io/latest/learn/async/', 'tool_call_id': 'call_xmAYDMXpqwCBNqZnAPq5edtg', 'name': '_web_search'}] using extract_content tool with args: {'url': 'https://exot.dev/'} using extract_content tool with args: {'url': 'https://developer.exotel.com/'} [{'role': 'user', 'content': 'What is Mirascope library?'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"queries":["Mirascope library","Mirascope Python library","Mirascope programming library","what is Mirascope library"]}', 'name': '_web_search'}, 'id': 'call_2olv5hg3ZzMFXMwwfIKvrFno'}]}, {'role': 'tool', 'content': 'https://github.com/mirascope/mirascope\n\nhttps://www.mirascope.io/\n\nhttps://pypi.org/project/mirascope/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/learn/\n\nhttps://github.com/mirascope/mirascope\n\nhttps://docs.mirascope.io/latest/WHY/\n\nhttps://github.com/mirascope/mirascope', 'tool_call_id': 'call_2olv5hg3ZzMFXMwwfIKvrFno', 'name': '_web_search'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"url": "https://www.mirascope.io/"}', 'name': 'extract_content'}, 'id': 'call_SugLNjl2qFQu6nDhdZ1eNzka'}, {'type': 'function', 'function': {'arguments': '{"url": "https://docs.mirascope.io/latest/learn/"}', 'name': 'extract_content'}, 'id': 'call_hDmkCB8DRPyvsZI1X82Wsyhh'}]}, {'role': 'tool', 'content': 'Join our beta list!\nGet updates and early access to try out new features as a beta tester.\nThank you! Your submission has been received!\nOops! Something went wrong while submitting the form.\nCore Components\nCalls\nStreams\nTools\nStreaming Tools\nStructured Outputs\nStructured Streaming\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\[email protected](\n"gpt-4o-mini"\n)\n4\n@prompt_template(\n"Recommend a {genre} book"\n)\n5\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n6\n7\nresponse = recommend_book(\n"fantasy"\n)\n8\nprint\n(response)\n9\n# > Sure! I would recommend The Name of the Wind by...\n1\[email protected](\n"gpt-4o-mini"\n, stream=\nTrue\n)\n2\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n3\n"""Recommend a {genre} book."""\n4\n5\nstream = recommend_book(\n"fantasy"\n)\n6\nfor\nchunk, _\nin\nstream:\n7\nprint\n(chunk, end=\n""\n, flush=\nTrue\n)\n8\n# > Sure! I would recommend...\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\ndef\nformat_book\n(\ntitle:\nstr\n, author:\nstr\n):\n4\nreturn\nf"\n{title}\nby\n{author}\n"\n5\n6\[email protected](\n7\n"gpt-4o-mini"\n,\n8\ntools=[format_book],\n9\ntool_choice=\n"required"\n,\n10\n)\n11\n@prompt_template(\n"Recommend a {genre} book"\n)\n12\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n13\n14\nresponse = recommend_book(\n"fantasy"\n)\n15\nif\ntool := response.tool\n16\nprint\n(tool.call())\n17\n# > The Name of the Wind by Patrick Rothfuss\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\n3\[email protected](\n4\n"gpt-4o-mini"\n,\n5\nstream=\nTrue\n,\n6\ntools=[format_book],\n7\ntool_choice=\n"required"\n8\n)\n9\n@prompt_template(\n"Recommend two (2) {genre} books"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n11\n12\nstream = recommend_book(\n"fantasy"\n)\n13\nfor\nchunk, tool\nin\nstream:\n14\nif\ntool:\n15\nprint\n(tool.call())\n16\nelse\n:\n17\nprint\n(chunk, end=\n""\n, flush=\nTrue\n)\n18\n# > The Name of the Wind by Patrick Rothfuss\n19\n# > Mistborn: The Final Empire by Brandon Sanderson\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\nfrom\npydantic\nimport\nBaseModel\n3\n4\nclass\nBook\n(\nBaseModel\n):\n5\ntitle:\nstr\n6\nauthor:\nstr\n7\n8\[email protected](\n"gpt-4o-mini"\n, response_model=Book\n)\n9\n@prompt_template(\n"Recommend a {genre} book"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n11\n12\nbook = recommend_book(\n"fantasy"\n)\n13\nassert\nisinstance\n(book, Book)\n14\nprint\n(book)\n15\n# > title=\'The Name of the Wind\' author=\'Patrick Rothfuss\'\n1\nfrom\nmirascope.core\nimport\nopenai, prompt_template\n2\nfrom\npydantic\nimport\nBaseModel\n3\n4\nclass\nBook\n(\nBaseModel\n):\n5\ntitle:\nstr\n6\nauthor:\nstr\n7\n8\[email protected](\n"gpt-4o-mini"\n, stream=\nTrue\n, response_model=Book\n)\n9\n@prompt_template(\n"Recommend a {genre} book"\n)\n10\ndef\nrecommend_book\n(\ngenre:\nstr\n):\n...\n11\n12\nbook_stream = recommend_book(\n"fantasy"\n)\n13\nfor\npartial_book\nin\nbook_stream:\n14\nprint\n(partial_book)\n15\n# > title=None author=None\n16\n# > title=\'The Name\' author=None\n17\n# > title=\'The Name of the Wind\' author=None\n18\n# > title=\'The Name of the Wind\' author=\'Patrick\'\n19\n# > title=\'The Name of the Wind\' author=\'Patrick Rothfuss\'\n...and more!\nExamples\nYou\'re in control\nSimple and transparent abstractions ensure you\'re in the driver\'s seat. Abstractions must be useful without getting in your way and blinding your view.\nType hints you\'ll love\nWe’ve taken care of all of the annoying Python typing so that you can have proper type hints in as simple an interface as possible.\nBuild\nyour\nway\nSimple, clean building blocks enable fast and reliable development without forcing you to do things our way. While we have recommendations and best practices, you\'re still always able to do things\nyour\nway to best suit\nyour\nneeds.\nTrusted by engineers building the next generation of AI-native applications\nThe Pydantic inspired LLM framework the space has been missing. Simple, modular, extensible...helps where you need it, stays out of your way when you don\'t.\nVince Trost\nCo-Founder / Plastic Labs\nMirascope\'s simplicity made it the natural next step from a provider\'s API (OpenAI) — all without fighting unnecessary complexity of other tools like LangChain. We have all the bells and whistles necessary for production while maintaining ease of use and easy onboarding of new team members.\nJake Duth\nCo-Founder &\xa0CTO / Reddy\nAs the author of Mirascope, I\'m certainly biased, but there\'s a reason I\'ve worked day and night on this package — it solves the problems I have with existing developer tools. Mirascope is simple and easy to use, and honestly building with Mirascope is the most fun I\'ve had coding in my life.\nWilliam Bakst\nFounder / Mirascope\nStart Building With Mirascope\nMirascope can help improve your application development experience no matter how simple or complex.\nIt will feel like writing the Python you already know.\nRead the Docs\nGitHub | ⭐ ${starCount}\nJoin the Community', 'tool_call_id': 'call_SugLNjl2qFQu6nDhdZ1eNzka', 'name': 'extract_content'}, {'role': 'tool', 'content': "Learn Mirascope\n¶\nThis section is designed to help you master Mirascope, a toolkit for building AI-powered applications with Large Language Models (LLMs).\nMirascope is a powerful, flexible, and user-friendly library that simplifies the process of working with LLMs. Whether you're building chatbots, content generators, or complex AI-driven agent systems, Mirascope provides the tools you need to streamline your development process and create powerful, robust applications.\nOur documentation is tailored for developers who have at least some experience with Python and LLMs. Whether you're coming from other development tool libraries or have worked directly with provider SDKs and APIs, Mirascope offers a familiar but enhanced experience.\nKey Features and Benefits\n¶\nType Safety and Superior Editor Support\n: We've prioritized proper type hints and type safety, ensuring you get world-class editor support. This means fewer errors, better autocomplete, and a smoother development experience in as simple and seamless an interface as possible.\nProvider-Agnostic Design\n: Mirascope works seamlessly with multiple LLM providers, allowing you to switch between them effortlessly or use multiple providers in the same project.\nSimplicity and Ease of Use\n: We've designed Mirascope with simplicity in mind. You get powerful features without unnecessary complexity, making it easy to get started and scale your projects.\nComprehensive Tooling\n: From prompt engineering to response parsing, Mirascope offers a complete suite of tools for every aspect of LLM application development.\nCore Components\n¶\nMirascope is built around the following core components, each designed to handle specific aspects of LLM interaction and application development. Here's a quick overview with links to detailed documentation:\nPrompts\n: Learn how to create and manage prompts effectively.\nCalls\n: Understand how to make calls to LLMs using Mirascope.\nStreams\n: Explore streaming responses for real-time applications.\nTools\n: Discover how to extend LLM capabilities with custom tools.\nDynamic Configuration\n: Learn to adjust LLM behavior at runtime.\nChaining\n: Understand the art of chaining multiple LLM calls for complex tasks.\nJSON Mode\n: Work with structured data responses from LLMs.\nResponse Models\n: Define and use structured output models with automatic validation.\nOutput Parsers\n: Process and transform custom LLM output structures effectively.\nAsync\n: Maximize efficiecy with asynchronous programming.\nEvals\n: Apply core components to build evaluation strategies for your LLM applications.\nAgents\n: Put everything together to build advanced AI agents using Mirascope.\nNext Steps\n¶\nWe encourage you to dive into each component's documentation to gain a deeper understanding of Mirascope's capabilities. Start with the topics that align most closely with your immediate needs, but don't hesitate to explore all areas – you might discover new ways to enhance your LLM applications!\nAs you progress, you'll find advanced topics and best practices throughout the documentation. These will help you optimize your use of Mirascope and build increasingly sophisticated AI-powered applications.\nHappy learning, and welcome to the world of development with Mirascope!\nBack to top", 'tool_call_id': 'call_hDmkCB8DRPyvsZI1X82Wsyhh', 'name': 'extract_content'}, {'role': 'assistant', 'content': 'The **Mirascope library** is a powerful toolkit designed for developers working with **Large Language Models (LLMs)**. It streamlines the process of building AI-powered applications, such as chatbots, content generators, and complex AI-driven systems. Here are some key aspects of Mirascope:\n\n### Key Features and Benefits:\n1. **User-Friendly Design**: Mirascope is engineered for simplicity, providing necessary functionalities without unnecessary complexity. It aims to create a seamless development experience that feels familiar to Python developers.\n \n2. **Type Safety**: The library emphasizes type hints and type safety, improving editor support and reducing potential errors during development.\n\n3. **Provider-Agnostic**: Mirascope can interact with multiple LLM providers, enabling developers to switch between them easily or even use several providers within the same project.\n\n4. **Comprehensive Tooling**: It offers a complete suite of tools to handle all aspects of LLM application development—from prompt engineering to response parsing.\n\n5. **Modular and Extensible**: Mirascope is built with core components that allow developers to manage prompts, make API calls, handle streaming responses, and much more, all while maintaining the ability to customize and extend functionalities.\n\n### Core Components:\n- **Prompts**: Facilitate efficient creation and management.\n- **Calls**: Handle interactions with LLMs.\n- **Streams**: Support real-time applications with streaming responses.\n- **Tools**: Allow for extending LLM capabilities.\n- **Chaining**: Enable coordinating multiple LLM calls for complex tasks.\n- **Output Parsing**: Simplify the processing and transformation of responses.\n\n### Application Scenarios:\nMirascope can be utilized for a range of applications like chatbots, content generation, and AI-driven agents, helping developers integrate sophisticated AI functionalities into their products.\n\n### Conclusion:\nMirascope serves as an essential tool for modern developers looking to harness AI capabilities efficiently and effectively, encouraging advanced application development while ensuring ease of use and flexibility.\n\nFor more detailed information, you can refer to the [official Mirascope documentation](https://docs.mirascope.io/latest/learn/) or visit its [homepage](https://www.mirascope.io/).'}, {'role': 'user', 'content': 'exot'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"queries":["exot programming","exot library","exot software","exot application","exot API","Mirascope exot connection"]}', 'name': '_web_search'}, 'id': 'call_xmAYDMXpqwCBNqZnAPq5edtg'}]}, {'role': 'tool', 'content': 'https://stackoverflow.com/questions/73663/how-do-i-terminate-a-script\n\nhttps://www.lifewire.com/how-to-force-quit-a-program-in-windows-2625781\n\nhttps://www.neb.com/en/products/m0265-exonuclease-t\n\nhttps://www.fsis.usda.gov/inspection/import-export/import-export-library\n\nhttps://exot.dev/\n\nhttps://exoft.net/\n\nhttps://learn.microsoft.com/en-us/dotnet/api/system.windows.forms.application.exit?view=windowsdesktop-8.0\n\nhttps://stackoverflow.com/questions/12977924/how-do-i-properly-exit-a-c-sharp-application\n\nhttps://developer.exotel.com/\n\nhttps://developer.exotel.com/api\n\nhttps://docs.mirascope.io/latest/\n\nhttps://docs.mirascope.io/latest/learn/async/', 'tool_call_id': 'call_xmAYDMXpqwCBNqZnAPq5edtg', 'name': '_web_search'}, {'role': 'assistant', 'content': '', 'tool_calls': [{'type': 'function', 'function': {'arguments': '{"url": "https://exot.dev/"}', 'name': 'extract_content'}, 'id': 'call_elkn1QwLzWadl9HgeSSud0r1'}, {'type': 'function', 'function': {'arguments': '{"url": "https://developer.exotel.com/"}', 'name': 'extract_content'}, 'id': 'call_bSmKZTKtQ7ZOdxRpD3zKu9hk'}]}, {'role': 'tool', 'content': 'Exot Inspector\nApplication inspector for Node, Bun and Deno, equipped with powerful tools for monitoring, tracing and debugging.\nGet Started\nOpen Inspector\nMonitoring\nEasily monitor performance metrics, create custom dashboards, and gain real-time insights into your application. No sign-ups required for a streamlined experience.\nDebugging\nDebug and analyze applications seamlessly with logging, tracing, and real-time monitoring of outgoing HTTP requests, ensuring privacy in your debugging process.\nSecurity\nCapture errors and exceptions, store metrics, and data in your preferred storage – all without external services or databases. Own and secure your data with complete control, emphasizing privacy throughout your application’s lifecycle.\nGet Started\nExot Inspector is a free, self-hosted software. Follow the guides to\nget started\n.', 'tool_call_id': 'call_elkn1QwLzWadl9HgeSSud0r1', 'name': 'extract_content'}, {'role': 'tool', 'content': 'API Docs\nVoice\nSMS\nExophones\nHeartbeat\nCall Campaigns\nSMS Campaigns\nLead\xa0Assist\nUsers\nExoVerify\nWhatsapp\nURL shortening\nMore...\nAPI Console\nVoice\nSMS\nExophones\nHeartbeat\nCall Campaigns\nSMS Campaigns\nLeadAssist\nMore...\nApplets Docs\nGreeting\nConnect\nPassthru\nTransfer\nSMS\nEmail\nMore\nUse Cases\nFAQs\nCreate Account', 'tool_call_id': 'call_bSmKZTKtQ7ZOdxRpD3zKu9hk', 'name': 'extract_content'}] ### Exot Overview **Exot** appears to be associated with two main contexts: a software monitoring and debugging tool, **Exot Inspector**, and **Exotel**, a cloud communication platform. Here’s a detailed look at both: #### 1. Exot Inspector: Exot Inspector is a development tool tailored for applications built with **Node.js**, **Bun**, and **Deno**. It provides powerful capabilities for monitoring, debugging, and tracing applications. - **Key Features**: - **Monitoring**: Offer performance metrics and create custom dashboards for real-time insights into application performance. - **Debugging**: Allows for seamless debugging and analysis of applications through logging, tracing, and monitoring outgoing HTTP requests. - **Security**: Users maintain complete control over their data, capturing errors and metrics without relying on external services. This focus on privacy is important for managing sensitive information. - **Usage**: Exot Inspector is self-hosted and free to use, making it accessible for developers seeking to enhance their application's oversight without additional costs. For more information, you can visit the [Exot Inspector website](https://exot.dev/). --- #### 2. Exotel: Exotel is a cloud-based communication platform that provides a suite of services aimed at enhancing businesses' communication capabilities. - **Key Features**: - **APIs for Communication**: Exotel offers APIs for voice and SMS communications, which include functionalities like call campaigns, lead assistance, and verification services. - **Integration Capabilities**: The platform allows developers to integrate communication features into their applications easily, making it suitable for businesses aiming to streamline customer interactions. - **Use Cases**: Exotel supports various use cases, highlighting the versatility of its communication solutions. For more details, you can explore the [Exotel Developer Portal](https://developer.exotel.com/). --- ### Conclusion Overall, "Exot" seems to refer to a versatile ecosystem of tools catering to application debugging and cloud communication needs. Whether you are looking to enhance your application's monitoring capabilities with Exot Inspector or improve your business's communication streams with Exotel, there are robust options available.
Search Queries:
- Mirascope library
- what is Mirascope
- Mirascope Python library
- Mirascope library documentation
- Mirascope library features
- Mirascope library use cases
- Mirascope library tutorial
The LLM can gather information regarding the Mirascope library but has no context beyond that.
Let's take a look at what happens when we call the user queries together.
await WebAssistantBaseWithStream().run()
Search Queries:
- Mirascope library
- Mirascope LLM development
- Mirascope open source
- Mirascope Python library
- LLM tools Mirascope
By giving the LLM search history, these search queries now connect the Mirascope library specifically to LLM development tools, providing a more cohesive set of results.
We can now create our _step
and run
functions which will call our _stream
and _step
functions respectively.
class WebAssistant(WebAssistantBaseWithStream):
async def _step(self, question: str):
print(self.messages)
response = await self._stream(question)
tools_and_outputs = []
async for chunk, tool in response:
if tool:
print(f"using {tool._name()} tool with args: {tool.args}")
tools_and_outputs.append((tool, tool.call()))
else:
print(chunk.content, end="", flush=True)
if response.user_message_param:
self.messages.append(response.user_message_param)
self.messages.append(response.message_param)
if tools_and_outputs:
self.messages += response.tool_message_params(tools_and_outputs)
await self._step("")
The run
function will keep running until the LLM feels that the users question can be answered.
web_assistant = WebAssistant()
await web_assistant.run()
Note that by giving the LLM the current date, it'll automatically search for the most up-to-date information.
Check out Evaluating Web Search Agent for an in-depth guide on how we evaluate the quality of our agent.
Additional Real-World Applications
Advanced Research Assistant
- Stay updated on latest developments in rapidly evolving fields
Personalized Education
- Create customized learning materials based on current curricula
Business Intelligence
- Assist in data-driven decision making with real-time insights
Technical Support and Troubleshooting
- Assist in debugging by referencing current documentation
Travel Planning
- Provide updates on travel restrictions, local events, and weather
Journalism and Fact-Checking
- Help identify and combat misinformation
Environmental Monitoring
- Track and analyze current climate data
When adapting this recipe, consider:
- Optimizing the search by utilizing
async
to increase parallelism. - When targeting specific websites for scraping purposes, use
response_model
to extract the specific information you're looking for across websites with similar content. - Implement a feedback loop so the LLM can rewrite the query for better search results.
- Reduce the number of tokens used by storing the extracted webpages as embeddings in a vectorstore and retrieving only what is necessary.
- Make a more specific web search agent for your use-case rather than a general purpose web search agent.