tutorials/36_Building_Fallbacks_with_Conditional_Routing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IR5wivW8THt7"
   },
   "source": [
    "# Tutorial: Building Fallbacks to Websearch with Conditional Routing\n",
    "\n",
    "- **Level**: Intermediate\n",
    "- **Time to complete**: 10 minutes\n",
    "- **Components Used**: [`ConditionalRouter`](https://docs.haystack.deepset.ai/docs/conditionalrouter), [`SerperDevWebSearch`](https://docs.haystack.deepset.ai/docs/serperdevwebsearch), [`ChatPromptBuilder`](https://docs.haystack.deepset.ai/docs/chatpromptbuilder), [`OpenAIChatGenerator`](https://docs.haystack.deepset.ai/docs/openaichatgenerator)\n",
    "- **Prerequisites**: You must have an [OpenAI API Key](https://platform.openai.com/api-keys) and a [Serper API Key](https://serper.dev/api-key) for this tutorial\n",
    "- **Goal**: After completing this tutorial, you'll have learned how to create a pipeline with conditional routing that can fallback to websearch if the answer is not present in your dataset.\n",
    "\n",
    "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F-a-MAMVat-o"
   },
   "source": [
    "## Overview\n",
    "\n",
    "When developing applications using **retrieval augmented generation ([RAG](https://www.deepset.ai/blog/llms-retrieval-augmentation))**, the retrieval step plays a critical role. It serves as the primary information source for **large language models (LLMs)** to generate responses. However, if your database lacks the necessary information, the retrieval step's effectiveness is limited. In such scenarios, it may be practical to use the web as a fallback data source for your RAG application. By implementing a conditional routing mechanism in your system, you gain complete control over the data flow, enabling you to design a system that can leverage the web as its data source under some conditions.\n",
    "\n",
    "In this tutorial, you will learn how to create a pipeline with conditional routing that directs the query to a **web-based RAG** route if the answer is not found in the initially given documents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LSwNKkeKeq0f"
   },
   "source": [
    "## Development Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eGJ7GmCBas4R"
   },
   "source": [
    "### Prepare the Colab Environment\n",
    "\n",
    "- [Enable GPU Runtime in Colab](https://docs.haystack.deepset.ai/docs/enabling-gpu-acceleration)\n",
    "- [Set logging level to INFO](https://docs.haystack.deepset.ai/docs/setting-the-log-level)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FwIgIpE2XqpO"
   },
   "source": [
    "### Install Haystack\n",
    "\n",
    "Install Haystack with `pip`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uba0mntlqs_O"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "pip install haystack-ai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WBkJ7d3hZkOJ"
   },
   "source": [
    "### Enable Telemetry\n",
    "\n",
    "Knowing you're using this tutorial helps us decide where to invest our efforts to build a better product but you can always opt out by commenting the following line. See [Telemetry](https://docs.haystack.deepset.ai/docs/telemetry) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "HvrOixzzZmMi"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from haystack.telemetry import tutorial_running\n",
    "\n",
    "tutorial_running(36)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QfECEAy2Jdqs"
   },
   "source": [
    "### Enter API Keys\n",
    "\n",
    "Enter API keys required for this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "13U7Z_k3yE-F",
    "outputId": "6ec48553-12d2-4c89-ca13-fc5d34fbc625"
   },
   "outputs": [],
   "source": [
    "from getpass import getpass\n",
    "import os\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass(\"Enter OpenAI API key:\")\n",
    "if \"SERPERDEV_API_KEY\" not in os.environ:\n",
    "    os.environ[\"SERPERDEV_API_KEY\"] = getpass(\"Enter Serper Api key: \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i_AlhPv1T-4t"
   },
   "source": [
    "## Creating a Document\n",
    "\n",
    "Create a Document about Munich, where the answer to your question will be initially searched:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "5CHbQlLMyVbg"
   },
   "outputs": [],
   "source": [
    "from haystack.dataclasses import Document\n",
    "\n",
    "documents = [\n",
    "    Document(\n",
    "        content=\"\"\"Munich, the vibrant capital of Bavaria in southern Germany, exudes a perfect blend of rich cultural\n",
    "                                heritage and modern urban sophistication. Nestled along the banks of the Isar River, Munich is renowned\n",
    "                                for its splendid architecture, including the iconic Neues Rathaus (New Town Hall) at Marienplatz and\n",
    "                                the grandeur of Nymphenburg Palace. The city is a haven for art enthusiasts, with world-class museums like the\n",
    "                                Alte Pinakothek housing masterpieces by renowned artists. Munich is also famous for its lively beer gardens, where\n",
    "                                locals and tourists gather to enjoy the city's famed beers and traditional Bavarian cuisine. The city's annual\n",
    "                                Oktoberfest celebration, the world's largest beer festival, attracts millions of visitors from around the globe.\n",
    "                                Beyond its cultural and culinary delights, Munich offers picturesque parks like the English Garden, providing a\n",
    "                                serene escape within the heart of the bustling metropolis. Visitors are charmed by Munich's warm hospitality,\n",
    "                                making it a must-visit destination for travelers seeking a taste of both old-world charm and contemporary allure.\"\"\"\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zMNy0tjtUh_L"
   },
   "source": [
    "## Creating the Initial Pipeline Components\n",
    "\n",
    "First, define a prompt instructing the LLM to respond with the text `\"no_answer\"` if the provided documents do not offer enough context to answer the query. Next, initialize a [ChatPromptBuilder](https://docs.haystack.deepset.ai/docs/chatpromptbuilder) with that prompt. `ChatPromptBuilder`accepts prompts inf the form of `ChatMessage`. It's crucial that the LLM replies with `\"no_answer\"` as you will use this keyword to indicate that the query should be directed to the fallback web search route.\n",
    "\n",
    "As the LLM, you will use an [OpenAIChatGenerator](https://docs.haystack.deepset.ai/docs/openaichatgenerator) with the `gpt-4o-mini` model.\n",
    "\n",
    "> The provided prompt works effectively with the `gpt-4o-mini` model. If you prefer to use a different [ChatGenerator](https://docs.haystack.deepset.ai/docs/generators), you may need to update the prompt to provide clear instructions to your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "nzhn2kDfqvbs"
   },
   "outputs": [],
   "source": [
    "from haystack.components.builders import ChatPromptBuilder\n",
    "from haystack.dataclasses import ChatMessage\n",
    "from haystack.components.generators.chat import OpenAIChatGenerator\n",
    "\n",
    "prompt_template = [\n",
    "    ChatMessage.from_user(\n",
    "        \"\"\"\n",
    "Answer the following query given the documents.\n",
    "If the answer is not contained within the documents reply with 'no_answer'\n",
    "Query: {{query}}\n",
    "Documents:\n",
    "{% for document in documents %}\n",
    "  {{document.content}}\n",
    "{% endfor %}\n",
    "\"\"\"\n",
    "    )\n",
    "]\n",
    "\n",
    "prompt_builder = ChatPromptBuilder(template=prompt_template)\n",
    "llm = OpenAIChatGenerator(model=\"gpt-4o-mini\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LepACkkWPsBx"
   },
   "source": [
    "## Initializing the Web Search Components\n",
    "\n",
    "Initialize the necessary components for a web-based RAG application. Along with a `ChatPromptBuilder` and an `OpenAIChatGenerator`, you will need a [SerperDevWebSearch](https://docs.haystack.deepset.ai/docs/serperdevwebsearch) to retrieve relevant documents for the query from the web.\n",
    "\n",
    "> If desired, you can use a different [ChatGenerator](https://docs.haystack.deepset.ai/docs/generators) for the web-based RAG branch of the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "VEYchFgQPxZ_"
   },
   "outputs": [],
   "source": [
    "from haystack.components.websearch.serper_dev import SerperDevWebSearch\n",
    "\n",
    "prompt_for_websearch = [\n",
    "    ChatMessage.from_user(\n",
    "        \"\"\"\n",
    "Answer the following query given the documents retrieved from the web.\n",
    "Your answer shoud indicate that your answer was generated from websearch.\n",
    "\n",
    "Query: {{query}}\n",
    "Documents:\n",
    "{% for document in documents %}\n",
    "  {{document.content}}\n",
    "{% endfor %}\n",
    "\"\"\"\n",
    "    )\n",
    "]\n",
    "\n",
    "websearch = SerperDevWebSearch()\n",
    "prompt_builder_for_websearch = ChatPromptBuilder(template=prompt_for_websearch)\n",
    "llm_for_websearch = OpenAIChatGenerator(model=\"gpt-4o-mini\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vnacak_tVWqv"
   },
   "source": [
    "## Creating the ConditionalRouter\n",
    "\n",
    "[ConditionalRouter](https://docs.haystack.deepset.ai/docs/conditionalrouter) is the component that handles data routing on specific conditions. You need to define a `condition`, an `output`, an `output_name` and an `output_type` for each route. Each route that the `ConditionalRouter` creates acts as the output of this component and can be connected to other components in the same pipeline.  \n",
    "\n",
    "In this case, you need to define two routes:\n",
    "- If the LLM replies with the `\"no_answer\"` keyword, the pipeline should perform web search. It means that you will put the original `query` in the output value to pass to the next component (in this case the next component will be the `SerperDevWebSearch`) and the output name will be `go_to_websearch`.\n",
    "- Otherwise, the given documents are enough for an answer and pipeline execution ends here. Return the LLM reply in the output named `answer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qyE9rGcawX3F"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages/pypdf/_crypt_providers/_cryptography.py:32: CryptographyDeprecationWarning: ARC4 has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.ARC4 and will be removed from this module in 48.0.0.\n",
      "  from cryptography.hazmat.primitives.ciphers.algorithms import AES, ARC4\n"
     ]
    }
   ],
   "source": [
    "from haystack.components.routers import ConditionalRouter\n",
    "\n",
    "routes = [\n",
    "    {\n",
    "        \"condition\": \"{{'no_answer' in replies[0].text}}\",\n",
    "        \"output\": \"{{query}}\",\n",
    "        \"output_name\": \"go_to_websearch\",\n",
    "        \"output_type\": str,\n",
    "    },\n",
    "    {\n",
    "        \"condition\": \"{{'no_answer' not in replies[0].text}}\",\n",
    "        \"output\": \"{{replies[0].text}}\",\n",
    "        \"output_name\": \"answer\",\n",
    "        \"output_type\": str,\n",
    "    },\n",
    "]\n",
    "\n",
    "router = ConditionalRouter(routes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wdyko78oXb5a"
   },
   "source": [
    "## Building the Pipeline\n",
    "\n",
    "Add all components to your pipeline and connect them. `go_to_websearch` output of the `router` should be connected to the `websearch` to retrieve documents from the web and also to `prompt_builder_for_websearch` to use in the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4sCyBwc0oTVs",
    "outputId": "fd2347d4-9363-45e0-e734-87e4a160f741"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<haystack.core.pipeline.pipeline.Pipeline object at 0x31c6acc10>\n",
       "🚅 Components\n",
       "  - prompt_builder: ChatPromptBuilder\n",
       "  - llm: OpenAIChatGenerator\n",
       "  - router: ConditionalRouter\n",
       "  - websearch: SerperDevWebSearch\n",
       "  - prompt_builder_for_websearch: ChatPromptBuilder\n",
       "  - llm_for_websearch: OpenAIChatGenerator\n",
       "🛤️ Connections\n",
       "  - prompt_builder.prompt -> llm.messages (List[ChatMessage])\n",
       "  - llm.replies -> router.replies (List[ChatMessage])\n",
       "  - router.go_to_websearch -> websearch.query (str)\n",
       "  - router.go_to_websearch -> prompt_builder_for_websearch.query (str)\n",
       "  - websearch.documents -> prompt_builder_for_websearch.documents (List[Document])\n",
       "  - prompt_builder_for_websearch.prompt -> llm_for_websearch.messages (List[ChatMessage])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from haystack import Pipeline\n",
    "\n",
    "pipe = Pipeline()\n",
    "pipe.add_component(\"prompt_builder\", prompt_builder)\n",
    "pipe.add_component(\"llm\", llm)\n",
    "pipe.add_component(\"router\", router)\n",
    "pipe.add_component(\"websearch\", websearch)\n",
    "pipe.add_component(\"prompt_builder_for_websearch\", prompt_builder_for_websearch)\n",
    "pipe.add_component(\"llm_for_websearch\", llm_for_websearch)\n",
    "\n",
    "pipe.connect(\"prompt_builder.prompt\", \"llm.messages\")\n",
    "pipe.connect(\"llm.replies\", \"router.replies\")\n",
    "pipe.connect(\"router.go_to_websearch\", \"websearch.query\")\n",
    "pipe.connect(\"router.go_to_websearch\", \"prompt_builder_for_websearch.query\")\n",
    "pipe.connect(\"websearch.documents\", \"prompt_builder_for_websearch.documents\")\n",
    "pipe.connect(\"prompt_builder_for_websearch\", \"llm_for_websearch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d0HmdbUJKJ_9"
   },
   "source": [
    "### Visualize the Pipeline\n",
    "\n",
    "To understand how you formed this pipeline with conditional routing, use [draw()](https://docs.haystack.deepset.ai/docs/drawing-pipeline-graphs) method of the pipeline. If you're running this notebook on Google Colab, the generated file will be saved in \\\"Files\\\" section on the sidebar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "svF_SUK4rFwv",
    "outputId": "60894eea-2cec-4be8-d13c-83d2c81656f4"
   },
   "outputs": [],
   "source": [
    "pipe.draw(\"pipe.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jgk1z6GGYH6J"
   },
   "source": [
    "## Running the Pipeline!\n",
    "\n",
    "In the `run()`, pass the query to the `prompt_builder` and the `router`. In real life applications, `documents` will be provided by a [Retriever](https://docs.haystack.deepset.ai/docs/retrievers) but to keep this example simple, you will provide the defined `documents` to the `prompt_builder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d_l4rYmCoVki",
    "outputId": "3bd7956a-7612-4bc1-c3e5-a7a51be8981f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Munich is located in southern Germany.\n"
     ]
    }
   ],
   "source": [
    "query = \"Where is Munich?\"\n",
    "\n",
    "result = pipe.run({\"prompt_builder\": {\"query\": query, \"documents\": documents}, \"router\": {\"query\": query}})\n",
    "\n",
    "# Print the `answer` coming from the ConditionalRouter\n",
    "print(result[\"router\"][\"answer\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dBN8eLSKgb16"
   },
   "source": [
    "✅ The answer to this query can be found in the defined document.\n",
    "\n",
    "Now, try a different query that doesn't have an answer in the given document and test if the web search works as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_v-WdlSy365M",
    "outputId": "603c9346-8718-427e-d232-4cc71799a2bb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ChatMessage(content='As of May 31, 2024, Munich has a population of approximately 1,594,632 residents, making it the third-largest city in Germany, after Berlin and Hamburg. The metro area population is about 1,585,000, reflecting a modest increase from the previous year. Additionally, estimates suggest that the city has consistently had over 1.5 million inhabitants in recent years. This information has been gathered from various web sources.', role=<ChatRole.ASSISTANT: 'assistant'>, name=None, meta={'model': 'gpt-4o-mini-2024-07-18', 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 91, 'prompt_tokens': 401, 'total_tokens': 492, 'prompt_tokens_details': {'cached_tokens': 0, 'audio_tokens': 0}, 'completion_tokens_details': {'reasoning_tokens': 0, 'audio_tokens': 0, 'accepted_prediction_tokens': 0, 'rejected_prediction_tokens': 0}}})]\n"
     ]
    }
   ],
   "source": [
    "query = \"How many people live in Munich?\"\n",
    "\n",
    "result = pipe.run({\"prompt_builder\": {\"query\": query, \"documents\": documents}, \"router\": {\"query\": query}})\n",
    "\n",
    "# Print the `replies` generated using the web searched Documents\n",
    "print(result[\"llm_for_websearch\"][\"replies\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wUkuXoWnHa5c"
   },
   "source": [
    "If you check the whole result, you will see that `websearch` component also provides links to Documents retrieved from the web:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_EYLZguZGznY",
    "outputId": "df49a576-9961-44b4-e89d-2c5195869360"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'websearch': {'links': ['https://en.wikipedia.org/wiki/Munich',\n",
       "   'https://www.macrotrends.net/cities/204371/munich/population',\n",
       "   'https://www.britannica.com/place/Munich-Bavaria-Germany',\n",
       "   'https://en.wikipedia.org/wiki/Demographics_of_Munich',\n",
       "   'https://www.statista.com/statistics/505774/munich-population/',\n",
       "   'https://www.citypopulation.de/en/germany/bayern/m%C3%BCnchen_stadt/09162000__m%C3%BCnchen/',\n",
       "   'https://eurocities.eu/cities/munich/',\n",
       "   'https://www.coe.int/en/web/interculturalcities/munich',\n",
       "   'https://www.worldometers.info/world-population/germany-population/']},\n",
       " 'llm_for_websearch': {'replies': [ChatMessage(content='As of May 31, 2024, Munich has a population of approximately 1,594,632 residents, making it the third-largest city in Germany, after Berlin and Hamburg. The metro area population is about 1,585,000, reflecting a modest increase from the previous year. Additionally, estimates suggest that the city has consistently had over 1.5 million inhabitants in recent years. This information has been gathered from various web sources.', role=<ChatRole.ASSISTANT: 'assistant'>, name=None, meta={'model': 'gpt-4o-mini-2024-07-18', 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 91, 'prompt_tokens': 401, 'total_tokens': 492, 'prompt_tokens_details': {'cached_tokens': 0, 'audio_tokens': 0}, 'completion_tokens_details': {'reasoning_tokens': 0, 'audio_tokens': 0, 'accepted_prediction_tokens': 0, 'rejected_prediction_tokens': 0}}})]}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6nhdYK-vHpNM"
   },
   "source": [
    "## What's next\n",
    "\n",
    "🎉 Congratulations! You've built a pipeline with conditional routing! You can now customize the condition for your specific use case and create a custom Haystack 2.0 pipeline to meet your needs.\n",
    "\n",
    "If you liked this tutorial, there's more to learn about Haystack 2.0:\n",
    "- [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline)\n",
    "- [Model-Based Evaluation of RAG Pipelines](https://haystack.deepset.ai/tutorials/35_model_based_evaluation_of_rag_pipelines)\n",
    "\n",
    "To stay up to date on the latest Haystack developments, you can [sign up for our newsletter](https://landing.deepset.ai/haystack-community-updates) or [join Haystack discord community](https://discord.gg/haystack).\n",
    "\n",
    "Thanks for reading!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}