tutorials/17_Audio.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "Dne2XSNzB3SK"
   },
   "source": [
    "# Tutorial: Make Your QA Pipelines Talk!\n",
    "\n",
    "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n",
    ">\n",
    "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n",
    "\n",
    "- **Level**: Intermediate\n",
    "- **Time to complete**: 15 minutes\n",
    "- **Nodes Used**: `InMemoryDocumentStore`, `BM25Retriever`, `FARMReader`, `AnswerToSpeech`\n",
    "- **Goal**: After completing this tutorial, you'll have created a extractive question answering system that can read out the answer."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "Question answering works primarily on text, but Haystack provides some features for audio files that contain speech as well.\n",
    "\n",
    "In this tutorial, we're going to see how to use `AnswerToSpeech` to convert answers into audio files."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "4UBjfz4LB3SS"
   },
   "source": [
    "## Preparing the Colab Environment\n",
    "\n",
    "- [Enable GPU Runtime in Colab](https://docs.haystack.deepset.ai/docs/enabling-gpu-acceleration#enabling-the-gpu-in-colab)\n",
    "- [Set logging level to INFO](https://docs.haystack.deepset.ai/docs/log-level)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "nBvGUPVKN2oJ"
   },
   "source": [
    "## Installing Haystack\n",
    "\n",
    "To start, let's install the latest release of Haystack with `pip`. In this tutorial, we'll use components from [text2speech](https://github.com/deepset-ai/haystack-extras/tree/main/nodes/text2speech) which contains some extra Haystack components, so we'll install `farm-haystack-text2speech`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QsY0HC8JB3Sc"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "pip install --upgrade pip\n",
    "pip install farm-haystack[colab,preprocessing,inference]\n",
    "pip install farm-haystack-text2speech"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enabling Telemetry\n",
    "\n",
    "Knowing you're using this tutorial helps us decide where to invest our efforts to build a better product but you can always opt out by commenting the following line. See [Telemetry](https://docs.haystack.deepset.ai/docs/telemetry) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from haystack.telemetry import tutorial_running\n",
    "\n",
    "tutorial_running(17)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "pbGu92rAB3Sl"
   },
   "source": [
    "## Indexing Documents\n",
    "\n",
    "We will populate the document store with a simple indexing pipeline. See [Tutorial: Build Your First Question Answering System](https://haystack.deepset.ai/tutorials/01_basic_qa_pipeline) for more details about these steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eWYnP3nWB3So",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from haystack.document_stores import InMemoryDocumentStore\n",
    "from haystack.utils import fetch_archive_from_http\n",
    "from haystack.pipelines import Pipeline\n",
    "from haystack.nodes import FileTypeClassifier, TextConverter, PreProcessor\n",
    "\n",
    "# Initialize the DocumentStore\n",
    "document_store = InMemoryDocumentStore(use_bm25=True)\n",
    "\n",
    "# Get the documents\n",
    "documents_path = \"data/tutorial17\"\n",
    "s3_url = \"https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt17.zip\"\n",
    "fetch_archive_from_http(url=s3_url, output_dir=documents_path)\n",
    "\n",
    "# List all the paths\n",
    "file_paths = [p for p in Path(documents_path).glob(\"**/*\")]\n",
    "\n",
    "# NOTE: In this example we're going to use only one text file from the wiki\n",
    "file_paths = [p for p in file_paths if \"Stormborn\" in p.name]\n",
    "\n",
    "# Prepare some basic metadata for the files\n",
    "files_metadata = [{\"name\": path.name} for path in file_paths]\n",
    "\n",
    "# Makes sure the file is a TXT file (FileTypeClassifier node)\n",
    "classifier = FileTypeClassifier()\n",
    "\n",
    "# Converts a file into text and performs basic cleaning (TextConverter node)\n",
    "text_converter = TextConverter(remove_numeric_tables=True)\n",
    "\n",
    "# - Pre-processes the text by performing splits and adding metadata to the text (Preprocessor node)\n",
    "preprocessor = PreProcessor(clean_header_footer=True, split_length=200, split_overlap=20)\n",
    "\n",
    "# Here we create a basic indexing pipeline\n",
    "indexing_pipeline = Pipeline()\n",
    "indexing_pipeline.add_node(classifier, name=\"classifier\", inputs=[\"File\"])\n",
    "indexing_pipeline.add_node(text_converter, name=\"text_converter\", inputs=[\"classifier.output_1\"])\n",
    "indexing_pipeline.add_node(preprocessor, name=\"preprocessor\", inputs=[\"text_converter\"])\n",
    "indexing_pipeline.add_node(document_store, name=\"document_store\", inputs=[\"preprocessor\"])\n",
    "\n",
    "# Then we run it with the documents and their metadata as input\n",
    "indexing_pipeline.run(file_paths=file_paths, meta=files_metadata)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "zW5qaqn1B3St"
   },
   "source": [
    "## Creating a QA Pipeline with AnswerToSpeech\n",
    "   \n",
    "Now we will create a pipeline very similar to the basic `ExtractiveQAPipeline` of [Tutorial: Build Your First Question Answering System](https://haystack.deepset.ai/tutorials/01_basic_qa_pipeline), with the addition of a node that converts our answers into audio files: AnswerToSpeech. Once the answer is retrieved, we can also listen to the audio version of the document where the answer came from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "m_oecui1B3Sw"
   },
   "outputs": [],
   "source": [
    "from haystack.nodes import BM25Retriever, FARMReader\n",
    "from text2speech import AnswerToSpeech\n",
    "\n",
    "retriever = BM25Retriever(document_store=document_store)\n",
    "reader = FARMReader(model_name_or_path=\"deepset/roberta-base-squad2\", use_gpu=True)\n",
    "answer2speech = AnswerToSpeech(\n",
    "    model_name_or_path=\"espnet/kan-bayashi_ljspeech_vits\", generated_audio_dir=Path(\"./audio_answers\")\n",
    ")\n",
    "\n",
    "audio_pipeline = Pipeline()\n",
    "audio_pipeline.add_node(retriever, name=\"Retriever\", inputs=[\"Query\"])\n",
    "audio_pipeline.add_node(reader, name=\"Reader\", inputs=[\"Retriever\"])\n",
    "audio_pipeline.add_node(answer2speech, name=\"AnswerToSpeech\", inputs=[\"Reader\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "oV1KHzXGB3Sy"
   },
   "source": [
    "## Asking a question!\n",
    "\n",
    "Use the pipeline `run()` method to ask a question. The query argument is where you type your question. Additionally, you can set the number of documents you want the Reader and Retriever to return using the `top-k` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "S-ZMUBzpB3Sz",
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "prediction = audio_pipeline.run(\n",
    "    query=\"Who is the father of Arya Stark?\", params={\"Retriever\": {\"top_k\": 10}, \"Reader\": {\"top_k\": 5}}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vpFSxtNNB3S1"
   },
   "outputs": [],
   "source": [
    "# Now you can print prediction\n",
    "from pprint import pprint\n",
    "\n",
    "pprint(prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Xg6BN4v8N2oM"
   },
   "outputs": [],
   "source": [
    "# The document the first answer was extracted from\n",
    "original_document = [doc for doc in prediction[\"documents\"] if doc.id == prediction[\"answers\"][0].document_ids[0]][0]\n",
    "pprint(original_document)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "FXf-kTn4B3S6"
   },
   "source": [
    "## Hear Answers out!\n",
    "\n",
    "Let's hear the answers and the context they are extracted from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "cJJVpT7dB3S7"
   },
   "outputs": [],
   "source": [
    "from IPython.display import display, Audio\n",
    "import soundfile as sf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "usGVf1N6B3S8"
   },
   "outputs": [],
   "source": [
    "# The first answer in isolation\n",
    "\n",
    "print(\"Answer: \", prediction[\"answers\"][0].meta[\"answer_text\"])\n",
    "\n",
    "speech, _ = sf.read(prediction[\"answers\"][0].answer)\n",
    "display(Audio(speech, rate=24000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yTFwNJqtB3S9"
   },
   "outputs": [],
   "source": [
    "# The context of the first answer\n",
    "\n",
    "print(\"Context: \", prediction[\"answers\"][0].meta[\"context_text\"])\n",
    "\n",
    "speech, _ = sf.read(prediction[\"answers\"][0].context)\n",
    "display(Audio(speech, rate=24000))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🎉 Congratulations! You've learned how to create a extactive QA system that can read out the answer."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}