L1-Document_loading.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "adf0e2f4",
   "metadata": {},
   "source": [
    "# Document Loading"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3d1f5a4",
   "metadata": {},
   "source": [
    "## Note to students.\n",
    "During periods of high load you may find the notebook unresponsive. It may appear to execute a cell, update the completion number in brackets [#] at the left of the cell but you may find the cell has not executed. This is particularly obvious on print statements when there is no output. If this happens, restart the kernel using the command under the Kernel tab."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3038ef6a",
   "metadata": {},
   "source": [
    "## Retrieval augmented generation\n",
    "In retrieval augmented generation (RAG), an LLM retrieves contextual documents from an external dataset as part of its execution.\n",
    "\n",
    "This is useful if we want to ask question about specific documents (e.g., our PDFs, a set of videos, etc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "41284636",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! pip install langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9e0196b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import openai\n",
    "import sys\n",
    "sys.path.append('../..')\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv()) # read local .env file\n",
    "\n",
    "openai.api_key  = os.environ['OPENAI_API_KEY']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd8c5b67",
   "metadata": {},
   "source": [
    "## PDFs\n",
    "Let's load a PDF [transcript](https://see.stanford.edu/materials/aimlcs229/transcripts/MachineLearning-Lecture01.pdf) from Andrew Ng's famous CS229 course! These documents are the result of automated transcription so words and sentences are sometimes split unexpectedly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e21479aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The course will show the pip installs you would need to install packages on your own machine.\n",
    "# These packages are already installed on this platform and should not be run again.\n",
    "# ! pip install pypdf "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8c29cf24",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import PyPDFLoader\n",
    "loader = PyPDFLoader(\"docs/cs229_lectures/MachineLearning-Lecture01.pdf\")\n",
    "pages = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df8d3b72",
   "metadata": {},
   "source": [
    "Each page is a Document.\n",
    "\n",
    "A Document contains text (page_content) and metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "10979ef4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c498c1f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "page = pages[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e86cbec5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MachineLearning-Lecture01  \n",
      "Instructor (Andrew Ng):  Okay. Good morning. Welcome to CS229, the machine \n",
      "learning class. So what I wanna do today is ju st spend a little time going over the logistics \n",
      "of the class, and then we'll start to  talk a bit about machine learning.  \n",
      "By way of introduction, my name's  Andrew Ng and I'll be instru ctor for this class. And so \n",
      "I personally work in machine learning, and I' ve worked on it for about 15 years now, and \n",
      "I actually think that machine learning i\n"
     ]
    }
   ],
   "source": [
    "print(page.page_content[0:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "57c0dacc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 0}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2817343a",
   "metadata": {},
   "source": [
    "## YouTube"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b848b13f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders.generic import GenericLoader\n",
    "from langchain.document_loaders.parsers import OpenAIWhisperParser\n",
    "from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4bac025b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! pip install yt_dlp\n",
    "# ! pip install pydub"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41ea326e",
   "metadata": {},
   "source": [
    "**Note**: This can take several minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "358b0603",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[youtube] Extracting URL: https://www.youtube.com/watch?v=jGwO_UgTS7I\n",
      "[youtube] jGwO_UgTS7I: Downloading webpage\n",
      "[youtube] jGwO_UgTS7I: Downloading ios player API JSON\n",
      "[youtube] jGwO_UgTS7I: Downloading android player API JSON\n",
      "[youtube] jGwO_UgTS7I: Downloading m3u8 information\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: [youtube] Incomplete data received in embedded initial data; re-fetching using API.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[youtube] jGwO_UgTS7I: Downloading initial data API JSON\n",
      "[info] jGwO_UgTS7I: Downloading 1 format(s): 140\n",
      "[download] Destination: docs/youtube//Stanford CS229： Machine Learning Course, Lecture 1 - Andrew Ng (Autumn 2018).m4a\n",
      "[download] 100% of   69.76MiB in 00:00:02 at 29.04MiB/s    \n",
      "[FixupM4a] Correcting container of \"docs/youtube//Stanford CS229： Machine Learning Course, Lecture 1 - Andrew Ng (Autumn 2018).m4a\"\n",
      "[ExtractAudio] Not converting audio docs/youtube//Stanford CS229： Machine Learning Course, Lecture 1 - Andrew Ng (Autumn 2018).m4a; file is already in target format m4a\n",
      "Transcribing part 1!\n",
      "Transcribing part 2!\n",
      "Transcribing part 3!\n",
      "Transcribing part 4!\n"
     ]
    }
   ],
   "source": [
    "url=\"https://www.youtube.com/watch?v=jGwO_UgTS7I\"\n",
    "save_dir=\"docs/youtube/\"\n",
    "loader = GenericLoader(\n",
    "    YoutubeAudioLoader([url],save_dir),\n",
    "    OpenAIWhisperParser()\n",
    ")\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "caa99538",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Welcome to CS229 Machine Learning. Uh, some of you know that this is a class that's taught at Stanford for a long time. And this is often the class that, um, I most look forward to teaching each year because this is where we've helped, I think, several generations of Stanford students become experts in machine learning, got- built many of their products and services and startups that I'm sure, many of you or probably all of you are using, uh, uh, today. Um, so what I want to do today was spend s\""
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[0].page_content[0:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d02460",
   "metadata": {},
   "source": [
    "## URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "09434f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import WebBaseLoader\n",
    "\n",
    "loader = WebBaseLoader(\"https://github.com/basecamp/handbook/blob/master/37signals-is-you.md\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "668d6e1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b7708cdb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "handbook/37signals-is-you.md at master · basecamp/handbook · GitHub\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Skip to content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Toggle navigation\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "            Sign up\n",
      "          \n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "        Product\n",
      "        \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Actions\n",
      "        Automate any workflow\n",
      "      \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Packages\n",
      "        Host and manage packages\n",
      "      \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Security\n",
      "        Find and fix vulnerabilities\n",
      "      \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Cod\n"
     ]
    }
   ],
   "source": [
    "print(docs[0].page_content[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d90673a7",
   "metadata": {},
   "source": [
    "## Notion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f70a079e",
   "metadata": {},
   "source": [
    "Follow steps [here](https://python.langchain.com/docs/modules/data_connection/document_loaders/integrations/notion) for an example Notion site such as [this one](https://yolospace.notion.site/Blendle-s-Employee-Handbook-e31bff7da17346ee99f531087d8b133f):\n",
    "\n",
    "- Duplicate the page into your own Notion space and export as Markdown / CSV.\n",
    "- Unzip it and save it as a folder that contains the markdown file for the Notion page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcd29f62",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import NotionDirectoryLoader\n",
    "loader = NotionDirectoryLoader(\"docs/Notion_DB\")\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "415bc771",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(docs[0].page_content[0:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "899134d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs[0].metadata"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}