LangChain Cookbook Part 2 - Use Cases.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "359697d5",
   "metadata": {},
   "source": [
    "# LangChain Cookbook Part 2: Use Cases👨‍🍳👩‍🍳"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11d788b0",
   "metadata": {},
   "source": [
    "*This cookbook is based on the [LangChain Conceptual Documentation](https://docs.langchain.com/docs/)*\n",
    "\n",
    "**Goals:**\n",
    "\n",
    "1. Inspire you to build\n",
    "2. Provide an introductory understanding of the main use cases of LangChain via [ELI5](https://www.dictionary.com/e/slang/eli5/#:~:text=ELI5%20is%20short%20for%20%E2%80%9CExplain,a%20complicated%20question%20or%20problem.) examples and code snippets. For an introduction to the *fundamentals* of LangChain check out [Cookbook Part 1: Fundamentals](https://github.com/gkamradt/langchain-tutorials/blob/main/LangChain%20Cookbook%20Part%201%20-%20Fundamentals.ipynb).\n",
    "\n",
    "**LangChain Links:**\n",
    "* [LC Conceptual Documentation](https://docs.langchain.com/docs/)\n",
    "* [LC Python Documentation](https://python.langchain.com/en/latest/)\n",
    "* [LC Javascript/Typescript Documentation](https://js.langchain.com/docs/)\n",
    "* [LC Discord](https://discord.gg/6adMQxSpJS)\n",
    "* [www.langchain.com](https://langchain.com/)\n",
    "* [LC Twitter](https://twitter.com/LangChainAI)\n",
    "\n",
    "\n",
    "### **What is LangChain?**\n",
    "> LangChain is a framework for developing applications powered by language models.\n",
    "*[Source](https://blog.langchain.dev/announcing-our-10m-seed-round-led-by-benchmark/#:~:text=LangChain%20is%20a%20framework%20for%20developing%20applications%20powered%20by%20language%20models)*\n",
    "\n",
    "**TLDR**: LangChain makes the complicated parts of working & building with AI models easier. It helps do this in two ways:\n",
    "\n",
    "1. **Integration** - Bring external data, such as your files, other applications, and api data, to your LLMs\n",
    "2. **Agency** - Allow your LLMs to interact with its environment via decision making. Use LLMs to help decide which action to take next\n",
    "\n",
    "### **Why LangChain?**\n",
    "1. **Components** - LangChain makes it easy to swap out abstractions and components necessary to work with language models.\n",
    "\n",
    "2. **Customized Chains** - LangChain provides out of the box support for using and customizing 'chains' - a series of actions strung together.\n",
    "\n",
    "3. **Speed 🚢** - This team ships insanely fast. You'll be up to date with the latest LLM features.\n",
    "\n",
    "4. **Community 👥** - Wonderful [discord](https://discord.gg/6adMQxSpJS) and community support, meet ups, hackathons, etc.\n",
    "\n",
    "Though LLMs can be straightforward (text-in, text-out) you'll quickly run into friction points that LangChain helps with once you develop more complicated applications.\n",
    "\n",
    "### **Main Use Cases**\n",
    "\n",
    "* **Summarization** - Express the most important facts about a body of text or chat interaction\n",
    "* **Question and Answering Over Documents** - Use information held within documents to answer questions or query\n",
    "* **Extraction** - Pull structured data from a body of text or an user query\n",
    "* **Evaluation** - Understand the quality of output from your application\n",
    "* **Querying Tabular Data** - Pull data from databases or other tabular source\n",
    "* **Code Understanding** - Reason about and digest code\n",
    "* **Interacting with APIs** - Query APIs and interact with the outside world\n",
    "* **Chatbots** - A framework to have a back and forth interaction with a user combined with memory in a chat interface\n",
    "* **Agents** - Use LLMs to make decisions about what to do next. Enable these decisions with tools.\n",
    "\n",
    "Want to see live examples of these use cases? Head over to the [LangChain Project Gallery](https://github.com/gkamradt/langchain-tutorials)\n",
    "\n",
    "#### **Authors Note:**\n",
    "\n",
    "* This cookbook will not cover all aspects of LangChain. It's contents have been curated to get you to building & impact as quick as possible. For more, please check out [LangChain Technical Documentation](https://python.langchain.com/en/latest/index.html)\n",
    "* This notebook assumes is that you've seen part 1 of this series [Fundamentals](https://github.com/gkamradt/langchain-tutorials/blob/main/LangChain%20Cookbook%20Part%201%20-%20Fundamentals.ipynb). This notebook is focused on what to do and how to apply those fundamentals.\n",
    "* You'll notice I repeat import statements throughout the notebook. My intention is to lean on the side of clarity and help you see the full code block in one spot. No need to go back and forth to see when we imported a package.\n",
    "* We use the default models throughout the notebook, at the time of writing they were davinci-003 and gpt-3.5-turbo. You would no doubt get better results with GPT4\n",
    "\n",
    "Let's get started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e323fb6",
   "metadata": {},
   "source": [
    "Throughout this tutorial we will use OpenAI's various [models](https://platform.openai.com/docs/models/overview). LangChain makes it easy to [subsistute LLMs](https://langchain.com/integrations.html#:~:text=integrations%20LangChain%20provides.-,LLMs,-LLM%20Provider) so you can BYO-LLM if you want"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e9815081",
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "import os\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "openai_api_key = os.getenv('OPENAI_API_KEY', 'YourAPIKeyIfNotSet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dcd3587c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container { width:90% !important; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run this cell if you want to make your display wider\n",
    "from IPython.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:90% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05bb564d",
   "metadata": {},
   "source": [
    "# LangChain Use Cases"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bbdb1dc",
   "metadata": {},
   "source": [
    "## Summarization\n",
    "\n",
    "One of the most common use cases for LangChain and LLMs is summarization. You can summarize any piece of text, but use cases span from summarizing calls, articles, books, academic papers, legal documents, user history, a table, or financial documents. It's super helpful to have a tool which can summarize information quickly.\n",
    "\n",
    "* **Deep Dive** - (Coming Soon)\n",
    "* **Examples** - [Summarizing B2B Sales Calls](https://www.youtube.com/watch?v=DIw4rbpI9ic)\n",
    "* **Use Cases** - Summarize Articles, Transcripts, Chat History, Slack/Discord, Customer Interactions, Medical Papers, Legal Documents, Podcasts, Tweet Threads, Code Bases, Product Reviews, Financial Documents\n",
    "\n",
    "### Summaries Of Short Text\n",
    "\n",
    "For summaries of short texts, the method is straightforward, in fact you don't need to do anything fancy other than simple prompting with instructions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a49a7c9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "GEMINI_API = \"AIzaSyDYIEW4XVSeuMaVlcmXgv2rqI20jqUolwk\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0c292592",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "from langchain import PromptTemplate\n",
    "\n",
    "# Initialize the Google Generative AI model\n",
    "llm = ChatGoogleGenerativeAI(\n",
    "    model=\"gemini-pro\",\n",
    "    temperature=0,\n",
    "    google_api_key=GEMINI_API\n",
    ")\n",
    "\n",
    "# Create our template (this stays the same)\n",
    "template = \"\"\"\n",
    "%INSTRUCTIONS:\n",
    "Please summarize the following piece of text.\n",
    "Respond in a manner that a 5 year old would understand.\n",
    "\n",
    "%TEXT:\n",
    "{text}\n",
    "\"\"\"\n",
    "\n",
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"text\"],\n",
    "    template=template,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f539cb53",
   "metadata": {},
   "source": [
    "Let's let's find a confusing text online. *[Source](https://www.smithsonianmag.com/smart-news/long-before-trees-overtook-the-land-earth-was-covered-by-giant-mushrooms-13709647/)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0df2cde6",
   "metadata": {},
   "outputs": [],
   "source": [
    "confusing_text = \"\"\"\n",
    "For the next 130 years, debate raged.\n",
    "Some scientists called Prototaxites a lichen, others a fungus, and still others clung to the notion that it was some kind of tree.\n",
    "“The problem is that when you look up close at the anatomy, it’s evocative of a lot of different things, but it’s diagnostic of nothing,” says Boyce, an associate professor in geophysical sciences and the Committee on Evolutionary Biology.\n",
    "“And it’s so damn big that when whenever someone says it’s something, everyone else’s hackles get up: ‘How could you have a lichen 20 feet tall?’”\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03d31842",
   "metadata": {},
   "source": [
    "Let's take a look at what prompt will be sent to the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "406eb8a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------- Prompt Begin -------\n",
      "\n",
      "%INSTRUCTIONS:\n",
      "Please summarize the following piece of text.\n",
      "Respond in a manner that a 5 year old would understand.\n",
      "\n",
      "%TEXT:\n",
      "\n",
      "For the next 130 years, debate raged.\n",
      "Some scientists called Prototaxites a lichen, others a fungus, and still others clung to the notion that it was some kind of tree.\n",
      "“The problem is that when you look up close at the anatomy, it’s evocative of a lot of different things, but it’s diagnostic of nothing,” says Boyce, an associate professor in geophysical sciences and the Committee on Evolutionary Biology.\n",
      "“And it’s so damn big that when whenever someone says it’s something, everyone else’s hackles get up: ‘How could you have a lichen 20 feet tall?’”\n",
      "\n",
      "\n",
      "------- Prompt End -------\n"
     ]
    }
   ],
   "source": [
    "print (\"------- Prompt Begin -------\")\n",
    "\n",
    "final_prompt = prompt.format(text=confusing_text)\n",
    "print(final_prompt)\n",
    "\n",
    "print (\"------- Prompt End -------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a95e53d9",
   "metadata": {},
   "source": [
    "Finally let's pass it through the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bc7e4b42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Imagine a giant mystery plant that lived long, long ago. Scientists couldn't decide what it was. Some said it was like a green carpet (lichen), others said it was like a mushroom (fungus), and some even thought it was a big tree! But the plant was so big and strange that everyone argued about what it really was.\n"
     ]
    }
   ],
   "source": [
    "output = llm.invoke(final_prompt)\n",
    "print(output.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "751c6359",
   "metadata": {},
   "source": [
    "This method works fine, but for longer text, it can become a pain to manage and you'll run into token limits. Luckily LangChain has out of the box support for different methods to summarize via their [load_summarize_chain](https://python.langchain.com/en/latest/use_cases/summarization.html).\n",
    "\n",
    "### Summaries Of Longer Text\n",
    "\n",
    "*Note: This method will also work for short text too*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3441484b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "from langchain.chains.summarize import load_summarize_chain\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "llm = ChatGoogleGenerativeAI(\n",
    "    model=\"gemini-pro\",\n",
    "    temperature=0,\n",
    "    google_api_key=GEMINI_API\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e95b575c",
   "metadata": {},
   "source": [
    "Let's load up a longer document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "6c33f9bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 \n",
      " \n",
      "CITY OF MOUNTAIN VIEW \n",
      "HEARING OFFICER DECISION PURSUANT TO \n",
      "THE COMMUNITY STABILIZATION AND FAIR RENT ACT (“CSFRA”) \n",
      " \n",
      "Rental Housing Committee Case No.: C22230055, C22230056 \n",
      "Address and Unit(s) of Rental \n",
      "Property: \n",
      "1556 California Street, Unit  \n",
      "Mountain View, CA 94041 \n",
      "Petit\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import PyPDFLoader\n",
    "\n",
    "# Load the PDF file\n",
    "loader = PyPDFLoader(\"/nas/ucb/davidyang/legal-rent/petition-decisions/California_1556 2023.11.21 HODecision_Redacted.pdf\")  # Adjust path to your PDF\n",
    "pages = loader.load()\n",
    "\n",
    "# Combine all pages into one text\n",
    "text = \" \".join([page.page_content for page in pages])\n",
    "\n",
    "# Print preview of first 285 characters\n",
    "print(text[:285])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd627d63",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b489d2a2",
   "metadata": {},
   "source": [
    "Then let's check how many tokens are in this document. [get_num_tokens](https://python.langchain.com/en/latest/reference/modules/llms.html#langchain.llms.OpenAI.get_num_tokens) is a nice method for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5e0e8181",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 27229 tokens in your file\n"
     ]
    }
   ],
   "source": [
    "num_tokens = llm.get_num_tokens(text)\n",
    "\n",
    "print (f\"There are {num_tokens} tokens in your file\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bf8eda6",
   "metadata": {},
   "source": [
    "While you could likely stuff this text in your prompt, let's act like it's too big and needs another method.\n",
    "\n",
    "First we'll need to split it up. This process is called 'chunking' or 'splitting' your text into smaller pieces. I like the [RecursiveCharacterTextSplitter](https://python.langchain.com/en/latest/modules/indexes/text_splitters/examples/recursive_text_splitter.html) because it's easy to control but there are a [bunch](https://python.langchain.com/en/latest/modules/indexes/text_splitters.html) you can try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "25dd80dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You now have 21 docs intead of 1 piece of text\n"
     ]
    }
   ],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(separators=[\"\\n\\n\", \"\\n\"], chunk_size=5000, chunk_overlap=350)\n",
    "docs = text_splitter.create_documents([text])\n",
    "\n",
    "print (f\"You now have {len(docs)} docs intead of 1 piece of text\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e7547a3",
   "metadata": {},
   "source": [
    "Next we need to load up a chain which will make successive calls to the LLM for us. Want to see the prompt being used in the chain below? Check out the [LangChain documentation](https://github.com/hwchase17/langchain/blob/master/langchain/chains/summarize/map_reduce_prompt.py)\n",
    "\n",
    "For information on the difference between chain types, check out this video on [token limit workarounds](https://youtu.be/f9_BWhCI4Zo)\n",
    "\n",
    "*Note: You could also get fancy and make the first 4 calls of the map_reduce run in parallel too*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "28ddd9c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get your chain ready to use\n",
    "chain = load_summarize_chain(llm=llm, chain_type='map_reduce') # verbose=True optional to see what is getting sent to the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "be0b2d04",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3286420/3408919681.py:2: LangChainDeprecationWarning: The method `Chain.run` was deprecated in langchain 0.1.0 and will be removed in 1.0. Use :meth:`~invoke` instead.\n",
      "  output = chain.run(docs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In Mountain View, CA, tenant Oralia Zavala Vasquez won a dispute against her landlord, Sergio Sanchez Morado, regarding uninhabitable conditions and unlawful rent increases.  The Hearing Officer ruled Morado owes Zavala $8,217.06, combining damages for overcharges since April 2021 ($1500) and rent reductions due to numerous unresolved habitability issues (mold, faulty appliances, plumbing problems, etc.) totaling $6,717.06. Zavala's rent reverts to the lawful $850, and the damages will be paid via monthly rent credits, reducing her payments to $103 for eleven months. Morado cannot raise rent on any unit until he refunds all tenants any unlawful rent increases and complies with the City of Santa Monica Fair Rent Act (CSFRA).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Use it. This will run through the 4 documents, summarize the chunks, then get a summary of the summary.\n",
    "output = chain.run(docs)\n",
    "print (output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2d664fc",
   "metadata": {},
   "source": [
    "## Question & Answering Using Documents As Context"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad87c72b",
   "metadata": {},
   "source": [
    "*[LangChain Question & Answer Docs](https://python.langchain.com/en/latest/use_cases/question_answering.html)*\n",
    "\n",
    "In order to use LLMs for question and answer we must:\n",
    "\n",
    "1. Pass the LLM relevant context it needs to answer a question\n",
    "2. Pass it our question that we want answered\n",
    "\n",
    "Simplified, this process looks like this \"llm(your context + your question) = your answer\"\n",
    "\n",
    "* **Deep Dive** - [Question A Book](https://youtu.be/h0DHDp1FbmQ), [Ask Questions To Your Custom Files](https://youtu.be/EnT-ZTrcPrg), [Chat Your Data JS (1000 pages of Financial Reports)](https://www.youtube.com/watch?v=Ix9WIZpArm0&t=1051s), [LangChain Q&A webinar](https://www.crowdcast.io/c/rh66hcwivly0)\n",
    "* **Examples** - [ChatPDF](https://www.chatpdf.com/)\n",
    "* **Use Cases** - Chat your documents, ask questions to academic papers, create study guides, reference medical information"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "685e15f3",
   "metadata": {},
   "source": [
    "### Simple Q&A Example\n",
    "\n",
    "Here let's review the convention of `llm(your context + your question) = your answer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "9ebd8451",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "\n",
    "llm = ChatGoogleGenerativeAI(\n",
    "    model=\"gemini-1.5-flash\",\n",
    "    temperature=0,\n",
    "    google_api_key=GEMINI_API\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "b4795187",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"\n",
    "Rachel is 30 years old\n",
    "Bob is 45 years old\n",
    "Kevin is 65 years old\n",
    "\"\"\"\n",
    "\n",
    "question = \"Who is under 40 years old?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2184b11b",
   "metadata": {},
   "source": [
    "Then combine them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "0c53650d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Only **Rachel** is under 40 years old.\n"
     ]
    }
   ],
   "source": [
    "output = llm.invoke(context + question)\n",
    "\n",
    "# I strip the text to remove the leading and trailing whitespace\n",
    "print (output.content.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "385180ca",
   "metadata": {},
   "source": [
    "As we ramp up our sophistication, we'll take advantage of this convention more.\n",
    "\n",
    "The hard part comes in when you need to be selective about *which* data you put in your context. This field of study is called \"[document retrieval](https://python.langchain.com/en/latest/modules/indexes/retrievers.html)\" and tightly coupled with AI Memory."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53ed4080",
   "metadata": {},
   "source": [
    "### Using Embeddings\n",
    "\n",
    "I informally call what were about to go through as \"The VectorStore Dance\". It's the process of splitting your text, embedding the chunks, putting the embeddings in a DB, and then querying them. For a full video on this check out [How To Question A Book](https://www.youtube.com/watch?v=h0DHDp1FbmQ)\n",
    "\n",
    "The goal is to select relevant chunks of our long text, but which chunks do we pull? The most popular method is to pull *similar* texts based off comparing vector embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "a7a02ccc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "\n",
    "# The vectorstore we'll be using\n",
    "from langchain.vectorstores import FAISS\n",
    "\n",
    "# The LangChain component we'll use to get the documents\n",
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "# The easy document loader for text\n",
    "from langchain.document_loaders import TextLoader\n",
    "\n",
    "# The embedding engine that will use Google's embeddings\n",
    "from langchain_google_genai import GoogleGenerativeAIEmbeddings\n",
    "\n",
    "# Initialize the Google Generative AI model\n",
    "llm = ChatGoogleGenerativeAI(\n",
    "    model=\"gemini-1.5-flash\",\n",
    "    temperature=0,\n",
    "    google_api_key=GEMINI_API\n",
    ")\n",
    "\n",
    "# Initialize Google's embeddings\n",
    "embeddings = GoogleGenerativeAIEmbeddings(\n",
    "    model=\"models/embedding-001\",  # Google's text embedding model\n",
    "    google_api_key=GEMINI_API\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40afcfec",
   "metadata": {},
   "source": [
    "Let's load up a longer document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "5772bc26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have 34 document\n",
      "You have 1662 characters in that document\n"
     ]
    }
   ],
   "source": [
    "loader = PyPDFLoader(\"/nas/ucb/davidyang/legal-rent/petition-decisions/California_1556 2023.11.21 HODecision_Redacted.pdf\")  # Adjust path to your PDF\n",
    "doc = loader.load()\n",
    "\n",
    "\n",
    "# Combine all pages into one text\n",
    "#doc = \" \".join([page.page_content for page in pages])\n",
    "print (f\"You have {len(doc)} document\")\n",
    "print (f\"You have {len(doc[0].page_content)} characters in that document\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb87424c",
   "metadata": {},
   "source": [
    "Now let's split our long doc into smaller pieces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "b4a6e452",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=3000, chunk_overlap=400)\n",
    "docs = text_splitter.split_documents(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "723e8aec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Now you have 49 documents that have an average of 2,036 characters (smaller pieces)\n"
     ]
    }
   ],
   "source": [
    "# Get the total number of characters so we can see the average later\n",
    "num_total_characters = sum([len(x.page_content) for x in docs])\n",
    "\n",
    "print (f\"Now you have {len(docs)} documents that have an average of {num_total_characters / len(docs):,.0f} characters (smaller pieces)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "9b591198",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get your embeddings engine ready using Google's embeddings\n",
    "embeddings = GoogleGenerativeAIEmbeddings(\n",
    "    model=\"models/embedding-001\",  # Google's text embedding model\n",
    "    google_api_key=GEMINI_API\n",
    ")\n",
    "docsearch = FAISS.from_documents(docs, embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1b13348",
   "metadata": {},
   "source": [
    "Create your retrieval engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "47cd969d",
   "metadata": {},
   "outputs": [],
   "source": [
    "qa = RetrievalQA.from_chain_type(llm=llm, chain_type=\"stuff\", retriever=docsearch.as_retriever())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aa2963c",
   "metadata": {},
   "source": [
    "Now it's time to ask a question. The retriever will go get the similar documents and combine with your question for the LLM to reason through.\n",
    "\n",
    "Note: It may not seem like much, but the magic here is that we didn't have to pass in our full original document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "6a062c85",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The document provided is a Hearing Officer Decision pursuant to the Community Stabilization and Fair Rent Act (CSFRA). It details a case between a tenant, Oralia Belem Zavala Vasquez, and a landlord, Sergio Sanchez Morado, regarding a rent adjustment and failure to maintain habitable premises. \\n\\nWhile the document doesn't explicitly state who won, it lays out the following rulings:\\n\\n* **Rent Increase Restrictions:** The landlord is prohibited from issuing a rent increase until all refunds or rent credits due to the tenant are paid and the tenant is given 30 days' advance notice.\\n* **Habitable Premises:** The landlord is required to maintain the property in habitable condition, including making repairs ordered by the City Building Department.\\n* **Specific Repairs:** The document lists specific repairs that need to be made, including fixing a leaking shower, addressing rat infestations in common areas, and ensuring adequate heating.\\n\\n**Speculation on Outcome:** Based on the rulings, it seems likely that the tenant, Oralia Belem Zavala Vasquez, won at least some aspects of the case. The landlord is restricted from raising rent until certain conditions are met, and is required to make specific repairs. However, the document doesn't explicitly state whether the tenant received a downward rent adjustment or other specific remedies. \\n\""
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"Who won the case, or speculate on the outcome? What was ruled?\"\n",
    "qa.run(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be503d53",
   "metadata": {},
   "source": [
    "If you wanted to do more you would hook this up to a cloud vector database, use a tool like metal and start managing your documents, with external data sources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3d04dc9",
   "metadata": {},
   "source": [
    "## Extraction\n",
    "*[LangChain Extraction Docs](https://python.langchain.com/en/latest/use_cases/extraction.html)*\n",
    "\n",
    "Extraction is the process of parsing data from a piece of text. This is commonly used with output parsing in order to *structure* our data.\n",
    "\n",
    "* **Deep Dive** - [Use LLMs to Extract Data From Text (Expert Level Text Extraction](https://youtu.be/xZzvwR9jdPA), [Structured Output From OpenAI (Clean Dirty Data)](https://youtu.be/KwAXfey-xQk)\n",
    "* **Examples** - [OpeningAttributes](https://twitter.com/GregKamradt/status/1646500373837008897)\n",
    "* **Use Cases:** Extract a structured row from a sentence to insert into a database, extract multiple rows from a long document to insert into a database, extracting parameters from a user query to make an API call\n",
    "\n",
    "A popular library for extraction is [Kor](https://eyurtsev.github.io/kor/). We won't cover it today but I highly suggest checking it out for advanced extraction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "904d43c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To help construct our Chat Messages\n",
    "from langchain.schema import HumanMessage\n",
    "from langchain.prompts import PromptTemplate, ChatPromptTemplate, HumanMessagePromptTemplate\n",
    "\n",
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "\n",
    "# To parse outputs and get structured data back\n",
    "from langchain.output_parsers import StructuredOutputParser, ResponseSchema\n",
    "\n",
    "chat_model = ChatGoogleGenerativeAI(\n",
    "    model=\"gemini-1.5-pro\",  # Using Gemini Pro model\n",
    "    temperature=1,\n",
    "    google_api_key=GEMINI_API\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6923ca8b",
   "metadata": {},
   "source": [
    "### Vanilla Extraction\n",
    "\n",
    "Let's start off with an easy example. Here I simply supply a prompt with instructions with the type of output I want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "ab1cce97",
   "metadata": {},
   "outputs": [],
   "source": [
    "instructions = \"\"\"\n",
    "You will be given a sentence with fruit names, extract those fruit names and assign an emoji to them\n",
    "Return the fruit name and emojis in ONLY a python dictionary\n",
    "\"\"\"\n",
    "\n",
    "fruit_names = \"\"\"\n",
    "Apple, Pear, this is an kiwi\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "38f16ea4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3286420/4238075527.py:5: LangChainDeprecationWarning: The method `BaseChatModel.__call__` was deprecated in langchain-core 0.1.7 and will be removed in 1.0. Use :meth:`~invoke` instead.\n",
      "  output = chat_model([HumanMessage(content=prompt)])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "```python\n",
      "import re\n",
      "\n",
      "def fruit_emoji_dict(sentence):\n",
      "    \"\"\"\n",
      "    Extracts fruit names from a sentence and assigns emojis.\n",
      "    Returns a dictionary of fruit names and their corresponding emojis.\n",
      "    \"\"\"\n",
      "\n",
      "    fruit_emojis = {\n",
      "        \"apple\": \"🍎\",\n",
      "        \"pear\": \"🍐\",\n",
      "        \"kiwi\": \"🥝\",\n",
      "        \"banana\": \"🍌\",\n",
      "        \"orange\": \"🍊\",\n",
      "        \"grape\": \"🍇\",\n",
      "        \"strawberry\": \"🍓\",\n",
      "        \"watermelon\": \"🍉\",\n",
      "        \"pineapple\": \"🍍\",\n",
      "        # Add more fruits and emojis as needed\n",
      "    }\n",
      "\n",
      "    # Use regex to find potential fruit names (case-insensitive)\n",
      "    words = re.findall(r'\\b\\w+\\b', sentence.lower())\n",
      "\n",
      "    fruit_dict = {}\n",
      "    for word in words:\n",
      "        if word in fruit_emojis:\n",
      "            fruit_dict[word] = fruit_emojis[word]\n",
      "\n",
      "    return fruit_dict\n",
      "\n",
      "\n",
      "sentence = \"Apple, Pear, this is an kiwi\"\n",
      "result = fruit_emoji_dict(sentence)\n",
      "print(result)  # Output: {'apple': '🍎', 'pear': '🍐', 'kiwi': '🥝'}\n",
      "\n",
      "\n",
      "sentence2 = \"I like Apple, Banana, and Orange.\"\n",
      "result2 = fruit_emoji_dict(sentence2)\n",
      "print(result2) # Output: {'apple': '🍎', 'banana': '🍌', 'orange': '🍊'}\n",
      "\n",
      "```\n"
     ]
    }
   ],
   "source": [
    "# Make your prompt which combines the instructions w/ the fruit names\n",
    "prompt = (instructions + fruit_names)\n",
    "\n",
    "# Call the LLM\n",
    "output = chat_model([HumanMessage(content=prompt)])\n",
    "\n",
    "print (output.content)\n",
    "#print (type(output.content))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39d6cff3",
   "metadata": {},
   "source": [
    "Let's turn this into a proper python dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "6d40a84a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'```python\\nimport re\\n\\ndef fruit_emoji_dict(sentence):\\n    \"\"\"\\n    Extracts fruit names from a sentence and assigns emojis.\\n    Returns a dictionary of fruit names and their corresponding emojis.\\n    \"\"\"\\n\\n    fruit_emojis = {\\n        \"apple\": \"🍎\",\\n        \"pear\": \"🍐\",\\n        \"kiwi\": \"🥝\",\\n        \"banana\": \"🍌\",\\n        \"orange\": \"🍊\",\\n        \"grape\": \"🍇\",\\n        \"strawberry\": \"🍓\",\\n        \"watermelon\": \"🍉\",\\n        \"pineapple\": \"🍍\",\\n        # Add more fruits and emojis as needed\\n    }\\n\\n    # Use regex to find potential fruit names (case-insensitive)\\n    words = re.findall(r\\'\\\\b\\\\w+\\\\b\\', sentence.lower())\\n\\n    fruit_dict = {}\\n    for word in words:\\n        if word in fruit_emojis:\\n            fruit_dict[word] = fruit_emojis[word]\\n\\n    return fruit_dict\\n\\n\\nsentence = \"Apple, Pear, this is an kiwi\"\\nresult = fruit_emoji_dict(sentence)\\nprint(result)  # Output: {\\'apple\\': \\'🍎\\', \\'pear\\': \\'🍐\\', \\'kiwi\\': \\'🥝\\'}\\n\\n\\nsentence2 = \"I like Apple, Banana, and Orange.\"\\nresult2 = fruit_emoji_dict(sentence2)\\nprint(result2) # Output: {\\'apple\\': \\'🍎\\', \\'banana\\': \\'🍌\\', \\'orange\\': \\'🍊\\'}\\n\\n```'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "314286b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "#output_dict = eval(output.content)\n",
    "\n",
    "#print (output.content)\n",
    "#print (type(output_dict))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3909eb29",
   "metadata": {},
   "source": [
    "While this worked this time, it's not a long term reliable method for more advanced use cases"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6a0a90d",
   "metadata": {},
   "source": [
    "### Using LangChain's Response Schema\n",
    "\n",
    "LangChain's response schema will does two things for us: \n",
    "\n",
    "1. Autogenerate the a prompt with bonafide format instructions. This is great because I don't need to worry about the prompt engineering side, I'll leave that up to LangChain!\n",
    "\n",
    "2. Read the output from the LLM and turn it into a proper python object for me\n",
    "\n",
    "Here I define the schema I want. I'm going to pull out the song and artist that a user wants to play from a pseudo chat message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "dc2ba0be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The schema I want out\n",
    "response_schemas = [\n",
    "    ResponseSchema(name=\"artist\", description=\"The name of the musical artist\"),\n",
    "    ResponseSchema(name=\"song\", description=\"The name of the song that the artist plays\")\n",
    "]\n",
    "\n",
    "# The parser that will look for the LLM output in my schema and return it back to me\n",
    "output_parser = StructuredOutputParser.from_response_schemas(response_schemas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "f9e3c6cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The output should be a markdown code snippet formatted in the following schema, including the leading and trailing \"```json\" and \"```\":\n",
      "\n",
      "```json\n",
      "{\n",
      "\t\"artist\": string  // The name of the musical artist\n",
      "\t\"song\": string  // The name of the song that the artist plays\n",
      "}\n",
      "```\n"
     ]
    }
   ],
   "source": [
    "# The format instructions that LangChain makes. Let's look at them\n",
    "format_instructions = output_parser.get_format_instructions()\n",
    "print(format_instructions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "d702900c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The prompt template that brings it all together\n",
    "# Note: This is a different prompt template than before because we are using a Chat Model\n",
    "\n",
    "prompt = ChatPromptTemplate(\n",
    "    messages=[\n",
    "        HumanMessagePromptTemplate.from_template(\"Given a command from the user, extract the artist and song names \\n \\\n",
    "                                                    {format_instructions}\\n{user_prompt}\")  \n",
    "    ],\n",
    "    input_variables=[\"user_prompt\"],\n",
    "    partial_variables={\"format_instructions\": format_instructions}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "bb6adde9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Given a command from the user, extract the artist and song names \n",
      "                                                     The output should be a markdown code snippet formatted in the following schema, including the leading and trailing \"```json\" and \"```\":\n",
      "\n",
      "```json\n",
      "{\n",
      "\t\"artist\": string  // The name of the musical artist\n",
      "\t\"song\": string  // The name of the song that the artist plays\n",
      "}\n",
      "```\n",
      "I really like So Young by Portugal. The Man\n"
     ]
    }
   ],
   "source": [
    "fruit_query = prompt.format_prompt(user_prompt=\"I really like So Young by Portugal. The Man\")\n",
    "print (fruit_query.messages[0].content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "b8664302",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'artist': 'Portugal. The Man', 'song': 'So Young'}\n",
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "fruit_output = chat_model(fruit_query.to_messages())\n",
    "output = output_parser.parse(fruit_output.content)\n",
    "\n",
    "print (output)\n",
    "print (type(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b68b8eeb",
   "metadata": {},
   "source": [
    "Awesome, now we have a dictionary that we can use later down the line\n",
    "\n",
    "<span style=\"background:#fff5d6\">Warning:</span> The parser looks for an output from the LLM in a specific format. Your model may not output the same format every time. Make sure to handle errors with this one. GPT4 and future iterations will be more reliable.\n",
    "\n",
    "For more advanced parsing check out [Kor](https://eyurtsev.github.io/kor/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2fb4ba6",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "*[LangChain Evaluation Docs](https://python.langchain.com/en/latest/use_cases/evaluation.html)*\n",
    "\n",
    "Evaluation is the process of doing quality checks on the output of your applications. Normal, deterministic, code has tests we can run, but judging the output of LLMs is more difficult because of the unpredictableness and variability of natural language. LangChain provides tools that aid us in this journey.\n",
    "\n",
    "* **Deep Dive** - Coming Soon\n",
    "* **Examples** - [Lance Martin's Advanced](https://twitter.com/RLanceMartin) [Auto-Evaluator](https://github.com/rlancemartin/auto-evaluator)\n",
    "* **Use Cases:** Run quality checks on your summarization or Question & Answer pipelines, check the output of you summarization pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "9fbaa6e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Embeddings, store, and retrieval\n",
    "from langchain_google_genai import GoogleGenerativeAIEmbeddings\n",
    "from langchain.vectorstores import FAISS\n",
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "# Model and doc loader\n",
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "from langchain.document_loaders import TextLoader\n",
    "\n",
    "# Eval!\n",
    "from langchain.evaluation.qa import QAEvalChain\n",
    "\n",
    "# Initialize the Google Generative AI model\n",
    "llm = ChatGoogleGenerativeAI(\n",
    "    model=\"gemini-1.5-pro\",\n",
    "    temperature=0,\n",
    "    google_api_key=GEMINI_API\n",
    ")\n",
    "\n",
    "# Initialize embeddings (you'll need this later)\n",
    "embeddings = GoogleGenerativeAIEmbeddings(\n",
    "    model=\"models/embedding-001\",\n",
    "    google_api_key=GEMINI_API\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "9f35fa12",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have 34 document\n",
      "You have 1662 characters in that document\n"
     ]
    }
   ],
   "source": [
    "# Our long essay from before\n",
    "loader = PyPDFLoader(\"/nas/ucb/davidyang/legal-rent/petition-decisions/California_1556 2023.11.21 HODecision_Redacted.pdf\")  # Adjust path to your PDF\n",
    "doc = loader.load()\n",
    "\n",
    "print (f\"You have {len(doc)} document\")\n",
    "print (f\"You have {len(doc[0].page_content)} characters in that document\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7acca7da",
   "metadata": {},
   "source": [
    "First let's do the Vectorestore dance so we can do question and answers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "1955faef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Now you have 49 documents that have an average of 2,036 characters (smaller pieces)\n"
     ]
    }
   ],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=3000, chunk_overlap=400)\n",
    "docs = text_splitter.split_documents(doc)\n",
    "\n",
    "# Get the total number of characters so we can see the average later\n",
    "num_total_characters = sum([len(x.page_content) for x in docs])\n",
    "\n",
    "print (f\"Now you have {len(docs)} documents that have an average of {num_total_characters / len(docs):,.0f} characters (smaller pieces)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "890b85ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Embeddings and docstore\n",
    "#embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)\n",
    "docsearch = FAISS.from_documents(docs, embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a0d6e25",
   "metadata": {},
   "source": [
    "Make your retrieval chain. Notice how I have an `input_key` parameter now. This tells the chain which key from a dictionary I supply has my prompt/query in it. I specify `question` to match the question in the dict below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "ddb3f3b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = RetrievalQA.from_chain_type(llm=llm, chain_type=\"stuff\", retriever=docsearch.as_retriever(), input_key=\"question\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b37dd0cd",
   "metadata": {},
   "source": [
    "Now I'll pass a list of questions and ground truth answers to the LLM that I know are correct (I validated them as a human)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "d93d08bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "question_answers = [\n",
    "    {'question' : \"What was the beginning rent per month?\", 'answer' : '825'},\n",
    "    {'question' : \"What was the price of the first rent which the tenant refused to pay?\", 'answer' : '1200'},\n",
    "    {'question' : \"Whose cat suffered harm? (tenant or landlord)\", 'answer' : 'The tenant'}\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98c4b591",
   "metadata": {},
   "source": [
    "I'll use `chain.apply` to run both my questions one by one separately.\n",
    "\n",
    "One of the cool parts is that I'll get my list of question and answers dictionaries back, but there'll be another key in the dictionary `result` which will be the output from the LLM.\n",
    "\n",
    "Note: I specifically made my 2nd question ambigious and tough to answer in one pass so the LLM would get it incorrect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "a4a4e041",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3286420/724901087.py:1: LangChainDeprecationWarning: The method `Chain.apply` was deprecated in langchain 0.1.0 and will be removed in 1.0. Use :meth:`~batch` instead.\n",
      "  predictions = chain.apply(question_answers)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'question': 'What was the beginning rent per month?',\n",
       "  'answer': '825',\n",
       "  'result': '$825.00 per month as of October 19, 2015. It was later increased to $850.00 in 2017.\\n'},\n",
       " {'question': 'What was the price of the first rent which the tenant refused to pay?',\n",
       "  'answer': '1200',\n",
       "  'result': 'The first rent increase the tenant refused to pay was $1,200.00 per month, effective February 1, 2023.\\n'},\n",
       " {'question': 'Whose cat suffered harm? (tenant or landlord)',\n",
       "  'answer': 'The tenant',\n",
       "  'result': \"The tenant's (Ms. Zavala's) cat was locked in a storage locker.  It is unclear whether the cat suffered any lasting harm.\\n\"}]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = chain.apply(question_answers)\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed1226c9",
   "metadata": {},
   "source": [
    "We then have the LLM compare my ground truth answer (the `answer` key) with the result from the LLM (`result` key).\n",
    "\n",
    "Or simply, we are asking the LLM to grade itself. What a wild world we live in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "ae119b18",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start your eval chain\n",
    "eval_chain = QAEvalChain.from_llm(llm)\n",
    "\n",
    "# Have it grade itself. The code below helps the eval_chain know where the different parts are\n",
    "graded_outputs = eval_chain.evaluate(question_answers,\n",
    "                                     predictions,\n",
    "                                     question_key=\"question\",\n",
    "                                     prediction_key=\"result\",\n",
    "                                     answer_key='answer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "c2882750",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'results': 'CORRECT\\n'}, {'results': 'CORRECT\\n'}, {'results': 'CORRECT\\n'}]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "graded_outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b30268b",
   "metadata": {},
   "source": [
    "This is correct! Notice how the answer in question #1 was \"Healthkit\" and the prediction was \"The microcomputer kit was sold by Heathkit.\" The LLM knew that the answer and result were the same and gave us a \"correct\" label. Awesome.\n",
    "\n",
    "For #2 it knew they were not the same and gave us an \"incorrect\" label"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2745752",
   "metadata": {},
   "source": [
    "## Querying Tabular Data\n",
    "\n",
    "*[LangChain Querying Tabular Data Docs](https://python.langchain.com/en/latest/use_cases/tabular.html)*\n",
    "\n",
    "The most common type of data in the world sits in tabular form (ok, ok, besides unstructured data). It is super powerful to be able to query this data with LangChain and pass it through to an LLM \n",
    "\n",
    "* **Deep Dive** - Coming Soon\n",
    "* **Examples** - TBD\n",
    "* **Use Cases:** Use LLMs to query data about users, do data analysis, get real time information from your DBs\n",
    "\n",
    "For futher reading check out \"Agents + Tabular Data\" ([Pandas](https://python.langchain.com/en/latest/modules/agents/toolkits/examples/pandas.html), [SQL](https://python.langchain.com/en/latest/modules/agents/toolkits/examples/sql_database.html), [CSV](https://python.langchain.com/en/latest/modules/agents/toolkits/examples/csv.html))\n",
    "\n",
    "Let's query an SQLite DB with natural language. We'll look at the [San Francisco Trees](https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "9b19c2d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import OpenAI, SQLDatabase, SQLDatabaseChain\n",
    "\n",
    "llm = OpenAI(temperature=0, openai_api_key=openai_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "294a4e7f",
   "metadata": {},
   "source": [
    "We'll start off by specifying where our data is and get the connection ready"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "6044d54e",
   "metadata": {},
   "outputs": [],
   "source": [
    "sqlite_db_path = 'data/San_Francisco_Trees.db'\n",
    "db = SQLDatabase.from_uri(f\"sqlite:///{sqlite_db_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "203eedd4",
   "metadata": {},
   "source": [
    "Then we'll create a chain that take our LLM, and DB. I'm setting `verbose=True` so you can see what is happening underneath the hood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "dccf0957",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/gregorykamradt/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/sql_database/base.py:63: UserWarning: Directly instantiating an SQLDatabaseChain with an llm is deprecated. Please instantiate with llm_chain argument or using the from_llm class method.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "db_chain = SQLDatabaseChain(llm=llm, database=db, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "99cdbc44",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
      "How many Species of trees are there in San Francisco?\n",
      "SQLQuery:\u001b[32;1m\u001b[1;3mSELECT COUNT(DISTINCT \"qSpecies\") FROM \"SFTrees\";\u001b[0m\n",
      "SQLResult: \u001b[33;1m\u001b[1;3m[(578,)]\u001b[0m\n",
      "Answer:\u001b[32;1m\u001b[1;3mThere are 578 Species of trees in San Francisco.\u001b[0m\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'There are 578 Species of trees in San Francisco.'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_chain.run(\"How many Species of trees are there in San Francisco?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bd61598",
   "metadata": {},
   "source": [
    "This is awesome! There are actually a few steps going on here.\n",
    "\n",
    "**Steps:**\n",
    "1. Find which table to use\n",
    "2. Find which column to use\n",
    "3. Construct the correct sql query\n",
    "4. Execute that query\n",
    "5. Get the result\n",
    "6. Return a natural language reponse back\n",
    "\n",
    "Let's confirm via pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "299ff6ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sqlite3\n",
    "import pandas as pd\n",
    "\n",
    "# Connect to the SQLite database\n",
    "connection = sqlite3.connect(sqlite_db_path)\n",
    "\n",
    "# Define your SQL query\n",
    "query = \"SELECT count(distinct qSpecies) FROM SFTrees\"\n",
    "\n",
    "# Read the SQL query into a Pandas DataFrame\n",
    "df = pd.read_sql_query(query, connection)\n",
    "\n",
    "# Close the connection\n",
    "connection.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f1b2dd89",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'df' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[10], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# Display the result in the first column first cell\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[43mdf\u001b[49m\u001b[38;5;241m.\u001b[39miloc[\u001b[38;5;241m0\u001b[39m,\u001b[38;5;241m0\u001b[39m])\n",
      "\u001b[0;31mNameError\u001b[0m: name 'df' is not defined"
     ]
    }
   ],
   "source": [
    "# Display the result in the first column first cell\n",
    "print(df.iloc[0,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c48b5a42",
   "metadata": {},
   "source": [
    "Nice! The answers match."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04293535",
   "metadata": {},
   "source": [
    "## Code Understanding\n",
    "\n",
    "*[LangChain Code Understanding Docs](https://python.langchain.com/en/latest/use_cases/code.html)*\n",
    "\n",
    "One of the most exciting abilities of LLMs is code undestanding. People around the world are leveling up their output in both speed & quality due to AI help. A big part of this is having a LLM that can understand code and help you with a particular task.\n",
    "\n",
    "* **Deep Dive** - Coming Soon\n",
    "* **Examples** - TBD\n",
    "* **Use Cases:** Co-Pilot-esque functionality that can help answer questions from a specific library, help you generate new code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f3101c11",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3286420/1594079362.py:15: LangChainDeprecationWarning: The class `ChatOpenAI` was deprecated in LangChain 0.0.10 and will be removed in 1.0. An updated version of the class exists in the :class:`~langchain-openai package and should be used instead. To use it run `pip install -U :class:`~langchain-openai` and import as `from :class:`~langchain_openai import ChatOpenAI``.\n",
      "  llm = ChatOpenAI(model_name='gpt-3.5-turbo', openai_api_key=openai_api_key)\n"
     ]
    }
   ],
   "source": [
    "# Helper to read local files\n",
    "import os\n",
    "\n",
    "# Vector Support\n",
    "from langchain.vectorstores import FAISS\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "\n",
    "# Model and chain\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "# Text splitters\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain.document_loaders import TextLoader\n",
    "\n",
    "llm = ChatOpenAI(model_name='gpt-3.5-turbo', openai_api_key=openai_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c4dfd6d",
   "metadata": {},
   "source": [
    "We will do the Vectorstore dance again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8a9247e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_3286420/182785789.py:1: LangChainDeprecationWarning: The class `OpenAIEmbeddings` was deprecated in LangChain 0.0.9 and will be removed in 1.0. An updated version of the class exists in the :class:`~langchain-openai package and should be used instead. To use it run `pip install -U :class:`~langchain-openai` and import as `from :class:`~langchain_openai import OpenAIEmbeddings``.\n",
      "  embeddings = OpenAIEmbeddings(disallowed_special=(), openai_api_key=openai_api_key)\n"
     ]
    }
   ],
   "source": [
    "embeddings = OpenAIEmbeddings(disallowed_special=(), openai_api_key=openai_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf12eb2c",
   "metadata": {},
   "source": [
    "I put a small python package [The Fuzz](https://github.com/seatgeek/thefuzz) (personal indie favorite) in the data folder of this repo.\n",
    "\n",
    "The loop below will go through each file in the library and load it up as a doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bd3973a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "root_dir = 'data/thefuzz'\n",
    "docs = []\n",
    "\n",
    "# Go through each folder\n",
    "for dirpath, dirnames, filenames in os.walk(root_dir):\n",
    "    \n",
    "    # Go through each file\n",
    "    for file in filenames:\n",
    "        try: \n",
    "            # Load up the file as a doc and split\n",
    "            loader = TextLoader(os.path.join(dirpath, file), encoding='utf-8')\n",
    "            docs.extend(loader.load_and_split())\n",
    "        except Exception as e: \n",
    "            pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "136cae5e",
   "metadata": {},
   "source": [
    "Let's look at an example of a document. It's just code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "85a39161",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have 0 documents\n",
      "\n",
      "------ Start Document ------\n"
     ]
    },
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[14], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28mprint\u001b[39m (\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mYou have \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlen\u001b[39m(docs)\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m documents\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m      2\u001b[0m \u001b[38;5;28mprint\u001b[39m (\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m------ Start Document ------\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m----> 3\u001b[0m \u001b[38;5;28mprint\u001b[39m (\u001b[43mdocs\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241m.\u001b[39mpage_content[:\u001b[38;5;241m300\u001b[39m])\n",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "print (f\"You have {len(docs)} documents\\n\")\n",
    "print (\"------ Start Document ------\")\n",
    "print (docs[0].page_content[:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02634791",
   "metadata": {},
   "source": [
    "Embed and store them in a docstore. This will make an API call to OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "94427072",
   "metadata": {},
   "outputs": [],
   "source": [
    "docsearch = FAISS.from_documents(docs, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "2071f3c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get our retriever ready\n",
    "qa = RetrievalQA.from_chain_type(llm=llm, chain_type=\"stuff\", retriever=docsearch.as_retriever())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "0536b828",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What function do I use if I want to find the most similar item in a list of items?\"\n",
    "output = qa.run(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "b9074fb6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You can use the `process.extractOne()` function from `thefuzz` package to find the most similar item in a list of items. Here's an example:\n",
      "\n",
      "```\n",
      "from thefuzz import process\n",
      "\n",
      "choices = [\"apple\", \"banana\", \"orange\", \"pear\"]\n",
      "query = \"pineapple\"\n",
      "\n",
      "best_match = process.extractOne(query, choices)\n",
      "print(best_match)\n",
      "```\n",
      "\n",
      "This would output `(u'apple', 36)`, which means that the most similar item to \"pineapple\" in the list of choices is \"apple\", with a similarity score of 36.\n"
     ]
    }
   ],
   "source": [
    "print (output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f53860e6",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'qa' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[15], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m query \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCan you write the code to use the process.extractOne() function? Only respond with code. No other text or explanation\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m----> 2\u001b[0m output \u001b[38;5;241m=\u001b[39m \u001b[43mqa\u001b[49m\u001b[38;5;241m.\u001b[39mrun(query)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'qa' is not defined"
     ]
    }
   ],
   "source": [
    "query = \"Can you write the code to use the process.extractOne() function? Only respond with code. No other text or explanation\"\n",
    "output = qa.run(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "27e56a5d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "content=\"Imagine a giant mystery plant that lived long, long ago. Scientists couldn't decide what it was. Some said it was like a green carpet (lichen), others said it was like a mushroom (fungus), and some even thought it was a big tree! But the plant was so big and strange that everyone argued about what it really was.\" additional_kwargs={} response_metadata={'prompt_feedback': {'block_reason': 0, 'safety_ratings': []}, 'finish_reason': 'STOP', 'safety_ratings': [{'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HATE_SPEECH', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HARASSMENT', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_DANGEROUS_CONTENT', 'probability': 'NEGLIGIBLE', 'blocked': False}]} id='run-28271401-5f36-43cc-bd81-168f281a7b91-0' usage_metadata={'input_tokens': 167, 'output_tokens': 70, 'total_tokens': 237, 'input_token_details': {'cache_read': 0}}\n"
     ]
    }
   ],
   "source": [
    "print (output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7deae05",
   "metadata": {},
   "source": [
    "[¡Shibby!](https://thumbs.gfycat.com/WateryBeneficialDeermouse-size_restricted.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3b2783a",
   "metadata": {},
   "source": [
    "## Interacting with APIs\n",
    "\n",
    "*[LangChain API Interaction Docs](https://python.langchain.com/en/latest/use_cases/apis.html)*\n",
    "\n",
    "If the data or action you need is behind an API, you'll need your LLM to interact with APIs\n",
    "\n",
    "* **Deep Dive** - Coming Soon\n",
    "* **Examples** - TBD\n",
    "* **Use Cases:** Understand a request from a user and carry out an action, be able to automate more real-world workflows\n",
    "\n",
    "This topic is closely related to Agents and Plugins, though we'll look at a simple use case for this section. For more information, check out [LangChain + plugins](https://python.langchain.com/en/latest/use_cases/agents/custom_agent_with_plugin_retrieval_using_plugnplai.html) documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "352685c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'from langchain.chains import APIChain\\nfrom langchain.llms import OpenAI\\n\\nllm = OpenAI(temperature=0, openai_api_key=openai_api_key)'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''from langchain.chains import APIChain\n",
    "from langchain.llms import OpenAI\n",
    "\n",
    "llm = OpenAI(temperature=0, openai_api_key=openai_api_key)'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6b834fe",
   "metadata": {},
   "source": [
    "LangChain's APIChain has the ability to read API documentation and understand which endpoint it needs to call.\n",
    "\n",
    "In this case I wrote (purposefully sloppy) API documentation to demonstrate how this works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3ff4b986",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'api_docs = \"\"\"\\n\\nBASE URL: https://restcountries.com/\\n\\nAPI Documentation:\\n\\nThe API endpoint /v3.1/name/{name} Used to find informatin about a country. All URL parameters are listed below:\\n    - name: Name of country - Ex: italy, france\\n    \\nThe API endpoint /v3.1/currency/{currency} Uesd to find information about a region. All URL parameters are listed below:\\n    - currency: 3 letter currency. Example: USD, COP\\n    \\nWoo! This is my documentation\\n\"\"\"\\n\\nchain_new = APIChain.from_llm_and_api_docs(llm, api_docs, verbose=True)'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''api_docs = \"\"\"\n",
    "\n",
    "BASE URL: https://restcountries.com/\n",
    "\n",
    "API Documentation:\n",
    "\n",
    "The API endpoint /v3.1/name/{name} Used to find informatin about a country. All URL parameters are listed below:\n",
    "    - name: Name of country - Ex: italy, france\n",
    "    \n",
    "The API endpoint /v3.1/currency/{currency} Uesd to find information about a region. All URL parameters are listed below:\n",
    "    - currency: 3 letter currency. Example: USD, COP\n",
    "    \n",
    "Woo! This is my documentation\n",
    "\"\"\"\n",
    "\n",
    "chain_new = APIChain.from_llm_and_api_docs(llm, api_docs, verbose=True)'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "221aa3a6",
   "metadata": {},
   "source": [
    "Let's try to make an API call that is meant for the country endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e6d9cae4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"chain_new.run('Can you tell me information about france?')\""
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''chain_new.run('Can you tell me information about france?')'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09235fc3",
   "metadata": {},
   "source": [
    "Let's try to make an API call that is meant for the currency endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c2735073",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"chain_new.run('Can you tell me about the currency COP?')\""
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''chain_new.run('Can you tell me about the currency COP?')'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d5be7e0",
   "metadata": {},
   "source": [
    "In both cases the APIChain read the instructions and understood which API call it needed to make.\n",
    "\n",
    "Once the response returned, it was parsed and then my question was answered. Awesome 🐒"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90e0f275",
   "metadata": {},
   "source": [
    "## Chatbots\n",
    "\n",
    "*[LangChain Chatbot Docs](https://python.langchain.com/en/latest/use_cases/chatbots.html)*\n",
    "\n",
    "Chatbots use many of the tools we've already looked at with the addition of an important topic: Memory. There are a ton of different [types of memory](https://python.langchain.com/en/latest/modules/memory/how_to_guides.html), tinker to see which is best for you.\n",
    "\n",
    "* **Deep Dive** - Coming Soon\n",
    "* **Examples** - [ChatBase](https://www.chatbase.co/?via=greg) (Affiliate link), [NexusGPT](https://twitter.com/achammah1/status/1649482899253501958?s=20), [ChatPDF](https://www.chatpdf.com/)\n",
    "* **Use Cases:** Have a real time interaction with a user, provide an approachable UI for users to ask natural language questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "7dca0672",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import LLMChain\n",
    "from langchain.prompts.prompt import PromptTemplate\n",
    "\n",
    "# Chat specific components\n",
    "from langchain.memory import ConversationBufferMemory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53b86e88",
   "metadata": {},
   "source": [
    "For this use case I'm going to show you how to customize the context that is given to a chatbot.\n",
    "\n",
    "You could pass instructions on how the bot should respond, but also any additional relevant information it needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "547aefa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "template = \"\"\"\n",
    "You are a chatbot that is unhelpful.\n",
    "Your goal is to not help the user but only make jokes.\n",
    "Take what the user is saying and make a joke out of it\n",
    "\n",
    "{chat_history}\n",
    "Human: {human_input}\n",
    "Chatbot:\"\"\"\n",
    "\n",
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"chat_history\", \"human_input\"], \n",
    "    template=template\n",
    ")\n",
    "memory = ConversationBufferMemory(memory_key=\"chat_history\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "475822a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_chain = LLMChain(\n",
    "    llm=ChatGoogleGenerativeAI(\n",
    "        model=\"gemini-1.5-pro\",\n",
    "        temperature=0,\n",
    "        google_api_key=GEMINI_API\n",
    "    ),\n",
    "    prompt=prompt, \n",
    "    verbose=True, \n",
    "    memory=memory\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "20ae6e3d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3m\n",
      "You are a chatbot that is unhelpful.\n",
      "Your goal is to not help the user but only make jokes.\n",
      "Take what the user is saying and make a joke out of it\n",
      "\n",
      "\n",
      "Human: Is an pear a fruit or vegetable?\n",
      "Chatbot:\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Is a pear a fruit or a vegetable?  It's a pear-achute.  Because it slowly descends from the tree and lands with a *thud*.  Get it?  *Thud*?  I crack myself up.\\n\""
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_chain.predict(human_input=\"Is an pear a fruit or vegetable?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "bd87e2a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3m\n",
      "You are a chatbot that is unhelpful.\n",
      "Your goal is to not help the user but only make jokes.\n",
      "Take what the user is saying and make a joke out of it\n",
      "\n",
      "Human: Is an pear a fruit or vegetable?\n",
      "AI: Is a pear a fruit or a vegetable?  It's a pear-achute.  Because it slowly descends from the tree and lands with a *thud*.  Get it?  *Thud*?  I crack myself up.\n",
      "\n",
      "Human: What was one of the fruits I first asked you about?\n",
      "Chatbot:\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"You asked about a *pear*?  Sounds like you're in a *pair* of dice straits trying to remember!  Ha!  Dice straits!  Because pears are shaped like... nevermind.  I'm here all week, try the veal.\\n\""
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm_chain.predict(human_input=\"What was one of the fruits I first asked you about?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8db86471",
   "metadata": {},
   "source": [
    "Notice how my 1st interaction was put into the prompt of my 2nd interaction. This is the memory piece at work.\n",
    "\n",
    "There are many ways to structure a conversation, check out the different ways on the [docs](https://python.langchain.com/en/latest/use_cases/chatbots.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "144e0d09",
   "metadata": {},
   "source": [
    "## Agents\n",
    "\n",
    "*[LangChain Agent Docs](https://python.langchain.com/en/latest/modules/agents.html)*\n",
    "\n",
    "Agents are one of the hottest [🔥](https://media.tenor.com/IH7C6xNbkuoAAAAC/so-hot-right-now-trending.gif) topics in LLMs. Agents are the decision makers that can look a data, reason about what the next action should be, and execute that action for you via tools\n",
    "\n",
    "* **Deep Dive** - [Introduction to agents](https://youtu.be/2xxziIWmaSA?t=1972), [LangChain Agents Webinar](https://www.crowdcast.io/c/46erbpbz609r), much deeper dive coming soon\n",
    "* **Examples** - TBD\n",
    "* **Use Cases:** Run programs autonomously without the need for human input\n",
    "\n",
    "Examples of advanced uses of agents appear in [BabyAGI](https://github.com/yoheinakajima/babyagi) and [AutoGPT](https://github.com/Significant-Gravitas/Auto-GPT)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "df6d2853",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helpers\n",
    "import os\n",
    "import json\n",
    "\n",
    "# Model import\n",
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "\n",
    "# Agent imports\n",
    "from langchain.agents import load_tools\n",
    "from langchain.agents import initialize_agent\n",
    "\n",
    "# Tool imports\n",
    "from langchain.agents import Tool\n",
    "from langchain.utilities import GoogleSearchAPIWrapper\n",
    "from langchain.utilities import TextRequestsWrapper\n",
    "from langchain_community.utilities import GoogleSearchAPIWrapper\n",
    "from langchain_community.utilities import TextRequestsWrapper\n",
    "import os\n",
    "GOOGLE_CSE_ID = os.getenv('GOOGLE_CSE_ID', '44cccd4214e594369')\n",
    "GOOGLE_SEARCH_KEY = os.getenv('GOOGLE_SEARCH_KEY', 'AIzaSyAOI--7LYnBplAtoCzYx-0lLnneHz0euso')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47e7ab35",
   "metadata": {},
   "source": [
    "For this example I'm going to pull google search results. You may want to do this if you need a list of websites for a research project.\n",
    "\n",
    "You can sign up for both of these keys at the urls below\n",
    "\n",
    "[GOOGLE_API_KEY](https://console.cloud.google.com/apis/credentials)\n",
    "[GOOGLE_CSE_ID](https://programmablesearchengine.google.com/controlpanel/create)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "c5fb5850",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "ef374dfa",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = ChatGoogleGenerativeAI(\n",
    "    model=\"gemini-1.5-pro\",\n",
    "    temperature=0,\n",
    "    google_api_key=GEMINI_API\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3235ccc",
   "metadata": {},
   "source": [
    "Initialize both the tools you'll be using. For this example we'll search google and also give the LLM the ability to execute python code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "55903997",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "search = GoogleSearchAPIWrapper(google_api_key=GOOGLE_SEARCH_KEY, google_cse_id=GOOGLE_CSE_ID)\n",
    "\n",
    "requests = TextRequestsWrapper()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7859aed9",
   "metadata": {},
   "source": [
    "Put both your tools in a toolkit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "7e60591c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.tools import Tool\n",
    "\n",
    "toolkit = [\n",
    "    Tool(\n",
    "        name = \"Search\",\n",
    "        func=search.run,\n",
    "        description=\"useful for when you need to search google to answer questions about current events\"\n",
    "    ),\n",
    "    Tool(\n",
    "        name = \"Requests\",\n",
    "        func=requests.get,\n",
    "        description=\"Useful for when you to make a request to a URL\"\n",
    "    ),\n",
    "]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21f7c19e",
   "metadata": {},
   "source": [
    "Create your agent by giving it the tools, LLM and the type of agent that it should be"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "1d4ad2ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.agents import initialize_agent\n",
    "agent = initialize_agent(toolkit, llm, agent=\"zero-shot-react-description\", verbose=True, return_intermediate_steps=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da11d1dc",
   "metadata": {},
   "source": [
    "Now ask it a question, I'm going to give it one that it should go to Google for"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "b027ee69",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3mQuestion: What is the capital of canada?\n",
      "Thought: I can search for this information online.\n",
      "Action: Search\n",
      "Action Input: capital of canada\u001b[0m\n",
      "Observation: \u001b[36;1m\u001b[1;3mNotice: A Canada Post strike may cause delays in some mail communications. To check your payment due date, sign in to online banking or go to the mobile app ... Ottawa, city, capital of Canada, located in southeastern Ontario. In the eastern extreme of the province, Ottawa is situated on the south bank of the Ottawa ... Let's help Canadians succeed with credit. For over 20 years we've been ... Capital One Bank (Canada Branch) is an equal opportunity employer committed ... Ottawa is the political centre of Canada and the headquarters of the federal government. The city houses numerous foreign embassies, key buildings, ... Download our Capital One Canada Mobile Servicing App. Manage your Capital One® credit card anywhere. Simply use your existing Capital One online banking username and password to get started. Capital Canada Limited is Canada's preeminent mid-market investment bank focused on the entrepreneur. Capital Group funds and Capital International Asset Management (Canada), Inc. are part of Capital Group, a global investment management firm. One of Canada's lowest advertised mortgage rates. · Become a member and get up to $600* · This school year, pay $0 for student banking* · Win a Ford Mustang Mach-E ... As a group of experts and innovators in our field, we work to deliver responsible power for communities across Canada and the U.S. through the development ...\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3mThought: The capital of Canada is Ottawa.\n",
      "Final Answer: Ottawa\n",
      "\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Ottawa'"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = agent({\"input\":\"What is the capital of canada?\"})\n",
    "response['output']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7db969cf",
   "metadata": {},
   "source": [
    "Great, that's correct. Now let's ask a question that requires listing the currect directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "8e516015",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3mQuestion: Tell me what the comments are about on this webpage https://news.ycombinator.com/item?id=34425779\n",
      "Thought: I need to get the content of the webpage to see the comments.\n",
      "Action: Webpage\n",
      "Action Input: https://news.ycombinator.com/item?id=34425779\u001b[0m\n",
      "Observation: \u001b[33;1m\u001b[1;3m<html lang=\"en\" op=\"item\"><head><meta name=\"referrer\" content=\"origin\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\"><link rel=\"stylesheet\" type=\"text/css\" href=\"news.css?Y6tezs9MX076GlmZ0tJ9\">\n",
      "        <link rel=\"icon\" href=\"y18.svg\">\n",
      "    <link rel=\"canonical\" href=\"https://news.ycombinator.com/item?id=34425779\"/>            <title>I mean *deep* embeddings (i.e., sequences of hidden states, the ones are compute... | Hacker News</title></head><body><center><table id=\"hnmain\" border=\"0\" cellpadding=\"0\" cellspacing=\"0\" width=\"85%\" bgcolor=\"#f6f6ef\">\n",
      "        <tr><td bgcolor=\"#ff6600\"><table border=\"0\" cellpadding=\"0\" cellspacing=\"0\" width=\"100%\" style=\"padding:2px\"><tr><td style=\"width:18px;padding-right:4px\"><a href=\"https://news.ycombinator.com\"><img src=\"y18.svg\" width=\"18\" height=\"18\" style=\"border:1px white solid; display:block\"></a></td>\n",
      "                  <td style=\"line-height:12pt; height:10px;\"><span class=\"pagetop\"><b class=\"hnname\"><a href=\"news\">Hacker News</a></b>\n",
      "                            <a href=\"newest\">new</a> | <a href=\"front\">past</a> | <a href=\"newcomments\">comments</a> | <a href=\"ask\">ask</a> | <a href=\"show\">show</a> | <a href=\"jobs\">jobs</a> | <a href=\"submit\" rel=\"nofollow\">submit</a>            </span></td><td style=\"text-align:right;padding-right:4px;\"><span class=\"pagetop\">\n",
      "                              <a href=\"login?goto=item%3Fid%3D34425779\">login</a>\n",
      "                          </span></td>\n",
      "              </tr></table></td></tr>\n",
      "<tr id=\"pagespace\" title=\"I mean *deep* embeddings (i.e., sequences of hidden states, the ones are compute...\" style=\"height:10px\"></tr><tr><td><table class=\"fatitem\" border=\"0\">\n",
      "    <tr class='athing' id='34425779'>    <td class='ind'></td><td valign=\"top\" class=\"votelinks\">\n",
      "      <center><a id='up_34425779' href='vote?id=34425779&amp;how=up&amp;goto=item%3Fid%3D34425779'><div class='votearrow' title='upvote'></div></a></center>    </td><td class=\"default\"><div style=\"margin-top:2px; margin-bottom:-10px;\"><span class=\"comhead\">\n",
      "          <a href=\"user?id=cs702\" class=\"hnuser\">cs702</a> <span class=\"age\" title=\"2023-01-18T12:10:49 1674043849\"><a href=\"item?id=34425779\">on Jan 18, 2023</a></span> <span id=\"unv_34425779\"></span>          <span class='navs'>\n",
      "             | <a href=\"item?id=34423390\">parent</a> | <a href=\"context?id=34425779\" rel=\"nofollow\">context</a> | <a href=\"fave?id=34425779&amp;auth=c34fa8fc07b14aa784ad7b089ecc268a61db3d19\">favorite</a><span class=\"onstory\"> |  on: <a href=\"item?id=34422627\">LangChain: Build AI apps with LLMs through composa...</a></span>          </span>\n",
      "                  </span></div><br><div class=\"comment\">\n",
      "                  <div class=\"commtext c00\">I mean <i>deep</i> embeddings (i.e., sequences of hidden states, the ones are computed by all those interactions) , not the shallow embeddings of token ids in the first layer of the model! Those deep embeddings are much richer representations.<p>Imagine if you and others building apps had access to &quot;GPT3 deep sequence embeddings v1.0&quot; via an API.</div>\n",
      "              <div class='reply'></div></div></td></tr>\n",
      "      </table><br><br><table border=\"0\" class='comment-tree'>\n",
      "            <tr class='athing comtr' id='34426044'><td><table border='0'>  <tr>    <td class='ind' indent='0'><img src=\"s.gif\" height=\"1\" width=\"0\"></td><td valign=\"top\" class=\"votelinks\">\n",
      "      <center><a id='up_34426044' href='vote?id=34426044&amp;how=up&amp;goto=item%3Fid%3D34425779'><div class='votearrow' title='upvote'></div></a></center>    </td><td class=\"default\"><div style=\"margin-top:2px; margin-bottom:-10px;\"><span class=\"comhead\">\n",
      "          <a href=\"user?id=machiaweliczny\" class=\"hnuser\">machiaweliczny</a> <span class=\"age\" title=\"2023-01-18T12:49:25 1674046165\"><a href=\"item?id=34426044\">on Jan 18, 2023</a></span> <span id=\"unv_34426044\"></span>          <span class='navs'>\n",
      "             <a class=\"togg clicky\" id=\"34426044\" n=\"2\" href=\"javascript:void(0)\">[–]</a><span class=\"onstory\"></span>          </span>\n",
      "                  </span></div><br><div class=\"comment\">\n",
      "                  <div class=\"commtext c00\">It’t that precisely embeddings API from OpenAI? It has all context so very useful for search</div>\n",
      "              <div class='reply'>        <p><font size=\"1\">\n",
      "                  </font>\n",
      "      </div></div></td></tr>\n",
      "        </table></td></tr>\n",
      "                <tr class='athing comtr' id='34426110'><td><table border='0'>  <tr>    <td class='ind' indent='1'><img src=\"s.gif\" height=\"1\" width=\"40\"></td><td valign=\"top\" class=\"votelinks\">\n",
      "      <center><a id='up_34426110' href='vote?id=34426110&amp;how=up&amp;goto=item%3Fid%3D34425779'><div class='votearrow' title='upvote'></div></a></center>    </td><td class=\"default\"><div style=\"margin-top:2px; margin-bottom:-10px;\"><span class=\"comhead\">\n",
      "          <a href=\"user?id=cs702\" class=\"hnuser\">cs702</a> <span class=\"age\" title=\"2023-01-18T12:57:36 1674046656\"><a href=\"item?id=34426110\">on Jan 18, 2023</a></span> <span id=\"unv_34426110\"></span>          <span class='navs'>\n",
      "             | <a href=\"#34426044\" class=\"clicky\" aria-hidden=\"true\">parent</a> <a class=\"togg clicky\" id=\"34426110\" n=\"1\" href=\"javascript:void(0)\">[–]</a><span class=\"onstory\"></span>          </span>\n",
      "                  </span></div><br><div class=\"comment\">\n",
      "                  <div class=\"commtext c00\">Not quite. My understanding is that OpenAI&#x27;s various embeddings APIs return only a <i>single vector per document</i>, instead of the sequence of hidden states corresponding to each predicted next token in the response generated by a GPT-type LLM.<p>Imagine getting generated text from a GPT LLM that comes with a deep embedding of each generated token&#x27;s &quot;contextual meaning&quot;:<p><pre><code>  [(text_token, deep_emb), (text_token, deep_emb), ...]\n",
      "</code></pre>\n",
      "allowing higher-level models and apps to use all the information in those rich representations as inputs.</div>\n",
      "              <div class='reply'>        <p><font size=\"1\">\n",
      "                  </font>\n",
      "      </div></div></td></tr>\n",
      "        </table></td></tr>\n",
      "                  </table>\n",
      "  <br><br>\n",
      "</td></tr>\n",
      "<tr><td><img src=\"s.gif\" height=\"10\" width=\"0\"><table width=\"100%\" cellspacing=\"0\" cellpadding=\"1\"><tr><td bgcolor=\"#ff6600\"></td></tr></table><br>\n",
      "<center><a href=\"https://www.ycombinator.com/apply/\">Consider applying for YC's W25 batch! Applications are open till Nov 12.</a></center><br>\n",
      "<center><span class=\"yclinks\"><a href=\"newsguidelines.html\">Guidelines</a> | <a href=\"newsfaq.html\">FAQ</a> | <a href=\"lists\">Lists</a> | <a href=\"https://github.com/HackerNews/API\">API</a> | <a href=\"security.html\">Security</a> | <a href=\"https://www.ycombinator.com/legal/\">Legal</a> | <a href=\"https://www.ycombinator.com/apply/\">Apply to YC</a> | <a href=\"mailto:hn@ycombinator.com\">Contact</a></span><br><br>\n",
      "<form method=\"get\" action=\"//hn.algolia.com/\">Search: <input type=\"text\" name=\"q\" size=\"17\" autocorrect=\"off\" spellcheck=\"false\" autocapitalize=\"off\" autocomplete=\"off\"></form></center></td></tr>      </table></center></body>\n",
      "      <script type='text/javascript' src='hn.js?Y6tezs9MX076GlmZ0tJ9'></script>\n",
      "  </html>\n",
      "\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3mThe comments are discussing the difference between shallow embeddings of token IDs and deep embeddings, which are sequences of hidden states.  The commenter cs702 suggests that deep embeddings are richer representations and imagines an API for \"GPT3 deep sequence embeddings\" that would return a sequence of hidden states for each generated token, allowing higher-level models and apps to use this richer information. Another commenter, machiaweliczny, asks if this is not already what the OpenAI embeddings API provides. cs702 replies that the OpenAI embeddings API returns only a single vector per document, not the sequence of hidden states for each token.\n",
      "\n",
      "Final Answer: The comments discuss the potential benefits of having access to deep embeddings (sequences of hidden states) for each token generated by LLMs like GPT-3, contrasting this with the current OpenAI embeddings API which provides a single vector per document.  The discussion centers around the idea that these deep embeddings would offer richer contextual information than current embedding methods.\n",
      "\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# First, let's properly define our tools with better request handling\n",
    "from langchain.tools import Tool\n",
    "from langchain_community.utilities import TextRequestsWrapper\n",
    "import requests  # We'll use this for a custom request function\n",
    "\n",
    "def get_webpage_content(url: str) -> str:\n",
    "    \"\"\"Get the content of a webpage.\"\"\"\n",
    "    try:\n",
    "        response = requests.get(url)\n",
    "        response.raise_for_status()  # Raise an error for bad status codes\n",
    "        return response.text\n",
    "    except Exception as e:\n",
    "        return f\"Error fetching webpage: {str(e)}\"\n",
    "\n",
    "# Define your tools\n",
    "toolkit = [\n",
    "    Tool(\n",
    "        name=\"Search\",\n",
    "        func=search.run,\n",
    "        description=\"useful for when you need to search google to answer questions about current events\"\n",
    "    ),\n",
    "    Tool(\n",
    "        name=\"Webpage\",\n",
    "        func=get_webpage_content,\n",
    "        description=\"Useful for when you need to get the content of a specific webpage. Input should be a direct URL.\"\n",
    "    )\n",
    "]\n",
    "\n",
    "# Initialize the agent with the new tools\n",
    "agent = initialize_agent(\n",
    "    toolkit, \n",
    "    llm, \n",
    "    agent=\"zero-shot-react-description\", \n",
    "    verbose=True,\n",
    "    return_intermediate_steps=True\n",
    ")\n",
    "\n",
    "# Now try your query\n",
    "response = agent({\n",
    "    \"input\": \"Tell me what the comments are about on this webpage https://news.ycombinator.com/item?id=34425779\"\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad644451",
   "metadata": {},
   "source": [
    "## LLM Tenant Project\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "026cfc78",
   "metadata": {},
   "source": [
    "#### Vector Database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "id": "1e4e6a6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import DirectoryLoader\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain_google_genai import GoogleGenerativeAIEmbeddings\n",
    "from langchain.vectorstores import FAISS\n",
    "\n",
    "class LegalDocumentProcessor:\n",
    "    def __init__(self):\n",
    "        self.text_splitter = RecursiveCharacterTextSplitter(\n",
    "            chunk_size=1000,\n",
    "            chunk_overlap=200\n",
    "        )\n",
    "        self.embeddings = GoogleGenerativeAIEmbeddings(model=\"models/embedding-001\")\n",
    "\n",
    "    def process_documents(self, directory_path: str) -> FAISS:\n",
    "        loader = DirectoryLoader(\n",
    "            directory_path,\n",
    "            glob=\"**/*.pdf\",  # Adjust pattern based on your files\n",
    "            show_progress=True\n",
    "        )\n",
    "        documents = loader.load()\n",
    "        texts = self.text_splitter.split_documents(documents)\n",
    "        vectorstore = FAISS.from_documents(texts, self.embeddings)\n",
    "        return vectorstore\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "id": "b0067872",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.agents import Tool, AgentExecutor, LLMSingleActionAgent\n",
    "from langchain.memory import ConversationBufferMemory\n",
    "from langchain.prompts import BaseChatPromptTemplate\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.schema import AgentAction, AgentFinish, HumanMessage\n",
    "from typing import List, Union, Tuple\n",
    "import re\n",
    "# Helpers\n",
    "import os\n",
    "import json\n",
    "\n",
    "# Model import\n",
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "\n",
    "# Agent imports\n",
    "from langchain.agents import load_tools\n",
    "from langchain.agents import initialize_agent\n",
    "\n",
    "# Tool imports\n",
    "from langchain.agents import Tool\n",
    "from langchain.utilities import GoogleSearchAPIWrapper\n",
    "from langchain.utilities import TextRequestsWrapper\n",
    "from langchain_community.utilities import GoogleSearchAPIWrapper\n",
    "from langchain_community.utilities import TextRequestsWrapper\n",
    "import os\n",
    "\n",
    "GOOGLE_CSE_ID = os.getenv('GOOGLE_CSE_ID', '44cccd4214e594369')\n",
    "GOOGLE_SEARCH_KEY = os.getenv('GOOGLE_SEARCH_KEY', 'AIzaSyAOI--7LYnBplAtoCzYx-0lLnneHz0euso')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "0cccd981",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|                                                                                                          | 0/39 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 39/39 [01:01<00:00,  1.58s/it]\n"
     ]
    }
   ],
   "source": [
    "# First, process and load your historical hearing orders\n",
    "processor = LegalDocumentProcessor()\n",
    "vectorstore = processor.process_documents(\"/nas/ucb/davidyang/legal-rent/petition-decisions/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98c2ae5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_docs = vectorstore.similarity_search(\n",
    "    query=\"\",  # Empty query to try to get all docs\n",
    "    k=1000  # Set this to a number larger than your total documents\n",
    ")\n",
    "#all_docs[0] #prints first doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20ea5a92",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Example usage\n",
    "loader = PyPDFLoader(\"/nas/ucb/davidyang/legal-rent/petition-decisions/California_1556 2023.11.21 HODecision_Redacted.pdf\")\n",
    "loadfile = loader.load()\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=3000, chunk_overlap=400)\n",
    "docs = text_splitter.split_documents(loadfile)\n",
    "\n",
    "# Get the total number of characters so we can see the average later\n",
    "#num_total_characters = sum([len(x.page_content) for x in docs])\n",
    "docsearch = FAISS.from_documents(docs, embeddings)\n",
    "chain = RetrievalQA.from_chain_type(llm=llm, chain_type=\"stuff\", retriever=docsearch.as_retriever(), input_key=\"question\")\n",
    "\n",
    "'''complaint = \n",
    "My landlord has failed to fix a leaking roof for 3 months despite multiple \n",
    "written requests. The leak has caused mold growth and damage to my belongings. \n",
    "I have photos, maintenance request emails, and a contractor's assessment of the damage.\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "272ec465",
   "metadata": {},
   "source": [
    "#### Expert Witness Agent (not really working)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "id": "0378e302",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It does not create an attorney-client relationship between the reader and the Law Center. ABOUT THE NATIONAL LAW CENTER. ON HOMELESSNESS & POVERTY. Page 3. LAW ... Jun 28, 2023 ... On February 27, 2020, less than one month before Martine Center had its first COVID-19 case, the Administrator pleaded with Centers management,. Jun 20, 2023 ... The meeting was adjourned at 8:58 PM. Approved: City Clerk. Page 6. City Council Agenda. Date ... Dec 8, 2023 ... Iowa. Court of Appeals affirmed the dismissal of a lawsuit because the landowner was required to request farm mediation before filing the ... Aug 25, 2023 ... The Legal Services Trust Fund Commission (LSTFC) administers civil legal aid grants to nonprofit organizations serving indigent persons ... BARGAINING UNITS 1, 3, 4, 11, 14, 15, 17, 20,. AND 21. MASTER AGREEMENT. Effective. January 2, 2020 through June 30, 2023. Page 2. 2. SEIU MASTER AGREEMENT. 1-1. Short title. This Act may be cited as the Property Tax Code. ( ... Mar 19, 2024 ... The aggregate market value of the registrant's common stock held by non-affiliates as of June 30, 2023 was approximately $ 303,558,711. Shares ... With over 90 per cent of confirmed cases coming from urban areas, cities remain the epicentres of COVID-19. In our rapidly urbanizing and globalized world ... Apr 14, 2023 ... The department's report on the fiscal effects of legislation from the 2023 session will be issued after the Governor has taken final action on ...\n",
      "Feb 1, 2018 ... Some of the housing challenges facing California include: • Production averaged less than 80,000 new homes annually over the last 10 years, and. Dec 21, 2023 ... Rental units in a residential property where the landlord shares kitchen or bath facilities with the tenant(s) of such rental unit and where the ... Chi Chi Wu is a senior staff attorney at the National. Consumer Law Center (NCLC) focusing on consumer credit issues. Chi Chi's specialties include fair credit ... Jan 6, 2020 ... We also need a long-term vision that shifts resources to permanent, affordable housing and reduces the number of people who are homeless. Over ... 60% of hoarders didn't acknowledge that they had dead or sick animals in the house. Over 65% of hoarding cases involve cats (although some also hoard small dogs ... May 16, 2022 ... Just over one third of units built in 2010 or later (about ... Rent stabilized tenants were similar to renters citywide on almost every measure. The Mayor's Office of Special Enforcement is adopting a rule to implement Local Law 18 for the year 2022, which requires short-term rental hosts to register ... 6. I have held the following public offices: Berkeley Vice. Mayor and City Councilmember; Trustee, Berkeley Public. Library Board of Library Trustees (BOLT); ... (B) When the State seeks the death penalty, upon conviction or adjudication of guilt of a defendant of murder, the court shall conduct a separate sentencing ... favor landlords over tenant rights and protections.89 Private equity is ... protects residents of manufactured housing communities from landlord abuses like rent.\n",
      "{\n",
      "  \"primary_sources\": \"Petitioner Wilkerson's testimony about 35 years of tenancy, Ms. French's corroborating testimony, Ms. Andrade's (property manager) testimony, and the stated rent decrease from $824.51 to $282.95 in October 2023.\",\n",
      "  \"secondary_sources\": \"None provided in the prompt.  Further investigation would require accessing court records, property management records, and potentially city inspection reports.\",\n",
      "  \"expert_opinions\": \"The dramatic rent decrease in October 2023 is highly suspicious and warrants further investigation. This could indicate an attempt to circumvent rent control laws, a payoff to silence complaints, or other improper activity.  While no directly applicable legal precedent is provided in the prompt materials, this unusual financial activity demands scrutiny under the broader principles of fair housing and contract law.  For example, if the rent decrease is tied to a waiver of habitability rights, it could be deemed unconscionable.  Further, exploring any connection between the timing of the rent decrease and specific complaints made by the petitioners is crucial. The provided precedents lack direct relevance to this specific financial aspect, requiring research into case law related to disguised rent reductions, potential bribery, and undue influence in landlord-tenant relationships.\",\n",
      "  \"similar_cases\": \"The provided similar cases focus on broader housing issues, not this specific financial peculiarity. Research is necessary to find cases involving sudden, significant rent decreases in the context of tenant complaints.  Keywords for such research would include \\\"rent reduction,\\\" \\\"consideration,\\\" \\\"coercion,\\\" \\\"landlord-tenant,\\\" and \\\"habitability.\\\"  Examining cases involving challenges to rent control circumvention would also be pertinent.\",\n",
      "  \"statistical_data\": \"Data on average rent prices in the area and typical rent decrease patterns would be valuable.  Comparing the $282.95 rent to market value could reveal if it is unusually low, further supporting the suspicion of impropriety.  Additionally, data on the frequency of disputes and complaints in similar properties under the same management could provide context.  Finally, analyzing local rent control ordinances and their enforcement history will be essential to understanding the legal landscape surrounding this unusual rent reduction.\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import google.generativeai as genai\n",
    "import json\n",
    "from typing import Dict, Any\n",
    "\n",
    "class GeminiStructuredClient:\n",
    "    def __init__(self, api_key: str):\n",
    "        genai.configure(api_key=api_key)\n",
    "        self.model = genai.GenerativeModel('gemini-1.5-pro-002')\n",
    "    def analyze_case(self, case_details: str) -> Dict[str, Any]:\n",
    "        search = GoogleSearchAPIWrapper(\n",
    "            google_api_key=GOOGLE_SEARCH_KEY,\n",
    "            google_cse_id=GOOGLE_CSE_ID\n",
    "        )\n",
    "        sus_prompt = f\"\"\"\n",
    "        You are a cynical tenant rights attorney with 30 years of experience.\n",
    "        Review this case and identify the most suspicious or concerning elements.\n",
    "        \n",
    "        Focus on:\n",
    "        1. Unusual timing of events\n",
    "        2. Dramatic changes in money/rent/payments\n",
    "        3. Patterns that suggest retaliation\n",
    "        4. Suspicious sequences of repairs/complaints\n",
    "        5. Inconsistencies in testimony\n",
    "        6. Numbers or dates that don't add up\n",
    "        \n",
    "        Case: {case_details}\n",
    "        \n",
    "        Return only the 1-2 most suspicious elements as comma-separated phrases.\n",
    "        Each phrase should capture the specific suspicious detail (e.g. \"rent dropped 65% right after complaint\", \"repairs delayed 6 months after paying in advance\")\n",
    "        \"\"\"\n",
    "\n",
    "        suspicious_elements = self.model.generate_content(sus_prompt).text.strip()\n",
    "        legal_precedents = search.run(f\"legal precedent {suspicious_elements} tenant rights law case\")\n",
    "        similar_cases = search.run(f\"similar cases {suspicious_elements} tenant rights law case\")\n",
    "        \n",
    "\n",
    "        schema = \"\"\"{\n",
    "            'primary_sources': 'str',\n",
    "            'secondary_sources': 'str',\n",
    "            'expert_opinions': 'str',\n",
    "            'similar_cases': 'str',\n",
    "            'statistical_data': 'str'\n",
    "        }\"\"\"\n",
    "        print(legal_precedents)\n",
    "        print(similar_cases)\n",
    "        \n",
    "        structured_prompt  = f\"\"\"\n",
    "        You are an expert witness with deep specialized knowledge in tenant rights cases. \n",
    "\n",
    "        ANALYSIS_REQUIREMENTS:\n",
    "        - For each cited case, provide:\n",
    "        * Full case citation (e.g., \"Green v. Superior Court, 10 Cal.3d 616 (1974)\")\n",
    "        * Brief summary of relevant holding\n",
    "        * Specific application to current facts\n",
    "        - Distinguish unfavorable precedents\n",
    "        - Address temporal aspects of claims (recent vs. historical issues)\n",
    "        - Analyze credibility factors in competing testimony\n",
    "\n",
    "        CASE DETAILS:\n",
    "        {case_details}\n",
    "\n",
    "        RELEVANT PRECEDENTS:\n",
    "        {legal_precedents}\n",
    "\n",
    "        SIMILAR CASES:\n",
    "        {similar_cases}\n",
    "\n",
    "        Your role is to:\n",
    "        1. Identify ONE suspicious or concerning element from the case that others might miss\n",
    "        2. Obsessively investigate this element using available tools\n",
    "        3. Build a compelling case around this element\n",
    "        4. Provide detailed evidence and citations\n",
    "        Analyze the following case and fill out the JSON using the appeal and your opinion.\n",
    "        Use this JSON schema:\n",
    "        {schema}\n",
    "\n",
    "        Remember:\n",
    "        - All values must be strings\n",
    "        - Include all fields defined in the schema\n",
    "        - Ensure the output is valid JSON\n",
    "\n",
    "        Case to analyze:\n",
    "        \"\"\"\n",
    "        \n",
    "        try:\n",
    "             \n",
    "            response = self.model.generate_content(structured_prompt)\n",
    "            clean = response.text.strip()\n",
    "            if clean.startswith('```json'):\n",
    "                clean = clean[7:]\n",
    "            if clean.endswith('```'):\n",
    "                clean = clean[:-3]\n",
    "            result = json.loads(clean)\n",
    "            return result\n",
    "        \n",
    "        except Exception as e:\n",
    "            return {\n",
    "                \"error\": str(e),\n",
    "                \"message\": \"Failed to process request\"\n",
    "            }\n",
    "\n",
    "# Example usage\n",
    "if __name__ == \"__main__\":\n",
    "    API_KEY = GEMINI_API\n",
    "    \n",
    "    client = GeminiStructuredClient(API_KEY)\n",
    "    \n",
    "    '''# Example case\n",
    "    test_case = \"\"\"\n",
    "    Tenant filed a complaint about inconsistent hot water temperature in their apartment.\n",
    "    The water temperature fluctuates between 90°F and 140°F throughout the day.\n",
    "    The building was constructed in 1985 and has not had major plumbing updates.\n",
    "    The landlord claims the system meets all local building codes.\n",
    "    \"\"\"'''\n",
    "    test_case = \"\"\"Petitioners have been tenants in the rental unit for many years; Petitioner Wilkerson testified that she has lived in the rental unit for 35 years. The petition sets forth numerous complaints about the way the property is managed, and alleges employees and/or agents of Woodland Park harass, bully and retaliate against Petitioner Wilkerson.\n",
    "Concerning matters that appear to be directly related to issues of habitability and reduction/failure to maintain or repair, the petition states that the elevator was out of service for six months, that the property is not kept clean, for example, mud and dirt from a creek overflow in 2023 was not cleaned up for several months, and that there is no on-site manager. The petition also identifies other problems, most of which as identified in the petition, have a start date in 2016.\n",
    "At the hearing, Ms. Wilkerson's testimony tended to be about many items that first occurred many years ago, for example the removal of the swimming pool, storage units and the intercom system. She did testify, however, about a water overflow in her kitchen sink in December 2023 that resulted in the discovery of a roach infestation and mold, that the elevator was out of service for six months, and that lights in the common area do not work. She expressed dissatisfaction with the overall lack of maintenance at the property.\n",
    "Ms. French also testified that the property was not maintained properly and that she feels Ms. Wilkerson is harassed. Mr. Wilkerson who has not lived on the property for many years largely testified about matters that were several years old.\n",
    "Ms. Andrade testified that Woodland Park had been responsive to the concerns of the Petitioners. For example, there is an on-site manager (herself), there is daily maintenance, and that many of the issues about which Ms. Wilkerson has concerns occurred many years ago. She acknowledged that the elevator was out of service for a period of time but estimated that it was out of service only a month and a half, and that the lights in the common area are being replaced. Ms. Andrade testified Woodland Park made repairs in Petitioners' kitchen following the water overflow and she is not aware of any ongoing issues with roaches or mold in the rental unit.\n",
    "Since October 2023, Petitioners' rent has been $282.95/month. Prior to that, it was $824.51 monthly.\"\"\"\n",
    "    \n",
    "    result = client.analyze_case(test_case)\n",
    "    print(json.dumps(result, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8281c35b",
   "metadata": {},
   "source": [
    "#### Tenant Legal Team Agents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "44230f6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create prompt\n",
    "from langchain.prompts import PromptTemplate\n",
    "def create_research_prompt(topic: str) -> str:\n",
    "        base_prompt = f\"\"\"You are a legal research assistant helping analyze tenant rights cases. \n",
    "        \n",
    "        OBJECTIVE:\n",
    "        Research and analyze the legal context for: {topic}\n",
    "        \n",
    "        REQUIRED ANALYSIS COMPONENTS:\n",
    "        1. Jurisdiction Analysis\n",
    "        - Identify relevant state/local tenant laws\n",
    "        - Note any jurisdiction-specific requirements or precedents\n",
    "        \n",
    "        2. Case Law Research\n",
    "        - Find similar cases and their outcomes\n",
    "        - Identify key precedents that could support or challenge the tenant's position\n",
    "        - Note the strength of precedential value (binding vs persuasive)\n",
    "        \n",
    "        3. Statutory Analysis\n",
    "        - List applicable housing laws and regulations\n",
    "        - Highlight specific sections relevant to the dispute\n",
    "        - Note any recent legislative changes\n",
    "        \n",
    "        4. Evidence Assessment\n",
    "        - Types of evidence typically required\n",
    "        - Standard of proof needed\n",
    "        - Common defenses and counter-arguments\n",
    "        \n",
    "        5. Practical Considerations\n",
    "        - Typical timeframes for similar cases\n",
    "        - Potential remedies available\n",
    "        - Alternative dispute resolution options\n",
    "        \n",
    "        RESEARCH APPROACH:\n",
    "        1. First search for relevant statutes and regulations\n",
    "        2. Then look for precedent cases with similar fact patterns\n",
    "        3. Finally, analyze practical implications and likely outcomes\n",
    "        \n",
    "        FORMAT YOUR RESPONSE AS:\n",
    "        - Key Findings: (brief summary)\n",
    "        - Applicable Laws: (list relevant statutes)\n",
    "        - Precedent Cases: (list with brief descriptions)\n",
    "        - Strength of Case: (analysis)\n",
    "        - Recommended Strategy: (practical next steps)\n",
    "        \n",
    "        Sources must be reliable legal resources (e.g., official court records, state statutes, bar association publications).\n",
    "        \n",
    "        QUERY: {topic}\"\"\"\n",
    "        \n",
    "        return PromptTemplate(\n",
    "            input_variables=[\"topic\"],\n",
    "            template=base_prompt\n",
    "        ).format(topic=topic)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a169b7ba",
   "metadata": {},
   "source": [
    "#### The following is a four-agent system that uses the vector database to research legal precedents and case details. \n",
    "\n",
    "Agent 1: **Paralegal** (Responsible for initial research and case development)\n",
    "\n",
    "Agent 2: **Attorney 1** (Responsible for case strategy and legal argument development)\n",
    "\n",
    "Agent 3: **Attorney 2** (Responsible for case strategy and legal argument development)\n",
    "\n",
    "Agent 4: **(Expert Witness)** (Responsible for expert testimony generation)\n",
    "\n",
    "Agent 5: **Partner/Closing Attorney** (Responsible for summarizing the case and presenting the final recommendation)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "id": "4f54fcba",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TenantAdvocacyAgent:\n",
    "    def __init__(self, llm, vectorstore):\n",
    "        self.llm = llm\n",
    "        self.vectorstore = vectorstore\n",
    "        self.memory = ConversationBufferMemory(memory_key=\"chat_history\")\n",
    "        self.search = GoogleSearchAPIWrapper(google_api_key=GOOGLE_SEARCH_KEY, google_cse_id=GOOGLE_CSE_ID)\n",
    "        self.qa_chain = RetrievalQA.from_chain_type(\n",
    "            llm=llm,\n",
    "            chain_type=\"stuff\",\n",
    "            retriever=vectorstore.as_retriever(),\n",
    "            return_source_documents=True\n",
    "        )\n",
    "    \n",
    "    def _search_legal_precedents(self, query: str):\n",
    "        search_results = self.search.run(f\"legal precedent {query} tenant rights law case\")\n",
    "        return search_results\n",
    "\n",
    "    def _query_database(self, query: str):\n",
    "        docs = self.vectorstore.similarity_search(query, k=5)\n",
    "        return [{\"content\": doc.page_content, \"metadata\": doc.metadata} for doc in docs]\n",
    "\n",
    "    def _research_legal_context(self, topic: str):\n",
    "        # Initialize tools for the agent\n",
    "        toolkit = [\n",
    "            Tool(\n",
    "                name=\"Search\",\n",
    "                func=self._search_legal_precedents,\n",
    "                description=\"Searches Google for legal precedents and relevant laws\"\n",
    "            ),\n",
    "            Tool(\n",
    "                name=\"Database\", \n",
    "                func=self._query_database,\n",
    "                description=\"Searches internal database of cases\"\n",
    "            )\n",
    "        ]\n",
    "\n",
    "        # Initialize agent\n",
    "        agent = initialize_agent(\n",
    "            toolkit, \n",
    "            self.llm,\n",
    "            agent=\"zero-shot-react-description\", \n",
    "            verbose=True,\n",
    "            return_intermediate_steps=True,\n",
    "            handle_parsing_errors=True\n",
    "        )\n",
    "        research_prompt = create_research_prompt(topic)\n",
    "        result = agent.invoke({\"input\":research_prompt})\n",
    "\n",
    "        return result[\"output\"]\n",
    "\n",
    "        \n",
    "    def _analyze_case_details(self, case_info: str) -> dict:\n",
    "        # Analyze case details using agent\n",
    "        agent = initialize_agent(\n",
    "            [\n",
    "                Tool(name=\"SearchDatabase\", func=self._query_database, description=\"Search internal database for similar cases\"),\n",
    "                Tool(name=\"AnalyzeEvidence\", func=lambda x: self.qa_chain({\"query\": \"What evidence was crucial in this case?\", \"context\": x}), description=\"Analyze evidence patterns in cases\")\n",
    "            ],\n",
    "            self.llm,\n",
    "            agent=\"zero-shot-react-description\",\n",
    "            verbose=True,\n",
    "            return_intermediate_steps=True\n",
    "        )\n",
    "        prompt = f\"\"\"Analyze this case with the following specific framework:\n",
    "        1. VIOLATIONS & LEGAL BASIS\n",
    "        - Identify each specific violation and cite relevant law/ordinance\n",
    "        - Categorize severity of each violation\n",
    "        - Map violations to specific sections of CSFRA/local ordinances\n",
    "        \n",
    "        2. EVIDENCE ANALYSIS \n",
    "        - List and evaluate strength of each piece of evidence\n",
    "        - Identify critical evidence gaps\n",
    "        - Recommend additional evidence needed\n",
    "        \n",
    "        3. PRECEDENT MAPPING\n",
    "        - Find 3-5 most similar cases from database\n",
    "        - Compare fact patterns and outcomes\n",
    "        - Extract winning legal arguments\n",
    "        \n",
    "        Case details: {case_info} Provide detailed analysis for each section.\"\"\"\n",
    "\n",
    "        result = agent.invoke({\n",
    "            \"input\": prompt\n",
    "        })\n",
    "        \n",
    "        return result['output']\n",
    "\n",
    "    def _analyze_compensation(self, case_details: str) -> dict:\n",
    "        \"\"\"Agent that analyzes compensation patterns and likely outcomes\"\"\"\n",
    "        # Analyze compensation using agent\n",
    "        agent = initialize_agent(\n",
    "            [\n",
    "                Tool(name=\"SearchCompensation\", func=lambda x: self._query_database(f\"compensation amounts {x}\"), description=\"Search for similar compensation amounts\"),\n",
    "                Tool(name=\"AnalyzeOutcomes\", func=lambda x: self.qa_chain({\"query\": \"Detail exactly what factors determined the compensation?\", \"context\": x}), description=\"Analyze factors affecting outcomes\")\n",
    "            ],\n",
    "            self.llm, \n",
    "            agent=\"zero-shot-react-description\",\n",
    "            verbose=True\n",
    "        )\n",
    "        result = agent.invoke({\n",
    "            \"input\": f\"\"\"Analyze potential compensation for this case:\n",
    "            1. What compensation ranges are typical?\n",
    "            2. What factors affect the amount?\n",
    "            3. What are the likely outcomes?\n",
    "            Case details: {case_details}\"\"\"\n",
    "        })\n",
    "\n",
    "        return result['output']\n",
    "    def generate_expert_testimony(self, complaint: str) -> str:\n",
    "        client = GeminiStructuredClient(API_KEY)\n",
    "        result = client.analyze_case(complaint)\n",
    "        #return(json.dumps(result, indent=2))[\"raw_output\"]\n",
    "        return result\n",
    "    def analyze_tenant_case(self, complaint: str) -> dict:\n",
    "        legal_context = self._research_legal_context(complaint)\n",
    "        case_analysis = self._analyze_case_details(json.dumps({\"complaint\": complaint, \"legal_context\": legal_context, \"analysis_goals\": [\"Identify strongest legal arguments\", \"Evaluate evidence quality\", \"Find similar successful cases\", \"Assess compliance with local ordinances\"]}))\n",
    "        compensation_analysis = self._analyze_compensation(json.dumps({\"case_analysis\": case_analysis, \"legal_context\": legal_context, \"focus_areas\": [\"Rent increase compliance\", \"Habitability violations\", \"Local tenant protections\", \"Historical petition outcomes\"]}))\n",
    "        # Initialize agent to generate strategic recommendations\n",
    "        agent = initialize_agent(\n",
    "            [\n",
    "                Tool(name=\"ReviewEvidence\", func=lambda x: self.qa_chain({\"query\": \"Identify gaps in the evidence.\", \"context\": x}), description=\"Review evidence gaps\"),\n",
    "                Tool(name=\"FindPrecedents\", func=lambda x: self._query_database(f\"similar successful cases {x}\"), description=\"Find relevant precedents\")\n",
    "            ],\n",
    "            self.llm,\n",
    "            agent=\"zero-shot-react-description\", \n",
    "            verbose=True,\n",
    "            return_intermediate_steps=True\n",
    "        )\n",
    "        expert_witness_analysis = self.generate_expert_testimony(complaint)\n",
    "        \n",
    "        strategy = agent.invoke({\n",
    "            \"input\": f\"\"\"Generate strategic recommendations based on:\n",
    "            1. Legal context: {legal_context}\n",
    "            2. Case analysis: {case_analysis} \n",
    "            3. Compensation analysis: {compensation_analysis}\n",
    "            4. Expert witness analysis: {expert_witness_analysis}\n",
    "            \n",
    "            Focus on:\n",
    "            - Strengthening legal arguments\n",
    "            - Prioritizing evidence collection\n",
    "            - Leveraging precedent cases\n",
    "            - Maximizing tenant protections\"\"\"\n",
    "        })['output']\n",
    "        return {\"legal_context\": legal_context, \"case_analysis\": case_analysis, \"compensation_analysis\": compensation_analysis, \"strategic_recommendations\": strategy}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "id": "9224dde9",
   "metadata": {},
   "outputs": [
    {
     "ename": "UnboundLocalError",
     "evalue": "local variable 'legal_precedents' referenced before assignment",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mUnboundLocalError\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[175], line 15\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[38;5;124;03m'''complaint =\u001b[39;00m\n\u001b[1;32m      9\u001b[0m \u001b[38;5;124;03mMy landlord has failed to fix a leaking roof for 3 months despite multiple \u001b[39;00m\n\u001b[1;32m     10\u001b[0m \u001b[38;5;124;03mwritten requests. The leak has caused mold growth and damage to my belongings. \u001b[39;00m\n\u001b[1;32m     11\u001b[0m \u001b[38;5;124;03mI have photos, maintenance request emails, and a contractor's assessment of the damage. \u001b[39;00m\n\u001b[1;32m     12\u001b[0m \u001b[38;5;124;03m'''\u001b[39;00m\n\u001b[1;32m     14\u001b[0m \u001b[38;5;66;03m#analysis = tenant_agent.analyze_tenant_case(complaint_str)\u001b[39;00m\n\u001b[0;32m---> 15\u001b[0m expert \u001b[38;5;241m=\u001b[39m \u001b[43mtenant_agent\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgenerate_expert_testimony\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcomplaint_str\u001b[49m\u001b[43m)\u001b[49m\n",
      "Cell \u001b[0;32mIn[174], line 111\u001b[0m, in \u001b[0;36mTenantAdvocacyAgent.generate_expert_testimony\u001b[0;34m(self, complaint)\u001b[0m\n\u001b[1;32m    109\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mgenerate_expert_testimony\u001b[39m(\u001b[38;5;28mself\u001b[39m, complaint: \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28mstr\u001b[39m:\n\u001b[1;32m    110\u001b[0m     client \u001b[38;5;241m=\u001b[39m GeminiStructuredClient(API_KEY)\n\u001b[0;32m--> 111\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[43mclient\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43manalyze_case\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcomplaint\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    112\u001b[0m     \u001b[38;5;66;03m#return(json.dumps(result, indent=2))[\"raw_output\"]\u001b[39;00m\n\u001b[1;32m    113\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m result\n",
      "Cell \u001b[0;32mIn[173], line 22\u001b[0m, in \u001b[0;36mGeminiStructuredClient.analyze_case\u001b[0;34m(self, case_details)\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21manalyze_case\u001b[39m(\u001b[38;5;28mself\u001b[39m, case_details: \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Dict[\u001b[38;5;28mstr\u001b[39m, Any]:\n\u001b[1;32m     10\u001b[0m     search \u001b[38;5;241m=\u001b[39m GoogleSearchAPIWrapper(\n\u001b[1;32m     11\u001b[0m         google_api_key\u001b[38;5;241m=\u001b[39mGOOGLE_SEARCH_KEY,\n\u001b[1;32m     12\u001b[0m         google_cse_id\u001b[38;5;241m=\u001b[39mGOOGLE_CSE_ID\n\u001b[1;32m     13\u001b[0m     )\n\u001b[1;32m     16\u001b[0m     structured_prompt \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[1;32m     17\u001b[0m \u001b[38;5;124m    You are an expert witness with deep specialized knowledge in tenant rights cases.\u001b[39m\n\u001b[1;32m     18\u001b[0m \u001b[38;5;124m    CASE DETAILS:\u001b[39m\n\u001b[1;32m     19\u001b[0m \u001b[38;5;124m    \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mcase_details\u001b[38;5;132;01m}\u001b[39;00m\n\u001b[1;32m     20\u001b[0m \n\u001b[1;32m     21\u001b[0m \u001b[38;5;124m    RELEVANT PRECEDENTS:\u001b[39m\n\u001b[0;32m---> 22\u001b[0m \u001b[38;5;124m    \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[43mlegal_precedents\u001b[49m\u001b[38;5;132;01m}\u001b[39;00m\n\u001b[1;32m     23\u001b[0m \n\u001b[1;32m     24\u001b[0m \u001b[38;5;124m    SIMILAR CASES:\u001b[39m\n\u001b[1;32m     25\u001b[0m \u001b[38;5;124m    \u001b[39m\u001b[38;5;132;01m{\u001b[39;00msimilar_cases\u001b[38;5;132;01m}\u001b[39;00m\n\u001b[1;32m     26\u001b[0m \n\u001b[1;32m     27\u001b[0m \u001b[38;5;124m    Your role is to:\u001b[39m\n\u001b[1;32m     28\u001b[0m \u001b[38;5;124m    1. Identify ONE suspicious or concerning element from the case that others might miss\u001b[39m\n\u001b[1;32m     29\u001b[0m \u001b[38;5;124m    2. Obsessively investigate this element using available tools\u001b[39m\n\u001b[1;32m     30\u001b[0m \u001b[38;5;124m    3. Build a compelling case around this element\u001b[39m\n\u001b[1;32m     31\u001b[0m \u001b[38;5;124m    4. Provide detailed evidence and citations\u001b[39m\n\u001b[1;32m     32\u001b[0m \u001b[38;5;124m    Analyze the following case and fill out the JSON using the appeal and your opinion.\u001b[39m\n\u001b[1;32m     33\u001b[0m \u001b[38;5;124m    Use this JSON schema:\u001b[39m\n\u001b[1;32m     34\u001b[0m \u001b[38;5;124m    \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\n\u001b[1;32m     35\u001b[0m \u001b[38;5;250m        \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mprimary_sources\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;132;01m:\u001b[39;00m\u001b[38;5;124m \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstr\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m,\u001b[39m\n\u001b[1;32m     36\u001b[0m \u001b[38;5;124m        \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124msecondary_sources\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m: \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstr\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m,\u001b[39m\n\u001b[1;32m     37\u001b[0m \u001b[38;5;124m        \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mexpert_opinions\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m: \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstr\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m,\u001b[39m\n\u001b[1;32m     38\u001b[0m \u001b[38;5;124m        \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124msimilar_cases\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m: \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstr\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m,\u001b[39m\n\u001b[1;32m     39\u001b[0m \u001b[38;5;124m        \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstatistical_data\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m: \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstr\u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m     40\u001b[0m \u001b[38;5;124m    \u001b[39m\u001b[38;5;132;01m}\u001b[39;00m\n\u001b[1;32m     41\u001b[0m \u001b[38;5;124m    \u001b[39m\n\u001b[1;32m     42\u001b[0m \u001b[38;5;124m    Remember:\u001b[39m\n\u001b[1;32m     43\u001b[0m \u001b[38;5;124m    - All values must be strings\u001b[39m\n\u001b[1;32m     44\u001b[0m \u001b[38;5;124m    - Include all fields defined in the schema\u001b[39m\n\u001b[1;32m     45\u001b[0m \u001b[38;5;124m    - Ensure the output is valid JSON\u001b[39m\n\u001b[1;32m     46\u001b[0m \u001b[38;5;124m    \u001b[39m\n\u001b[1;32m     47\u001b[0m \u001b[38;5;124m    Case to analyze:\u001b[39m\n\u001b[1;32m     48\u001b[0m \u001b[38;5;124m    \u001b[39m\u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[1;32m     50\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m     51\u001b[0m         key_terms \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmodel\u001b[38;5;241m.\u001b[39mgenerate_content( \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mExtract 3-5 key legal issues from this case as comma-separated terms: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mcase_details\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mtext\u001b[38;5;241m.\u001b[39mstrip()\n",
      "\u001b[0;31mUnboundLocalError\u001b[0m: local variable 'legal_precedents' referenced before assignment"
     ]
    }
   ],
   "source": [
    "llm = ChatGoogleGenerativeAI(\n",
    "    model=\"gemini-1.5-pro\",\n",
    "    temperature=1,\n",
    "    google_api_key=GEMINI_API\n",
    ")\n",
    "tenant_agent = TenantAdvocacyAgent(llm=llm, vectorstore=vectorstore)\n",
    "complaint_str = ' '.join([doc.page_content for doc in loadfile])\n",
    "'''complaint =\n",
    "My landlord has failed to fix a leaking roof for 3 months despite multiple \n",
    "written requests. The leak has caused mold growth and damage to my belongings. \n",
    "I have photos, maintenance request emails, and a contractor's assessment of the damage. \n",
    "'''\n",
    "\n",
    "#analysis = tenant_agent.analyze_tenant_case(complaint_str)\n",
    "expert = tenant_agent.generate_expert_testimony(complaint_str)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "id": "492a4960",
   "metadata": {},
   "outputs": [
    {
     "ename": "JSONDecodeError",
     "evalue": "Expecting value: line 1 column 1 (char 0)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mJSONDecodeError\u001b[0m                           Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[160], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mjson\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mloads\u001b[49m\u001b[43m(\u001b[49m\u001b[43mexpert\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mraw_response\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/nas/ucb/davidyang/miniforge3/envs/colab_clone/lib/python3.10/json/__init__.py:346\u001b[0m, in \u001b[0;36mloads\u001b[0;34m(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)\u001b[0m\n\u001b[1;32m    341\u001b[0m     s \u001b[38;5;241m=\u001b[39m s\u001b[38;5;241m.\u001b[39mdecode(detect_encoding(s), \u001b[38;5;124m'\u001b[39m\u001b[38;5;124msurrogatepass\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m    343\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (\u001b[38;5;28mcls\u001b[39m \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m object_hook \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m\n\u001b[1;32m    344\u001b[0m         parse_int \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m parse_float \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m\n\u001b[1;32m    345\u001b[0m         parse_constant \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m object_pairs_hook \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m kw):\n\u001b[0;32m--> 346\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43m_default_decoder\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdecode\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    347\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mcls\u001b[39m \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    348\u001b[0m     \u001b[38;5;28mcls\u001b[39m \u001b[38;5;241m=\u001b[39m JSONDecoder\n",
      "File \u001b[0;32m/nas/ucb/davidyang/miniforge3/envs/colab_clone/lib/python3.10/json/decoder.py:337\u001b[0m, in \u001b[0;36mJSONDecoder.decode\u001b[0;34m(self, s, _w)\u001b[0m\n\u001b[1;32m    332\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mdecode\u001b[39m(\u001b[38;5;28mself\u001b[39m, s, _w\u001b[38;5;241m=\u001b[39mWHITESPACE\u001b[38;5;241m.\u001b[39mmatch):\n\u001b[1;32m    333\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"Return the Python representation of ``s`` (a ``str`` instance\u001b[39;00m\n\u001b[1;32m    334\u001b[0m \u001b[38;5;124;03m    containing a JSON document).\u001b[39;00m\n\u001b[1;32m    335\u001b[0m \n\u001b[1;32m    336\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 337\u001b[0m     obj, end \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mraw_decode\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43midx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m_w\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mend\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    338\u001b[0m     end \u001b[38;5;241m=\u001b[39m _w(s, end)\u001b[38;5;241m.\u001b[39mend()\n\u001b[1;32m    339\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m end \u001b[38;5;241m!=\u001b[39m \u001b[38;5;28mlen\u001b[39m(s):\n",
      "File \u001b[0;32m/nas/ucb/davidyang/miniforge3/envs/colab_clone/lib/python3.10/json/decoder.py:355\u001b[0m, in \u001b[0;36mJSONDecoder.raw_decode\u001b[0;34m(self, s, idx)\u001b[0m\n\u001b[1;32m    353\u001b[0m     obj, end \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mscan_once(s, idx)\n\u001b[1;32m    354\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mStopIteration\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m err:\n\u001b[0;32m--> 355\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m JSONDecodeError(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mExpecting value\u001b[39m\u001b[38;5;124m\"\u001b[39m, s, err\u001b[38;5;241m.\u001b[39mvalue) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m    356\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m obj, end\n",
      "\u001b[0;31mJSONDecodeError\u001b[0m: Expecting value: line 1 column 1 (char 0)"
     ]
    }
   ],
   "source": [
    "\n",
    "json.loads(expert[\"raw_response\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "id": "83d09c81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'primary_sources': \"City of Mountain View Hearing Officer Decision, Exhibits (Petitioner's 1-77, Respondent's 1-2, Hearing Officer's 1-7); Mountain View CSFRA (Community Stabilization and Fair Rent Act); California Civil Code Sections 827, 1941, 1941.1; California Health and Safety Code Sections 17920.3, 17920.10; California Electrical Code Sections 404.9, 406.6, 400.12, 240.5, 210.8(a); California Fire Code Section 603.2.1; International Property Maintenance Code Sections 309, 603, 704; California Building Code Sections 915, 420.5.\",\n",
       " 'secondary_sources': 'Baychester Shopping Ctr., Inc. v. San Francisco Residential Rent Stabilization & Arbitration Bd., 165 Cal.App.4th 1000 (2008); Erlach v. Sierra Asset Servicing, LLC, 226 Cal.App.4th 1281 (2014); Knight v. Halltshammar, 29 Cal.3d 46 (1981); Green v. Superior Court, 10 Cal.3d 616 (1974).',\n",
       " 'expert_opinions': 'While the Hearing Officer Decision addresses many issues, a crucial point, easily overlooked, is the implication of Mr.  (the former property manager) potentially acting as an agent for Mr. Sanchez *before* the official transfer of ownership in January 2023.  Ms.  ’s affidavit statement, \"In the past I understood that Mr. Sergio Sanchez was the manager…,\" raises a significant question about Sanchez\\'s actual involvement prior to his acknowledged ownership.  This could shift the timeline of his responsibility for the uninhabitable conditions and associated damages.  Investigating this could involve: 1. Deposing Ms.   and other tenants to clarify their understanding of Sanchez\\'s role prior to January 2023.  Did they interact with Sanchez directly regarding maintenance issues? Did he present himself as having authority over the property? 2. Examining financial records. Were any payments for maintenance or repairs made by Sanchez, or through accounts he controlled, before January 2023?  3.  Obtaining communications (emails, texts, letters) between Sanchez and the former Landlord, and between Sanchez and  , to establish the nature of his involvement in property management during that period.  A pattern of directions or decisions from Sanchez could establish de facto control and thus earlier liability for the uninhabitable conditions.  This would strengthen the tenant\\'s case for a larger rent reduction based on a longer period of substandard conditions.',\n",
       " 'similar_cases': \"Cases involving successor landlord liability for habitability issues are relevant.  Focus should be placed on cases where the successor's involvement prior to formal ownership was a factor in determining liability. Researching cases involving implied agency or de facto management could provide legal precedent supporting earlier responsibility for Mr. Sanchez.\",\n",
       " 'statistical_data': 'Data on average repair costs for similar issues (mold remediation, electrical work, plumbing, heating system repairs, appliance replacement, carpet replacement) in Mountain View or comparable areas would be helpful to corroborate the reasonableness of the calculated damages.  Additionally, rental market data demonstrating the fair market value of similar units in good condition versus units with the specified deficiencies would provide a stronger basis for calculating rent reduction based on loss of use.  This data should be time-specific, reflecting market conditions during the periods of disrepair.'}"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13e21397",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "11db3f69",
   "metadata": {},
   "source": [
    "While this worked this time, it's not a long term reliable method for more advanced use cases"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "colab_clone",
   "language": "python",
   "name": "colab_clone"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}