In this video, we will discuss how to load your private documents into Lanchain.
This is the first step in building the question -answering application.
The purpose of this project is to ask questions about the content of your
custom or private data, which is stored in different formats, such as Pandas
DataFrames, PDFs, CSV or JSON files, HTML or Office documents.
Lanchain provides with document loaders which load the data into documents.
In the Lanchain terminology, a document is a fancy way of saying some piece of text.
These document loaders are aimed at making this task easy.
There are two types of document loaders, transform loaders and public and
proprietary data set or service loaders.
Transform loaders will transform the data
from a specific format into the Lanchain document format.
This is what you will use most of the time for formats such as PDFs, CSV and
JSON files, Office documents, and many more.
You can see the entire list in the
Lanchain documentation.
The public data set and service loaders
are used to query and search for data on public websites such as Project
Gutenberg, Hacker News, Wikipedia, and so on.
The proprietary data set and service
loaders are used for private or proprietary services such as Azure, AWS
S3, Google Cloud, Discord, Twitter, and many more.
In this video, I will show you how to load your data from PDFs, which is a very
common data format.
In the next video, we will generalize it
for other formats as well.
I am loading the environment variables
used for authentication with OpenAI and Pinecon.
To load PDF files, I am going to install
another library called pypdf that is required.
So pip install pypdf I want my code to be
modular so I am defining a function.
So df load document of file The function
will take as an argument a PDF file and return its text content.
I am importing the required libraries.
From lanchain .documentloaders I am
importing the pypdf loader class.
By the way, the standard recommendation
is to put import statements at the top of the file.
Here.
However, there are cases when putting
import statements inside the function is even better.
This will prevent circular dependencies and you will also benefit from a more
reliable refactoring of your code.
When you move a function from one module
to another, you will know that the function will continue to work because it
contains everything inside it.
This function loads the PDFs using a
library called pypdf into an array of documents where each document contains
the page content and metadata with a page number.
I am printing a message for the user.
Loading and file.
And the loader equals pypdf loader of file.
Note that it is also able to load online PDFs.
Just pass a URL to a PDF to pypdf loader.
The next step is to call loader .load.
This will return a list of lang chain documents.
One document for each page.
And I am returning the data.
The list.
This will be the section for the running code.
Let's test the function.
I am loading a local PDF file.
In the current directory, there is a subdirectory called files that contains a
PDF called us -constitution.
This one.
I am loading it.
So data equals load document of and the
argument is files and us -constitution.
The data was split by pages and you can
use indexes to display a specific page.
For example, print data of 1.
This is the second page because it starts from 0.
That page content.
I am running the cell that contains the function.
And this one.
And I've got the page.
If you want to see the metadata of this document, use the metadata attribute.
For example, data of 10 .metadata.
This is a dictionary.
Let's check how many pages there are.
You have length of data, pages, in your data.
There are 41 pages.
And if you want to see how many
characters are in one page, you can do print.
There are curly braces, length of data
of, let's say, 20.
That page content.
Characters in the page.
If you want to load remote PDFs, just
pass the URL to the function as an argument.
We are done.
In this lecture, I showed you how to load PDFs into Lanchain documents.
We'll take a quick break.
And in the next video, I'll show you how
to load other document formats into Lanchain.
Bye Bye.