Welcome back.
In this video, we'll embed each chunk of
text into numeric vectors and insert them into a Pinecone index.
I'm importing the necessary libraries and initializing the Pinecone client.
Import Pinecone from LangchainCommunity .VectorStores import the Pinecone class
and I'm initializing the client.
Pay attention, there are two classes
named Pinecone, one in the Pinecone module and another one in
LangchainCommunity .VectorStores.
The next step is to create a Pinecone index.
You can also use an existing index if you want.
Note that if you are currently using the
free plan, you are limited to one index and one project and if you already have
one, you will get an error if you try to create a second one.
So it's a good idea to delete all existing indexes before creating a new one.
For i in PC .listIndex .names, PC .deleteIndex of i.
And it's always a good idea to display a message for the user.
Deleting all indexes and print, done.
I'm also adding end equals empty string.
Good.
I'm creating a new index for these
embeddings called ChurchillSpeech.
indexName equals ChurchillSpeech.
If indexName not in PC .listIndex .names, if the index does not exist, I will
create it.
Creating index and indexName in curly braces.
And PC .createIndex and the arguments are name equals indexName, dimension equals
1536, metric equals cosine and the spec equals pinecone .podSpec of environment
equals gcpStarter.
After creating the index, I am printing done.
I am running the code.
Note that indexNames must consist of
lowercase alphanumeric characters or dashes and must start and end with an
alphanumeric character.
Next, we will upload the vectors to
pinecone using langchain.
We will call pinecone .fromDocuments.
pinecone .fromDocuments.
This method takes three arguments,
chunks, embeddings and the indexName.
Chunks is a list of text documents that
have been obtained by calling the recursive character text splitter method.
These smaller chunks will be indexed in pinecone to make it easier to search and
retrieve relevant information later on.
Embeddings is an instance of the OpenAI
Embeddings class which is responsible for converting text data into embeddings
using OpenAI's embedding model.
These embeddings will be stored in the
pinecone index and used for similarity search.
And the indexName is a string
representing the name of the pinecone index.
This name is used to identify the index
in pinecones database and must already exist.
We have defined all these objects earlier.
Chunks, embeddings and indexName.
The pinecone .fromDocuments method also
returns a vectorStore object initialized from documents and embeddings.
vectorStore equals.
In a nutshell, this method processes the
input documents, generates embeddings using the provided OpenAI Embeddings
instance and returns a new pinecone vectorStore.
The resulting vectorStore object can
perform similarity searches and retrieve relevant documents based on user queries.
I am running it.
Ok, if you get this error, use a keyword
argument for indexName.
So, indexName equals indexName.
I am running it again and there are no errors.
Let's take a look at the pinecone dashboard.
We can see that the index contains 300 vectors.
These vectors were inserted by this code.
The 300 vectors represent the numeric representation of the 300 chunks of text
we saw earlier.
Take a look here.
We have 300 chunks of text.
If you want to specify a specific
embedding model, add the model argument to OpenAI Embeddings.
For example, the recommended embedding model is a text embedding 3, small or large.
With this, we have successfully embedded the text into vectors and inserted them
into a pinecone index.
You will do this only once when split the
document into chunks and embed those chunks into pinecone.
Once you populated your index with your embeddings, you will just query the index.
If you call pinecone .fromDocuments again, it will insert the vectors again
and creating double entries.
To load the vector store from an existing
index, do vector store equals pinecone .fromExistingIndex and the arguments are
indexName equals the name of the index, ChurchillSpeech, and embedding equals embeddings.
This is loading the vector store from an existing index.
Very well.
Now, let's take a break.
In the next video, we will demonstrate how to use similarity search to ask
questions about the content of this custom document.