We have loaded the data into long chain documents and split them into smaller chunks.
Next, we will embed the chunks and upload both the chunks and the embeddings to a
pine cone vector store for fast retrieval and similarity search.
I am defining a function called insertOrFetchEmbeddings.
This function will create an index if the
index doesn't exist, embed the chunks and add both the chunks and embeddings into
the pine cone index.
If the index already exists, the function
will just load the embeddings from that index.
This function will have two arguments,
indexName and chunks.
I am importing the necessary libraries.
I am importing the pine cone library and from long chain community vector stores,
I am importing the pine cone class.
From long chain OpenAI, I am importing
OpenAI embeddings and from pine cone, I am importing the pod spec class.
This is necessary to create an index.
Next, I will initialize the pine cone client.
To do this, you will need the pine cone API key.
We covered this in the last section where we discussed pine cone for the first time.
If you skipped that video, I can tell you in just a few hours what you have to do now.
Go to pinecone .io and create a free pine cone account.
Log into your pine cone account, go to API keys and generate one.
Copy the key to keyboard and paste it in the .env file where the OpenAI API key is to.
This is my .env file that contains both the OpenAI API key and the pine cone API key.
I am creating a pine cone object, pc equals pinecone .pinecone.
The constructor expects an environment variable called pine cone API key.
If you have already created and loaded such a variable into memory as we did,
the authentication is automatically handled.
However, if you haven't loaded the
environment variable, you should explicitly pass an argument named API key
with your keys value to the pine cone constructor like this.
API underline key equals and your API key.
Next, I will create an instance of the
OpenAI embeddings class which will be used to embed text into vectors.
Embeddings equals OpenAI embeddings and the arguments model equals text embedding
3small and dimensions 1536.
If the OpenAI API key is loaded as an
environment variable, it is not necessary to specify it as an argument.
If the pine cone index which is the functions argument already exists, you
can load the embeddings like this.
First, I am checking that the index
already exists.
If indexName in PC .list indexes .names,
I am printing a message and loading the embeddings index indexName in curly
braces already exists and loading embeddings and equals empty string and
vectorStore equals pineCone .fromExistingIndex of indexName and
embeddings, the OpenAI embeddings object.
I am also printing ok, but if the index
does not exist, create the index and insert the embeddings.
I am printing a message for the user.
Creating index, indexName in curly braces
and embeddings.
I am creating a new index, PC
.createIndex and the arguments name equals indexName, the functions
parameter, dimensions equals 1536, metric equals cosine and the spec equals podSpec
of environment equals gcpStarter and if you want to create a serverless index
instead of a pod based one, use this configuration instead.
I am calling pineCone .fromDocuments, vectorStore equals the pineCone class
.fromDocuments and the arguments are chunks, embeddings and the indexName
equals indexName.
This method is processing the input
documents, the chunks, generating the embeddings using the provided OpenAI
embeddings instance, inserting the embeddings into the index and returning a
new pineCone vectorStore object.
I am also printing ok and return vectorStore.
That is it for insert or fetch embeddings.
I will also create a function that
deletes a pineCone index or all the indexes because the pineCone free tier
supports only one index, it could be necessary to delete the existing index frequently.
So, def deletePineConeIndex and the function will have one argument indexName
with the default value all.
I am importing the pineCone library and
the pc equals pineCone .pineCone.
By the way, pineCone written in lowercase
is the library and the pineCone written with an uppercase p is a class contained
in this library and if indexName equals equals all, the default value,
I am deleting all the indexes.
So, indexes equals pc .listIndexes
.names, I am displaying a message for the user deleting all indexes and a for loop
for index in indexes colon pc .deleteIndex of index and I am printing
ok, else if the user provided an argument, a name for the indexName, I
will delete only that index, I am displaying a message.
Deleting index, indexName in curly braces and end equals empty string and I am
calling deleteIndex, pc .deleteIndex of indexName, that is it.
I am running the code in these two cells.
Let's test the functions.
To make sure there are no pineCone indexes, I will remove all indexes first.
I am using the pineCone free tier and I want to avoid getting an error when I try
to create a new index.
So, delete pineCone index.
Before running the function, I will check my on pineCone .io.
There is an index called ask a document.
Very well.
I am calling the function.
It is deleting all the indexes.
It is done and I am refreshing this page.
There is no index on pineCone.
Next, I will create an index, embed the chunks and upload both the chunks and the
embeddings to pineCone.
Let's say indexName equals ask a document
and the vector store equals insert or fetch embeddings.
The function I have just defined.
The arguments are indexName and chunks.
I have already created the chunks by calling chunkData.
I am running the code in this cell again just to be sure that the data was chunked.
I am running the code and I have got an error.
The correct name is dimensions not
dimensions and I noticed another mistake.
It is GCP, Google Cloud Platform.
I am running the code again.
All right, the index is being created.
It is done.
Let's check it on pineCone.
I am refreshing the page and we notice a new index called ask a document.
We can also notice that the index contains 190 vectors.
These vectors are the chunks that were embedded.
Great, we have loaded the data into
LangStream documents, split the documents into chunks, embedded the chunks and
uploaded them to the pineCone index.
We have accomplished a lot of work.
In the next video, we will start asking questions and getting answers based on
similarity search.