Welcome back.
Up until now, we've loaded our custom or
private data into Lanchain documents.
The next step is to split or chunk the
documents into smaller parts.
In the context of building LLM -related
applications, chunking is the process of breaking down large pieces of text into
smaller segments.
It's an essential technique that helps
optimize the relevance of the content we get back from a vector database once we
use the LLM to embed content.
As you know, we need to first embed any
content that we index in Pinecon.
The main reason for chunking is to make
sure that we are embedding a piece of content with as little noise as possible
while still keeping it semantically relevant.
For example, in a semantic search, we
index a corpus of documents.
Each document contains valuable
information on a specific topic.
By applying an effective chunking
strategy, we can make sure that our search results accurately capture the
essence of the user's query.
If our chunks are too small or too large,
it may lead to imprecise search results or missed opportunities to surface
relevant content.
As a rule of thumb, if a chunk of text
makes sense without the surrounding context to a human, it will make sense to
the language model as well.
Therefore, finding the optimal chunk size
for the documents in the corpus is crucial to ensure that the search results
are accurate and relevant.
I am defining a function called chunkData
.df chunkData of data.
Lanchain provides many text splitters,
but recursive character text splitter is the recommended one for generic text.
I am importing it from Lanchain.
textSplitter import recursive character
text splitter By default, the characters it tries to
split on are double backslash n, backslash n, and whitespace.
I have already explained to you in the previous section how recursive character
text splitter works, and now I am just writing the code.
When a sentence is embedded, the resulting vector focuses on the
sentence's specific meaning.
The comparison would naturally be done on
that level when compared to other sentence embeddings.
This also implies that embedding may miss out a broader contextual information
found in a paragraph.
When a full paragraph or document is
embedded, the embedding process considers both the overall context and the
relationships between the sentences and the phrases within the text.
This can result in a more comprehensive vector representation that captures the
broader meaning of the text.
To be flexible, I'll add a second
argument with a default value for the chunk size, instead of this hard -coded value.
So, chunkSize equals to 5, 6.
And here chunkSize equals chunkSize the
function's parameter.
It's better now.
And chunks equals textSplitter .splitDocumentsOfData.
It returns a list of documents.
Use createDocuments instead of splitDocuments when it's not already
splitted in pages.
This depends on how you have loaded the data.
And I'm returning the chunks.
I'm running the cell.
Let's test it in the running code part.
Ok.
I'm gonna load the US Constitution.
I'm commenting out the other code.
I'm running this cell.
I'm loading it.
And I'm running chunks equals chunkDataOfData.
Data was returned by loadDocument.
Let's see how many chunks do I have.
printLen of chunks.
And I'm running it.
There are 190 chunks.
And if you want to see one chunk of data, you can do print chunks of 10 .pageContent.
This is the chunk at index 10.
Good.
We'll be using OpenAI's Text Embedding ADA -002, which has a cost.
It's prudent to calculate the embedding costs in advance to avoid any surprises.
We'll utilize the TickToken library for this.
As I've already shown you how to do this
in the first part of the course, I'll be simply copy -pasting the code.
I'm running the cell.
And checking the cost.
So print embedding cost of chunks.
Ok.
It's really cheap.
After chunking the document, the next
step is to embed the chunks and upload them to a vector database like Pinecon.
We'll take a break, and in the next video, we'll discuss embedding and
uploading to a vector database.