Now a really nice thing to do at this point is to potentially save this information as a file, because 1
we've done all this work reading all the individual files, extracting the email bodies, cleaning up our 2
data and creating this nice structure here that we're going to work with going forward, 3
it's probably a good idea to save this dataframe in some sort of format as its own file. 4
That way we can simply load this one file and skip all the work that we've done beforehand. So I'm going to 5
insert a markdown cell here and I'm going to write "Save to File using Pandas", add a few more rows and 6
here we're gonna write the code that saves our dataframe to the disk. 7
Now, we could be saving this file in a number of different formats, for example like a CSV file or 8
a text file or what have you. 9
And most of these libraries like pandas or numpy or what have you, 10
they'll have the functionality to create a file locally on your disk. 11
The method that I want to show you in this lesson is called "to_json", "data.to_json()" 12
will be the method that we're going to use. If I hit Shift+Tab, I 13
get a quick description 14
and here you can see that the key information that we need to supply is a path or a buffer. 15
Now this is just a string with a relative path and a file name. 16
We've actually worked with something very, very similar already with our practice email. 17
So let's create another constant up here and call it "DATA_ 18
JSON_FILE" and set that equal to single quotes, and then I'm going to copy this bit here, because 19
I'm very, very lazy and I don't want to make any typos and I'm going to paste it in here and then just 20
tack on my file name as well. I'm going to call this "email-text-data.json"; "json" here is the 21
file extension. Let me hit Shift+Enter on this. Now I can come down here and simply call on this constant. 22
Now before I hit Shift+Enter on the cell, I've pulled up my Processing folder here on the left so you 23
can see what's happening when I actually execute this code. 24
You should see this new JSON File being created. And here we go. 25
There it is. 26
Now if you've done some web development or you've worked with APIs, then you're probably familiar with 27
the JSON file format, but if you're not and you're seeing this for the first time, 28
let me quickly show you what the JSON file format looks like. 29
So what I'm going to do is I'm going to open this file in my text editor. 30
Now, one thing to be aware of is that this file is actually huge. 31
This is a file that has 17.7 megabytes in it. 32
So it's a quite a large file with a lot of text, as you can imagine from parsing about 5800 33
emails. When you open this file, 34
then you see something that looks like this. 35
You see a curly brace, then the word "CATEGORY", then '"0":1, "1":1" and so 36
on, right. 37
This formatting here looks very, very, very unfriendly. 38
Right. 39
But I can show you this in a lot more of a readable fashion. 40
If I select my entire JSON and paste it into a website like jsonmate.com, we can beautify our 41
JSON and make it easier to read. After clicking "Beautify" and letting this website run for a little 42
while, we can see that the formatting here is a lot more friendly now. Scrolling down to the JSON editor, 43
we start to recognize our column names from the dataframe as three different keys in this JSON. 44
We've got the CATEGORY key, we've got the MESSAGE key and we've got the FILE_NAME key. 45
The category was simply 1 or 0 for spam or non-spam and these numbers here highlighted in blue, those are 46
my document IDs, so document number 60 was a spam email. 47
Similarly, if I expand the messages, I again have my document ID here, 0, 1, 2, 3, 4, 5, 6, 7 and so on, all the 48
way to 5795 49
and I have the text of all the spam emails here. 50
In other words, even though the file formatting does look kind of scary on the JSON, especially for a 51
massive file like this, it still preserves the overall structure of our dataframe. 52
And JSON in fact is one of the most common ways that messages are passed around the web, 53
so every time you hit up a server and request some weather data or what have you, the information that 54
you're likely to get back is going to be in this JSON format. 55
All right. 56
So that pretty much wraps up the data cleaning that we're going to do. 57
We've also managed not to just read a bunch of files from our disk. 58
We've also managed to save all the work that we've done to a JSON file. 59
The next step in our workflow is exploring and visualizing the data in our dataset. Looking forward to 60
seeing you in the next lesson. 61
Take care.