Hey, Cloud Gurus, in this lesson,
we'll be talking about encoding and decoding data.
To start off with, we'll talk about what the problem is.
Why do we even need to encode and decode data?
We'll look at some solutions, new and old.
Demo--how to go about this in Azure Data Factory.
And then, wrap everything up with a review.
To briefly talk about the problem,
the format of source data depends
on the encoding options of the system.
And this may vary from that of the sink.
This can obviously create issues
when performing ETL or ELT operations.
Systems that are going to use the data from the source
may depend on it being in a different format.
Normally things are in UTF-8 or UTF-16,
but that's not always the case.
So, if your source system is using an encoding format
other than the one you'll eventually need it to be in,
you're going to have to go through
some type of encoding/decoding process.
In the past, some of the ways you might have gone
about this are manually exporting the file.
You can download it to your local computer, convert it
using a text editor, and then upload it back to the cloud.
This is easy, but obviously not optimal.
There's a lot of overhead, both on network resources
and on your time to accomplish this.
You could use a virtual machine and download the file,
run some scripts to convert it, and then upload it again.
This keeps it in the cloud and is a bit more automated,
but still has the overhead of downloading
and uploading the file.
An improvement to that would be to leave
the file in Azure Storage and use a cloud VM
to convert it in a similar way using scripts.
This avoids the problem of downloading
and uploading the file, leaving it there in Azure Storage,
but it's still not an ideal situation.
Luckily, there are new ways to go about this.
We can use the copy activity, setting encoding options
on dataset properties for your source and sink.
Or we can use Azure Functions for those complex scenarios
where simply setting those properties is not sufficient.
You can insert an Azure Function activity into the pipeline
and use custom code to go about the conversion.
The great thing is that both of these
are available in the Azure Data Factory.
And so, they can be included as part of your pipelines.
Let's jump into the demo and take a look
at a simple example of encoding and decoding data.
Before we jump over to the Azure portal,
I just wanted to show you the encoding format
of a file we'll be working with.
This is just an example JSON file.
And if you'll point your attention down here to the bottom,
we have our encoding, in this case, ISO 8859-7.
Now, I want you to note that as we go forward.
Over in Azure Data Factory Studio,
I have a demo pipeline open.
This is a very simple pipeline with just one copy activity.
As you can see, its source is a json1 dataset
and its sink is a json2 dataset.
If we take a look at json1, you can see this
is pointing to my Azure Data Lake Storage linked service,
and it's referencing that example.json file.
I've simply uploaded that same file
to my Azure Data Lake container.
I have set the encoding as that same ISO 8859-7,
so that Azure Data Factory knows what to expect.
For my target, my json2, I have another linked service
to Azure Data Lake, and it's pointing
at a second file decode.json.
And that is set to use the default UTF-8 format.
And so, when we come over to our pipeline and trigger this,
it will simply copy that file from the source to the sink,
and change the format in doing so.
I'll go ahead and kick that off.
And then I'll rejoin you over at my text editor
to look at what its encoding format is.
Back over in my text editor, I have the decode.json file.
And, again, I simply downloaded that
from the container it was copied over to.
You can see we are on a different file,
just so you know I'm not playing tricks.
And if you take note down, at the bottom,
the format is now Unicode UTF-8.
So we've copied a file from the source system
that was using one format, and during
the copy converted it to a more usable format.
By way of review, the source system determines
the encoding options for your incoming data.
Again, normally you'll be working with UTF-8 or UTF-16,
but that's not always the case.
Your data projects will involve a variety of formats
and so, sometimes you'll need to be able to convert those.
Azure Data Factory allows you to do many types
of conversions simply by setting dataset properties
to the desired values and using the copy activity.
And that's what we just demoed: setting the original format
on the source, and then our desired format on the sink.
Of course, it won't always be that easy.
And so, for complex scenarios,
you can use the Azure Function Activity in your pipeline
and use custom code to perform the conversion
before passing it along down the pipeline.
That's it, Gurus, thank you for joining me.
When you're ready, I'll see you in the next video.