Hey, Cloud Gurus.
Welcome back to our series,
Introducing the Tools of the Trade.
In this video, we're talking about Scala.
So we'll start with an overview.
We'll then look at some Scala code
and then wrap everything up with a review.
For the exam, you don't have to be a Scala expert.
You more or less need to be able to recognize
that it is Scala syntax you are looking at
and be able to fill in some holes using dropdown boxes.
So for example, you'll have 4 options
and you'll have to decide which of those
slots into the Scala syntax you're looking at.
For that reason, we're going to go over some examples
so you recognize it on site,
but we're not going in-depth in how to be a Scala expert.
So what even is Scala?
Well, the full definition taken from their website is
Scala combines object-oriented and functional programming
in one concise, high-level language. Scala's
static types help avoid bugs in complex applications,
and its JVM and JavaScript runtimes
let you build high-performance systems
with easy access to huge ecosystems of libraries.
And that's from www.scalalang.org.
For our purposes, it's a programming language
leveraged in Azure Databricks
for ETL and data analysis operations.
It's not the only type of code you'll see
come across the exam.
So let's take a look at some examples,
so that you can easily recognize it.
All of these examples can be executed within a notebook
in your Azure Databricks workspace.
And for our first example,
this is loading a sample JSON file as a data frame.
You'll see our val df for data frame.
And we're reading in a JSON file at the path listed.
In order to then display the contents of that data frame,
we could use df.show.
And we now have the contents of that JSON file
in a Scala data frame so that we can work with it.
If we wanted to move from there and transform the data,
we could use a query like this.
We're specifying a new value called specificColumnsDf
and we're taking that same data frame, df,
and selecting from it,
df.select and pulling the columns
first name, last name, gender, location, and level.
Again, to output the results of that,
we can use the .show mechanism.
And this time we say specific columns df.
And of course you can do more than just limit
to specific columns.
To further transform the data, we can rename a column
such as changing the level column to subscription_type.
By this point, you're probably noticing a pattern.
We define a value renamedColumnsDf.
And then we reference the last results that we had,
specificColumnsDf, and use .withColumnRenamed.
And again, as you might expect, to output those results,
we can simply say renamedcolumnsdf.show.
Eventually, you'll probably need to load the data.
There are many options for that
and you can use Scala to set the configuration.
Such as using spark.conf.set
to set spark.sql.parquet.writelegacyformat
equal to true.
And when set to true, this particular setting
will cause data to be written in the manner of
Spark 1.4 and earlier.
To actually load the data, we can use a query such as this.
Here, we're loading the transformed data frame
of renamedColumnsDf and writing it as a table
in Azure Synapse.
So you can see we specify .write.
There are several options passed in here,
which you would have set as variables earlier on.
Such as the URL, the tempDir, and so forth.
But we're creating a table called sample table
to load our data frame into.
You can see we have an option here
forward_spark_azure_storage_credentials.
Man, what a mouthful.
But this causes Azure Synapse
to access data from Blob storage using an access key.
And this is really the only supported method
of authentication for this type of scenario.
By way of review, Scala is a high-level programming language
used within Azure Databricks.
You can use Scala notebooks
in your Azure Databricks workspace to carry out ETL,
configuration, and analysis operations.
And again, this is by no means even the tip of the iceberg
with how to use Scala.
I just want you to be able to recognize the general syntax
and pattern so that you can identify it on site
and know that it's Scala you're working with.
I have placed a link in the lesson resources
for further information about Scala data frames.
So be sure to check that out.
There's no way we could cover every facet of Scala
without creating an entire course dedicated to it.
So I would suggest some outside reading
until you have a basic familiarity with the syntax.
Remember that for the exam,
you don't have to be a Scala expert.
You just have to be able to recognize
the language you're working with
and then make intelligent decisions
about which options should go into certain portions
of the statement.
But that's it for this lesson.
In the next video, we'll cover one last tool
for you to place in your data engineering tool belt.
When you're ready, I'll see you there.