So now that we've introduced the players,
in this lesson, we are going to be talking about
designing a stream processing solution.
So in this lesson, we are going to take a look
and focus on a quick review of Azure Stream Analytics.
Then, we're going to see Azure Stream Analytics in action.
I wanted to introduce it
by actually jumping into the Azure portal
and kind of walking through just a very high-level scenario
and showing you a little bit of how it works. I
think that's going to help you as we move through the
lessons to have a little bit better understanding
as we talk about concepts.
So with that, welcome back to Azure Stream Analytics.
Azure Stream Analytics, if you remember from Microsoft,
is the fully managed real-time analytics service
designed to help you analyze
and process fast moving streams.
And you can read the rest.
Basically, it is the tool
that we're going to focus on the most for the DP-203
to move data in a streaming fashion through Azure,
as well as do some cleanup and transformation on that data
as it passes through Azure Stream Analytics.
At its core, you can have Event Hubs, IoT Hubs,
Blob storage, you have some sort of input,
then, you're going to do some sort of transformation,
which is our second piece,
and that's going to be using a query language
in Azure Stream Analytics.
So we have our input, pull data through,
do some transformation on it
as it's passing through Stream Analytics,
and then, we have our output.
We are going to send it somewhere to store
and save the results or pass it
into some other process in our cloud journey.
So at its core,
you need to think about that for Azure Stream Analytics.
Everything comes down to input, query, and output.
Now, with that very brief refresher, let's see it in action.
Let's just go ahead and jump into the portal
and we will talk more about Azure Stream Analytics.
So I've opened up a portal
and created a Stream Analytics job,
and we are looking now at that Stream Analytics job.
Hasn't run yet, haven't configured any inputs or outputs,
and there is no query as well,
so it looks pretty barren.
So when we start off,
we're always going to take a look at our inputs.
And so when you think about creating a Stream Analytics job,
a good way to consider that is honestly
just to get like a whiteboard,
kind of mock up where all the inputs
are going to be coming in from the various IoT sensors
or whatever it is that you have.
Think about what you need to do to get that data
into a format where it's usable,
either for a report or for another step in your process.
And then, just simply define your outputs.
Once you've done that, on your whiteboard,
you're ready to start writing the queries,
ready to start building the inputs and outputs,
so that you can test.
So we're going to start by building our input.
We're going to click on Input, add a stream input.
And if you remember we have 3 choices,
Event Hub, IoT Hub, and Blob storage.
We're going to choose Blob storage now.
For my input alias, I just need to name this thing.
So let's just go ahead and call it testinput1.
We're going to choose our storage account,
and I'm just picking one that I've already created.
And we can use our existing containers,
and that's just fine.
Now, for our authentication mode,
if you have trouble with our authentication,
you would go through and follow the manual steps
to grant authentication here.
And I'm going to leave the rest of this alone
because I'm just trying to show you how this would work.
Now, when I click on Save,
it's going to fail because I haven't done
the manual authentication steps,
but for what we're doing now, that's really okay.
Yeah, and you can see here,
here is our manual setup request for our Blob storage,
but it's going to go ahead and create that test input,
so that is created now.
And so you would do that for all of your inputs.
Then, you would come in to Query
and you would build your query language.
And I'm going to go ahead
and just paste a query language in here for now.
And you can see that as I do that,
it's going to update my inputs and my outputs,
so you can see what's going on there.
And so for my input,
I can go ahead and change that to testinput1.
There we go, and let's go ahead
and change this one, and then I'll show you
what is happening here.
So to start off with, I am selecting everything,
and the INTO is actually where we're sending it to.
So what we're doing is we're taking all of our files
that are in this testinput1 input that I created,
and we are just simply going to pass
all of those things through to TestOutput,
which it's going to show you right here.
And we haven't defined that yet,
so we would need to create our output as well.
But that is our very first query.
Select everything, pass it from testinput1 into TestOutput.
And let's go ahead and get rid of those capitals.
There we go. And then, we're going to do another query
and we're going to have a select statement here.
And so what we're going to do is just do a simple count,
and we're going to take our simple count again,
and we are going to pass that into TestOutput2.
So we're going to take this testinput1,
here, just like we did at the top,
and we're going to send it to testoutput2,
while doing a simple summation statement.
And then, what we're going to do as well
is we're going to group by a tumbling window.
So in the last lesson,
I talked a little bit about windows
and how it's important to look at your data,
so we're going to be talking a lot more about windows.
So what we're doing here with our window
is we're defining a length of time
and we're going to pass data through
that length of time or that window,
and we're going to use that to figure out
where we should do our summation statements.
Okay. So we'll talk more about that later,
but it's a very important concept to think
about your data as it passes through time,
and figure out what your windows are going to look like
and how you're going to aggregate your data
so that you can run your queries.
All right. So this is a super simple example
of a query in Azure Stream Analytics.
And what I would simply do is save that,
and then I could test my query.
I'm not going to test it now,
because we didn't finish our input.
And then the final thing is
is we would come in and we can add our outputs as well.
So I could choose Cosmos DB, for example,
and I could just say that this is testoutput1.
And I could have my account here,
and this is a Cosmos account that I created earlier.
So it's just going to go ahead and add our test output.
And so when we look at Azure Stream Analytics,
it's really that simple.
We're going to think about our input,
our query, and then our output.
And so you can see here, we now have our testoutput1,
so I could go ahead
and update my query to match my testoutput1.
So I could say testoutput1.
There we go, got rid of that,
and now, it's going to be sending it to that Cosmos DB.
So with that, I'm going to go ahead
and jump back into our lesson
and run through a couple of quick reviews.
So first, stream processing.
As you can see here,
it's really all about our input, query, and output.
So keep that in mind as you think
about designing for Stream Analytics.
You also need to understand how it works.
And I'm going to have quite a few more demos,
as well as some labs,
and so you need to make sure that you understand
and are comfortable with Azure Stream Analytics,
because it is a very important concept on the DP-203.
Finally, windowing. Windowing definitely matters.
And in the next lesson, we're going to dive in
and talk a lot about the different types of windows
that you can encounter,
but make sure as you look at design,
that you're considering windowing
and how you want to handle queries of your data.
All right, with that, I'm going to go ahead, end this
lesson, and we'll pick up in the next
and talk a lot more about windows.
I'll see you there.