>> In this video, we'll use
our Splunk search skills to create alerts.
Splunk alerts are based on saved searches.
When the conditions you've specified are met,
you can set an alert to trigger.
In this example,
we've got a search that looks on an index called
firewall for events where
threat name equals SQL injection.
Then organizes the results into a table with source IP,
destination IP and message.
We can save this search as an alert to run every hour
and send an email if an event
matches that search in the time frame.
In this other example,
we could have the search
continuously run and trigger an alert when
there have been more than five events that have
triggered for this in the last 10 minutes.
These saved searches for alerts
can be scheduled to run every hour,
every day, or on certain days.
You can also schedule them using cron expressions.
If you're unfamiliar with cron expressions,
I recommend using a site like
Crontab.guru to help you schedule it.
This will give you a lot of flexibility for
scheduling and help you to stagger your alerts.
It's a good idea to schedule alerts at
different times to spread out
the impact of running a search.
You can also do searches in real-time,
where they're continuously running.
Typically, it's better to do
even a really frequent scheduled search
rather than real-time searches.
You can set up various actions for an alert,
including multiple actions in
response to a triggered alert.
You can, for example,
send an email notification when an alert triggers.
You could also use webhooks to update a web resource,
write the results to a CSV file,
add alerts to a list of recently triggered alerts.
There are also a lot of actions that come with apps.
For example, a popular ticketing tool called
ServiceNow has an app that when you set it up,
it allows you to easily open a ticket.
Phantom, Splunk's new purchase,
is an automation and orchestration tool.
It can take events and kick-off
more complex actions in response to alerts.
To receive phone calls, texts notifications,
and push notifications,
you could use Splunk's other
recent acquisition, VictorOps.
There are also other good tools for
this that integrate with Splunk like PagerDuty.
With that, I think it's time to
go in and create a practice alert.
I've got two machines that
I'll be working with in this video.
I have my Splunk search head
hosted on this Linux machine,
and I have a Windows machine with
a universal forwarder installed.
The Windows machine will serve to
generate new events for us to alert on.
I'm looking at the search and
reporting app on my search head here.
I've searched for Windows event logs that have
an ID of 4625,
which is a code for an account failing to log on.
I've further specified that I want events where the field
for failure reason equals
unknown username or bad password,
and then I've run statistics
on the returned events where I'm getting a count of
events by computer name
and have included other values in the display.
Right now, I've got one event.
One event might be pretty reasonable in this environment,
but maybe I want alerted if there have been
more than three of these in the last 60 minutes.
I'm going to add a pipe here and say,
where count is greater than three,
and changes from verbose mode to
smart mode because I don't need the full event data.
I just want what's going to show up in these dots.
Nope, I might have left to go too long.
We're going to do last four hours.
Oh, I didn't leave it wrong.
If you remember, I said there's only one event there,
so something you can do
to check and make sure your search is working,
is lower the count you need.
I'm getting results when I lower the count.
If I want to change that back up to three,
I can save this as an alert.
I'm just going to go up here to Save As
and I'm going to call this authentication failures.
For description, I'll just say this is in progress.
I want to see how this alert is doing.
I could set this to run every week,
every hour, every day.
I'm going to put it on a Cron schedule so I
don't have to wait too long to get my results.
To do that, I'm just going to say,
looks like it's 54 right now.
I'm going to give myself a few minutes for this
to run and say I want this to
happen every hour at the 59th minute.
This is going to check the last 60 minutes.
The results of this alert will expire in 24 hours.
But that doesn't mean the data will disappear.
If I want to, I can go back and run this search
again and specify this time frame of today,
during these hours and then I can still get
the results but I would need to
rerun the search to have those.
I want this alert to trigger when
the number of results is greater than zero.
I've already specified in
the search that I want the count to be higher than three.
I only want one alert.
Then for alert actions, I'm going to go ahead and say,
add to trigger alerts with a severity of medium.
I'm also going to want to send an email.
I'll fill out an email to send it to here,
and leave that priority normal.
If you see this name with the dollar signs around it,
this is called a token and it's going to
fill in the name of this alert in here.
If I want to, I could add extra information.
Like, if this
triggers call Bob or whatever I want in there.
There's going to be a link to the alert and
the results in the body of the email.
But I'm also going to want to put the results in line.
It will show this that we had in the search back
here in the body of the email.
Also, say attach a PDF with the results and with that,
I'm going to save it.
I can view the alert if I want.
Perfect. I've got my two actions here.
It's currently enabled and now I'm going to hop over
to my Windows machine and try and
trigger some authentication failures.
I have a user called Hermione.
That didn't work. I don't want it to work.
Just type it away. That's a good password.
But that's not her password.
I'm just going to keep trying to generate
these events so we can get a higher count here.
If I want to take a look at my alert,
I can go to settings,
search as reports and alerts.
I think I called this authentication failures.
This tells me the next scheduled time,
which should be in a minute here.
There's currently no alerts for it
and in a minute we should see that count go up.
Great. I've got an alert here now,
if I want to take a look at,
I can go to view recent.
Here's this alert.
This is when the results were created.
It ran really quickly and the status is complete.
If I click on this here,
I can see the results.
Perfect. We've got results
here matching the search.
This will run every hour at,
what did I have it set for?
It tells me the next time here.
It's going to run again in just less than an hour.
The other thing I can do is go under activity and look at
triggered alerts and it'll also show up there as well.
It gives it a severity here.
I can view the results from here too if I want to.
With that, we've successfully created
an alert and it's time for a quiz.
When an alert triggers, Splunk can blank.
There's a lot of options here, I'm going to help you out.
Splunk could send an email,
it could write a script at the alert to triggered events.
Save the results to the machine.
Other app actions, do HTTP POST requests.
With that, we've completed this video.
In the next video,
we'll be looking at reports and dashboards.