>> Module 4, talking about
the different ways to get data into Splunk.
In this video, we'll discuss
some examples of the many data sources
Splunk can work with.
We'll talk about ways to get data,
goes through creating an index,
add some data by uploading a file,
talk about source types,
and create some field extractions.
Then we'll finish off with the quiz.
Here's a nice image of examples of
what Splunk can index from splunk.com.
This highlights different types of data such as
Windows event logs, Linux command results,
events from Cloud services,
weblogs, database queries,
NetFlow data, clickstream data,
power consumption information, and more.
Down at the bottom here are some common sources of data.
In the middle we see popular forms of data.
For example, you might retrieve metrics on webblogs or
ingest data from tickets
opened by intrusion detection systems.
All these types of information can be
transformed into events that are
searchable and usable in different ways.
There are many ways to get these types of data.
You can, for example,
monitor files and directories,
upload data, run scripts and collect the results,
listening on our network ports,
including listening for sys log messages,
collect events using WMI,
run queries against connected databases,
perform API calls, and other methods.
We'll be uploading a file for
a simple demonstration in this video.
As a review, a Splunk index is a data repository.
When raw data is turned into events,
it gets put into an index.
These indexes are helpful for running efficient searches.
When you're able to search across a specific index,
that can speed up your searches.
For example, you might have an index called Cisco
ASA that just contains Cisco ASA logs.
When you're looking for ASA logs,
you wouldn't want to search across
all your Windows Event logs
while trying to find something.
Indexes can also help
you apply more control to your data.
For example, if you know you need
to keep authentication logs for
six months but only need to
keep application logs for one month,
you can apply those different kinds of
retention policies by index.
Additionally, you can easily limit
users to certain types of data but only
allowing them to search across
specific indexes that apply to their jobs.
Source types are used to identify the structure of
events and Splunk uses
these to format the data while indexing.
You might have multiple source types in the same index.
For example, you might collect
all your web sphere logs and an index called WebSphere,
but WebSphere activity logs are
formatted differently and are marked with
a different source type than WebSphere system error logs.
You can also use source types to
narrow down your searches in Splunk.
Field extraction is pulling out fields from event data.
Splunk automatically recognizes fields for
some source types and you can
also manually extract fields on your data.
In this example, we have
a NetScreen firewall event and
then we have lots of
potential fields including an action field.
In this case, the field name could be
action and the field value would be deny.
In another event, the field name would still be action,
but the field value could be allow.
The field name does not have to
be specified in the event.
This Jun at the beginning could be
extracted and have a field name of
month and a field value of Jun or June.
With that, we're going to jump into a basic example.
I have my Splunk server up and on it I have
a file field with some example exchanging mail logs.
Normally we probably wouldn't want to
upload a file to get these types of events,
but we'll do it for this example.
I'm going to my Splunk web interface.
I'm going to click on "Settings" and "Add Data".
I also have the option on my main page here.
From here, I'm going to scroll down and click "Upload".
Then I'm going to select the file we were just
looking at. Click "Next".
Splunk automatically did a good job of breaking
these events out and identifying the timestamps.
If I click on source type here,
I can try some of the pre-trained
source types Splunk has.
Under email, for example,
if I click "Proximal" as a source type,
it no longer breaks that out into
events and leaves them in a clump.
That's obviously not the right source type for this.
I'm going to go back to the default
here and make my own source type.
There are some other options
under here that you can play with.
But I'm just going to click "Save As".
I'm going to call this exchange logs
and put it in the email category.
Now, if I want to upload
the same type of file in the future,
I can pick the source type.
I'm going to click "Next" on this.
From here, I have to decide what the host is.
Since I uploaded the file,
leaving as my hostname makes sense,
so I'm just going to keep that as is.
It's also currently set to go to the default index.
It's often a good idea to put things into
the default index until you make sure things are working.
But I'm going to create a new index for
this data just by clicking here.
For index name, I'm going to call it exchange.
Leave as events and leave all the rest of
the settings as default for now.
I'm going to save this,
click "Review" up here, and "Submit".
From here, I can run a search across my new data.
It's already picked out
my index here that I assigned it,
and we've got all these events.
If you notice, here's the timeline of these events.
I can look at one of these and it's
broken out some of the fields such as
the source type and a long the side here it's
got things like hour and day and minute broken out.
But I'd like to have more fields.
I'm going to click here and extract new fields.
Here I'll just select
a sample event to work with. Click "Next".
Now I can decide between using a regular expression
or delimiters to break out my field.
Delimiters would be a good option if my fields are
separated by something like a tab or a special character.
In this case, I'm going to
pick regular expression and hit "Next".
You have the option of writing your own rejects
here or by trying to make Splunk to do the work.
To try to get an extraction,
I'm going to highlight a field.
I'm going to look at this first IP and click on it.
I'm going to name this field IP and hit "Add Extraction".
Now down at the bottom we can go and see
how this works with other events.
If we see here on different events,
it's pulled out this IP field even with different values.
I can click here to see if there's
anything that doesn't match,
which there's nothing, I think
we're in pretty good shape for this.
Now, if I want another field,
I can do the same thing.
I'm going to look at this two email address.
I'm going to call this recipient and add the extraction.
Now if I look at the sample events,
I'm going to remove that.
I got a notification that failed.
I think I missed that first letter there.
Let's see if this does better.
Recipient. That worked.
But if I scroll down here, there's a problem.
I call it recipient because it said
to here and it made sense for the sample event I had.
But now I'm also getting it for from.
I probably want to work more on this field extraction.
If I wanted to, I could click "Show
regular expression" and write my own.
But I'm just going to remove it to keep this video
short and we'll keep this IP field.
Click "Next". I'll review it and this all looks good.
It's pulling out the IPs on
all these different events. Click "Next".
Everything looks good here, and "Finish".
Now I can explore
the field that I just created and search.
Now if we were to rerun that search,
there should be a new field value
there that we could use.
We had index equals
exchange.
I need all times since there were
events that were older than the last 24 hours.
Now, if I look on the left side here,
there's a new field, that's our IP field.
It gives us a count, so we've got
12 counts of this 205 IP.
We could also do other things like specifically search
for a result and see the events related to this.
If I just wanted to see events for
that, we can pull that out.
Or I could do things like run statistics by
IP and lot of
other types of searches that
we'll get into in the next module.
Now that we've successfully added
data to our Splunk index, is quiz time.
True or false, you should keep all
of your data in the same index.
The answer is false.
Breaking out your data
into different indexes can help you run
searches more efficiently and apply
different rules to different types of data.
In our next video,
we'll add even more data to Splunk by modifying
a config file on the machine where we
installed the universal forwarder.