>> In this video, we'll discuss the parts of Splunk.
First, we'll talk about the data pipeline,
then go into Splunk components
and have a little bit of a discussion
on distributed versus
non-distributed versus clustered environments.
The data pipeline as Splunk sees it,
is made up of input,
parsing, indexing, and searching.
The input part of that is just what you might think,
Splunk is getting data.
At this stage, there's also metadata
added like source, host,
and source type, but the main focus is getting input,
those data coming in.
Each of these stages
corresponds to different actual Splunk components.
We'll talk about each of these,
but input goes along with forwarders,
universal or heavy forwarders
and it can be done at the indexer level.
At the next stage, we have parsing.
Data is getting turned into events at this stage.
This could be line breaks happening or
data being transformed based on certain rules.
This can occur on an indexer or heavy forwarder.
Indexing is taking the parsed events
and putting them into an index for later use.
When you get to the searching stage,
there's some interaction between
the search head and indexers.
The search head is responsible for search management.
This is where you would go to run a search
and that search request gets sent to an indexer.
Then the results get sent back to
the search head for you to view and work with.
At the searching level,
you can have scheduled searches,
alerts and, dashboards.
Along the side here is a common setup you might see.
I have UF for a universal forwarder.
You can think of a universal forwarder
as something like an agent,
maybe installed on the server and setup to
collect Windows event logs as getting input,
taking those Windows event logs and
sending them onto the indexer.
On the indexer, you have
parsing indexing is taking the data,
breaking it into events,
and organizing it in a place
the search head can easily send requests for.
Search heads are what users typically interact with.
They perform search management.
You can, for example,
go into this box and run a basic search,
which the search head then distributes as a request to
the different indexers and
then displays the results form.
You can do things like have custom dashboards,
alerts, and reports.
Indexers receive indexed and stored data.
They can provide the search head with needed information.
There's a bit of complexity around the term index,
so I wanted to break down the different definitions.
Index as a noun is a data repository.
By breaking up data into different indexes,
you can improve performance,
apply different data retention policies,
and limit access to different sets of data.
For example, if you're collecting firewall logs,
you may have an index titled firewall logs,
where logs from multiple firewalls gets stored.
This can make it easier to limit
your searches to the type of information you're
looking for and if you have another team, say,
a help desk, that needs to look at authentication logs,
but maybe not web traffic,
you can easily limit them from viewing this data.
If you need acute firewall logs
for a set amount of time for audits,
you can specify this on the data retention policies.
Index as a verb is the processing of raw data,
as in taking the data and handling and organizing it.
An indexer is
a particular Splunk instance that indexes data.
This sentence might help you
remember the different meanings.
An indexer indexes data and puts it in an index.
Forwarders, like I mentioned,
you can think of them like agents you install on a host.
They send data onward.
There are several different types of forwarders.
A light forwarder is deprecated,
meaning there aren't newer versions of it,
but it does exist.
Universal forwarders are typically
what you want to install when possible.
They have a pretty light footprint and
mostly just work to send data onward.
You can do some filtering with universal forwarders
such as by blacklisting certain event types,
but if you want to do any more complex filtering,
you're probably going to need to
set up a heavy forwarder.
There are also different server roles.
We aren't going to talk about
them too much in this course,
but I want you to know they exist.
For example, things like a deployment server can help you
manage forwarders and send apps by groups.
Distributed environments are basically
ones where different components of Splunk are broken out.
The setup in this course will be
a simple non-distributed environment where
a search head indexer and
licensed master are all combined.
For larger companies or if you're handling a lot of data,
you'll probably need to separate these pieces out.
This is sometimes thought of as horizontal scaling.
As you grow, you can add
different parts to scale the environment.
With the idea of different deployment skills,
if you have a very small office working with
less than 20 gigs a day with fear than a 100 forwarders,
you could probably get away with
a non-distributed environment like
we're doing for this course.
For a larger company,
you're probably going to need a distributed environment.
Clustering is a more advanced topic,
but you should know what it is.
At a basic level,
it replicates data between different components to create
redundancy so that there is
duplicate data across multiple instances.
This is good to look at if you can't
have any downtime in your environment,
or if you're worried about disaster
recovery or the potential of
losing data. Question time.
A universal forwarder deals with
the blank part of the data pipeline.
The answer is input.
A universal forwarder helps to
bring data into the Splunk environment.
As a review, forwarders send data,
indexers turn data into
events and place them in indexes,
search heads send,
search requests, and display data.
A large company will
likely need a distributed environment.
For this course, we will set up
a simple non-distributed environment.
Clustering also won't be covered in
this course as it is a more advanced topic,
but you should know that it provides redundancy and is
a good option for
high availability and disaster recovery options.
In the next up videos,
we will install Splunk.