Hi everybody and welcome to the video about metrics so metrics are a depiction of a large amount of
data into an easily digestible and actionable dataset.
So the example on the screen is taking a phishing campaign of a lot of people that opened or did not
open a specific email and converting them to a percentage of opened an unopened so why is this type
of data important why are the metrics important.
It provides non team members a way to measure us.
So the network team the application team the security engineering team.
So at any any other team can measure us as whether we're being successful or not based on these metrics
that we're providing so metrics should be easily digestible output of the team's performance at a high
level.
Now metrics are not going to go into play of every single thing that you do on a daily basis but as
a high level for a team it shows the performance of that team.
Most enterprise organizations do require some sort of metrics.
These are normally presented to staff or executives which will go over in a second.
But in a general sense it's required for enterprise level.
Now in smaller organizations a quick PowerPoint talking about the progress of the team might be enough.
Even those you tend to show a metric or two.
So what is each staff.
So each staff is the executive staff such as the CEO Cisco CIO CFO although the chief level this particular
group provides all approvals on budget staffing anything around what what a team might need.
Anything money wise.
Essentially the metrics are presented to each staff on a regular basis depending on the organization
it could be monthly yearly quarterly.
In a general sense it's it's once per month they get a quick recap and a monthly meeting that could
be an hour or two hours long where they everybody from different parts of the organization all come
together and explain what they need to do.
What's been improved where they're at such a so presenting to each staff comes with its own challenges.
So in a general sense a TPM or technical project slash program manager will present these in most cases
you as an individual on the red team will almost never have to present to staff a metric.
Now on smaller teams that may be the case where a TPM does not exist at the moment where you may have
to present that in place of them following that your manager in all other cases should be the one presenting
it.
However a lead on the team still may present it on the manager's behalf.
But when you do present to each staff make sure you watch your colors.
Green is good it's never really looked frowned upon.
When you start getting into the yellow to the oranges things start to look like they need improvement.
And when you start getting into the reds or any darker shades of color you're starting to look at.
There's a massive issue when each step looks at metrics and you have something in the red or the orange
it's going to be called out.
So you need to have a reason and all of the details behind it ready and available.
Once it gets called out if you don't have it prepare for a lot of headache around a few executives running
to you and trying to figure out what happened.
Because as far as they're concerned red means some massive issue.
So even if it may be a change of metric which we'll get in later in this video and change of metric
should not indicate a red dip like that especially when it goes from let's say 90 percent to 60 percent
and 50 is the breaking point for it being red.
So you don't want giant dips either unless there is an actual issue.
And if that's the case this is your way of presenting its e f whether that's budget concerns or maybe
you need more headcount in staffing or you need different types of tools.
So a metric layout in a general sense is going to be a percentage scale of zero to 100.
Most people tend to stay around that type of scale.
There's a few other options.
But I would strictly stay around a percentage mark something you can measure in a percentage which is
normally done by normalizing data which we will go through and a little bit.
You want to keep at least two to four metrics per team depending on the size of the team.
Now you won't present all four of these to each staff.
You're going to pick the ones that most accurately reflect your team and how your team is doing.
Sometimes this is done depending on what the team needs.
So if all metrics are good but you have one that's lacking and you've only been showing the first three
maybe showing the other one for a few months might get better input on whether you need more budget
concerns so when it comes to changing the metrics like we just talked about we might want to change
them for a variety of reasons.
This could be anything from the metrics were bad beforehand we generated them pretty roughly they were
they were asked to quickly we didn't put enough prep time into it.
Were they just small improvements you want to avoid any massive changes to a metric it provides too
much of a fluctuation for staff without them noticing that there's that there may be an issue which
will cause a massive investigation a lot of wasted resources you're your best chance is a slow rollout
and if you have to massively change a metric you need to be able to articulate that in a very clear
way to e after in that particular meeting.
So they start understanding what's required going forward.
So what makes a good measure so a good metric is a metric that accurately describes how much work is
being done what your team is doing that can be done on a weekly monthly or quarterly or yearly basis.
Most teams do a weekly metric count that they send out each week so you can get a better idea of how
things are flowing week to week.
E F tends to get the monthly one and there may be requirements for the organization to provide a overall
metric which is done yearly in this particular case you won't have to handle anything yearly unless
your organization demands it.
You tend to want to stay on a percentage scale it's the quickest easiest to identify whether there's
an issue as long as you set a threshold of what you're actually looking for.
So for example scanning 100 percent of your infrastructure is great but having zero percent issues or
1 percent issues identified as a high level issue is good.
So depending on which scale you need to be working with it needs to be clearly defined.
You're going to want at least one metric per program so that's phishing password cracking the automation
and scripting whether you're doing a pivot escalation team whatever program inside of your red team
is designed you will need a new metric for each one it tends to be better to have a few per program
that you can pick from.
So this as an overall is just a minimum but it's always good to have some more.
So normalizing data normalizing data is the idea of taking a large set of information and putting it
in between a range of two numbers.
So this could be tends to be between zero and one hundred and zero to one.
So depending on what you're doing 0 3 one might make more sense zero to one hundred is pretty common.
So normalizing that is just getting it into a more concise focused format that you know the output of
for example 1 percent to 100 percent so we're gonna do a quick demo of taking raw fishing metrics and
normalizing it into a metric that you may present
so in this particular case we now have a quick list of 20 email addresses and whether they opened or
did not open that particular phishing e-mail.
Now keep in mind this is just an example they're not really email addresses.
I will have a above this video you will see a raw data link raw metrics link so you can go ahead and
download this and that's going to be part of the homework for this video is take that Excel document
download it and reproduce the same metrics.
Now we'll go over later in the course around different you'll see different ways you could generate
metrics and I'll make sure I bring it up when we hit certain things.
So in this particular example we have a pretty easy way to normalize the data.
So we have a yes or a no so index.
Now you can do this in the command line but for sake of us not having gone over working command line
in Linux or Windows yet which will be the following modules I'm just going to do this in excel so anybody
can follow along without having any knowledge on the command line and piping stuff around.
So what we're going to want to do is we're going to want to look for something called count.
If so we're going to count.
If this particular dataset has
a loops that's not right as a yes.
Right.
So we're looking so how many have been open.
So this is currently set as a percentage ignore that So in a general sense you will see a general number
here.
So in this case it's nine.
So we're counting nine yeses if we come in here and change this to a yes we should see 10.
So we've got our count.
Right.
So the next thing we want to do is we want to grab the percentage.
Right.
We want to do another count if we want to grab the percentage of the total amount there.
Like I said we're getting an average.
So we're gonna get the total amount and we're going to grab anything in there so star in this particular
case means any text at all.
Right.
So we had a point 4 7 3.
So we come in and form out the cell to be a percentage for decimal points.
Now you get forty seven percent right.
So if we wanted the opposite metric we can come in to not open changed the Y to an end.
And now we have our full metric.
So as we can see now we have a forty seven percent opened at fifty two percent did not so if we come
in here we can change around the metric a little bit.
So now we have 31 percent opened and 68 didn't.
So depending on how your engagements go maybe you want to take this is for example month 1 Maybe you
want to take month 2 and 3 and then get the average of these three which has done the same way grab
all the percentages that were opened.
Divide by how many months now you have your overall average percentage.
So this is how you normalize data in a basic format to provide some sort of metric for Red Team program.
Now again this will change for this will change a little bit depending on what you're doing what you're
looking for but we'll hit up different different aspects and how to normalize different data further
and of course.