[Autogenerated] right requirement 11.3 ___________ testing
is actually probably one of my favorites.
It is.
It is one of the good ones, and it's one of my favorites.
Because, you know, there are two sorts of ___________ test.
There's the ___________ test you pay for.
On that finds vulnerabilities in your system that you can learn from on.
There's a ___________ test that you don't pay for, which is called the criminal.
Doing that to got to security incidents.
Yeah, I always think it's better to do it yourself.
The rely on having free ones now Standard change quite a lot recently got a
new requirement in ___________ testing to first of all,
have a ___________ test methodology.
Yes, I mean, what what's what's what's what's that look like to an assessor?
I know it looks like to to why, why we wrote it a certain way.
But in practice, what's it look like?
It's an assessor.
Well,
I think it's one of its major purposes is intended to
sort of ____ out lousy pen tests.
___________ testing can cover a wide spread of activities,
right,
and there are lots of good pen tests out there that a thorough and careful and,
uh, you know, considered.
And then there are lots of sort of lousy tests that error
basically glorified vulnerability Scans with fancy cover
sheets stable to the front.
Uh,
and and that has been the case for for at least 20 years on
will probably be the case for 20 more.
Uh, so having a methodology, right?
Having having at least some capacity to understand
what you are buying right to say, Does this cover our scope?
Does this cover all the stuff we need to address?
Uh, does it test our environment with sufficient rigor?
Do we make sure that our tester is able to,
you know, understand our environment and and examine it thoroughly?
Right?
Uh,
if we need to do things like test in a pre production environment because
we're worried about adversely affecting our real environment,
you know,
do we have some means of making sure that that pre
production environment is a useful replica?
Uh, you I can look at, you know,
some of the existing penetrations testing methodology standards out there,
like the NIST Special publication 800 at 1 15,
or or any of the other standards.
And there at least worth a look to get an idea of what they say.
Whose writings methodology?
You, you the organization of the pen tester.
So I think you the organization need at least some of that right now.
Maybe some of what you're doing is saying and you know,
some some of what we have to do is rely on the
expertise of this third-party tester, right?
If we don't have that in house,
then you know we have to do that then Then I think maybe your
methodology is a little bit more about evaluating the quality of the
tester on making sure that they're testing your scope.
One of the problems I see that with that is often clients
don't really know how to buy a pen test.
So asking the client to have a pen test methodology is quite hard It is.
And is this where the USA could help?
The USA could absolutely help you with that.
These que se could get a sense of of what the tester is capable of doing.
Yeah, I could get a sense of what some of their reports look like.
Uh,
it's a if your potential pen tester is unwilling to share with
you at least a sample or redacted report.
Uh,
that showcases sort of the kinds of things they look
for and how they do some testing.
Uh, that that should be a concern.
Okay, on, then, if they do share something with you,
you know,
you ought to be able to tell at least something about their approach right on.
And there's some obvious tells to look out for right if they don't
really have much sort of introductory stuff,
for it looks very, very boilerplate.
Uh,
if they don't expressly described the scope or some capacity to
try to understand what's in the environment,
Uh, if they talk about their methodology by listing a bunch of tools,
you know,
usually of the sort that you could download from spurious website circa 2006.
Uh, those error all warning signs that that you're not really,
uh, sort of getting the best quality tester.
I had a colleague who was a very good protestor.
Give a really good analogy about that, right?
A lot.
You know,
he was complaining that a lot of clients and there are FPs wanted
you to list what tools you were using right now.
when you have a conversation with a surgeon about,
you know, not coming procedure.
Do you want to know what tools that surgeon is using?
Or do you want to understand his or her
qualifications as a medical professional,
Right.
I mean, I know what my answer is on.
And so just so with your pen tester, right,
it's not about whether or not you downloaded the
latest version of Better Cap right?
But do you actually have some idea of how things work?
Who should do your pants?
Or should it be the QoS iPhone?
Because I know a lot of QoS iPhone do pan testing as well it can, right?
It's not necessarily the case that that you're QoS a from needs to do.
The pent is.
You could get an outside for him.
You could do it internally if you have the requisite expertise.
Um,
there's often some benefit in in having the same firm do
that and that you have a higher degree,
or at least a higher potential degree of coordination.
Uh, and you know, that could be of use, right?
Uh, some people would argue that, uh,
that could work the other way when maybe you have a
little bit more of a conflict of interest.
Uh and I suppose either argument is valid, right?
But that's why the requirements don't sort of force
you to do that one way or the other.
And the assessor can't compel you to use their pen testing practice.
And that's a really important point.
The assessor can't compel you to use a pen test from them.
No, they absolutely cannot.
But if they say things like, Well,
if you get us through the pan test will make you pass your CSS thing.
That's another serious red flag.
Yeah, e mean, personally, I never, ever let the QoS iPhone do the pan test.
A lot of organizations,
you absolutely take it the same way that the QoS a
should be as independent as a c P.
A auditor, right?
So I want them to come in and assess me, right?
I might want someone else from the company to give me some advice occasionally,
but in terms of delivering services like manage security
services or or actually doing a pen test,
I want that to come from somebody else because otherwise
they're marking their own homework.
I'm not getting any assurance.
Okay, I get a check, but I'm not getting insurance, right.
That's a fair concern, right?
And a lot of organizations do that right where they say we
want a pen test from a different firm.
And while I find relatively few clients want to very their assessor firm,
I find many more.
Clients want to vary their pen tester.
You should change your pen tester like you change your e.
Can't find an analogy something which some garment?
No, no.
But you probably wanna change your pen tester annually.
Uh, there is There is There's a reasonable justification for for that idea.
Yeah, I don't want the same even if you use the same firm because,
well, I tend to find is that firms have set ways of doing pan testing.
There's the this company way of doing a pen test.
They followed this type of methodology and they try these type of techniques.
But if you go to a different firm,
they try a whole different set of techniques and they find
vulnerabilities that they wouldn't have found right well,
and and some of that is kind of the sort of philosophy of how you approach
this right there is the sort of breadth first versus depth,
first kind of approach, right?
And so the breadth first is the idea of I'm going to sort of grab the
vulnerability tree and shake it to see what falls loose,
right?
And that could be useful, right?
I found 50 individual vulnerabilities that I think you should resolve.
Right.
And the depth first is a little more sort of capture the flag
in the sense of what can I do with this?
Can I string these together to accomplish some objective,
like compromise a particular system or get cardholder data or,
uh, you know, whatever and And they both have their merits,
right?
You get a broader spread of potential things that you need to worry about,
right?
But having a pen test report full of kind of abstracted
vulnerabilities with kind of abstracted severity levels with
kind of abstracted exploit cases, um, is sometimes less compelling,
particularly to executives that don't understand how these things work.
Whereas the sort of depth first approach of let's play capture the flag.
Can I accomplish a particular goal?
And then you could go to some executive and say,
Hey, the pence is firm was able to NuGet cardholder data was able to compromise.
You know, this important system was able to,
you know, adversely affect you know,
our our financial statements or something like that,
right?
That that is potentially more compelling.
Right?
And then the executives say, Well, how do I fix that?
Well, they did this and this and this and this right?
And then that you get those things fixed, but maybe not necessarily others.
E One of the problems with that is and I thinks about pen testers.
The way they work is a lot of pen testers like to get roots.
It's like I got root.
IT got to admin privileges.
Right now I've given up, so you tend to sometimes get these limited pen test off.
Well,
I got this in a day and then I stopped because there's no point going on snow.
I've got roots like, Yeah,
but I want to know how many different ways you could do that.
I want to know where my vulnerabilities are because it's not a
matter of it's not a matter of you winning and me losing you
getting roots and mean less and get rows.
It's not a matter of me getting a mean the worst pen
testers apprenticed with no findings because I just paid
a bundle of money for learning.
Yeah, yeah.
Uh, yeah,
I think that can happen on I think that that rears its
sort of ugly head in other ways, in the sense of like,
oh, I found a bunch of cross site scripting vulnerabilities.
And I'm going to show you that by doing this sort of classic
cross site scripting testing where I get the little dialog box
pop up with some snarky comment, right?
And then now again,
if you try to go to somebody who doesn't really
understand how web attacks working, you say,
Look, I've got a dialog box to pop up, right?
That's not calling argument for why they need to fix cross site scripting,
right?
But they need to fix their cross site scripting issues, right?
And, you know,
cross that scripting is the sort of cockroach of
Web application vulnerabilities.
Right?
If you see one there, 10 more that you don't on.
So if you're finding a bunch of these errors in your site,
um, that that the flag was lots of other things.
So there are lots of other problems, you know, and and to me is the assessor.
When the pen tester says,
I found a bunch of cross site scripting errors in this Web
application that tells me that all the stuff in the
Requirement six is probably broken, and I need to go back and re evaluate that.