If you've been around coding and development for any amount of time,
you've probably heard the term regular expression,
or the more common regex, probably the topic that frustrates people the most.
So I'm not going to dive too deep and bore you,
but just enough to unlock the capabilities to help you understand
how it may be useful in your day‑to‑day life.
The full documentation will by far help you more than I can in
a few short minutes explaining this topic.
There are people who have written and delivered
full‑day training courses just on regex.
So, what is it?
Well, more or less pattern recognition.
Does the next line in the log file start with a certain character?
Does this file contain a specific signature for a virus?
There are many use cases for using regex in security.
Most notably,
I want to know if certain files on our servers
contain a signature for a new zero day,
or do we have the ability to locate and pull out notable actions in a log file?
Just recently,
I used regex to audit a data file from a production server our
development teams were using for testing purposes.
I was looking for customer personal information including email,
phone numbers, and addresses,
spot checking whether information was obfuscated in
order to use the data as per our policy.
The functions of the re module are important,
obviously,
but I will start with the regular expression syntax that includes
the special character and special sequences and can be found
under the regular expression syntax function of the python.org
documents of the re module.
These characters and sequences are at the heart of
how regular expression operates.
With regex,
you are basically doing a string comparison and laying out the
format to say if this string matches this pattern,
do something.
If not, move on.
Standard ASCII, numbers, and letters are the simplest form of comparison.
Does this string contain this specific character or characters in this order?
The special characters, also referred to as metacharacters,
are a layer of rules that sit on top of the patterns you are trying to find.
For example,
the question mark placed beside the pattern states this
pattern must be present 0 or 1 time only,
but not more.
A plus sign states the pattern must be present at least one time in the string,
and the star states that the pattern is basically optional to the string.
I will show you a few examples to hopefully curve this up in a second here,
but first I want to also talk about special sequences.
The backslash plays a significant role in regex.
It tells the tool to escape,
meaning instead of literally looking for that
lowercase D character in the string,
instead it will match this pattern if it encounters any digit 0 to 9.
In this example, I'm looking for the pattern \d\d, meaning two digits.
But,
the special character attached to the second sequence tells
the system how to treat that second digit.
The asterisk states the character pattern is optional,
yet okay if it's repetitive, and will return the full number.
The question mark only wants the pattern if it's specifically one character,
or digit in that case, and we'll pull the 30 out as it has two digits,
then the double 0.
Finally, the plus sign wants one or more,
so it pulls the full part of the string that lines up.
So what's the difference between the asterisk and the plus symbol, you ask?
Well, looking at the new short string with this same pattern,
the asterisk returns it as it requires 0 or more,
as does the question mark, but the plus symbol,
which requires a minimum of one match to fit the pattern,
will not return anything.
Regex can be so confusing, no doubt.
Finding the right pattern match will take practice and a lot of testing.
Notice the r at the front of the string.
Well, that's to signify I want the raw text to test the pattern against.
You don't want Python functions escaping characters and not
displaying what it is you think you are getting.
The r tells Python to display the back slashes and anything
else that may accidentally change the input.
The syntax makes up all the patterns which are at the heart of the regex.
How those patterns are used to find and manipulate the information,
well, you'll need the functions.
Compile is when you're using a pattern in multiple areas.
You basically write your pattern once, drop it into the compile function,
and assign it a pattern variable,
which can then be used in various search functions.
The next three, search, match, and findall,
tell Python how to find that pattern.
Match searches at the start of the string in question,
search will look for the pattern anywhere in that string,
but only returns the first match,
while findall returns all the matches of the pattern.
Definitely 100% ways of manipulating even that,
and there are functions like split and replace that fall
more on the string manipulation side.
But for security,
you're going to focus more on matching and searching
than any type of manipulation action.
All right, jumping into this regex demo.
We're going to take a look at some of the more important
functions of the regular expression.
One of the first things you need to do is import the re module.
That's what brings all these functions into the script.
I did include the string that we're going to be parsing today.
Normally this would be more of a dynamic input coming
from a log file or some type of input,
but I thought it was important that we had the string here
in front of us so that we can take a look at it and
understand what we're trying to do, what we're trying to parse out,
what that looks like in the string itself.
So the two patterns we're going to be looking at.
The first patterns are a result pattern,
we're going to be looking for the word success
followed by that \w and the asterisk.
So the word success is quite literal, the word success.
Th \w says I want to get all the alphanumeric characters
that come after that word success,
and then the asterisk means however many there are.
So this is going to pull the word success,
it's going to pull the word successful, it's going to pull the word successfully,
all different common ways of describing an accepted logging attempt.
The next one is the date pattern.
So the \d means I'm looking for a digit.
So I'm looking for three digits, all separated by a slash,
and they're all going to have that little plus symbol on the end.
It's signifying that I don't care how many digits there are.
So it could be one or two or more, but I'm basically saying this could be in day,
month, year format, it could be in month,
day, year format or it could be in year,
day, month or year, month, date, however the format of the log is set up.
This should be able to take into account all those different combinations.
We're then going to take those result patterns and we're going to compile
them down so that we can store these patterns into a variable of their own
so that they can be used within the functions.
So this first section we're going to be looking at,
we're going to be running through a match function.
So match, as you recall, looks at the very start of the string,
and it's going to say does this word success show up at the start of the string?
And of course it does not, so I fully expect this,
and the way I handled this is to basically if it does come back as None,
we're going to gracefully just print out the word None.
Otherwise, we may get an error, the page may fail,
the script may fail, so we don't want that to happen.
So we're just going to print out the word None,
which I fully anticipate it happening.
We're then going to run through the search function.
So the search function looks anywhere within that string,
not just at the start,
but it basically starts from left to right and looks for that
word success anywhere within the string.
It then returns did it find it, what did it find there,
it returns where it found it, and then also the date.
So we're also using that second pattern of the date
pattern to also display the date.
So let's run this real fast and see what happens.
So we do in fact have the match returned a None,
which we were anticipating, but then the search returned the name of the string,
it returned the word successful, so that's more than just the success.
That's showing that we actually found the full response from the log.
It started at character 35,
so as you recall it started at character 0 all the way down.
So right here is where 35 would be.
And then date, we were looking at the dates,
which is in that first set of characters.
It was able to find that pattern as well and return it.
So jumping back here into the script,
we're going to go through and clean this up a little bit.
So we're going to change some of our comments around so we
can comment that top part out again.
We're going to look at this bottom example now. So, similar to what we
just did, we looked for that pattern, we pulled out the pattern of the
date, we pulled out the pattern of the results.
Another way of doing this is to split that string.
So if you recall the string at the top had a date followed by
a space, then a time followed by a space.
If we were able to split this whole string up by those spaces and
assign each of these different parts of the string into a list,
basically what we're doing here is saying print out all the tokens,
so each individual part of this list.
We're then going to say give me token number 4.
So start at the beginning, token 0 is the date,
token 1 is the time, token 2 is the user, token 3 is the
page, and token 4 would be the result.
And then followed by that we would want to go back to
token 0, which was, as you know, the date.
And then finally after that, we're going to do a findall.
So similar to the search, I'm looking for this pattern, in this case I'm
looking for a single digit pattern throughout the string.
So anywhere that this pattern is found, it's going to drop
that into a list and return all the results.
So this is good when you have something you're looking for
multiple times within a single string.
So in this case we're just looking at the digits, so I'm fully
expecting it to return all the digits from the date and all the
digits from the time when it returns.
So let's save this, jump over here and we run it again.
So when we do our split by the spaces, you'll see that all the tokens
prints out each individual part of the string into its own separate
item in the list. Item number 4 is, in fact, the word Successful, and
item number 0 is, in fact, the date. Then down below, I'm splitting
out that string into a list and giving me every single digit in its
own list item.