Welcome back.
We were in the course now in this lecture, I will talk about robots, that 60 file, which is a very,
very important file for search engines like search engines, actually get the instructions from this
file.
And after that, they crawl to our side, indexed different Web pages.
So let's talk about this Web site.
We have created not by default, we have not edited any robots or text file, and the default file is
available there.
OK, so let's go.
And if we check this Web site for a robot, that text file we just have to write after the Eurail dot
com backlist robots that tsd.
So because this is a text file.
So if you see here, there are only two lines by the four in the two lines.
Means user Desh agent Kahlan.
And then you can see this sign multiplication sign.
This sign means that this website has allowed all the search engines to index everything on this Web
site index.
Everything means what?
Look, let me just open this Web site here.
Like, this is our Web site right now.
This is our Web site.
And this content is for the users to see, to read, to watch, like they can click any of the links.
They can open any up the post here or any to the content available on this site is for the users.
OK.
So this should be indexed in crawl.
That makes sense.
But how can you allow the searching gender of boards to actually go into the the directories in all
the folders of your site?
For example, if I go to my C panel, my Web site's Web hosting account, and then I go to click the
folder of this Web site skimmer list dot com.
There you can see different folders like WP Edman.
This is the folder for the air domain from where we exist.
This dashboard for actually managing our site on what price?
So this is basically they're different folders.
You can see like the Edman folder.
They include in different scripts like, you know, different JavaScript files and different other scripts.
So this shouldn't be a unique role by such in search engine crawlers.
They if they come and spend crawling all the folders and all the pages, it will take a lot of time
for them.
They will vist resources.
And sometimes when you have a big Web site like an e-commerce Web site, you have to optimize the crowd
rate time, basically.
This is very important because if you are allowing the robots to actually crawl and index everything,
then you are just limiting them and you are just wasting their time by actually crawling all the pages.
So you should increase and optimize the crawling rate by not allowing them to index everything in a
cloud, everything.
So they will just go and crawl the important pages of your site, which are for visitors and not for
the demands.
Okay.
So what we can do, because this is our WordPress Web site.
So this is a little bit different, you know, but you can use, you know, this technique for all other
Web sites as well.
But for WordPress, I have this file.
I can show you how this file looks like this.
OK.
This is the same.
I just created Rahbar Stack to 60 file.
OK.
And the North Page.
And after that, I have date site map to the top.
Why this is important.
You can just rightside map then column.
And then the exact.
You are all up your site map.
OK.
You can't replace this with your site.
I have provided this file in the resources section of this video.
So this is the site map you URL where whenever a search engine prowler comes toward this file.
So they will see the updated site map and will get the site map crawl from this.
You are it.
And then we have this all instructions to four different and for all search engine crawlers.
First of all, this user agent in this sign means all all user agents.
OK.
And then disallow disallow means like this allowed this folder.
I can show you this folder in my seat panel.
You can see CGI.
Ben, this is not really important folder for the visitors.
Then WP, Edman and other scripts.
So I can once again show you do those includes a domain bin or chaos.
This sign when somebody searches.
So are you oralism politically created in the browser?
So I'm saying like, don't crawl.
This question marks your ends as well.
And there's also, you know, in the outer pages than comments, Pete.
And we can when we disallow something, we can once again override that by allowing something like here
we say use it.
Agents, media partners of Google is allowed to index our Web site, you know, and different things.
But we are not allowing these sections.
And this way, we're optimizing our crawling rate because Google, Googles, KRAEUTLER will come in
only index what is instructed in this file.
So what we need to do simply we go to, you know, simply we go to the.
This page, this folder in the file manager of our C panel will air this Roboshark text file to the
main home page, OK?
The main folder where we already edit the Google and Bing estimate file, we uploaded that here.
You can see the robots at the 60 file is already there, but we'll delete that one.
Really lead that permanently, because we are just going to click applaud and applaud the new one which
is optimized.
OK.
So I will just go to the folder where this file is located.
On my computer.
And I will just click it here and click open so it will be applauded.
And then I go back is already uploaded.
I can see it here or this is robot at 60 plus 60.
The name is wrong.
So what I will do, I will just to remove the dart.
These things, you know.
The extra I need to remove that.
So that's it.
Now I have to once again check my side for this file.
Robots that the 60.
And it is not indexed because.
OK.
Let me open this file in a new incognito window here.
Sometimes it is not working.
So here you can see in the incognito window, it is updated already.
OK, because here this is not working because we have the cash plug in, you know, if we clear altercation.
Sometimes when we change something, you should always clear your caché to take effect of that right
now, if I just refresh, it should be shown here as well.
But it is fine.
It is already there.
OK.
So there is no issue.
This should be a simple file robot's activity file.
Now, you can also test this using Google, Google.
Also have a tool for robots, a text file.
You can just write in Google.
OK.
Just write in Google robots that text file test derogate so you can test this.
Using the Google tool, Google has a tool here, you can see open reports, a TFT tester.
When you open the tool, it will ask you which Web site you want to test this.
So I will just select my Web site, camera list, dot com.
And it will automatically open there.
Okay.
Right now you can see this.
This is the file, which is actually the old file.
Or aborts dot DSTO K.
Begin aided here and then click test.
So here you can see allowed allowed means Google board is allowed, you can click Google image and test
the Eurail Google board bought images also allowed.
So all of the Google crawler, these are different Google crowds are like about Google News, Google
image.
Google views, mobile and media partner, stuff like that.
So you can allow all of them.
I have allowed like three to four of them, but you can allow all of them in the robot's text file.