1 00:00:00,930 --> 00:00:01,470 Welcome back. 2 00:00:01,500 --> 00:00:06,900 We were in the course now in this lecture, I will talk about robots, that 60 file, which is a very, 3 00:00:06,900 --> 00:00:12,870 very important file for search engines like search engines, actually get the instructions from this 4 00:00:12,870 --> 00:00:13,410 file. 5 00:00:13,740 --> 00:00:17,880 And after that, they crawl to our side, indexed different Web pages. 6 00:00:18,270 --> 00:00:20,520 So let's talk about this Web site. 7 00:00:20,550 --> 00:00:28,260 We have created not by default, we have not edited any robots or text file, and the default file is 8 00:00:28,260 --> 00:00:28,890 available there. 9 00:00:28,980 --> 00:00:30,410 OK, so let's go. 10 00:00:30,420 --> 00:00:38,640 And if we check this Web site for a robot, that text file we just have to write after the Eurail dot 11 00:00:38,640 --> 00:00:41,550 com backlist robots that tsd. 12 00:00:41,910 --> 00:00:43,860 So because this is a text file. 13 00:00:44,280 --> 00:00:49,540 So if you see here, there are only two lines by the four in the two lines. 14 00:00:49,560 --> 00:00:52,700 Means user Desh agent Kahlan. 15 00:00:52,770 --> 00:00:56,340 And then you can see this sign multiplication sign. 16 00:00:56,760 --> 00:01:04,070 This sign means that this website has allowed all the search engines to index everything on this Web 17 00:01:04,070 --> 00:01:05,340 site index. 18 00:01:05,400 --> 00:01:07,140 Everything means what? 19 00:01:07,870 --> 00:01:12,150 Look, let me just open this Web site here. 20 00:01:13,860 --> 00:01:17,450 Like, this is our Web site right now. 21 00:01:17,600 --> 00:01:18,780 This is our Web site. 22 00:01:18,800 --> 00:01:26,150 And this content is for the users to see, to read, to watch, like they can click any of the links. 23 00:01:26,180 --> 00:01:32,670 They can open any up the post here or any to the content available on this site is for the users. 24 00:01:32,690 --> 00:01:33,080 OK. 25 00:01:33,770 --> 00:01:36,830 So this should be indexed in crawl. 26 00:01:37,250 --> 00:01:38,510 That makes sense. 27 00:01:38,900 --> 00:01:47,120 But how can you allow the searching gender of boards to actually go into the the directories in all 28 00:01:47,120 --> 00:01:48,450 the folders of your site? 29 00:01:48,470 --> 00:01:55,550 For example, if I go to my C panel, my Web site's Web hosting account, and then I go to click the 30 00:01:55,550 --> 00:01:57,850 folder of this Web site skimmer list dot com. 31 00:01:58,280 --> 00:02:01,350 There you can see different folders like WP Edman. 32 00:02:01,760 --> 00:02:04,640 This is the folder for the air domain from where we exist. 33 00:02:04,670 --> 00:02:09,050 This dashboard for actually managing our site on what price? 34 00:02:09,410 --> 00:02:11,780 So this is basically they're different folders. 35 00:02:11,780 --> 00:02:14,330 You can see like the Edman folder. 36 00:02:14,390 --> 00:02:21,560 They include in different scripts like, you know, different JavaScript files and different other scripts. 37 00:02:21,770 --> 00:02:27,470 So this shouldn't be a unique role by such in search engine crawlers. 38 00:02:28,190 --> 00:02:34,940 They if they come and spend crawling all the folders and all the pages, it will take a lot of time 39 00:02:34,940 --> 00:02:35,600 for them. 40 00:02:36,080 --> 00:02:37,760 They will vist resources. 41 00:02:38,000 --> 00:02:43,720 And sometimes when you have a big Web site like an e-commerce Web site, you have to optimize the crowd 42 00:02:43,720 --> 00:02:44,730 rate time, basically. 43 00:02:44,930 --> 00:02:52,880 This is very important because if you are allowing the robots to actually crawl and index everything, 44 00:02:52,880 --> 00:03:00,510 then you are just limiting them and you are just wasting their time by actually crawling all the pages. 45 00:03:00,530 --> 00:03:09,080 So you should increase and optimize the crawling rate by not allowing them to index everything in a 46 00:03:09,080 --> 00:03:10,010 cloud, everything. 47 00:03:10,250 --> 00:03:17,090 So they will just go and crawl the important pages of your site, which are for visitors and not for 48 00:03:17,090 --> 00:03:17,690 the demands. 49 00:03:17,840 --> 00:03:18,110 Okay. 50 00:03:18,770 --> 00:03:21,920 So what we can do, because this is our WordPress Web site. 51 00:03:21,950 --> 00:03:28,100 So this is a little bit different, you know, but you can use, you know, this technique for all other 52 00:03:28,100 --> 00:03:29,030 Web sites as well. 53 00:03:29,360 --> 00:03:31,820 But for WordPress, I have this file. 54 00:03:31,940 --> 00:03:35,610 I can show you how this file looks like this. 55 00:03:35,630 --> 00:03:35,790 OK. 56 00:03:35,870 --> 00:03:36,500 This is the same. 57 00:03:36,500 --> 00:03:39,560 I just created Rahbar Stack to 60 file. 58 00:03:39,640 --> 00:03:39,930 OK. 59 00:03:39,980 --> 00:03:41,030 And the North Page. 60 00:03:41,500 --> 00:03:45,410 And after that, I have date site map to the top. 61 00:03:45,620 --> 00:03:46,790 Why this is important. 62 00:03:47,180 --> 00:03:49,370 You can just rightside map then column. 63 00:03:49,460 --> 00:03:50,510 And then the exact. 64 00:03:50,510 --> 00:03:51,980 You are all up your site map. 65 00:03:52,030 --> 00:03:52,320 OK. 66 00:03:52,580 --> 00:03:54,650 You can't replace this with your site. 67 00:03:54,710 --> 00:03:59,150 I have provided this file in the resources section of this video. 68 00:03:59,810 --> 00:04:08,060 So this is the site map you URL where whenever a search engine prowler comes toward this file. 69 00:04:08,530 --> 00:04:14,990 So they will see the updated site map and will get the site map crawl from this. 70 00:04:14,990 --> 00:04:15,410 You are it. 71 00:04:15,590 --> 00:04:22,460 And then we have this all instructions to four different and for all search engine crawlers. 72 00:04:22,910 --> 00:04:28,070 First of all, this user agent in this sign means all all user agents. 73 00:04:28,100 --> 00:04:28,280 OK. 74 00:04:28,730 --> 00:04:33,410 And then disallow disallow means like this allowed this folder. 75 00:04:33,800 --> 00:04:36,110 I can show you this folder in my seat panel. 76 00:04:36,500 --> 00:04:37,610 You can see CGI. 77 00:04:37,640 --> 00:04:42,620 Ben, this is not really important folder for the visitors. 78 00:04:42,650 --> 00:04:46,460 Then WP, Edman and other scripts. 79 00:04:46,730 --> 00:04:51,770 So I can once again show you do those includes a domain bin or chaos. 80 00:04:52,340 --> 00:04:54,140 This sign when somebody searches. 81 00:04:54,380 --> 00:04:58,520 So are you oralism politically created in the browser? 82 00:04:58,550 --> 00:05:00,470 So I'm saying like, don't crawl. 83 00:05:00,620 --> 00:05:03,950 This question marks your ends as well. 84 00:05:04,430 --> 00:05:08,150 And there's also, you know, in the outer pages than comments, Pete. 85 00:05:08,750 --> 00:05:16,760 And we can when we disallow something, we can once again override that by allowing something like here 86 00:05:16,850 --> 00:05:17,840 we say use it. 87 00:05:17,840 --> 00:05:23,950 Agents, media partners of Google is allowed to index our Web site, you know, and different things. 88 00:05:23,960 --> 00:05:26,990 But we are not allowing these sections. 89 00:05:27,500 --> 00:05:33,620 And this way, we're optimizing our crawling rate because Google, Googles, KRAEUTLER will come in 90 00:05:33,680 --> 00:05:36,730 only index what is instructed in this file. 91 00:05:36,770 --> 00:05:42,680 So what we need to do simply we go to, you know, simply we go to the. 92 00:05:44,540 --> 00:05:53,780 This page, this folder in the file manager of our C panel will air this Roboshark text file to the 93 00:05:54,290 --> 00:05:55,450 main home page, OK? 94 00:05:55,550 --> 00:06:02,360 The main folder where we already edit the Google and Bing estimate file, we uploaded that here. 95 00:06:02,660 --> 00:06:07,510 You can see the robots at the 60 file is already there, but we'll delete that one. 96 00:06:08,880 --> 00:06:15,870 Really lead that permanently, because we are just going to click applaud and applaud the new one which 97 00:06:15,870 --> 00:06:16,820 is optimized. 98 00:06:16,880 --> 00:06:17,190 OK. 99 00:06:17,730 --> 00:06:22,650 So I will just go to the folder where this file is located. 100 00:06:24,190 --> 00:06:25,140 On my computer. 101 00:06:26,930 --> 00:06:33,900 And I will just click it here and click open so it will be applauded. 102 00:06:34,040 --> 00:06:37,250 And then I go back is already uploaded. 103 00:06:38,640 --> 00:06:42,350 I can see it here or this is robot at 60 plus 60. 104 00:06:42,390 --> 00:06:43,230 The name is wrong. 105 00:06:43,560 --> 00:06:47,370 So what I will do, I will just to remove the dart. 106 00:06:49,530 --> 00:06:50,520 These things, you know. 107 00:06:51,560 --> 00:06:53,720 The extra I need to remove that. 108 00:06:54,810 --> 00:06:55,410 So that's it. 109 00:06:55,830 --> 00:07:00,300 Now I have to once again check my side for this file. 110 00:07:04,100 --> 00:07:05,420 Robots that the 60. 111 00:07:07,360 --> 00:07:10,630 And it is not indexed because. 112 00:07:11,500 --> 00:07:11,730 OK. 113 00:07:11,800 --> 00:07:17,470 Let me open this file in a new incognito window here. 114 00:07:19,210 --> 00:07:21,760 Sometimes it is not working. 115 00:07:22,060 --> 00:07:25,880 So here you can see in the incognito window, it is updated already. 116 00:07:25,930 --> 00:07:34,240 OK, because here this is not working because we have the cash plug in, you know, if we clear altercation. 117 00:07:35,210 --> 00:07:41,320 Sometimes when we change something, you should always clear your caché to take effect of that right 118 00:07:41,330 --> 00:07:44,750 now, if I just refresh, it should be shown here as well. 119 00:07:44,780 --> 00:07:45,650 But it is fine. 120 00:07:45,650 --> 00:07:46,560 It is already there. 121 00:07:46,580 --> 00:07:46,910 OK. 122 00:07:47,450 --> 00:07:48,320 So there is no issue. 123 00:07:48,350 --> 00:07:50,810 This should be a simple file robot's activity file. 124 00:07:51,170 --> 00:07:55,030 Now, you can also test this using Google, Google. 125 00:07:55,080 --> 00:07:57,440 Also have a tool for robots, a text file. 126 00:07:57,830 --> 00:07:59,840 You can just write in Google. 127 00:08:01,390 --> 00:08:01,570 OK. 128 00:08:01,640 --> 00:08:07,490 Just write in Google robots that text file test derogate so you can test this. 129 00:08:08,490 --> 00:08:14,630 Using the Google tool, Google has a tool here, you can see open reports, a TFT tester. 130 00:08:15,060 --> 00:08:21,300 When you open the tool, it will ask you which Web site you want to test this. 131 00:08:21,300 --> 00:08:24,000 So I will just select my Web site, camera list, dot com. 132 00:08:25,470 --> 00:08:27,130 And it will automatically open there. 133 00:08:27,210 --> 00:08:27,560 Okay. 134 00:08:27,980 --> 00:08:29,800 Right now you can see this. 135 00:08:29,940 --> 00:08:32,790 This is the file, which is actually the old file. 136 00:08:36,770 --> 00:08:38,940 Or aborts dot DSTO K. 137 00:08:41,770 --> 00:08:44,650 Begin aided here and then click test. 138 00:08:47,820 --> 00:08:55,140 So here you can see allowed allowed means Google board is allowed, you can click Google image and test 139 00:08:55,140 --> 00:08:58,980 the Eurail Google board bought images also allowed. 140 00:08:59,010 --> 00:09:03,900 So all of the Google crawler, these are different Google crowds are like about Google News, Google 141 00:09:03,900 --> 00:09:04,200 image. 142 00:09:04,230 --> 00:09:08,040 Google views, mobile and media partner, stuff like that. 143 00:09:08,280 --> 00:09:09,970 So you can allow all of them. 144 00:09:10,230 --> 00:09:15,290 I have allowed like three to four of them, but you can allow all of them in the robot's text file.