{"id":24140,"date":"2022-02-02T14:21:55","date_gmt":"2022-02-02T14:21:55","guid":{"rendered":"https:\/\/1-grid.com\/knowledge\/cpanel-how-do-i-create-a-robots-txt-file\/"},"modified":"2023-10-19T12:15:22","modified_gmt":"2023-10-19T12:15:22","slug":"cpanel-how-do-i-create-a-robots-txt-file","status":"publish","type":"ht_kb","link":"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/","title":{"rendered":"cPanel: How do I create a robots.txt file ?"},"content":{"rendered":"<blockquote><p><span style=\"color: #000000;\">This guide provide you with information on how to manage which internet bots crawl your site.<\/span><\/p><\/blockquote>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<h1><span style=\"color: #000000;\"><strong><span style=\"font-family: Helvetica, sans-serif; font-size: 30px;\">Why create a Robots.txt file ?<\/span><\/strong><\/span><\/h1>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-size: 16px; font-family: Helvetica, sans-serif;\">A web robot\u2019s primary job is to scan websites and pages for information. They work tirelessly to collect data on behalf of search engines and other applications.<\/span><\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-size: 16px; font-family: Helvetica, sans-serif;\">Whether you want to fine-tune access to your site or want to work on a development site without showing up Google results.<\/span><\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-size: 16px; font-family: Helvetica, sans-serif;\">Once implemented the robots.txt file lets web crawlers know which parts they can collect information.\u00a0<\/span><\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-family: 'Courier New', Courier, monospace; font-size: 15px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 300; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #f7f4f2; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;\">User-agent: *<\/span><br style=\"box-sizing: border-box; color: #646464; font-family: 'Courier New', Courier, monospace; font-size: 15px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 300; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;\" \/><span style=\"font-family: 'Courier New', Courier, monospace; font-size: 15px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 300; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #f7f4f2; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;\">Disallow: \/buildinglayout.png<\/span>\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-size: 16px; font-family: Helvetica, sans-serif;\">Let\u2019s break down the code above \u201cuser-agent\u201d pertains to the web crawlers and the * sign means all web crawlers. The \u00a0first line grabs attention by saying \u201cListen up all web crawlers!\u201d .\u00a0<\/span><\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-family: Helvetica,sans-serif;\">\u00a0<\/span><\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-family: Helvetica,sans-serif;\"><span style=\"font-size: 16px;\">We move onto our second line which lets the web crawler know its direction. The forward slash (\/) stops the bots from searching all the pages on your site. You can also discourage information collected for one specific page, in this case.<\/span><\/span><\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-family: Helvetica,sans-serif;\">\u00a0<\/span><\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-family: Helvetica,sans-serif;\">\u00a0<\/span><\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-size: 16px; font-family: Helvetica, sans-serif;\">There are many types of web crawlers (aka user-agents) that can be specified. Below is a chart of the most popular web crawlers followed by as their associations.<\/span><\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\"><strong><span style=\"font-size: 24px;\">The top 10 bots you should know about.<\/span><\/strong><span style=\"font-family: Helvetica,sans-serif;\"><br \/>\n<\/span><\/span><\/p>\n<p><span style=\"color: #000000;\"><span style=\"font-family: Helvetica,sans-serif;\"><img decoding=\"async\" class=\"fr-fic fr-dib aligncenter\" style=\"width: 488px;\" src=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/33038670981\/original\/T9sRvmtnpMg8-GwkjBoeEtZyD4OSkG30dw.png?1585063443\" height=\"330\" data-attachment=\"[object Object]\" data-id=\"33038670981\" \/><\/span><\/span><\/p>\n<hr \/>\n<h1><span style=\"color: #000000;\"><span style=\"font-size: 30px;\"><strong><span style=\"font-family: Helvetica, sans-serif;\">How to create a robots.txt<\/span><\/strong><\/span><\/span><\/h1>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p style=\"box-sizing: border-box; margin: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #fefefe; text-decoration-style: initial; text-decoration-color: initial;\"><span style=\"color: #000000;\"><span style=\"box-sizing: border-box; font-size: 24px;\"><strong style=\"box-sizing: border-box; font-weight: bold;\"><span style=\"font-family: Helvetica, sans-serif;\">Step 1- \u00a0<\/span><\/strong><span style=\"font-family: Helvetica, sans-serif;\">Login to Cpanel<\/span><\/span><\/span><\/p>\n<ul style=\"box-sizing: border-box; margin: 8px 0px 4px; padding: 0px 0px 0px 40px; list-style: outside disc; line-height: 17px; color: #333333; font-family: Tahoma, Verdana, Segoe, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #fefefe; text-decoration-style: initial; text-decoration-color: initial;\">\n<li style=\"box-sizing: border-box; line-height: 20.8px; font-size: 16px !important; margin: 0px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif !important;\"><span style=\"color: #000000;\"><span style=\"font-family: Helvetica,sans-serif;\"><span style=\"box-sizing: border-box; font-size: 16px;\">Please see our article on logging into cPanel <a style=\"box-sizing: border-box; background-color: transparent; color: #00c1dc; text-decoration: none; font-family: Tahoma, Verdana, Segoe, sans-serif !important;\" href=\"https:\/\/1-grid.com\/knowledge\/how-to-log-in-to-cpanel-via-your-1-grid-customer-zone\/\" target=\"_blank\" rel=\"noreferrer noopener\">here.<\/a><\/span><\/span><\/span><\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"fr-fic fr-fil fr-dib aligncenter\" src=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/33092595781\/original\/clggMLCUSVYYYMSyIbo_7xc04IrhUoYIXA.png?1678165611\" width=\"190\" height=\"295\" data-attachment=\"[object Object]\" data-id=\"33092595781\" \/><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\"><span style=\"box-sizing: border-box; font-size: 24px;\"><strong style=\"box-sizing: border-box; font-weight: bold;\"><span style=\"font-family: Helvetica, sans-serif;\">Step 2 &#8211; \u00a0<\/span><\/strong><span style=\"font-family: Helvetica, sans-serif;\">Go to File Manager<\/span><\/span><\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\"><span style=\"box-sizing: border-box; font-size: 18px;\"><span style=\"font-family: Helvetica, sans-serif;\"><img decoding=\"async\" class=\"fr-fic fr-dib fr-bordered aligncenter\" style=\"width: 444px;\" src=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/33038671016\/original\/hgmI_9Nvhqp4vM8I9pylMM6KP44qHZGqhQ.png?1585063463\" height=\"209\" data-id=\"33038670854\" \/><\/span><\/span><br \/>\n<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\"><span style=\"box-sizing: border-box; font-size: 24px;\"><strong style=\"box-sizing: border-box; font-weight: bold;\"><span style=\"font-family: Helvetica, sans-serif;\">Step 2 &#8211; \u00a0<\/span><\/strong><span style=\"font-family: Helvetica, sans-serif;\">Select public_html and create a new file.<\/span><\/span><\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\"><span style=\"box-sizing: border-box; font-size: 18px;\"><span style=\"font-family: Helvetica, sans-serif;\"><img decoding=\"async\" class=\"fr-fic fr-dib fr-bordered aligncenter\" style=\"width: 471px;\" src=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/33038671173\/original\/vLLuKfVvW1rWD7ZBpkA9Gk9c0oF9XFN4fA.png?1585063656\" height=\"308\" data-id=\"33038671133\" \/><\/span><\/span><br \/>\n<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\"><span style=\"box-sizing: border-box; font-size: 24px;\"><strong style=\"box-sizing: border-box; font-weight: bold;\"><span style=\"font-family: Helvetica, sans-serif;\">Step 3 &#8211; \u00a0<\/span><\/strong><span style=\"font-family: Helvetica, sans-serif;\">Edit the new <strong>robots f<\/strong>ile to add your instructions and save.<\/span><\/span><\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\"><span style=\"box-sizing: border-box; font-size: 18px;\"><span style=\"font-family: Helvetica, sans-serif;\"><img decoding=\"async\" class=\"fr-fic fr-dib fr-bordered aligncenter\" style=\"width: 433px;\" src=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/33038671196\/original\/y6ILCfEwQxY_mtYcR5c3ndKn5TGRa0mLYQ.png?1585063691\" height=\"334\" data-attachment=\"[object Object]\" data-id=\"33038671196\" \/><\/span><\/span><br \/>\n<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\"><span style=\"box-sizing: border-box; font-size: 18px;\"><span style=\"font-family: Helvetica, sans-serif;\"><img decoding=\"async\" class=\"fr-fic fr-dib fr-bordered aligncenter\" style=\"width: 448px;\" src=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/33038671328\/original\/q36DFbogNRqEwa5beAs_rsliSO7mf8MxAQ.png?1585063801\" height=\"180\" data-attachment=\"[object Object]\" data-id=\"33038671328\" \/><\/span><\/span><br \/>\n<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<hr \/>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\"><strong>TIP:\u00a0<\/strong>Should you require more information regarding Google search please How to get <a href=\"https:\/\/1-grid.com\/knowledge\/how-to-get-google-to-index-your-site\/\">website displayed on Google.<\/a><\/span><\/p>\n<p style=\"box-sizing: border-box; margin-bottom: 0px; margin-left: 0px; font-size: 16px; line-height: 20.8px; word-break: normal; overflow-wrap: break-word; font-family: Tahoma, Verdana, Segoe, sans-serif; color: #333333; font-weight: 400; text-align: start; text-indent: 0px;\"><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n<p><span style=\"color: #000000;\">\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This guide provide you with information on how to manage which internet bots crawl your site. \u00a0 \u00a0 Why create a Robots.txt file ? \u00a0 A web robot\u2019s primary job is to scan websites and pages for information. They work tirelessly to collect data on behalf of search engines and&#8230;<\/p>\n","protected":false},"author":21,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"ht-kb-category":[1198],"ht-kb-tag":[233,238,247,403,470,480,864],"class_list":["post-24140","ht_kb","type-ht_kb","status-publish","format-standard","hentry","ht_kb_category-cpanel","ht_kb_tag-control-panel","ht_kb_tag-cpanel","ht_kb_tag-create","ht_kb_tag-file","ht_kb_tag-hosting","ht_kb_tag-how-to","ht_kb_tag-robots-txt"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.0 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Cpanel: How Do I Create A Robots.Txt File? | 1-grid<\/title>\n<meta name=\"description\" content=\"This guide provide you with information on how to manage which internet bots crawl your site. cPanel: How do I create a robots.txt file ?\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cpanel: How Do I Create A Robots.Txt File? | 1-grid\" \/>\n<meta property=\"og:description\" content=\"This guide provide you with information on how to manage which internet bots crawl your site. cPanel: How do I create a robots.txt file ?\" \/>\n<meta property=\"og:url\" content=\"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/\" \/>\n<meta property=\"og:site_name\" content=\"1-grid Knowledge Base\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/1grid.co.za\/\" \/>\n<meta property=\"article:modified_time\" content=\"2023-10-19T12:15:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/33038670981\/original\/T9sRvmtnpMg8-GwkjBoeEtZyD4OSkG30dw.png?1585063443\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@1grid_hosting\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/\",\"url\":\"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/\",\"name\":\"Cpanel: How Do I Create A Robots.Txt File? | 1-grid\",\"isPartOf\":{\"@id\":\"https:\/\/1grid.co.za\/knowledge\/#website\"},\"datePublished\":\"2022-02-02T14:21:55+00:00\",\"dateModified\":\"2023-10-19T12:15:22+00:00\",\"description\":\"This guide provide you with information on how to manage which internet bots crawl your site. cPanel: How do I create a robots.txt file ?\",\"breadcrumb\":{\"@id\":\"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/1grid.co.za\/knowledge\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"cPanel: How do I create a robots.txt file ?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/1grid.co.za\/knowledge\/#website\",\"url\":\"https:\/\/1grid.co.za\/knowledge\/\",\"name\":\"1-Grid Domain Hosting Knowledge Base\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/1grid.co.za\/knowledge\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/1grid.co.za\/knowledge\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/1grid.co.za\/knowledge\/#organization\",\"name\":\"1-Grid Domain Hosting Knowledge Base\",\"url\":\"https:\/\/1grid.co.za\/knowledge\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/1grid.co.za\/knowledge\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/1grid.co.za\/knowledge\/wp-content\/uploads\/2023\/01\/1-grid-logo-color.svg\",\"contentUrl\":\"https:\/\/1grid.co.za\/knowledge\/wp-content\/uploads\/2023\/01\/1-grid-logo-color.svg\",\"width\":100,\"height\":100,\"caption\":\"1-Grid Domain Hosting Knowledge Base\"},\"image\":{\"@id\":\"https:\/\/1grid.co.za\/knowledge\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/1grid.co.za\/\",\"https:\/\/twitter.com\/1grid_hosting\",\"https:\/\/www.instagram.com\/1grid_hosting\/\",\"https:\/\/za.linkedin.com\/company\/1-grid\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cpanel: How Do I Create A Robots.Txt File? | 1-grid","description":"This guide provide you with information on how to manage which internet bots crawl your site. cPanel: How do I create a robots.txt file ?","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/","og_locale":"en_US","og_type":"article","og_title":"Cpanel: How Do I Create A Robots.Txt File? | 1-grid","og_description":"This guide provide you with information on how to manage which internet bots crawl your site. cPanel: How do I create a robots.txt file ?","og_url":"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/","og_site_name":"1-grid Knowledge Base","article_publisher":"https:\/\/www.facebook.com\/1grid.co.za\/","article_modified_time":"2023-10-19T12:15:22+00:00","og_image":[{"url":"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/33038670981\/original\/T9sRvmtnpMg8-GwkjBoeEtZyD4OSkG30dw.png?1585063443"}],"twitter_card":"summary_large_image","twitter_site":"@1grid_hosting","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/","url":"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/","name":"Cpanel: How Do I Create A Robots.Txt File? | 1-grid","isPartOf":{"@id":"https:\/\/1grid.co.za\/knowledge\/#website"},"datePublished":"2022-02-02T14:21:55+00:00","dateModified":"2023-10-19T12:15:22+00:00","description":"This guide provide you with information on how to manage which internet bots crawl your site. cPanel: How do I create a robots.txt file ?","breadcrumb":{"@id":"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/1grid.co.za\/knowledge\/cpanel-how-do-i-create-a-robotstxt-file\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/1grid.co.za\/knowledge\/"},{"@type":"ListItem","position":2,"name":"cPanel: How do I create a robots.txt file ?"}]},{"@type":"WebSite","@id":"https:\/\/1grid.co.za\/knowledge\/#website","url":"https:\/\/1grid.co.za\/knowledge\/","name":"1-Grid Domain Hosting Knowledge Base","description":"","publisher":{"@id":"https:\/\/1grid.co.za\/knowledge\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/1grid.co.za\/knowledge\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/1grid.co.za\/knowledge\/#organization","name":"1-Grid Domain Hosting Knowledge Base","url":"https:\/\/1grid.co.za\/knowledge\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/1grid.co.za\/knowledge\/#\/schema\/logo\/image\/","url":"https:\/\/1grid.co.za\/knowledge\/wp-content\/uploads\/2023\/01\/1-grid-logo-color.svg","contentUrl":"https:\/\/1grid.co.za\/knowledge\/wp-content\/uploads\/2023\/01\/1-grid-logo-color.svg","width":100,"height":100,"caption":"1-Grid Domain Hosting Knowledge Base"},"image":{"@id":"https:\/\/1grid.co.za\/knowledge\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/1grid.co.za\/","https:\/\/twitter.com\/1grid_hosting","https:\/\/www.instagram.com\/1grid_hosting\/","https:\/\/za.linkedin.com\/company\/1-grid"]}]}},"_links":{"self":[{"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/ht-kb\/24140","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/ht-kb"}],"about":[{"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/types\/ht_kb"}],"author":[{"embeddable":true,"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/comments?post=24140"}],"version-history":[{"count":2,"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/ht-kb\/24140\/revisions"}],"predecessor-version":[{"id":31641,"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/ht-kb\/24140\/revisions\/31641"}],"wp:attachment":[{"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/media?parent=24140"}],"wp:term":[{"taxonomy":"ht_kb_category","embeddable":true,"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/ht-kb-category?post=24140"},{"taxonomy":"ht_kb_tag","embeddable":true,"href":"https:\/\/1grid.co.za\/knowledge\/wp-json\/wp\/v2\/ht-kb-tag?post=24140"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}