Goal
To properly create or optimize your robots.txt file
learn.llmdecode.com
Back to all resources
SOP-057
How to Setup the Perfect Robots.txt File
Create or optimize your robots.txt file and allow search engines to index your website exactly as you want them to
Type
SOPs · PDF
Execution time
30 mins
Last updated
26 March 2024
Who does this
The person responsible for SEO in your organization
Length
11 pages
Ideal outcome
You have an excellent robots.txt on your website that allows search engines to index your website exactly as you want them to
Why it matters
Your robots.txt sets the fundamental rules that most search engines will read and follow, once they start crawling your website. They tell search engines which parts of your website you don’t want (or they don’t need) to crawl
Before you start
• You need to have access to the Google Search Console property of the website you are working on. If you don’t have a Google Search Console property setup yet you can do so by following SOP 020 (web version)
Key steps 19 steps
What is inside 23 headings
1 Auditing your current Robots.txt 3
- Open your current robots.txt in your browser.
- Tick the following cheat-sheet and fix each issue if it exists:
- To make sure you didn’t miss any page after this checklist, perform the following test:
2 Creating a Robots.txt 2
- Create a .txt document on your computer.
- Allowing all crawlers to crawl your entire website
3 Adding the Robots.txt to your website 8
- Open your WordPress Admin.
- Click “SEO” → “Tools”.
- Click “SEO” → “Dashboard” → “Features”
- Click “SEO” → “Dashboard” → “Features”
- Click “Save changes”
- Click “File Editor”:
- You will see an input text box where you can add or edit the text of your robots.txt file → Paste the content of the robots.txt file you have just created → Click “Save Changes to Robots.txt”.
- To make sure everything went perfectly, on your browser open: “http://llmdecode.com/robots.txt”.
4 Validating your Robots.txt 6
- Open Google Search Console’s Robot.txt tester.
- If you have multiple properties in that Google Account select the right one:
- You will be sent to the robots.txt Tester tool where you will see the last robots.txt file that Google crawled.
- Open your actual robots.txt and make sure the version that you are seeing in Google Search Console’s tool is exactly the same as the current version you have live on your domain.
- If there are no syntax errors or logic warnings on your file you will see this message:
- If no issues were found on your robots.txt. Test your URLs by adding them to the text input box and clicking “Test”:
The document 11 pages
The file is locked
It comes with the SOP Library.
Bought from LLM Decode before? Sign in to pay ₹ 1,999.00.