How to get Robots.txt to help SEO

How does Robots.txt help you {{website designer}} if made correctly? If it is not correctly written search engines will just move past your {{website designer}}. Inevitably this means your {{website designer}} will not get ranked at all. The .txt is a file connection to your text file. Basically, it tells search engines where to go and not go. This means you NEED your robot.txt file to be written correctly.

Where do I put my Robots.txt

Google Robots.txt

Putting your Robots.txt in the correct location in ESSENTIAL. So where do you put the file? It must be in your root directory for it to serve its purpose otherwise it will be useless. For example, whatever your domain name is, depends on what would be included in your file. https://www.(your domain name).com/robots.txt. In addition, you must not change the name and keep it as robots.txt or again, it would be useless.

What should the file look like?

Robots.txt File

Each {{website designer}} with a Robots.txt file may look differently depending on how the {{website designer}} is configured. For a regular and default configured Robots.txt using the Yoast SEO WordPress Plug-In should look like this picture. Whenever you see the asterisk (*) means fully accessible for spiders. The User-Agent is obviously you, however, you can have more than one User-Agent. When you see “Disallow” that is indicating for those files NOT be indexed. Even still, if the search engine finds enough URL links to your {{website designer}} that will not prevent being seen by search engines. The only true 100% way for you to not have something indexed, you MUST use a “rel=noindex” tag.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.