What is the robots.txt for?

The HRPro setup package includes a robots.txt file, which serves as a simple text file for websites to communicate with web crawlers and robots. This file ensures that the HRPro website or Employee Portal is not indexed by search engines.

The content of the sample robots.txt is 

User-agent: *

Disallow: /

Here's what it does:

• User-agent: *: This means the rule applies to all web crawlers or bots, regardless of their type.

• Disallow: /: This tells all bots to avoid crawling any part of the website, effectively blocking the entire site from being indexed.

This file is usually placed at the root of the website (e.g., www.example.com/robots.txt) to ensure it can be accessed by web crawlers.

External Reference:

• MDN Web Docs: A detailed guide on robots.txt configuration and its practical implementation.

• Google Developers: Instructions on how to create and submit a robots.txt file, along with examples.

• Wikipedia: A comprehensive overview of the Robots Exclusion Protocol, including its history and usage.