Google's John Mueller said that since the robots.txt file is cached by Google for about 24-hours, it does not make much sense to dynamically update your robots.txt file throughout the day to control ...
Results that may be inaccessible to you are currently showing.
Hide inaccessible results
Feedback