Those running a public web server are frequently subject to bot scans. While using best practices is the proper approach, there may still be a lot of 404 log results from bots. Declutter the log by redirecting bot URLs to a single zero-byte empty page while still returning code 404.
Another choice is to return HTTP status codes. Netlify only allows a few _redirects HTTP status codes. We redirect bot URLs to /bot.html and return a 404 code using an empty file.
Empty files: since these files are legitimate, but may not be used by your site, we create empty files for these:
favicon.ico ads.txt app-ads.txt
Netlify _redirects file
should be in the site
For Hugo it may be “public/_redirects”.
We prefer the
_redirects file to netlify.toml, because the redirect syntax is much more concise in
to instantly test file syntax.
Here is our _redirects file for a Hugo blog, which may need editing to be suitable for your site.