From d33015844e9841fc94bb0abc1cf0ce5e688d3f7d Mon Sep 17 00:00:00 2001 From: Paul Duncan Date: Thu, 30 Jan 2025 22:10:02 -0500 Subject: content/articles/site-backend.md: document robots.txt --- content/articles/site-backend.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) (limited to 'content/articles/site-backend.md') diff --git a/content/articles/site-backend.md b/content/articles/site-backend.md index 3ea2b38..cd15a8d 100644 --- a/content/articles/site-backend.md +++ b/content/articles/site-backend.md @@ -112,6 +112,8 @@ Here are a few articles which cover guidelines that I follow: - [Why your website should be under 14kB in size][14kb] - [5 things you don't need JavaScript for][you-dont-need-js] + + ### Images Images are created as follows: @@ -552,6 +554,9 @@ I am investigated doing automated validation with [htmltest][], ## Other +The [robots.txt][] for this site is from [here][robotstxt-ai] and +excludes all known [LLM][] crawlers. + I do not store credentials (e.g., the [HMAC][] key for the deployment [web hook][]) in the [Git repository for this site][git-repo]. @@ -587,6 +592,7 @@ published. allowing `style-src-attr 'unsafe-inline'`. Without this exception inline [SVGs][svg] work fine, but viewing them directly just shows a black box. +- 2025-01-30: Added [robots.txt][] info to [Other](#other). [s-hugo-configuration]: #hugo-configuration "Hugo Configuration" @@ -800,3 +806,9 @@ published. "Uniform resource locator (URL)" [progressive enhancement]: https://developer.mozilla.org/en-US/docs/Glossary/Progressive_Enhancement "Web design which puts an emphasis on content first." +[robots.txt]: https://en.wikipedia.org/wiki/Robots.txt + "Robot exclusion protocol." +[robotstxt-ai]: https://robotstxt.com/ai + "AI / LLM User-Agents: Blocking Guide" +[llm]: https://en.wikipedia.org/wiki/Large_language_model + "Large Language Model" -- cgit v1.2.3