From d33015844e9841fc94bb0abc1cf0ce5e688d3f7d Mon Sep 17 00:00:00 2001
From: Paul Duncan <pabs@pablotron.org>
Date: Thu, 30 Jan 2025 22:10:02 -0500
Subject: content/articles/site-backend.md: document robots.txt

---
 content/articles/site-backend.md | 12 ++++++++++++
 1 file changed, 12 insertions(+)

(limited to 'content')

diff --git a/content/articles/site-backend.md b/content/articles/site-backend.md
index 3ea2b38..cd15a8d 100644
--- a/content/articles/site-backend.md
+++ b/content/articles/site-backend.md
@@ -112,6 +112,8 @@ Here are a few articles which cover guidelines that I follow:
 - [Why your website should be under 14kB in size][14kb]
 - [5 things you don't need JavaScript for][you-dont-need-js]
 
+
+
 ### Images
 
 Images are created as follows:
@@ -552,6 +554,9 @@ I am investigated doing automated validation with [htmltest][],
 
 ## Other
 
+The [robots.txt][] for this site is from [here][robotstxt-ai] and
+excludes all known [LLM][] crawlers.
+
 I do not store credentials (e.g., the [HMAC][] key for the deployment
 [web hook][]) in the [Git repository for this site][git-repo].
 
@@ -587,6 +592,7 @@ published.
   allowing `style-src-attr 'unsafe-inline'`.  Without this exception
   inline [SVGs][svg] work fine, but viewing them directly just shows a
   black box.
+- 2025-01-30: Added [robots.txt][] info to [Other](#other).
 
 [s-hugo-configuration]: #hugo-configuration
   "Hugo Configuration"
@@ -800,3 +806,9 @@ published.
   "Uniform resource locator (URL)"
 [progressive enhancement]: https://developer.mozilla.org/en-US/docs/Glossary/Progressive_Enhancement
   "Web design which puts an emphasis on content first."
+[robots.txt]:  https://en.wikipedia.org/wiki/Robots.txt
+  "Robot exclusion protocol."
+[robotstxt-ai]: https://robotstxt.com/ai
+  "AI / LLM User-Agents: Blocking Guide"
+[llm]: https://en.wikipedia.org/wiki/Large_language_model
+  "Large Language Model"
-- 
cgit v1.2.3