diff options
| author | Paul Duncan <pabs@pablotron.org> | 2025-01-30 22:10:02 -0500 |
|---|---|---|
| committer | Paul Duncan <pabs@pablotron.org> | 2025-01-30 22:10:02 -0500 |
| commit | d33015844e9841fc94bb0abc1cf0ce5e688d3f7d (patch) | |
| tree | 2f911d08f43f1324012f3c2d99a43cb155304a62 /content | |
| parent | 6257b5eccd471315f52363a10aa591770376f62e (diff) | |
| download | pablotron.org-d33015844e9841fc94bb0abc1cf0ce5e688d3f7d.tar.xz pablotron.org-d33015844e9841fc94bb0abc1cf0ce5e688d3f7d.zip | |
content/articles/site-backend.md: document robots.txt
Diffstat (limited to 'content')
| -rw-r--r-- | content/articles/site-backend.md | 12 |
1 files changed, 12 insertions, 0 deletions
diff --git a/content/articles/site-backend.md b/content/articles/site-backend.md index 3ea2b38..cd15a8d 100644 --- a/content/articles/site-backend.md +++ b/content/articles/site-backend.md @@ -112,6 +112,8 @@ Here are a few articles which cover guidelines that I follow: - [Why your website should be under 14kB in size][14kb] - [5 things you don't need JavaScript for][you-dont-need-js] + + ### Images Images are created as follows: @@ -552,6 +554,9 @@ I am investigated doing automated validation with [htmltest][], ## Other +The [robots.txt][] for this site is from [here][robotstxt-ai] and +excludes all known [LLM][] crawlers. + I do not store credentials (e.g., the [HMAC][] key for the deployment [web hook][]) in the [Git repository for this site][git-repo]. @@ -587,6 +592,7 @@ published. allowing `style-src-attr 'unsafe-inline'`. Without this exception inline [SVGs][svg] work fine, but viewing them directly just shows a black box. +- 2025-01-30: Added [robots.txt][] info to [Other](#other). [s-hugo-configuration]: #hugo-configuration "Hugo Configuration" @@ -800,3 +806,9 @@ published. "Uniform resource locator (URL)" [progressive enhancement]: https://developer.mozilla.org/en-US/docs/Glossary/Progressive_Enhancement "Web design which puts an emphasis on content first." +[robots.txt]: https://en.wikipedia.org/wiki/Robots.txt + "Robot exclusion protocol." +[robotstxt-ai]: https://robotstxt.com/ai + "AI / LLM User-Agents: Blocking Guide" +[llm]: https://en.wikipedia.org/wiki/Large_language_model + "Large Language Model" |