diff options
| author | Paul Duncan <pabs@pablotron.org> | 2025-01-30 22:14:07 -0500 |
|---|---|---|
| committer | Paul Duncan <pabs@pablotron.org> | 2025-01-30 22:14:07 -0500 |
| commit | 2b7bef795c4df94630b47b429a4629d9a1089441 (patch) | |
| tree | 1a9017a2676a3d0d6411f7ba34762b57138fd5d7 | |
| parent | 4607df6cd65a4f21bdd559ec4c7f1470a84f3309 (diff) | |
| download | pablotron.org-2b7bef795c4df94630b47b429a4629d9a1089441.tar.xz pablotron.org-2b7bef795c4df94630b47b429a4629d9a1089441.zip | |
README.md: add brief robots.txt documentation
| -rw-r--r-- | README.md | 7 |
1 files changed, 7 insertions, 0 deletions
@@ -159,6 +159,11 @@ carousel: height: 655 ``` +## Robots.txt + +The contents of `static/robots.txt` are from [this site][robotstxt-ai] +and used to prevent the entire site from being indexed by LLM crawlers. + ## Deploy Site To clone site repo: @@ -248,3 +253,5 @@ The [Bulma][] [SASS][] is: "Progressive enhancement picture and figure shortcodes for Hugo." [pngquant]: https://pngquant.org/ "Lossy compression of PNG images." +[robotstxt-ai]: https://robotstxt.com/ai + "AI / LLM User-Agents: Blocking Guide" |