AI crawlers
All artka.dev articles and notes tagged AI crawlers: hands-on breakdowns, code examples and lessons learned in production.
- 01
8 pages out of 185: an Astro blog indexing audit
A Search Console audit of an Astro blog: the bugs it found and why "Crawled — currently not indexed" does not explain the cause of exclusion.
- 02
robots.txt and AI bots: four tasks, not one
Which levers control regular search, generative answers, model training, and user-triggered page fetches. Current crawler names and an access check as of September 2026.