# Sanatana Rahasya
#
# ACCESS AND RIGHTS ARE TWO DIFFERENT LAYERS. THIS FILE GOVERNS ONLY THE FIRST.
#
# This file says who may FETCH a page. It is a request to well-behaved crawlers and it binds
# nobody else. Whether the fetched text may then be used to TRAIN a model is a rights
# question, and it is answered in three other places that have NOT changed: /terms/, the
# machine-readable reservation at /.well-known/tdmrep.json, and the
# <meta name="tdm-reservation"> that every page carries. Text and data mining remains
# expressly reserved. Asking is still the right thing to do; the address is on /terms/.
#
# CHANGED 14 Aug 2026, deliberately, reversing the position taken on 9 Aug. The earlier file
# refused the named training crawlers here as well as in the reservation. Two things decided
# it. First, the refusal was costing the thing it was meant to protect: most of what models
# say about Sanatana Dharma is thin, or arrives through a colonial-era secondary source, and
# this corpus exists to correct that. Material that is never read corrects nothing. Second,
# Cloudflare is retiring on 15 September the managed rule that enforced the block at the edge,
# so the position had to be restated rather than quietly inherited.
#
# WHY THERE ARE NO NAMED AGENT GROUPS BELOW. A crawler obeys the most specific group that
# matches it and ignores the rest, so a named group saying only "Allow: /" ALSO exempted that
# agent from the four Disallow lines -- OAI-SearchBot was never told to stay out of /control/.
# One "*" group is both simpler and stricter.
#
# THE TWO SURFACES THAT MUST NOT DRIFT ARE NOW /terms/ AND tdmrep.json + the BaseLayout meta.
# This file is deliberately no longer one of them. That separation IS the change.

User-agent: *
Allow: /
Disallow: /control/
Disallow: /engagement/
Disallow: /admin/
Disallow: /api/

Sitemap: https://www.sanatanarahasya.com/sitemap-index.xml