# robots.txt for https://www.uempistons.com/ # Platform: Drupal 8 User-agent: * # --- Drupal core paths (standard) --- Disallow: /core/ Disallow: /profiles/ Disallow: /README.txt Disallow: /web.config Disallow: /admin/ Disallow: /comment/reply/ Disallow: /filter/tips Disallow: /node/add/ Disallow: /search/ Disallow: /user/register Disallow: /user/password Disallow: /user/login Disallow: /user/logout Disallow: /index.php/admin/ Disallow: /index.php/search/ Disallow: /index.php/user/password Disallow: /index.php/user/register Disallow: /index.php/user/login Disallow: /index.php/user/logout # --- Ecommerce / account / transactional pages (site-specific) --- Disallow: /cartpage Disallow: /user/ Disallow: /contact_us Disallow: /dealer_inquiry # --- Internal search / query-string duplicate content --- Disallow: /*?*sort= Disallow: /*?*page= Disallow: /*?*combine= # -------Allow all CSS, JS, and image assets so pages render fully for crawlers------ Allow: /sites/default/files/ Allow: /themes/ Allow: /*.css$ Allow: /*.js$ # ----------Allow AI Engine crawlers and explicitly----------- User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Cohere-ai Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: /