# This file implements the "robot exclusion protocol". It tells robots # such as web crawlers and other search engines not to proceed below # the root of this database. # See # http://info.webcrawler.com/mak/projects/robots/norobots.html # for a description of the robot exclusion protocol and more information # on how this file works. # ---------------------------------------------------------------------- # Configuration Options # ---------------------------------------------------------------------- # # This disallows indexing on the entire site # User-agent: * # Disallow: / # # This allows indexing of the entire site # User-agent: * # Disallow: # # This excludes *only* certain directories on the site # NOTE: Only those directories listed are specifically excluded, other # directories above the excluded one is accessable and will be # indexed by the robot. IE: /global/sitename will be indexed # User-agent: * # Disallow: /global/ # Disallow: /globaltest/ # Disallow: /jt/ # Disallow: /bsc/ # Disallow: /q1/ # ---------------------------------------------------------------------- User-agent: * Disallow: