User-agent: * Disallow: /comment.php Disallow: /docs Disallow: /pingback.php Disallow: /submit.php Disallow: /trackback.php User-agent: twiceler Disallow: / User-agent: * Disallow: Disallow: /cgi-bin/