Задать вопрос
Ответы пользователя по тегу robots.txt
  • Как проверить ошибки в файле Robot.txt?

    @RETAND
    Имеем такой robots.txt для wordpress, индексирует отлично:

    User-agent: *
    Allow: /wp-content/uploads
    Disallow: /wp-login.php
    Disallow: /wp-register.php
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /trackback
    Disallow: */trackback
    Disallow: */*/trackback
    Disallow: */*/feed
    Disallow: */feed
    Disallow: /*?*
    Disallow: /comment-page-*
    Disallow: /*/?replytocom=*
    Disallow: /tag
    Disallow: /category/
    Disallow: /author/
    Disallow: /20*/*
    Disallow: /page/*

    User-agent: Yandex
    Allow: /wp-content/uploads
    Disallow: /wp-login.php
    Disallow: /wp-register.php
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/
    Disallow: /trackback
    Disallow: */trackback
    Disallow: */*/trackback
    Disallow: */*/feed
    Disallow: /*?*
    Disallow: */feed
    Disallow: /comment-page-*
    Disallow: /*/?replytocom=*
    Disallow: /tag
    Disallow: /category/
    Disallow: /author/
    Disallow: /js/
    Disallow: /20*/*
    Disallow: /page/*
    Host: www.famreal.com

    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/

    User-agent: YandexImages
    Allow: /wp-content/uploads/

    Sitemap: www.famreal.com/sitemap.xml
    Ответ написан