tispoint

Скажите, а глядя в лог
2016-12-18 13:12:02 [scrapy] INFO: Spider opened
2016-12-18 13:12:02 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 i
tems (at 0 items/min)
2016-12-18 13:12:02 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-12-18 13:12:03 [scrapy] DEBUG: Redirecting (302) to from
2016-12-18 13:12:05 [scrapy] DEBUG: Crawled (200) (referer: None)
https://ecom.elko.ru/Catalog/Category/SCO
2016-12-18 13:12:05 [scrapy] INFO: Closing spider (finished)
Это ведь последовательные операции одного экземпляра робота (не знаю как назвать точнее)?
То есть, я правильно думаю, что на странице https://ecom.elko.ru/Catalog/Category/SCO спайдер не находит ссылок и заканчивает работу?

Где у меня ошибка?

import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.item import Item, Field
from scrapy.contrib.loader import XPathItemLoader, ItemLoader
from scrapy.http import Request, FormRequest
from scrapy.contrib.loader.processor import TakeFirst

class ScrapyTestItem(scrapy.Item):
    title = Field()
    url = Field()

class Test03Loader(XPathItemLoader):
    default_output_processor = TakeFirst()

class ScrapyTestSpider(CrawlSpider):
    name = "catalog"

    rules = (
        Rule(LinkExtractor(
            allow=('https://ecom.elko.ru/Catalog/Product/')),
             callback='parse_item', follow=False),
        Rule(LinkExtractor(
            allow=('https://ecom.elko.ru/Catalog/Category/')),
             follow=True),
    )

    def start_requests(self):
        return [
            FormRequest(
                "https://ecom.elko.ru/Account/Login?ReturnUrl=%2fCatalog%2fCategory%2fSCO",
                formdata={"Username": "tiscom6", "Password": "6307860"}
            )]

    def parse(self, response):
          print(response.url)

    def parse_item(self, response):
        hxs = HtmlXPathSelector(response)
        l = Test03Loader(ScrapyTestItem(), hxs)
        l.add_xpath('title', "//h1/text()")
        l.add_value('url', response.url)
        return l.load_item()

Вот, кстати конфиг nginx

user  apache;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  2048;
    use epoll;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;


proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

client_max_body_size 100m;

gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_min_length 10;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon application/vnd.ms-fontobject font/opentype application/x-font-ttf;
gzip_vary on;
gzip_proxied any; 
gzip_disable msie6;


    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/vhosts/*/*.conf;
    server {
        server_name localhost;
 disable_symlinks if_not_owner;
        listen 80;
	include /etc/nginx/vhosts-includes/*.conf;
	location @fallback {
		error_log /dev/null crit;
		proxy_pass http://127.0.0.1:8080;
		proxy_redirect http://127.0.0.1:8080 /;
		proxy_set_header Host $host;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		access_log off ;
	}
    }
}

В логе, нам мой взгляд, все нормально:

2016-04-19 17:12:36 [scrapy] INFO: Scrapy 1.0.1 started (bot: intelxeon)
2016-04-19 17:12:36 [scrapy] INFO: Optional features available: ssl, http11
2016-04-19 17:12:36 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'intelxeon.spiders', 'FEED_URI': 'xeon.csv', 'SPIDER_MODULES': ['intelxeon.spiders'], 'BOT_NAME': 'intelxeon', 'LOG_STDOUT': True, 'FEED_FORMAT': 'csv', 'LOG_FILE': 'C:/log.txt'}
2016-04-19 17:12:36 [scrapy] INFO: Enabled extensions: CloseSpider, FeedExporter, TelnetConsole, LogStats, CoreStats, SpiderState
2016-04-19 17:12:37 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-04-19 17:12:37 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-04-19 17:12:37 [scrapy] INFO: Enabled item pipelines: 
2016-04-19 17:12:37 [scrapy] INFO: Spider opened
2016-04-19 17:12:37 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-04-19 17:12:37 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-04-19 17:12:38 [scrapy] DEBUG: Crawled (200) <GET http://ark.intel.com/products/family/78581> (referer: None)
2016-04-19 17:12:39 [scrapy] DEBUG: Crawled (200) <GET http://ark.intel.com/products/family/78581> (referer: http://ark.intel.com/products/family/78581)
...
2016-04-19 17:12:40 [scrapy] INFO: Closing spider (finished)
2016-04-19 17:12:40 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 16136,
 'downloader/request_count': 51,
 'downloader/request_method_count/GET': 51,
 'downloader/response_bytes': 697437,
 'downloader/response_count': 51,
 'downloader/response_status_count/200': 51,
 'dupefilter/filtered': 8,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 4, 19, 13, 12, 40, 99000),
 'log_count/DEBUG': 53,
 'log_count/INFO': 7,
 'request_depth_max': 1,
 'response_received_count': 51,
 'scheduler/dequeued': 51,
 'scheduler/dequeued/memory': 51,
 'scheduler/enqueued': 51,
 'scheduler/enqueued/memory': 51,

Если в командной строке указать для вывода -o c:\xeon.csv , то появляется ошибка такого плана:
2016-04-19 17:14:33 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'intelxeon.spiders', 'FEED_URI': 'c:\xeon.csv', 'SPIDER_MODULES': ['intelxeon.spiders'], 'BOT_NAME': 'intelxeon', 'LOG_STDOUT': True, 'FEED_FORMAT': 'csv', 'LOG_FILE': 'C:/log.txt'}
2016-04-19 17:14:33 [scrapy] ERROR: Unknown feed storage scheme: c

Насчет полей - я все убрал для чистоты эксперимента, оставил только
item['url'] = response.request.url
это ведь по-любому должно выводиться?

HTML я неправильно указал

<div class="char c1 ">
<div class="characteristic">Для процессоров </div>
<div class="value"> Intel  </div>
<div class="clear"/>
</div>
<div class="char c0 ">
<div class="characteristic">Серия процессора </div>
<div class="value"> Xeon  </div>
<div class="clear"/>
</div>

Это таблица. Она довольно длинная.
Надо чтобы искало по столбцу characteristic и возвращало значение из столбца value