Crawl and monitor +1000 websites
https://stackoverflow.com/questions/19618735/crawl...
И вот что-то подобное, судя по описанию - The Crawler dashboard view shows you a much more in depth view of your current Scrapy Crawlers. Here you see breakdowns of response codes, machine load balances, and successful outbound Kafka messages.
scrapy-cluster.readthedocs.io/en/latest/topics/adv... 