/
githubmirror
/
scrapy
Обзор
Документация
Войти
/
githubmirror
/
scrapy
Код
Запросы
0
Пакеты
0
Релизы
0
Аналитика
Безопасность
master
tests/AsyncCrawlerProcess/caching_hostname_resolver.py
36 строк
941 B
Andrey Rakhmatullin
More granular untyped defs checking in tests. (#7712)
08 июл 2026, 11:05
Не верифицирован
08 июл 2026, 11:05
61f99f2
Код
Авторство
О чём код?
import sys import scrapy from scrapy.crawler import AsyncCrawlerProcess class CachingHostnameResolverSpider(scrapy.Spider): """ Finishes in a finite amount of time (does not hang indefinitely in the DNS resolution) """ name = "caching_hostname_resolver_spider" url: str async def start(self): yield scrapy.Request(self.url) def parse(self, response): for _ in range(10): yield scrapy.Request( response.url, dont_filter=True, callback=self.ignore_response ) def ignore_response(self, response): self.logger.info(repr(response.ip_address)) if __name__ == "__main__": process = AsyncCrawlerProcess( settings={ "RETRY_ENABLED": False, "TWISTED_DNS_RESOLVER": "scrapy.resolver.CachingHostnameResolver", } ) process.crawl(CachingHostnameResolverSpider, url=sys.argv[1]) process.start()