2026-05-11 15:01:00 [scrapy.utils.log] INFO: Scrapy 2.12.0 started (bot: competitor_spider)
2026-05-11 15:01:00 [scrapy.utils.log] INFO: Versions: lxml 6.1.0.0, libxml2 2.14.6, cssselect 1.4.0, parsel 1.11.0, w3lib 2.4.1, Twisted 25.5.0, Python 3.11.15 (main, Apr  7 2026, 02:24:41) [GCC 14.2.0], pyOpenSSL 26.0.0 (OpenSSL 3.5.6 7 Apr 2026), cryptography 46.0.7, Platform Linux-6.12.30+-x86_64-with-glibc2.41
2026-05-11 15:01:00 [scrapy.addons] INFO: Enabled addons:
[]
2026-05-11 15:01:00 [scrapy.extensions.telnet] INFO: Telnet Password: 3a711226543960cb
2026-05-11 15:01:00 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2026-05-11 15:01:00 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'competitor_spider',
 'CLOSESPIDER_PAGECOUNT': 300,
 'CONCURRENT_REQUESTS': 2,
 'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
 'DEPTH_LIMIT': 5,
 'DOWNLOAD_DELAY': 2,
 'DOWNLOAD_TIMEOUT': 60,
 'LOG_FILE': '/var/lib/scrapyd/logs/competitors/competitor/159de4d84d4a11f1b1460242ac1f0002.log',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'competitor_spider.spiders',
 'RETRY_HTTP_CODES': [500, 502, 503, 504, 408, 429],
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['competitor_spider.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'ChalknPencilsBot/1.0 (+https://chalknpencils.com)'}
2026-05-11 15:01:00 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2026-05-11 15:01:00 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2026-05-11 15:01:00 [scrapy.middleware] INFO: Enabled item pipelines:
['competitor_spider.pipelines.WebhookPipeline']
2026-05-11 15:01:00 [scrapy.core.engine] INFO: Spider opened
2026-05-11 15:01:00 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-05-11 15:01:00 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6033
2026-05-11 15:01:00 [scrapy-playwright] INFO: Starting download handler
2026-05-11 15:01:00 [scrapy-playwright] INFO: Starting download handler
2026-05-11 15:01:01 [scrapy-playwright] INFO: Launching 1 startup context(s)
2026-05-11 15:01:01 [scrapy-playwright] INFO: Launching browser chromium
2026-05-11 15:01:01 [scrapy-playwright] INFO: Launching 1 startup context(s)
2026-05-11 15:01:01 [scrapy-playwright] INFO: Launching browser chromium
2026-05-11 15:01:01 [scrapy-playwright] INFO: Browser chromium launched
2026-05-11 15:01:01 [scrapy-playwright] INFO: Browser chromium launched
2026-05-11 15:01:01 [scrapy-playwright] INFO: Startup context(s) launched
2026-05-11 15:01:01 [scrapy-playwright] INFO: Startup context(s) launched
2026-05-11 15:01:05 [competitor] INFO: Starting crawl: Chalk N Pencils @ https://www.chalknpencils.com/
2026-05-11 15:02:00 [scrapy.extensions.logstats] INFO: Crawled 12 pages (at 12 pages/min), scraped 11 items (at 11 items/min)
2026-05-11 15:02:42 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.chalknpencils.com/corporate-team-building>: HTTP status code is not handled or not allowed
2026-05-11 15:02:46 [competitor_spider.pipelines] INFO: Webhook POST 20 items -> 200
2026-05-11 15:03:00 [scrapy.extensions.logstats] INFO: Crawled 24 pages (at 12 pages/min), scraped 22 items (at 11 items/min)
2026-05-11 15:03:11 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.chalknpencils.com/children-art/portfolio>: HTTP status code is not handled or not allowed
2026-05-11 15:03:25 [scrapy.core.engine] INFO: Closing spider (finished)
2026-05-11 15:03:25 [competitor_spider.pipelines] INFO: Webhook POST 4 items -> 200
2026-05-11 15:03:25 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (24 items) in: file:///var/lib/scrapyd/items/competitors/competitor/159de4d84d4a11f1b1460242ac1f0002.jl
2026-05-11 15:03:25 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 11957,
 'downloader/request_count': 28,
 'downloader/request_method_count/GET': 28,
 'downloader/response_bytes': 3007821,
 'downloader/response_count': 28,
 'downloader/response_status_count/200': 26,
 'downloader/response_status_count/404': 2,
 'dupefilter/filtered': 1415,
 'elapsed_time_seconds': 145.17926,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2026, 5, 11, 15, 3, 25, 788845, tzinfo=datetime.timezone.utc),
 'httpcompression/response_bytes': 50,
 'httpcompression/response_count': 1,
 'httperror/response_ignored_count': 2,
 'httperror/response_ignored_status_count/404': 2,
 'item_scraped_count': 24,
 'items_per_minute': None,
 'log_count/INFO': 28,
 'memusage/max': 128098304,
 'memusage/startup': 79544320,
 'playwright/context_count': 2,
 'playwright/context_count/max_concurrent': 1,
 'playwright/context_count/persistent/False': 2,
 'playwright/context_count/remote/False': 2,
 'playwright/page_count': 27,
 'playwright/page_count/closed': 27,
 'playwright/page_count/max_concurrent': 1,
 'playwright/request_count': 2752,
 'playwright/request_count/method/GET': 2709,
 'playwright/request_count/method/POST': 43,
 'playwright/request_count/navigation': 77,
 'playwright/request_count/resource_type/document': 77,
 'playwright/request_count/resource_type/font': 136,
 'playwright/request_count/resource_type/image': 585,
 'playwright/request_count/resource_type/script': 1646,
 'playwright/request_count/resource_type/stylesheet': 261,
 'playwright/request_count/resource_type/xhr': 47,
 'playwright/response_count': 2741,
 'playwright/response_count/method/GET': 2707,
 'playwright/response_count/method/POST': 34,
 'playwright/response_count/resource_type/document': 77,
 'playwright/response_count/resource_type/font': 136,
 'playwright/response_count/resource_type/image': 583,
 'playwright/response_count/resource_type/script': 1646,
 'playwright/response_count/resource_type/stylesheet': 261,
 'playwright/response_count/resource_type/xhr': 38,
 'request_depth_max': 3,
 'response_received_count': 28,
 'responses_per_minute': None,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 27,
 'scheduler/dequeued/memory': 27,
 'scheduler/enqueued': 27,
 'scheduler/enqueued/memory': 27,
 'start_time': datetime.datetime(2026, 5, 11, 15, 1, 0, 609585, tzinfo=datetime.timezone.utc)}
2026-05-11 15:03:25 [scrapy.core.engine] INFO: Spider closed (finished)
2026-05-11 15:03:25 [scrapy-playwright] INFO: Closing download handler
2026-05-11 15:03:25 [scrapy-playwright] INFO: Closing browser
2026-05-11 15:03:25 [scrapy-playwright] INFO: Closing download handler
2026-05-11 15:03:25 [scrapy-playwright] INFO: Closing browser