2026-05-07 16:00:05 [scrapy.utils.log] INFO: Scrapy 2.12.0 started (bot: competitor_spider)
2026-05-07 16:00:05 [scrapy.utils.log] INFO: Versions: lxml 6.1.0.0, libxml2 2.14.6, cssselect 1.4.0, parsel 1.11.0, w3lib 2.4.1, Twisted 25.5.0, Python 3.11.15 (main, Apr  7 2026, 02:24:41) [GCC 14.2.0], pyOpenSSL 26.0.0 (OpenSSL 3.5.6 7 Apr 2026), cryptography 46.0.7, Platform Linux-6.12.30+-x86_64-with-glibc2.41
2026-05-07 16:00:05 [competitor_spider.spiders.gebiz] INFO: GeBIZ spider starting in mode=open
2026-05-07 16:00:05 [scrapy.addons] INFO: Enabled addons:
[]
2026-05-07 16:00:05 [scrapy.extensions.telnet] INFO: Telnet Password: d2ee46a41eb82fb1
2026-05-07 16:00:05 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2026-05-07 16:00:05 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'competitor_spider',
 'CLOSESPIDER_PAGECOUNT': 500,
 'CONCURRENT_REQUESTS': 1,
 'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
 'DEPTH_LIMIT': 3,
 'DOWNLOAD_DELAY': 3,
 'DOWNLOAD_TIMEOUT': 60,
 'LOG_FILE': '/var/lib/scrapyd/logs/competitors/gebiz/cd2224684a2d11f1b1460242ac1f0002.log',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'competitor_spider.spiders',
 'RETRY_HTTP_CODES': [500, 502, 503, 504, 408, 429],
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['competitor_spider.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'ChalknPencilsBot/1.0 (+https://chalknpencils.com)'}
2026-05-07 16:00:05 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2026-05-07 16:00:05 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2026-05-07 16:00:05 [scrapy.middleware] INFO: Enabled item pipelines:
['competitor_spider.pipelines.WebhookPipeline']
2026-05-07 16:00:05 [scrapy.core.engine] INFO: Spider opened
2026-05-07 16:00:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-05-07 16:00:05 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2026-05-07 16:00:05 [scrapy-playwright] INFO: Starting download handler
2026-05-07 16:00:05 [scrapy-playwright] INFO: Starting download handler
2026-05-07 16:00:05 [scrapy-playwright] INFO: Launching 1 startup context(s)
2026-05-07 16:00:05 [scrapy-playwright] INFO: Launching browser chromium
2026-05-07 16:00:05 [scrapy-playwright] INFO: Launching 1 startup context(s)
2026-05-07 16:00:05 [scrapy-playwright] INFO: Launching browser chromium
2026-05-07 16:00:05 [scrapy-playwright] INFO: Browser chromium launched
2026-05-07 16:00:05 [scrapy-playwright] INFO: Browser chromium launched
2026-05-07 16:00:05 [scrapy-playwright] INFO: Startup context(s) launched
2026-05-07 16:00:05 [scrapy-playwright] INFO: Startup context(s) launched
2026-05-07 16:00:18 [competitor_spider.spiders.gebiz] INFO: GeBIZ page loaded. Title: GeBIZ | Opportunities
2026-05-07 16:00:18 [competitor_spider.spiders.gebiz] INFO: --- Searching for: 'art instructor' ---
2026-05-07 16:00:24 [competitor_spider.spiders.gebiz] INFO: Submitted via mojarra.ab(contentForm:j_idt183_searchBar_BUTTON-GO, render=contentForm:j_idt183)
2026-05-07 16:00:31 [competitor_spider.spiders.gebiz] INFO: Debug screenshot saved to /var/lib/scrapyd/gebiz_debug.png
2026-05-07 16:00:31 [competitor_spider.spiders.gebiz] INFO: After search — Title: GeBIZ | Opportunities, URL: https://www.gebiz.gov.sg/ptn/opportunity/BOListing.xhtml
2026-05-07 16:00:31 [competitor_spider.spiders.gebiz] INFO: 'art instructor': 68 opportunities found
2026-05-07 16:00:31 [competitor_spider.spiders.gebiz] INFO: Found 1 result card headers
2026-05-07 16:00:31 [competitor_spider.spiders.gebiz] INFO:   → MOESCHETQ26001810 | OPEN | Ministry of Education - Schools | Supply of Instructors and Robotics Resources for Teacher Professional 
2026-05-07 16:00:31 [competitor_spider.spiders.gebiz] INFO: 'art instructor': 0 detail pages to follow, 1 total QNs seen
2026-05-07 16:00:31 [competitor_spider.spiders.gebiz] INFO: --- Searching for: 'AMIS' ---
2026-05-07 16:00:37 [competitor_spider.spiders.gebiz] INFO: Submitted via mojarra.ab(contentForm:j_idt183_searchBar_BUTTON-GO, render=contentForm:j_idt183)
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO: After search — Title: GeBIZ | Opportunities, URL: https://www.gebiz.gov.sg/ptn/opportunity/BOListing.xhtml
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO: 'AMIS': 127 opportunities found
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO: Found 7 result card headers
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO:   → MOESCHETQ26001884 | OPEN | Ministry of Education - Schools | Supply of AMIS Intructor for P5 Ceramic Arts Programme
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO:   → MOESCHETQ26001865 | OPEN | Ministry of Education - Schools | Supply of Instructor for P3 Orff & P4 Ukulele AMIS Program in Ahmad Ib
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO:   → CDVHQ0ETT26000010 | OPEN | Ministry of Social and Development - Ministry Headquarter | Invitation to Tender for the Provision of Family Connection Sessions (
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO:   → CDVHQ0ETT26000009 | OPEN | Ministry of Social and Development - Ministry Headquarter | Invitation to Tender for Addition and Alteration (A&A) works at THK Ho
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO:   → MOESCHETQ26001872 | OPEN | Ministry of Education - Schools | Supply and Delivery of Foldable Desks for School Examination Purposes
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO:   → CDA000ETT26000005 | OPEN | Communicable Diseases Agency | (CDA-LOQ-ITT-2025-451) Invitation to Tender for the Provision of Gaseo
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO: 'AMIS': 0 detail pages to follow, 8 total QNs seen
2026-05-07 16:00:44 [competitor_spider.spiders.gebiz] INFO: --- Searching for: 'art club' ---
2026-05-07 16:00:49 [competitor_spider.spiders.gebiz] INFO: Submitted via mojarra.ab(contentForm:j_idt183_searchBar_BUTTON-GO, render=contentForm:j_idt183)
2026-05-07 16:00:56 [competitor_spider.spiders.gebiz] INFO: After search — Title: GeBIZ | Opportunities, URL: https://www.gebiz.gov.sg/ptn/opportunity/BOListing.xhtml
2026-05-07 16:00:56 [competitor_spider.spiders.gebiz] INFO: 'art club': 16 opportunities found
2026-05-07 16:00:56 [competitor_spider.spiders.gebiz] INFO: Found 10 result card headers
2026-05-07 16:00:56 [competitor_spider.spiders.gebiz] INFO: 'art club': 0 detail pages to follow, 18 total QNs seen
2026-05-07 16:00:58 [competitor_spider.spiders.gebiz] INFO: Found 6 result card headers
2026-05-07 16:00:58 [competitor_spider.spiders.gebiz] INFO: 'art club' page 2: +6 new QNs (total 24)
2026-05-07 16:00:58 [competitor_spider.spiders.gebiz] INFO: --- Searching for: 'CCA art' ---
2026-05-07 16:01:03 [competitor_spider.spiders.gebiz] INFO: Submitted via mojarra.ab(contentForm:j_idt183_searchBar_BUTTON-GO, render=contentForm:j_idt183)
2026-05-07 16:01:05 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 2 pages/min), scraped 7 items (at 7 items/min)
2026-05-07 16:01:10 [competitor_spider.spiders.gebiz] INFO: After search — Title: GeBIZ | Opportunities, URL: https://www.gebiz.gov.sg/ptn/opportunity/BOListing.xhtml
2026-05-07 16:01:10 [competitor_spider.spiders.gebiz] INFO: 'CCA art': 21 opportunities found
2026-05-07 16:01:10 [competitor_spider.spiders.gebiz] INFO: Found 10 result card headers
2026-05-07 16:01:10 [competitor_spider.spiders.gebiz] INFO: 'CCA art': 0 detail pages to follow, 26 total QNs seen
2026-05-07 16:01:12 [competitor_spider.spiders.gebiz] INFO: Found 10 result card headers
2026-05-07 16:01:12 [competitor_spider.spiders.gebiz] INFO: 'CCA art' page 2: +7 new QNs (total 33)
2026-05-07 16:01:13 [competitor_spider.spiders.gebiz] INFO: Found 1 result card headers
2026-05-07 16:01:13 [competitor_spider.spiders.gebiz] INFO: 'CCA art' page 3: +1 new QNs (total 34)
2026-05-07 16:01:13 [competitor_spider.spiders.gebiz] INFO: --- Searching for: 'visual art' ---
2026-05-07 16:01:19 [competitor_spider.spiders.gebiz] INFO: Submitted via mojarra.ab(contentForm:j_idt183_searchBar_BUTTON-GO, render=contentForm:j_idt183)
2026-05-07 16:01:26 [competitor_spider.spiders.gebiz] INFO: After search — Title: GeBIZ | Opportunities, URL: https://www.gebiz.gov.sg/ptn/opportunity/BOListing.xhtml
2026-05-07 16:01:26 [competitor_spider.spiders.gebiz] INFO: 'visual art': 4 opportunities found
2026-05-07 16:01:26 [competitor_spider.spiders.gebiz] INFO: Found 4 result card headers
2026-05-07 16:01:26 [competitor_spider.spiders.gebiz] INFO: 'visual art': 0 detail pages to follow, 38 total QNs seen
2026-05-07 16:01:26 [competitor_spider.spiders.gebiz] INFO: --- Searching for: 'painting' ---
2026-05-07 16:01:31 [competitor_spider.spiders.gebiz] INFO: Submitted via mojarra.ab(contentForm:j_idt183_searchBar_BUTTON-GO, render=contentForm:j_idt183)
2026-05-07 16:01:39 [competitor_spider.spiders.gebiz] INFO: After search — Title: GeBIZ | Opportunities, URL: https://www.gebiz.gov.sg/ptn/opportunity/BOListing.xhtml
2026-05-07 16:01:39 [competitor_spider.spiders.gebiz] INFO: 'painting': 13 opportunities found
2026-05-07 16:01:39 [competitor_spider.spiders.gebiz] INFO: Found 2 result card headers
2026-05-07 16:01:39 [competitor_spider.spiders.gebiz] INFO: 'painting': 0 detail pages to follow, 40 total QNs seen
2026-05-07 16:01:39 [competitor_spider.spiders.gebiz] INFO: --- Searching for: 'drawing' ---
2026-05-07 16:01:44 [competitor_spider.spiders.gebiz] INFO: Submitted via mojarra.ab(contentForm:j_idt183_searchBar_BUTTON-GO, render=contentForm:j_idt183)
2026-05-07 16:01:51 [competitor_spider.spiders.gebiz] INFO: After search — Title: GeBIZ | Opportunities, URL: https://www.gebiz.gov.sg/ptn/opportunity/BOListing.xhtml
2026-05-07 16:01:51 [competitor_spider.spiders.gebiz] INFO: 'drawing': 3 opportunities found
2026-05-07 16:01:51 [competitor_spider.spiders.gebiz] INFO: Found 3 result card headers
2026-05-07 16:01:51 [competitor_spider.spiders.gebiz] INFO: 'drawing': 0 detail pages to follow, 41 total QNs seen
2026-05-07 16:01:51 [competitor_spider.spiders.gebiz] INFO: --- Searching for: 'manga' ---
2026-05-07 16:01:56 [competitor_spider.spiders.gebiz] INFO: Submitted via mojarra.ab(contentForm:j_idt183_searchBar_BUTTON-GO, render=contentForm:j_idt183)
2026-05-07 16:02:03 [competitor_spider.spiders.gebiz] INFO: After search — Title: GeBIZ | Opportunities, URL: https://www.gebiz.gov.sg/ptn/opportunity/BOListing.xhtml
2026-05-07 16:02:03 [competitor_spider.spiders.gebiz] WARNING: 'manga': no result count found on page. Title: GeBIZ | Opportunities
2026-05-07 16:02:03 [competitor_spider.spiders.gebiz] INFO: Debug — text inputs: 3, first 20 link texts: ['Trusted websites', 'Home', 'Opportunities', 'Supplier Directory', 'SIGN UP', 'WHICH TO USE?', 'Home', 'Opportunities', 'Supplier Directory', 'Log in for Companies', 'Log in for Individuals', 'Log in for Foreigners w/o Singpass', 'WHICH TO USE?', 'SIGN UP', "Today's Opportunities", 'Advanced Search', 'Match Any', '\n                ', '\n              ', '\n                ']
2026-05-07 16:02:03 [competitor_spider.spiders.gebiz] INFO: Found 0 result card headers
2026-05-07 16:02:03 [competitor_spider.spiders.gebiz] INFO: 'manga': 0 detail pages to follow, 41 total QNs seen
2026-05-07 16:02:04 [scrapy.core.engine] INFO: Closing spider (finished)
2026-05-07 16:02:04 [competitor_spider.pipelines] INFO: Webhook POST 7 items -> 200
2026-05-07 16:02:04 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (7 items) in: file:///var/lib/scrapyd/items/competitors/gebiz/cd2224684a2d11f1b1460242ac1f0002.jl
2026-05-07 16:02:04 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1011,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 67754,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'elapsed_time_seconds': 118.969452,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2026, 5, 7, 16, 2, 4, 150105, tzinfo=datetime.timezone.utc),
 'item_scraped_count': 7,
 'items_per_minute': None,
 'log_count/INFO': 86,
 'log_count/WARNING': 1,
 'memusage/max': 128561152,
 'memusage/startup': 78741504,
 'playwright/context_count': 2,
 'playwright/context_count/max_concurrent': 1,
 'playwright/context_count/persistent/False': 2,
 'playwright/context_count/remote/False': 2,
 'playwright/page_count': 1,
 'playwright/page_count/max_concurrent': 1,
 'playwright/request_count': 1180,
 'playwright/request_count/method/GET': 1144,
 'playwright/request_count/method/POST': 36,
 'playwright/request_count/navigation': 17,
 'playwright/request_count/resource_type/document': 17,
 'playwright/request_count/resource_type/font': 110,
 'playwright/request_count/resource_type/image': 355,
 'playwright/request_count/resource_type/script': 492,
 'playwright/request_count/resource_type/stylesheet': 170,
 'playwright/request_count/resource_type/xhr': 36,
 'playwright/response_count': 1180,
 'playwright/response_count/method/GET': 1144,
 'playwright/response_count/method/POST': 36,
 'playwright/response_count/resource_type/document': 17,
 'playwright/response_count/resource_type/font': 110,
 'playwright/response_count/resource_type/image': 355,
 'playwright/response_count/resource_type/script': 492,
 'playwright/response_count/resource_type/stylesheet': 170,
 'playwright/response_count/resource_type/xhr': 36,
 'response_received_count': 2,
 'responses_per_minute': None,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2026, 5, 7, 16, 0, 5, 180653, tzinfo=datetime.timezone.utc)}
2026-05-07 16:02:04 [scrapy.core.engine] INFO: Spider closed (finished)
2026-05-07 16:02:04 [scrapy-playwright] INFO: Closing download handler
2026-05-07 16:02:04 [scrapy-playwright] INFO: Closing browser
2026-05-07 16:02:04 [scrapy-playwright] INFO: Closing download handler
2026-05-07 16:02:04 [scrapy-playwright] INFO: Closing browser