-
Notifications
You must be signed in to change notification settings - Fork 518
Description
This is where you provide the details of the bug. Use Markdown for formatting to make it easy to read.
Description
The BDJobs scraper is throwing a TypeError when scrape_jobs is called with the user_agent parameter. This indicates that the scraper's initialization method does not recognize this argument, even though user_agent is a valid parameter for the main scrape_jobs function.
Steps to Reproduce
Provide a minimal code snippet that reliably reproduces the error.
Python
from jobspy import scrape_jobs
jobs = scrape_jobs(
site_name=["bdjobs"],
search_term="software engineer",
location="Dhaka",
results_wanted=1,
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
)
Expected behavior
The scraper should run without a TypeError, either by handling the user_agent parameter correctly or by ignoring it if it's not a supported feature for that specific scraper.
Actual behavior
The script fails with the following traceback:
Traceback (most recent call last):
File "/path/to/your/script.py", line 4, in
jobs = scrape_jobs(
^^^^^^^^^^^^
File "/path/to/jobspy/init.py", line 116, in worker
site_val, scraped_info = scrape_site(site)
^^^^^^^^^^^^^^^^^
File "/path/to/jobspy/init.py", line 106, in scrape_site
scraper = scraper_class(proxies=proxies, ca_cert=ca_cert, user_agent=user_agent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: BDJobs.init() got an unexpected keyword argument 'user_agent'
The error seems to be isolated to the BDJobs scraper. The ZipRecruiter scraper also failed for unrelated reasons (rate limiting, which is expected).