Skip to content

TypeError: BDJobs scraper fails with unexpected keyword argument 'user_agent' #295

@krishianjan

Description

@krishianjan

This is where you provide the details of the bug. Use Markdown for formatting to make it easy to read.

Description
The BDJobs scraper is throwing a TypeError when scrape_jobs is called with the user_agent parameter. This indicates that the scraper's initialization method does not recognize this argument, even though user_agent is a valid parameter for the main scrape_jobs function.

Steps to Reproduce
Provide a minimal code snippet that reliably reproduces the error.

Python
from jobspy import scrape_jobs

jobs = scrape_jobs(
site_name=["bdjobs"],
search_term="software engineer",
location="Dhaka",
results_wanted=1,
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
)
Expected behavior
The scraper should run without a TypeError, either by handling the user_agent parameter correctly or by ignoring it if it's not a supported feature for that specific scraper.

Actual behavior
The script fails with the following traceback:

Traceback (most recent call last):
File "/path/to/your/script.py", line 4, in
jobs = scrape_jobs(
^^^^^^^^^^^^
File "/path/to/jobspy/init.py", line 116, in worker
site_val, scraped_info = scrape_site(site)
^^^^^^^^^^^^^^^^^
File "/path/to/jobspy/init.py", line 106, in scrape_site
scraper = scraper_class(proxies=proxies, ca_cert=ca_cert, user_agent=user_agent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: BDJobs.init() got an unexpected keyword argument 'user_agent'

The error seems to be isolated to the BDJobs scraper. The ZipRecruiter scraper also failed for unrelated reasons (rate limiting, which is expected).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions