A simple and efficient web crawler written in Go. This is designed for crawling web pages and following links to deepen exploration(BFS approch).
- Multi-threaded crawling for efficiency
- Bloom Filter for Duplicates URL
- Customizable depth and URL filtering
- Graceful handling of robots.txt
- Parsing HTML and extraction of links
- Added comments for easy work flow