**LLM ahead** I used this project to test working with an LLM so most of the code here is generated by one.
Kagi is a good product, but in my opinion, the company behind it has a few issues that make it impossible for me to use. However, I have identified two main features that I find most useful and that I can replicate in my own project.
The idea behind this feature is, that you can customize the results to match your liking and needs. You don’t like a certain site? Block it. You think Wikipedia is the best site ever? Give it more weight and rank it up.
When I started with this project I tried to write my own meta search engine and aggregate results from various engines. While this would be possible it is a pain to do. One has to stay up to date with the search engines because most of them don’t provide access that you can afford if they even provide access at all. So in the end you’re left scraping their sites for results.
I had another look at SearXNG and found that it was good enough for my use case so I decided to combine it with this project. The next step might be to integrate this project directly into SearXNG as a plugin.
Bangs are, as far as I know, a feature invented by DuckDuckGo.
They allow you to quickly forward a query to a specific site by prefixing the query with a designated string.
For example, !g foo will search for foo on Google.
This can be very powerful, especially on mobile devices, where it serves as a convenient shortcut. Having a site that supports this feature makes it particularly convenient, as you don’t have to configure each device to your preferences.
As not every user has the same preferences when it comes to search engines, the users can configure their default search engine. If the application finds a matching bang it will act on the bang. Else it will forward the query to the configured default search engine. In my case this will the an SearXNG instance.
To set up the development environment, you need:
- Nix (required) - Used for reproducible development environments
- direnv (recommended) - Automatically loads the development environment when entering the project directory
- Install Nix if you haven’t already: https://nixos.org/download.html
- Install direnv (optional but recommended): https://direnv.net/docs/installation.html
- Clone the repository and enter the directory:
git clone <repository-url> cd meta-search
- If using direnv, allow the environment:
direnv allow - If not using direnv, manually enter the Nix shell:
nix develop
Once in the development environment, you can run tests using pytest:
# Run all tests
pytest
# Run specific test file
pytest tests/test_meta_search.py
# Run tests in verbose mode
pytest -v- All the Python code has to be formatted with
black,docformatter,isort. The easiest way to do is, is via runningdev fmt. - For linting the following tools are used and configured in
pyproject.toml:- mypy
- pylint
- ruff
The easiest way is to get the docker-compose.yml from this repository and adjust the image: options to make sure it pulls the pre-built images from GitHub.
Afterwards use podman compose up to run it.
The site will then be accessible under: http://localhost:8000
The default admin user is: admin
The password is: password
In order to have a production ready system you need to tweak a few configurations.
ZWEILI_SEARCH_DOMAIN- The domain under which you search site should be accessible. E.g.
foo.example.org ZWEILI_SEARCH_DB_DIR- The path where you want to store the SQLite DB. By default it is
/var/lib/zweili_search
You can enter a search query right away and it will start searching on all the configured search engines. At the moment it is only DuckDuckGo.
When you go to https://your.domain.com/admin you can configure domains you want to block or setup custom bangs. They are always tied to a specific user so make sure you select the correct one otherwise they will not work or another will get your settings by accident. For the moment user specific settings aren’t planed from my side as I will only allow trusted users on my instance.