If files are ignored via an exception, remove them during the discovery/find_files process

[xydesa]

https://github.com/triboelectric/statick/blob/main/src/statick_tool/discovery_plugin.py#L41

This will remove them super early on and hopefully prevent processing lots of unwanted files.

[kogut]

In addition to ignoring, either this issue or a follow-on could also modify the workflows so that workflow artifacts don't end up in the repo tree unless necessary - for example mypy caches or tox venvs.

[axydes]

I tried moving the filter_file_exceptions_early call into the discovery_plugin.find_files function (which is only called once), but the timings of all discovery plugins (and that specific function) all increased after this change.

we already call that function in the scan function of most discovery plugins. That might be as good as we can get.

[tdenewiler]

One thing I tried in a forked branch (that I threw away) was to switch our os.walk() call in find_files() with os.scandir(). You can call os.scandir() recursively, since it returns the files and directories in the current directory it is looking at. See https://stackoverflow.com/questions/33135038/how-do-i-use-os-scandir-to-return-direntry-objects-recursively-on-a-directory for an example. The benefit I was hoping for was to pass a list of exceptions into find_files(). then along with each .is_dir(entry) call I added a check to make sure the entry was not in our list of exceptions.

The result I got was the scandir approach was 2-4 times slower than our current implementation. That's why I deleted my branch.