Allow setting a language's query path in the configuration

[TLATER]

Currently, to configure a new language, both TOPIARY_LANGUAGE_DIR and a block in the configuration file need to be added. This is a bit confusing, and not quite as flexible.

It would be nice if the language section of the configuration permitted a query key, e.g.:

nickel = {
  extensions = ["ncl"],
  grammar.source.path = "/path/to/compiled/grammar/file.so",
  query.path = "/path/to/query.ncl",
},

Original issue contents

**Is your feature request related to a problem? Please describe.**

I'd really like to be able to format languages without having to mess with creating separate repositories, just write a .scm file, compile it, add it to my topiary configuration, done. It'd make testing and working with formatters smoother, even if I eventually want to publish the formatter.

Describe the solution you'd like

Ideally a topiary compile command that just takes a .scm file and outputs the .so file I need to hand to my configuration.

Describe alternatives you've considered

Not using topiary, and instead just writing the parsing logic separately. This is tedious.

[Xophmeister]

Thanks for this, @TLATER

I'm not quite sure I follow: the .scm file defines the Tree-sitter queries that Topiary uses to format source code, but these are not compiled. The Tree-sitter grammar definitions are compiled, necessarily so, which is what topiary prefetch does (or topiary format does, on the fly, if the grammar isn't already cached). Topiary requires both the .scm file and the grammar to format a language.

When Topiary reads the .scm file, it does parse it into an internal representation that is kept in memory. This parsing does consume the majority of the Topiary start-up time. Is the serialisation of this parsed query file what you are referring to? (This wouldn't be an .so, though.)

[TLATER]

Ah, sorry, in that case I misunderstood the documentation and didn't read the code closely enough, sorry :)

So, if I understand this correctly, in the configuration I can specify a tree-sitter grammar like this:

nickel = {
  extensions = ["ncl"],
  grammar.source.path = "/path/to/compiled/grammar/file.so",
},

But this only provides the tree-sitter grammar. Where do the queries come from?

Re-reading other docs, is it the --query arg of the topiary format command, and it's hard-coded for languages supported upstream?

[TLATER]

Got it. So, given a configured grammar like the above, an associated <extension>.scm file must be present in the TOPIARY_LANGUAGE_DIR.

What I want is certainly possible, albeit not very obvious from the docs. Sorry about the confusion! I'll try to see if I can help at least improve the documentation here.

It'd still be nice, and a bit more intuitive, if the configuration file could specify the path to the .scm file as part of the language configuration.

[Xophmeister]

Exactly, you can either supply a precompiled path -- like in your example -- or you can provide a Git reference to a Tree-sitter grammar repo. In the latter case, Topiary will fetch the repo and build the .so for you. This is described in more detail here: https://topiary.tweag.io/book/cli/configuration.html#specifying-the-grammar

The query is taken from the query directory and matches the name of the configuration key. In your example, you've used nickel, so Topiary will look for nickel.scm in the query directory. (For the built-in language support, these are also baked directly into the binary, so this lookup isn't necessary. However, as you point out, the --query argument can be used to override this.)

[yannham]

There's one suggestion of @TLATER that hasn't really been addressed, and I do think it's a good one: making it possible to specify a query file directly in the configuration. It's true that the user experience is a bit inconsistent on this front: we can specify most of the configuration needed to format a language in the Nickel configuration file (mostly where is the grammar file), but we must not forget to set $TOPIARY_LANGUAGE_DIR otherwise so that it can find the query file (which must also be named in a certain way). We could wonder why we use this mechanism for queries, and a different one for grammars (hardcoded path in the configuration instead of a $TOPIARY_GRAMMAR_DIR).

In my (limited) experience writing queries for a custom language, you have one working query file, and it's annoying to have to set up the environment variable each time you log in and out (I forgot everytime). Environment variables are always a good knob to have, if only for usages like invoking topiary for a script, or if you have a dozen custom query files and you don't want to bother repeating the same path again and again, but it could still be interesting to have the option to overwrite it locally on a per-language basis through configuration:

nickel = {
  extensions = ["ncl"],
  grammar.source.path = "/path/to/compiled/grammar/file.so",
  queries.path = "/path/to/query/file.scm",
},

[Xophmeister]

Oh yes, I complete agree: Exposing the query path in the configuration is a great idea, which is why I didn't close this issue 👍

[TLATER]

Now it should be a bit clearer at a glance what this issue is ultimately about :)