Specify "critical data" for the incremental build

[AdrianGuery]

Hello !

I'm currently working on a big Gatsby site (10,000+ pages) and we're trying to optimize build times. Because currently we take more than 5 minutes to build it locally (on high-performance machines).
The incremental build works well, it allows us to optimize time of the build of HTML pages. We managed to get off in less than 4 minutes.

But that doesn't optimize "pages queries" at all, and it is this part that takes us the most time (160 seconds). This seems logical (if I understood correctly how the incremental build works) because the pages queries are run to see if their result change compared to the cache and then see if Gatsby is using the cache or needs to regenerate the page.

Would there be a way to tell to Gatsby, even before he does the pages queries, whether he should use the cache or not ?
For example, by specifying that if the "last_mod" data of the page (which is got during the Bootstrap phase) has changed so Gatsby must do the queries and regenerate the pages, otherwise Gatsby takes the cache and don't run pages queries .

Maybe you have another idea for the optimization?

Thanks a lot

[KyleAMathews]

Page queries are incremental as well. You can see that with e.g. the blog starter where you do a build, make a change to a markdown file, and rebuild and see that only the corresponding page query was run.

What you're seeing I'm guessing is that your pages query "connections" (i.e. allSomeType) — when this happens, Gatsby has to invalidate the page when any nodes of SomeType has changed. Another common reason for why many/most pages are invalidated is if you have schema customization and use findAll as this also creates a connection dependency.

Ways around this include:

• if you can isolate the connection queries to static queries that each page imports then this will mean only the static query gets re-run each time
• if you're not on v4 yet, queries are know run in parallel on multiple cores which speeds them up considerably.

[AdrianGuery]

Thank you for your very clear answer!
Indeed, we do a lot of allSomeType in query pages. And suddenly, even if the node which changes are not query thanks to the filters, it will still rerun the query page?

We cannot isolate them in static queries, precisely because filters are variables.
We will therefore try to enable the parallel queries.

[KyleAMathews]

We could possibly leverage filters to avoid invalidating as many pages but that research hasn't happened yet.

If you're on v3 still, you can also enable PQR via its flag #32389

[AdrianGuery]

It can be incredible !
Thanks