Feature Request: Support an Environment Variable to Control Use of Local Cache Directory

[hcoona]

Hi team,

In discussion #534, we talked about using a local cache directory to prepare the environment, enabling the use of the pkl tool in air-gapped environments.

Currently, each place that invokes pkl needs to explicitly include logic to determine whether to use a local cache directory or not. This creates repetitive code and increases the complexity of integration, especially in larger projects or multi-service environments.

Request:

It would be extremely helpful if pkl could support an environment variable (e.g., PKL_USE_CACHE_DIR or similar) that globally controls whether pkl should use the local cache directory. This would avoid the need for conditional logic scattered across various invocation points and provide a cleaner, centralized configuration mechanism.

Benefits:

1. Simplifies usage in air-gapped environments
2. Avoids duplicated environment checks in every code path
3. Provides a consistent and global configuration option
4. Improves maintainability for teams using pkl in multiple services or scripts

Thanks for considering this request!

[StefMa]

Hey,

I don't get the value of the environemnt variable.
By default, pkl is always loking at the cache, when resolving a dependency.
It is there -> Use it
If not -> Do an internet request

So by default, assuming your env variable exist, PKL_USE_CACHE_DIR would be true.
This is the case today. And this is what you would do in an air-gabbed environment.
You popluate the cache and pkl will only use these dependencies.
It will never look at the internet to resolve them. Everything is there.

If set PKL_USE_CACHE_DIR=false, you simply tell pkl to ignore the cache.
Always going to the internet and resolve your dependencies there.
Why should you ever do this? 🤔
Who benefits from that?

Note

You can always invoke pkl with --no-cache to do actually that.
If, for whatever reasons, you don't want to use the cache.

Does this help?

[hcoona]

Hi @StefMa,

Thank you so much for taking the time to respond and explain how the caching works! I really appreciate it.

I apologize. I realize now that my original message wasn't clear at all. Let me try to explain what I'm actually looking for.

You're absolutely right that caching is enabled by default and works great. What I didn't communicate well is that I'm actually hoping for an environment variable to specify the cache directory location (similar to the --cache-dir command-line flag), not to control whether caching is on or off.

Let me describe our typical workflow to clarify the use case:

1. During development (with internet access): Developers work normally with pkl using the default user-level cache. No vendor directory needed at this stage.
2. Before release/deployment: We create a vendor directory (e.g., ./vendor/pkl-cache) for the project
   We populate it with all required dependencies using the method mentioned in Can PklProjects be packaged with their dependencies? #534. This vendor directory is committed to the repository (potentially via Git LFS).
3. In production environments (air-gapped): DevOps pipelines read pkl configurations to adjust build processes based on target environments. Runtime systems generate configurations dynamically based on deployment conditions. All of this happens in completely offline, controlled environments.

For production use cases, we need isolation between different projects. We don't want multiple projects sharing a user-level cache because:

1. Different projects may have different dependency requirements
2. We want controlled, reproducible builds with vendored dependencies
3. We need to ensure no cross-project interference

I understand that Docker can provide this isolation in many cases, but we also have some constrained scenarios where Docker isn't available or practical.

Right now, we add --cache-dir ./vendor/pkl-cache to every pkl invocation throughout our codebase. While this works, it creates several issues:

1. Repetitive code across multiple scripts and tools
2. Maintenance burden when we have many invocation points
3. Complex conditional logic: Some scripts need to work in both online (development) and offline (production) environments. This means we have to add logic to check the environment and conditionally include --cache-dir only when needed.

The third point is particularly challenging because:

1. During development, developers may modify dependencies but haven't updated the vendor directory yet
   We only want to update the vendor directory during the pre-release process
2. If we always use --cache-dir pointing to the vendor directory during development, it would fail when dependencies change
3. So we need environment-detection logic scattered throughout our codebase to decide whether to add --cache-dir or not

This is particularly important for us because we're serving development teams and trying to introduce pkl as a new tool. We believe pkl brings significant benefits, but we can't force developers to adopt this new approach. The key to getting buy-in is minimizing the cognitive burden on developers.

If developers have to constantly think about whether they need to update the vendor directory, or worry about when to use which cache location, it creates significant friction that could prevent adoption. Making this transition transparent and seamless would be tremendously helpful in convincing teams to adopt pkl.

I understand that for most use cases, pkl works perfectly out of the box. However, we're in a somewhat unique position where we straddle the boundary between online (development) and offline (production) environments, and need to support both seamlessly.

An environment variable like PKL_CACHE_DIR that sets the cache directory location would allow us to:

1. Configure it once for production environments (e.g., in CI environment variables)
2. Leave it unset for development environments (using the default cache)
3. Avoid repeating --cache-dir flags and conditional logic everywhere
4. Maintain consistent project-level dependency isolation in production
5. Switch between default behavior (development) and vendored dependencies (production) seamlessly
6. Make the difference completely transparent to developers, removing a potential adoption barrier

This pattern is common in other ecosystems (like npm with NPM_CONFIG_CACHE, pip with PIP_CACHE_DIR, etc.).

I hope this makes the use case clearer! If you have any suggestions or alternative approaches, I'd love to hear them. And of course, if this doesn't align with pkl's design philosophy, I completely understand.

Thanks again for your help and for maintaining such a useful tool!

[bioball]

To address this specific ask, you can add this to your PklProject files:

amends "pkl:Project"

evaluatorSettings {
  moduleCacheDir = read?("env:PKL_CACHE_DIR")
}

Then, PKL_CACHE_DIR=./vendor-pkl-cache pkl eval ... will change the cache dir to be that project's relative path.
And, omitting that env var will cause Pkl to fall back to its default behavior.

I'm reading that your root question here is: "how do I make pkl eval work in air-gapped environments"?
We three solutions to this question; one is to use the cache dir (as addressed above).

Two other options:

1. Use an http proxy (introduced in Pkl 0.26)

   In this approach, you stand up an HTTP CONNECT proxy that can serve Pkl packages to you in your air-gapped environment.

   amends "pkl:settings"
   
   http {
     proxy {
       address = "http://my.proxy.address"
       noProxy {
        "127.0.0.1"
        "169.254.0.0/16"
        "localhost"
       }
     }
   }

2. Stand up a mirror server, and rewrite URL requests to the mirror (introduced in Pkl 0.29)

   In this approach, you maintain a mirror server that serves the same assets. You can then tell Pkl to rewrite HTTP requests to point to this mirror instead.
   In the below, for example, a request to https://pkg.pkl-lang.org/pkl-k8s/[email protected] will be rewritten to https://my.mirror/pkg.pkl-lang.org/pkl-k8s/[email protected], and you'd then have a mirror available at that URL to serve these assets.

   amends "pkl:settings"
   
   http {
     rewrites {
       ["https://pkg.pkl-lang.org/"] = "https://my.mirror/pkg.pkl-lang.org/"
       // github rewrites are also required in order to fetch package ZIP assets.
       ["https://github.com/"] = "https://my.mirror/github-releases/"
       ["https://www.github.com/"] = "https://my.mirror/github-releases/"
     }
   }

Both of these settings can be set at the project level (in your PklProject, or on the machine-user level (in ~/.pkl/settings.pkl).
So, your solution can be: in your production workload, configure your machine's ~/.pkl/settings.pkl with either an HTTP proxy or a mirror server, and your user-level code doesn't need to change!

[hcoona]

These 3 solutions are perfect to me! Thank you!

[bioball]

This is now the topic of our latest blog post! See https://pkl-lang.org/blog/using-packages-in-air-gapped-environments.html

[hcoona]

@bioball Thanks for sharing the new blog post. It’s great to have this documentation!

I did notice, however, that the approach you mentioned earlier using read?("env:PKL_CACHE_DIR") isn’t included in the article. As a newcomer, I didn’t immediately realize that PklProject itself is also a pkl-lang file and can make use of this technique. I think adding a brief explanation or example of this method would be very helpful for beginners.

Thanks again for the detailed guidance and for putting together the documentation!

[bioball]

Using an env var to set the cache dir is somewhat orthogonal here. You can use a env vars to influence any value inside a PklProject file, and not specific to cacheing!