[Bug]: salt-ssh source instrictions does not work, source file salt:// not found error

[syn4ps]

What happened?

A bug happened!

Then running salt-ssh test state.apply test7 , that will just copy static jinja template to tmp on target host, always got error Source file salt://test7.txt.jinja not found in saltenv 'base'.

salt-master file roots config is:

file_roots:
   base:
     - /opt/saltstack/conf

file test7.txt.jinja exist in /opt/saltstack/conf

test7.sls code

test7:
  file.managed:
    - name: /tmp/test.txt
    - source: salt://test7.txt.jinja
    - template: jinja
    - user: root
    - group: root
    - mode: 644

test7.txt.jinja just static text file.
for test i've tried salt-ssh test cp.get_template salt://test7.txt.jinja /tmp/test.txt and it copied file succefully.
i've also tried contents instead source and with contents everything worked.

debug file attached to post, i've looked on target host , and i've seed this file test7.txt.jinja exist inside tar archive, but was not unpacked to path that expected no files folder totally

[DEBUG   ] In saltenv 'base', ** considering ** path '/var/tmp/.root_3d5835_salt/running_data/var/cache/salt/minion/salt-ssh/test/files/base/test7.txt.jinja' to resolve 'salt://test7.txt.jinja'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://test7.txt.jinja'
[DEBUG   ] No dest file found

on target host errors in salt-call.log

2025-10-23 07:06:58,919 [salt.loader      :1270][ERROR   ][209679] Unable to write to grains cache file /var/tmp/.root_3d5835_salt/running_data/var/cache/salt/minion/grains.cache.p: [Errno 2] No such file or directory: '/var/tmp/.root_3d5835_salt/running_data/var/cache/salt/minion/grains.cache.p'
2025-10-23 07:07:00,658 [salt.state       :320 ][ERROR   ][209749] Source file salt://test7.txt.jinja not found in saltenv 'base'

saltstack onedir installed via bootstrap.sh script, started at version 3007.1 , update to 3007.8 does not solve issue.

salt-ssh-debug.txt

Type of salt install

other (please specify in bug details)

Major version

3007.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-12

salt --versions-report output

Salt Version:
          Salt: 3007.8

Python Version:
        Python: 3.10.18 (main, Sep  5 2025, 22:48:51) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: 1.7.2
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.14.1
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.20.6
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4.2
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: debian 12.12 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.8.12-15-pve
        system: Linux
       version: Debian GNU/Linux 12.12 bookworm

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here's some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Salt's Contributor Guide
• Join our Community Discord
• Salt Project YouTube channel
• GitHub Discussions

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject.pdl@broadcom.com. We're glad you've joined our community and look forward to doing awesome things with you!

[carsten-AEI]

@syn4ps what is the OS on your test target?

I do see the very same issue when my remote target is Debian trixie but not when it is bookworm. Locally, it happens for both 3006 and 3007 at the respective latest onedir versions.

[syn4ps]

yeah i have trixie on target , will try to test on bookworm today. but it's not normal too that on trixie it does not working, because as i think its basic feature that must working anyway.

[carsten-AEI]

Given how much Python modules were deprecated for 3.13, I wonder if any of these are the culprit.

(edit: removed probably false statement after restarting salt-mater and removing cache)

[syn4ps]

Given how much Python modules were deprecated for 3.13, I wonder if any of these are the culprit.

(edit: removed probably false statement after restarting salt-mater and removing cache)

I've tried to clean cache on master and restart it does not help, i saw this line on my logs too but i think it's just a cache. Already tried --extra-filerefs workaround from this issue #21370 not helped

[lkubb]

If the target has Python 3.12+ as the system Python, this is probably #66898 (there are several duplicates, see #68267 (comment), also for patch instructions).

[carsten-AEI]

Hmm, still no luck.

I've reinstalled salt-master, salt-common and salt-ssh on my salt-master (Debian trixie):

Salt Version:
          Salt: 3006.16

Python Version:
        Python: 3.10.18 (main, Sep  5 2025, 22:48:51) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.10.0
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.20.6
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 13 trixie
        locale: utf-8
       machine: x86_64
       release: 6.12.48+deb13-amd64
        system: Linux
       version: Debian GNU/Linux 13 trixie

Then, I'm using various patches, namely

• Remove use of swpd, removed module in py3.13 #67788
• Remove use of crypt, removed module in py3.13 #67789
• 9e81a35
• [BUG] file_tree files cause JSON serialization error with salt-ssh #67998 (my other issue with unicode file_tree)

which I streamlined a bit: patches.tar.gz

applied with minimal fuzz:

 cat *.patch | sudo patch -N -p1 -d /opt/saltstack/salt/lib/python3.10/site-packages
patching file salt/pillar/file_tree.py
Hunk #1 succeeded at 256 with fuzz 1.
patching file salt/utils/pycrypto.py
patching file salt/utils/url.py

restarted salt-master.

Starting with

grep . test.sls test/contents
test.sls:/tmp/bla:
test.sls:  file.managed:
test.sls:    - source: salt://test/contents
test.sls: 
test/contents:blabla

at file_roots's base, I get this for bookworm target:

salt-ssh vps1 state.apply test
          ID: /tmp/bla
    Function: file.managed
      Result: True
     Comment: File /tmp/bla updated
     Started: 11:38:01.024882
    Duration: 9.357 ms
     Changes:
              ----------
              diff:
                  New file
              mode:
                  0644

however, on the trixie target, I still get

salt-ssh vps2 state.apply test
          ID: /tmp/bla
    Function: file.managed
      Result: False
     Comment: Source file salt://test/contents not found in saltenv 'base'
     Started: 11:38:52.104105
    Duration: 4.223 ms
     Changes:

Probably, I'm missing something.

NB: on the trixie target I also need to run pip install --break-system-packages backports.ssl-match-hostname otherwise salt-ssh will just die not finding the backports module.

[carsten-AEI]

Tinkering a bit further... while for example pycrypto.py is patched on the master, on the target I still find an unpatched pyall/salt/utils/pycrypto.py under /var/tmp/.

running_data/salt-call.log contains

2025-10-23 15:50:24,010 [salt.loader      :1257][ERROR   ][3364] Unable to write to grains cache file /var/tmp/.adminhelper_303cbc_salt/running_data/var/cache/salt/minion/grains.cache.p: [Errno 2] No such file or directory: '/var/tmp/.adminhelper_303cbc_salt/running_data/var/cache/salt/minion/grains.cache.p'
2025-10-23 15:50:27,028 [salt.loader.lazy :809 ][ERROR   ][3436] Failed to import module bsd_shadow, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 788, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1248, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1263, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 680, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1207, in load_module
  File "<frozen importlib._bootstrap_external>", line 1031, in load_module
  File "<frozen importlib._bootstrap>", line 537, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 966, in _load
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/modules/bsd_shadow.py", line 21, in <module>
    import salt.utils.pycrypto
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/utils/pycrypto.py", line 105, in <module>
    methods = {m.name.lower(): m for m in crypt.methods}
                                          ^^^^^^^^^^^^^
AttributeError: module 'crypt' has no attribute 'methods'
2025-10-23 15:50:27,454 [salt.loader.lazy :809 ][ERROR   ][3436] Failed to import utils pycrypto, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 788, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1248, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1263, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 680, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1207, in load_module
  File "<frozen importlib._bootstrap_external>", line 1031, in load_module
  File "<frozen importlib._bootstrap>", line 537, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 966, in _load
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/utils/pycrypto.py", line 105, in <module>
    methods = {m.name.lower(): m for m in crypt.methods}
                                          ^^^^^^^^^^^^^
AttributeError: module 'crypt' has no attribute 'methods'
2025-10-23 15:50:27,660 [salt.loader.lazy :809 ][ERROR   ][3436] Failed to import module linux_shadow, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 788, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1248, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1263, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 680, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1207, in load_module
  File "<frozen importlib._bootstrap_external>", line 1031, in load_module
  File "<frozen importlib._bootstrap>", line 537, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 966, in _load
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/modules/linux_shadow.py", line 27, in <module>
    import salt.utils.pycrypto
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/utils/pycrypto.py", line 105, in <module>
    methods = {m.name.lower(): m for m in crypt.methods}
                                          ^^^^^^^^^^^^^
AttributeError: module 'crypt' has no attribute 'methods'
2025-10-23 15:50:27,750 [salt.loader.lazy :809 ][ERROR   ][3436] Failed to import module mod_random, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 788, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1248, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1263, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 680, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1207, in load_module
  File "<frozen importlib._bootstrap_external>", line 1031, in load_module
  File "<frozen importlib._bootstrap>", line 537, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 966, in _load
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/modules/mod_random.py", line 13, in <module>
    import salt.utils.pycrypto
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/utils/pycrypto.py", line 105, in <module>
    methods = {m.name.lower(): m for m in crypt.methods}
                                          ^^^^^^^^^^^^^
AttributeError: module 'crypt' has no attribute 'methods'
2025-10-23 15:50:27,872 [salt.loader.lazy :809 ][ERROR   ][3436] Failed to import module nxos, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 788, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1248, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1263, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 680, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1207, in load_module
  File "<frozen importlib._bootstrap_external>", line 1031, in load_module
  File "<frozen importlib._bootstrap>", line 537, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 966, in _load
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/modules/nxos.py", line 96, in <module>
    from salt.utils.pycrypto import gen_hash
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/utils/pycrypto.py", line 105, in <module>
    methods = {m.name.lower(): m for m in crypt.methods}
                                          ^^^^^^^^^^^^^
AttributeError: module 'crypt' has no attribute 'methods'
2025-10-23 15:50:28,055 [salt.loader.lazy :809 ][ERROR   ][3436] Failed to import module solaris_shadow, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 788, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1248, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/loader/lazy.py", line 1263, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 680, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 1207, in load_module
  File "<frozen importlib._bootstrap_external>", line 1031, in load_module
  File "<frozen importlib._bootstrap>", line 537, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 966, in _load
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/modules/solaris_shadow.py", line 30, in <module>
    import salt.utils.pycrypto
  File "/var/tmp/.adminhelper_303cbc_salt/pyall/salt/utils/pycrypto.py", line 105, in <module>
    methods = {m.name.lower(): m for m in crypt.methods}
                                          ^^^^^^^^^^^^^
AttributeError: module 'crypt' has no attribute 'methods'
2025-10-23 15:50:28,560 [salt.state       :318 ][ERROR   ][3436] Source file salt://services/openssh/files/sshd_config.jj not found in saltenv 'base'

[lkubb]

Did you regenerate the thin archive after applying the patches? IIRC salt-ssh --regen-thin ...

[carsten-AEI]

That's it and I apparently did not do that and/or I already had a cached version on the target.

Then it just bears the question, why restarting the salt-master does not rebuild thin/thin.tgz on its own...

I'll probably solve this locally for now with

cat /etc/systemd/system/salt-master.service.d/clear_cached_thin.conf

[Service]
ExecStartPre=rm -f /var/cache/salt/master/thin/thin.tgz

which should fix this issue for me forgetting about this in the future (obviously this is quick and dirty as I don't know how salt decides where the cache resides).

I do hope the patches will appear in one of the next releases!

[lkubb]

There's two parts to the salt-ssh deployment, the Salt thin archive (+ library dependencies, upcoming in 3008: + saltexts) and the extmods one. Both are sent from the master to the remote node on request.

IIRC the thin archive on the master is only regenerated automatically when the Salt version changes and includes a checksum, which is verified against the locally available one on the remote node, while the extmods archive is regenerated for each salt-ssh invocation (+ same checksum behavior to avoid network overhead).

I agree it would make some sense to automatically regenerate the thin tar when the master restarts, but since the overlap between Salt users that have a running master daemon and those that use salt-ssh is not the majority of Salt users (I guess), it shouldn't be the default behavior. (Edit: Maybe flushing the cache could be though, much less overhead).

The simplest solution might be to have a reactor or schedule that does it. The cache path is static, it's always in <cachedir>/thin/thin.tgz (it can be configured to be a .zip, not sure what the support status of that functionality is). It's probably enough to just run salt-run cache.flush thin. There's a runner module (salt-run thin.generate overwrite=true), but AFAIR it has been removed in 3008.