[3007.x][BUG] Non-root users can not call functions after upgrade to 3007.0

[lee-harmonic]

Description
After upgrading to 3007.0, non-root users in the salt group (and in publisher_acl) can not start jobs. Permissions to ipc prevent access and are automatically reset.

Error messages:

lee@host:~$ salt '*.*' test.ping     
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds -   File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 532, in salt_main
    client.run()

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 816, in cmd_cli
    self.pub_data = self.run_job(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 388, in run_job
    pub_data = self.pub(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1905, in pub
    if listen and not self.event.connect_pub(timeout=timeout):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
    self.subscriber = salt.utils.asynchronous.SyncWrapper(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 77, in __init__
    self.obj = cls(*args, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
    return publish_client(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
    return salt.transport.tcp.PublishClient(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 219, in __init__
    super().__init__(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in __init__
    super().__init__()

Authentication error occurred.

Group membership and permissions:

lee@host:~$ groups
lee salt
lee@host:~$ ls -l /var/run/salt/master/
total 0
srw------- 1 salt salt 0 Mar 12 09:53 master_event_pub.ipc
srw------- 1 salt salt 0 Mar 12 09:53 master_event_pull.ipc
srw------- 1 salt salt 0 Mar 12 09:53 publish_pull.ipc
srw------- 1 salt salt 0 Mar 12 09:53 workers.ipc

Setting permissions:

lee@host:~$ sudo chmod g+rw /var/run/salt/master/*
lee@host:~$ ls -l /var/run/salt/master/           
total 0
srw-rw---- 1 salt salt 0 Mar 12 09:53 master_event_pub.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 master_event_pull.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 publish_pull.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 workers.ipc
lee@host:~$ salt '*.*' test.ping
Authentication error occurred.

Logs for salt master now have:

[WARNING ] Authentication failure of type "user" occurred.

Restarting salt master:

lee@host:~$ sudo service salt-master restart
lee@host:~$ ls -l /var/run/salt/master/
total 0
srw------- 1 salt salt 0 Mar 14 10:30 master_event_pub.ipc
srw------- 1 salt salt 0 Mar 14 10:30 master_event_pull.ipc
srw------- 1 salt salt 0 Mar 14 10:30 publish_pull.ipc
srw------- 1 salt salt 0 Mar 14 10:30 workers.ipc

The log from the restart has the following line:

[ERROR   ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None

Setup
Contents of /etc/salt/master.d/auth.conf:

publisher_acl:
  lee:
    - .*

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
Follow instructions at https://docs.saltproject.io/salt/user-guide/en/latest/topics/security.html#publisher-acls to set up publisher-acl and directory permissions.

Expected behavior

Non-root user can start jobs such as test.ping and permissions to do so are not reset when (re)starting the salt-master service. Was working before upgrade to 3007.0.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3007.0
 
Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.5 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-18-amd64
        system: Linux
       version: Debian GNU/Linux 12.5 bookworm

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[david-pulkowski]

We started noticing issues with the Publisher_ACL since switching to salt 3006
#66067

[alexholodak]

I ran into this exact issue after upgrading from salt-master 3006.7 to 3007.0.
I think I may have stumbled upon a workaround that solved the problem for me:

I did not have the python-is-python3 package installed. This made it so python commands would run as python3. Previously only python3 commands would run.

On Debian/Ubuntu:
sudo apt install python-is-python3

This change seems to have fixed the issue. I no longer see the TCP Publish Client error anymore.
I can now run salt '*' test.ping and other commands fine.

@lee-harmonic - does this work for you?

I'm encountering the same issues described earlier. Reverting back to version 3005.5 enables me to utilize local or domain-signed-in users. I've experimented with versions 3006.7 and 3007.0, both with and without root user access. However, the 'publisher_acl' feature doesn't operate as anticipated in these versions.

Thanks for bringing it up, @alexholodak, but it doesn't address the problem we've discussed previously.

[dmurphy18]

@alexholodak python not being available, and having to use python3 is very much an OS issue, for example: arch has python has python3, but older OS's got rid of python since it implies python2, so don't see having to install python-is-python3 as a Salt issue but a user and their OS's in use issue, about forgetting python referred to python2.

But thanks for the tip, for those needing the work-around for their environments.

@lee-harmonic Wondering if this is related to 3006.x plus using user salt, rather than root as in earlier releases, and missed a place where the change to default user salt affects things. Wonder if you change to user: root in the /etc/salt/master and minion configuration files, and restart and see if the problem still occurs, if not, indicates issue is default user: salt in configuration file.

[dmurphy18]

@lee-harmonic Can you check if the problem occurs with user: root as previously mentioned, it would help a lot in identifying the cause.

[jamest-pin]

@dmurphy18
We are having this same issue after upgrading from 3005 to 3007. Our CI runner can no longer run salt commands.

Triple-checked all the settings as per https://docs.saltproject.io/en/latest/ref/publisheracl.html#publisher-acl-system

Everything works fine as salt user. We were already running the salt-master service as user salt prior to updating.

Installing python-is-python3 had no effect.

buildkite-agent@salt:/srv$ salt 'salt' test.ping
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds -   File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
    client.run()

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 815, in cmd_cli
    self.pub_data = self.run_job(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 387, in run_job
    pub_data = self.pub(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1904, in pub
    if listen and not self.event.connect_pub(timeout=timeout):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
    self.subscriber = salt.utils.asynchronous.SyncWrapper(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in __init__
    self.obj = cls(*args, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
    return publish_client(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
    return salt.transport.tcp.PublishClient(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 220, in __init__
    super().__init__(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in __init__
    super().__init__()

[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
TypeError: argument must be an int, or have a fileno() method.
Traceback (most recent call last):
  File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
    client.run()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 830, in cmd_cli
    for fn_ret in self.get_cli_event_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1653, in get_cli_event_returns
    for ret in self.get_iter_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1187, in get_iter_returns
    for raw in ret_iter:
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1104, in get_returns_no_block
    raw = self.event.get_event(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 651, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 553, in _get_event
    raw = self.subscriber.recv(timeout=wait)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 138, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 146, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 527, in run_sync
    return future_cell[0].result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 375, in recv
    events, _, _ = select.select([self._stream.socket], [], [], 0)
TypeError: argument must be an int, or have a fileno() method.
Traceback (most recent call last):
  File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 528, in salt_main
    client.run()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 830, in cmd_cli
    for fn_ret in self.get_cli_event_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1653, in get_cli_event_returns
    for ret in self.get_iter_returns(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1187, in get_iter_returns
    for raw in ret_iter:
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1104, in get_returns_no_block
    raw = self.event.get_event(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 651, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 553, in _get_event
    raw = self.subscriber.recv(timeout=wait)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 138, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 146, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 527, in run_sync
    return future_cell[0].result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 375, in recv
    events, _, _ = select.select([self._stream.socket], [], [], 0)
TypeError: argument must be an int, or have a fileno() method.
buildkite-agent@salt:/srv$

[jamest-pin]

We figured this out, turns out that the update messes up the permissions.

Before (not working, see above)

$ /var/run/salt/master$ ls -al
total 0
drwxr-xr-x 2 salt salt 120 Jul 24 08:50 .
drwxr-xr-x 4 root root  80 Jul 24 08:39 ..
srw------- 1 salt salt   0 Jul 24 08:50 master_event_pub.ipc
srw------- 1 salt salt   0 Jul 24 08:50 master_event_pull.ipc
srw------- 1 salt salt   0 Jul 24 08:50 publish_pull.ipc
srw------- 1 salt salt   0 Jul 24 08:50 workers.ipc

after (working):

$ /var/run/salt/master$ ls -al
total 0
drwxr-xr-x 2 salt salt 120 Jul 24 08:50 .
drwxr-xr-x 4 root root  80 Jul 24 08:39 ..
srw-rw---- 1 salt salt   0 Jul 24 08:50 master_event_pub.ipc
srw-rw---- 1 salt salt   0 Jul 24 08:50 master_event_pull.ipc
srw-rw---- 1 salt salt   0 Jul 24 08:50 publish_pull.ipc
srw-rw---- 1 salt salt   0 Jul 24 08:50 workers.ipc

The command, for anyone in doubt:

cd /var/run/salt/master
sudo chmod g+rw *.ipc