Updates to 'httpd' package recreate default config files removed by this module, which crashes httpd

[jlemire3]

Describe the Bug

I have a number of Oracle Linux 8 webservers that are managed completely by this module. When this module installs and configures Apache and the vhosts defined in my code, it sets up an initial environment that works great.

Since the puppetlabs-apache module manages the files in the /etc/httpd/conf.modules.d and /etc/httpd/conf.d directories, it removes all of the irrelevant files from those directories. The issue appears when updates for the httpd package are released. During updates, the httpd package reintroduces config files in those directories that were removed by this module, causing httpd to crash.

The order of operations I've noticed are:

1. httpd is working correctly, and a pending update is available for httpd.
2. I run updates on the affected servers, which installs the latest httpd package. This installation reintroduces config files that were managed and removed by the puppetlabs-apache module during its Puppet run.
3. The reintroduced config files conflict with other configurations made by the puppetlabs-apache module causing httpd to crash.
4. The next Puppet run will remove the following files (which may not be all inclusive), and httpd will start correctly again:

• '/etc/httpd/conf.modules.d/00-base.conf',
• '/etc/httpd/conf.modules.d/00-dav.conf',
• '/etc/httpd/conf.modules.d/00-lua.conf',
• '/etc/httpd/conf.modules.d/00-mpm.conf',
• '/etc/httpd/conf.modules.d/00-optional.conf',
• '/etc/httpd/conf.modules.d/00-proxy.conf',
• '/etc/httpd/conf.modules.d/00-ssl.conf',
• '/etc/httpd/conf.modules.d/00-systemd.conf',
• '/etc/httpd/conf.modules.d/01-cgi.conf',
• '/etc/httpd/conf.modules.d/README',

and

• '/etc/httpd/conf.d/autoindex.conf',
• '/etc/httpd/conf.d/ssl.conf',
• '/etc/httpd/conf.d/userdir.conf',
• '/etc/httpd/conf.d/welcome.conf',
• '/etc/httpd/conf.d/README',

During my efforts to try working around this issue, I have tried manually deleting affected files as I notice the issue. If I remove the /etc/httpd/conf.modules.d/00-mpm.conf file manually after an update (and before running Puppet again), I may get another error: AH00526: Syntax error on line 5 of /etc/httpd/conf.d/ssl.conf Cannot define multiple Listeners on the same IP:port. This error appears to be related to /etc/httpd/conf.d/ssl.conf, so I delete that file too. After deleting one or both of those files depending on the Apache config in use, httpd is able to start up successfully again in all of the cases I've tested so far.

In all cases, a subsequent run of the Puppet agent will fix the issue caused from the update without manual intervention as the puppetlabs-apache module takes over the default config again and sets it how the module expects it. However, updating the httpd package will always knock out the httpd service until the next Puppet run happens, causing downtime to all of my webservers managed by this module.

Expected Behavior

I expect to be able to update httpd without default configurations being created at install and crashing httpd. I expect that the default configuration applied by the puppetlabs-apache module to prevent these files from being created so that httpd doesn't crash, and so I don't have to make sure a Puppet run happens immediately after updates, hoping that nobody is trying to use a webserver in that brief period between an update and the Puppet run finishing.

I would expect the puppetlabs-apache module to maintain empty copies of the affected config files so that httpd doesn't try to recreate them during an update (or somehow prevent those files from being created during the update). Then, the puppetlabs-apache module can add or remove configurations from those files as needed. These files do not appear to be overwritten by htttpd during updates unless they are missing, so managing empty copies of those files seems to me like the easiest way to prevent this behavior.

Steps to Reproduce

Steps to reproduce the behavior:

1. Have a running Apache server managed by the puppetlabs-apache module on Oracle Linux 8
2. Update httpd with the Puppet Server package management feature or manually update it with dnf udpate or yum update OR you can downgrade the httpd package to observe the same behavior with dnf downgrade httpd.
3. Observe the state of the httpd service immediately after the package update or downgrade and it should be failed due to one of the above errors (or possibly other errors depending on what features are enabled in the puppetlabs-apache module based on the default configs that are recreated).
4. Run Puppet on the affected host with puppet agent -t --debug and observe the following logs, which remove the default config files that were created and successfully starts httpd again:

Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/README]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/README]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/README]: The container /etc/httpd/conf.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/autoindex.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/autoindex.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/autoindex.conf]: The container /etc/httpd/conf.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/ssl.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/ssl.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/ssl.conf]: The container /etc/httpd/conf.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/userdir.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/userdir.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/userdir.conf]: The container /etc/httpd/conf.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/welcome.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.d/welcome.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.d/welcome.conf]: The container /etc/httpd/conf.d will propagate my refresh event
Debug: /etc/httpd/conf.d: The container Class[Apache] will propagate my refresh event
Info: /etc/httpd/conf.d: Scheduling refresh of Class[Apache::Service]

and

Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-base.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-base.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-base.conf]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-dav.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-dav.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-dav.conf]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-lua.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-lua.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-lua.conf]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-mpm.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-mpm.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-mpm.conf]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-optional.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-optional.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-optional.conf]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-proxy.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-proxy.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-proxy.conf]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-ssl.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-ssl.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-ssl.conf]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-systemd.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-systemd.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-systemd.conf]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/01-cgi.conf]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/01-cgi.conf]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/01-cgi.conf]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/README]: Removing existing file for replacement with absent
Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/README]/ensure: removed (corrective)
Debug: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/README]: The container /etc/httpd/conf.modules.d will propagate my refresh event
Debug: /etc/httpd/conf.modules.d: The container Class[Apache] will propagate my refresh event
Info: /etc/httpd/conf.modules.d: Scheduling refresh of Class[Apache::Service]
Debug: Class[Apache]: The container Stage[main] will propagate my refresh event

Environment

• puppetlabs-apache: v12.3.1 (Earlier versions also exhibit this behavior, at least back to v10.1.1)
• Platform: Oracle Linux 8
• Puppet Server Version: v2023.8.5 (and older - unknown how far back this goes)
• Puppet Agent Version: 8.14.0 (and older - unknown how far back this goes)

Additional Context

I've been trying to work around this issue since at least November 2024 by managing default copies of the above mentioned Apache config files with all default configs commented out so that the httpd package doesn't try overwriting those files during updates, but somewhere in the Puppet code it's managing the files causing dependency loops depending on what features are enabled. For most of my webservers, managing those files has worked and it prevents the crashing issue. However, one of my webservers is using the apache::mod::auth_mellon class, which was causing a dependency loop for several of the files I was manually managing on my end. This is just one example, but for every file I stopped managing, another file would cause a different dependency loop stemming from the same root class:

Error: Found 1 dependency cycle:
(File[/etc/httpd/conf.d/ssl.conf] => Class[Profile::Webserver::Apache_default] => Class[Apache::Mod::Auth_mellon] => Apache::Mod[auth_mellon] => Package[mod_auth_mellon] => File[/etc/httpd/conf.d] => File[/etc/httpd/conf.d/ssl.conf])\nTry the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz

It's also possible I'm managing my code for the puppetlabs-apache module in a way that is adversely affecting the functionality of the module. I have a class that sets up my defaults for Apache on all my servers, with a few custom parameters I set up so I can selectively overwrite those defaults when I call it in my profiles. Here is my apache_default class, which is included by default as part of a role for ALL of my webservers:

class profile::webserver::apache_default (
  String                           $ensure                        = 'present',
  Optional[Variant[String, Undef]] $default_certificate_cn        = undef,
  Optional[String]                 $apache_group                  = $facts['os']['family'] ? {
    'RedHat'      => 'apache',
    'Debian'      => 'www-data',
    default       => 'apache',
  },
  Optional[Variant[String, Array]] $apache_additional_groups      = undef,
  Hash                             $apache_custom_params          = {},
  Hash                             $apache_rhel_override_params   = {},
  Hash                             $apache_debian_override_params = {},
) {
  require role::webserver::default_webserver
  $debug = false

  if $ensure == 'present' {
    $package_ensure = undef
  } else {
    $package_ensure = 'absent'
  }

  ####################################
  ######## USER CONFIGURATION ########
  ####################################

  group { $apache_group:
    ensure => $ensure,
    gid    => '48',
  }

  $apache_groups = [$apache_group] + $apache_additional_groups ? {
    Array  => $apache_additional_groups,
    String => [$apache_additional_groups],
    undef  => [],
  }

  # Create the Apache user
  user { 'apache':
    ensure     => $ensure,
    uid        => '48',
    groups     => $apache_groups,
    managehome => false,
    system     => true,
    shell      => '/sbin/nologin',
    password   => '!!',
  }

  ###################################
  ### APACHE PARAMETER GENERATION ###
  ###################################

  $apache_base_params = {
    'package_ensure'   => $package_ensure,
    'default_vhost'    => false,
    'manage_user'      => false,
    'manage_group'     => false,
    'servername'       => $facts['networking']['fqdn'],
    'serveradmin'      => 'hostmaster@domain.com',
  }

  # Whether to include default SSL params or not depending if a default certificate name was provided.
  if !($default_certificate_cn) {
    $apache_ssl_params = {}
  } else {
    $live_certificate_path = letsencrypt::letsencrypt_lookup($default_certificate_cn)
    $apache_ssl_params = {
      'default_ssl_cert' => "${live_certificate_path}/fullchain.pem",
      'default_ssl_key'  => "${live_certificate_path}/privkey.pem",
    }
  }

  $apache_defaults_with_ssl = merge($apache_base_params, $apache_ssl_params)
  if $debug {
    notify { "DEFAULT PARAMS WITH SSL: ${apache_defaults_with_ssl}": }
  }

  $apache_overridden_params = merge($apache_defaults_with_ssl,$apache_custom_params)
  if $debug {
    notify { "APACHE OVERRIDDEN CUSTOM PARAMS: ${apache_overridden_params}": }
  }

  case $facts['os']['family'] {
    'RedHat': {
      $apache_os_params = $apache_rhel_override_params
    }
    'Debian': {
      $apache_os_params = $apache_debian_override_params
    }
    default: {
      fail ('This manifest does not support anything other than RHEL and Debian based operating systems')
    }
  }

  # Merge the OS params over all the previous defaults and overrides to ensure OS specific parameters override defaults
  $apache_final_params = merge($apache_overridden_params, $apache_os_params)
  if $debug {
    notify { "BASE AND SSL PARAMS WITH CUSTOM AND OS OVERRIDES: ${apache_final_params}": }
  }

  # Call the Apache class
  class { 'apache':
    *         => $apache_final_params,
    subscribe => User['apache'],
  }
}

and here is an example of how its called in the profile for the server I've been gathering logs from:

######################################
######## APACHE CONFIGURATION ########
######################################

class { 'profile::webserver::apache_default':
  default_certificate_cn      => $facts['networking']['fqdn'],
}

# Installs Apache Module Auth_Mellon
# Apache Mellon Project - https://github.com/latchset/mod_auth_mellon
class { 'apache::mod::auth_mellon':
  require => Class['profile::webserver::apache_default'],
}

I have also tried making use of the defined types apache::mpm::disable_mpm_event, apache::mpm::disable_mpm_prefork and apache::mpm::disable_mpm_worker, but it doesn't stop the default config files from being created, and as such Apache still crashes after updates. It also doesn't prevent the issue I'm having with AH00526: Syntax error on line 5 of /etc/httpd/conf.d/ssl.conf Cannot define multiple Listeners on the same IP:port.

I understand that only Oracle Linux 7 is officially supported by this module, and OL 8 is "unsupported", but OL8 is RHEL based and other RHEL 8 based OSs are supported so I presume that OL8 is likely unofficially supported for most use-cases.