exporter is picking up bogus smart data from virtual devices #308
Description
Hi all,
I'm not quite sure where to file this issue, as it seems to be a combination of behaviors from longhorn, iscsi, udev and smartctl, so please forgive me if this is in the wrong place.
The problem is that logical volumes created by longhorn are being up by the exporter as real disks with smartctl. I think the process that's happening is as follows:
- longhorn creates a PV and PVC (physical volume and physical volume claim, respective), using iscsid to do the the backing
- iscsid creates new virtual scsi devices. As part of this, it emulates smartctl (smartctl output below)
- udev and/or systemd (I'm confused on what managed this these days), detects the new scsi devices and creates new block devices under /dev/sd[a-z] and /dev/longhorn
- smartctl_exporter runs smartctl scan, and finds the new fake block devices.
- smartctl_exporter pulls the limited (bogus?) smart data from the iscsi block devices
I suspect that there are potential remediations in the chain above, but I decided to come here first. =)
Versions
- smartctl_exporter 0.14.0 (as deployed by helm)
- Distro: Ubuntu 24.04.3 LTS
- Kernel: Linux k8smaster 6.8.0-79-generic
- open-iscsi 2.1.9-3ubuntu5.4
- kubernetes 1.33.4
-
- (helm) prometheus deployed kube-prometheus-stack-77.5.0 v0.85.0
- (helm) prometheus-smartctl deployed prometheus-smartctl-exporter-0.15.4 v0.14.0
block device directory listing
root@k8smaster:/dev# ls -ald /dev/sdb /dev/longhorn/*
brw-rw---- 1 root disk 8, 16 Sep 5 19:49 /dev/longhorn/pvc-5d58472e-9ce7-4bf0-abcc-f343d21f3774
brw-rw---- 1 root disk 8, 16 Sep 5 19:49 /dev/sdb
smartctl -a output from a take device:
root@k8smaster:/dev# smartctl -a /dev/sdb
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.8.0-79-generic] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: IET
Product: VIRTUAL-DISK
Revision: 0001
Compliance: SPC-3
User Capacity: 2,147,483,648 bytes [2.14 GB]
Logical block size: 512 bytes
LU is thin provisioned, LBPRZ=1
Logical Unit id: 0x30000001000000010x60000000000000000e00000000010001
Serial number: beaf11
Device type: disk
Local Time is: Sat Sep 6 04:42:55 2025 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Disabled
Temperature Warning: Disabled or Not Supported
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 0 C
Drive Trip Temperature: 0 C
Error Counter logging not supported
[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
Device does not support Self Test logging
/var/log/syslog | grep sdb
root@k8smaster:/dev# grep sdb /var/log/syslog
2025-09-01T06:35:25.633110+00:00 k8smaster kernel: EXT4-fs (sdb): unmounting filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487.
2025-09-01T06:35:33.939111+00:00 k8smaster kernel: EXT4-fs (sdb): mounted filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487 r/w with ordered data mode. Quota mode: none.
2025-09-03T21:59:14.982107+00:00 k8smaster kernel: EXT4-fs (sdb): unmounting filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487.
2025-09-03T21:59:16.231101+00:00 k8smaster kernel: EXT4-fs (sdb): mounted filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487 r/w with ordered data mode. Quota mode: none.
2025-09-03T22:01:02.915101+00:00 k8smaster kernel: EXT4-fs (sdb): unmounting filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487.
2025-09-03T22:01:08.753098+00:00 k8smaster kernel: EXT4-fs (sdb): mounted filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487 r/w with ordered data mode. Quota mode: none.
2025-09-03T22:23:15.657109+00:00 k8smaster kernel: EXT4-fs (sdb): unmounting filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487.
2025-09-03T22:23:22.765354+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] 4194304 512-byte logical blocks: (2.15 GB/2.00 GiB)
2025-09-03T22:23:22.765368+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Write Protect is off
2025-09-03T22:23:22.765371+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Mode Sense: 69 00 10 08
2025-09-03T22:23:22.765372+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
2025-09-03T22:23:22.815105+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Attached SCSI disk
2025-09-03T22:23:31.097120+00:00 k8smaster kernel: EXT4-fs (sdb): mounted filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487 r/w with ordered data mode. Quota mode: none.
2025-09-03T22:26:17.980141+00:00 k8smaster kernel: EXT4-fs (sdb): unmounting filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487.
2025-09-03T22:26:19.949116+00:00 k8smaster kernel: EXT4-fs (sdb): mounted filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487 r/w with ordered data mode. Quota mode: none.
2025-09-03T22:57:59.241198+00:00 k8smaster kernel: EXT4-fs (sdb): unmounting filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487.
2025-09-03T22:58:08.992112+00:00 k8smaster kernel: EXT4-fs (sdb): mounted filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487 r/w with ordered data mode. Quota mode: none.
2025-09-03T23:08:51.715111+00:00 k8smaster kernel: EXT4-fs (sdb): unmounting filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487.
2025-09-03T23:09:07.333196+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] 4194304 512-byte logical blocks: (2.15 GB/2.00 GiB)
2025-09-03T23:09:07.334280+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Write Protect is off
2025-09-03T23:09:07.334293+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Mode Sense: 69 00 10 08
2025-09-03T23:09:07.334296+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
2025-09-03T23:09:07.384681+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Attached SCSI disk
2025-09-03T23:09:15.380109+00:00 k8smaster kernel: EXT4-fs (sdb): mounted filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487 r/w with ordered data mode. Quota mode: none.
2025-09-05T19:40:12.451099+00:00 k8smaster kernel: EXT4-fs (sdb): unmounting filesystem 0efe3c5f-1792-47e9-8608-09bf48aed487.
2025-09-05T19:49:08.543097+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] 4194304 512-byte logical blocks: (2.15 GB/2.00 GiB)
2025-09-05T19:49:08.543109+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Write Protect is off
2025-09-05T19:49:08.543111+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Mode Sense: 69 00 10 08
2025-09-05T19:49:08.543112+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
2025-09-05T19:49:08.568274+00:00 k8smaster kernel: sd 4:0:0:1: [sdb] Attached SCSI disk
2025-09-05T19:49:17.228109+00:00 k8smaster kernel: EXT4-fs (sdb): mounted filesystem 0c9e80ca-ad45-444f-a41c-97dc145ba104 r/w with ordered data mode. Quota mode: none.
cat /var/log/syslog.1 /var/log/syslog | grep iscsi
This didn't seem to show anything interesting, just a bunch of connections and disconnections
root@k8smaster:/var/log# journalctl | grep smartd
Aug 28 12:29:22 k8smaster (smartd)[162319]: smartmontools.service: Referenced but unset environment variable evaluates to an empty string: smartd_opts
Aug 28 12:29:22 k8smaster smartd[162319]: smartd 7.4 2023-08-01 r5530 [x86_64-linux-6.8.0-79-generic] (local build)
Aug 28 12:29:22 k8smaster smartd[162319]: Opened configuration file /etc/smartd.conf
Aug 28 12:29:22 k8smaster smartd[162319]: Drive: DEVICESCAN, implied '-a' Directive on line 18 of file /etc/smartd.conf
Aug 28 12:29:22 k8smaster smartd[162319]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Aug 28 12:29:22 k8smaster smartd[162319]: Device: /dev/sda, type changed from 'scsi' to 'sat'
Aug 28 12:29:22 k8smaster smartd[162319]: Device: /dev/sda [SAT], opened
Aug 28 12:29:22 k8smaster smartd[162319]: Device: /dev/sda [SAT], MSATA 120GB SSD, S/N:Z2024040800021, FW:W0529A0, 120 GB
Aug 28 12:29:22 k8smaster smartd[162319]: Device: /dev/sda [SAT], not found in smartd database 7.3/5528.
Aug 28 12:29:22 k8smaster smartd[162319]: Device: /dev/sda [SAT], is SMART capable. Adding to "monitor" list.
Aug 28 12:29:22 k8smaster smartd[162319]: Device: /dev/sdb, opened
Aug 28 12:29:22 k8smaster smartd[162319]: Device: /dev/sdb, [IET VIRTUAL-DISK 0001], lu id: 0x30000001000000010x60000000000000000e00000000010001, S/N: beaf11, 2.14 GB
Aug 28 12:29:22 k8smaster smartd[162319]: Device: /dev/sdb, IE (SMART) not enabled, skip device
Aug 28 12:29:22 k8smaster smartd[162319]: Try 'smartctl -s on /dev/sdb' to turn on SMART features
Aug 28 12:29:22 k8smaster smartd[162319]: Monitoring 1 ATA/SATA, 0 SCSI/SAS and 0 NVMe devices
Aug 28 12:29:22 k8smaster smartd[162319]: Device: /dev/sda [SAT], state written to /var/lib/smartmontools/smartd.MSATA_120GB_SSD-Z2024040800021.ata.state
root@k8smaster:/var/log# journalctl | grep udev
Didn't have anything apparently useful