bitmap 'virtnbdbackup.X' not found in backing chain when switching to a new backup location

[patvdv-smo]

Hi,

We are using v2.30 and v2.37. We have a monthly backup scheme that consist of 1 full + 30 incrementals and at the first day of each month we start a new backup chain in a new location. This month we noticed that the initial month's backup failed because:

2025-10-01 01:00:03] INFO root check - targetDir [main]: Backup mode auto, target folder is empty: executing full backup.
[2025-10-01 01:00:03] INFO root virtnbdbackup - main [main]: Backup will save [1] attached disks.
[2025-10-01 01:00:03] INFO root virtnbdbackup - main [main]: Concurrent backup processes: [1]
[2025-10-01 01:00:03] INFO root checkpoint - create [main]: Loading checkpoints from: [/home/backup/sws-ha-01/20251001-0000/sws-ha-01.cpt]
[2025-10-01 01:00:03] INFO root checkpoint - redefine [main]: Loading checkpoint list from: [/home/backup/sws-ha-01/20251001-0000/checkpoints]
[2025-10-01 01:00:03] INFO root checkpoint - create [main]: Checkpoint handling.
[2025-10-01 01:00:03] ERROR root checkpoint - delete [main]: Error during checkpoint removal: [internal error: bitmap 'virtnbdbackup.4' not found in backing chain of 'vda']
[2025-10-01 01:00:03] ERROR root virtnbdbackup - main [main]: Failed to remove checkpoint

The backup correctly switched to full but is complaining about the missing checkpoints. They exist in the previous month's directory 20250901 but not 20251001 (obviously).

We could fix this by copying over the checkpoints subdirectory and .cpt file or by deleting the existing 30 or so (metadata) checkpoints in the running VM. We chose for the second option and then a new full backup started without problem.

This problem didn't happen in the previous month so I am confused as what may have caused this?

The issue happened with v2.30. We then upgraded to v2.37 but the behaviour didn't change. We still had to delete the metadata checkpoints.

[abbbi]

see readme:

https://github.com/abbbi/virtnbdbackup?tab=readme-ov-file#backup-fails-with-error-during-checkpoint-removal-internal-error-bitmap-xx-not-found-in-backing-chain-of-xx

most probably in these cases VM was powered off in a way it was not able to write the bitmaps to the qcow file.

[patvdv-smo]

Hi,

Thanks for the reply. I was aware of the README entry. But the reason I also posted the message is because of the comment:

Note:

Since version 2.30 virtnbdbackup will handle these situations automatically and falls back to a full backup in case it detects a broken bitmap chain.

Except it didn't, we had to manually remove all the existing checkpoints.

most probably in these cases VM was powered off in a way it was not able to write the bitmaps to the qcow file.

Do you mean at the point when the new backup chain was started (it was online) or sometime in the past month?

[abbbi]

well, it falls back in case an incremental backup is attempted.

In your case, an full backup was attempted but it failed to remove the existing checkpoints, because at least one bitmap tied to an old checkpoint was not existant in the qcow image anymore. This can happen for example if the host is powered off (power outage) or the qemu process for the virtual machine dies with another reason not allowing to flush the bitmaps to the qcow image. Or if the virtual machine qcow image was converted without keeping bitmaps.. or.. other reasons.

So you have to check what happend to the qcow image or the virtual machine after the checkpoint/bitmap "virtnbdbackup.4" was created for your incremental chain.

The more checkpoints exist, the more likely it is that some failure leads to this situation.

During a full backup, it allways attempts to remove all old checkpoints, if that doesnt work, you need to manually handle things.

It would be possible to implement fallback code that forcefully removes the inconsistent checkpoints, even if the bitmap is not existant anymore and issues an warning. But anyhow, in a well operated system these cases should not happen that often.

[abbbi]

Ive pushed some changes to this branch:

master...checkpoint_285

basically during full backup, checkpoint/bitmap consistency is now validated and if an inconsistency is detected, the
checkpoints are removed with the METADATA flag enabled (allowing removal "forcefully") and warning is issued.

As this is an patch working around a particular issue in your environment and nothing that troubles me much, i'd
consider merging after some sponsoring

[patvdv-smo]

Hi Michael,

Thanks for the explanation and the proposed fix. I am working on trying to get some sponsoring sorted.

[abbbi]

changes merged to new version 2.38