[gthvn1]

This patch fixes an issue where the HA statefile and metadata VDIs remain attached to dom0 after a host restart, even if HA has been disabled on the pool. Here is the scenario:

• HA is enabled on the pool
• A host is fenced, so it is shut down
• HA is disabled on the pool. ha_release_resources is called on live hosts but the fenced one doesn't receive the cleanup and keeps the statefile and metadata VDIs in /etc/xensource/static-vdis
• The fenced host restarts:
  • Its local database still has ha.armed set to true, so on_server_restart attached the static VDIs and tries to start the HA daemon
  • ha_start_daemon fails with MTC_EXIT_INVALID_POOL_STATE because the pool is no longer in HA mode. While failing, it sets ha.armed to false in the local database without releasing the static VDIs
  • Later resynchronise_ha_state sees HA disabled on both localhost and the pool. So the (true, false) case, which normally calls ha_release_resources, is not taken and no cleanup is done. At the end of the boot, the statefile and the metadata VDIs are still attached to dom0 and must be removed manually with "static-vdis detach/del".

So the patch adds release_local_ha_static_vdis that deactivates and detaches any HA statefile or metadata VDI still configured as a static VDI on the host, and calls it in the (false, false) case of resynchronise_ha_state. When nothing is left behind it does nothing.

ha_release_resources now also uses this helper. It used Db.Pool.get_ha_statefiles to find statefiles, but a statefile attached on the host always has a local static-vdis entry, so the local configuration is what this host actually need to release. Metadata VDIs were already found using the local configuration. Xha_metadata_vdi.deactivate_and_detach_existing is no longer used and is removed.

Tracking-reference: XCPNG-3914

[last-genius]

I believe this should be safe - despite appearances, ha_release_resources was intended to be local, not pool-wide.

[last-genius]

I'm still curious about this comment - did you test this, what happens in this case? Does the function just skip over non-local VDIs that are returned by Db.Pool.get_ha_statefiles?

[gthvn1]

I didn't test it explicitly but I can check. My current understanding from the code is that for a VDI that is not configured locally, the old code called SM vdi_deactivate (exception are ignored) , then static-vdis detach (fails with "Disk configuration not found"), and finally static vdis del (same error propagated and caught by log_exn_continue . So yes, non-local VDIs are kind of skipped (it fails and errors are ignored).
I will do more test.

[gthvn1]

With the new patch we start from local static-vdis configuration (/etc/xensource/static-vdis) which always contains statefile and metadata attached on this host. So we only release what is actually there.

[gthvn1]

So I was able to test it. I enabled HA so both locations are updated, I see static vdis in the local dir and also in the database of the pool. The I disabled HA. So now we are in the state where static VDIs are not there locally but still there in the Db.pool. I then call the host.ha_release_resources and I see with the old code:

Oct 20 14:17:02 xcp-gtn-ip14 xapi: [debug||2039 /var/lib/xcp/xapi|host.ha_release_resources R:c6f28847342f|helpers] /opt/xensource/bin/static-vdis detach 2ee943e7-1ec1-4648-a9dd-97797ce7177a exited with code 1 [stdout = ''; stderr = '
Traceback (most recent call last):\x0A  File "/opt/xensource/bin/static-vdis", line 413, in <module>\x0A    main()\x0A  File "/opt/xensource/bin/static-vdis", line 407, in main\x0A    detach(sys.argv[2])\x0A  File "/opt/xensource/bin/
static-vdis", line 372, in detach\x0A    raise Exception("Disk configuration not found")\x0AException: Disk configuration not found\x0A']
...
Oct 20 14:17:02 xcp-gtn-ip14 xapi: [debug||2039 /var/lib/xcp/xapi|host.ha_release_resources R:c6f28847342f|helpers] /opt/xensource/bin/static-vdis del 2ee943e7-1ec1-4648-a9dd-97797ce7177a exited with code 1 [stdout = ''; stderr = 'Tra
ceback (most recent call last):\x0A  File "/opt/xensource/bin/static-vdis", line 413, in <module>\x0A    main()\x0A  File "/opt/xensource/bin/static-vdis", line 402, in main\x0A    delete(sys.argv[2])\x0A  File "/opt/xensource/bin/sta
tic-vdis", line 222, in delete\x0A    raise Exception("Disk configuration not found")\x0AException: Disk configuration not found\x0A']
Oct 20 14:17:02 xcp-gtn-ip14 xapi: [debug||2039 /var/lib/xcp/xapi|host.ha_release_resources R:c6f28847342f|helpers] Ignoring exception: INTERNAL_ERROR: [ Subprocess exited with unexpected code 1; stdout = [  ]; stderr = [ Traceback (m
ost recent call last):\x0A  File "/opt/xensource/bin/static-vdis", line 413, in <module>\x0A    main()\x0A  File "/opt/xensource/bin/static-vdis", line 402, in main\x0A    delete(sys.argv[2])\x0A  File "/opt/xensource/bin/static-vdis"
, line 222, in delete\x0A    raise Exception("Disk configuration not found")\x0AException: Disk configuration not found\x0A ] ] while detaching statefile VDI uuid: 2ee943e7-1ec1-4648-a9dd-97797ce7177a

So yes, the old code skips non-local VDIs, but only because each step fails and the errors are ignored.

[gthvn1]

and it shows that Pool.ha_statefiles is not cleared when disabled.