Skip to content

Fix systemd-networkd recovery after interface failures - #19686

Open
Alex Cortelyou (acortelyou) wants to merge 1 commit into
microsoft:3.0-devfrom
acortelyou:backport-systemd-networkd-auto-reconfigure
Open

Alex Cortelyou (acortelyou) wants to merge 1 commit into
microsoft:3.0-devfrom
acortelyou:backport-systemd-networkd-auto-reconfigure

Conversation

@acortelyou

@acortelyou Alex Cortelyou (acortelyou) commented Oct 23, 2026 •

Copy link
Copy Markdown
Member
Merge Checklist

All boxes should be checked before merging the PR (just tick any boxes which don't apply to this PR)

  • The toolchain has been rebuilt successfully (or no changes were made to it)
  • The toolchain/worker package manifests are up-to-date
  • Any updated packages successfully build (or no packages were changed)
  • Packages depending on static components modified in this PR (Golang, *-static subpackages, etc.) have had their Release tag incremented.
  • Package tests (%check section) have been verified with RUN_CHECK=y for existing SPEC files, or added to new SPEC files
  • All package sources are available
  • cgmanifest files are up-to-date and sorted (./cgmanifest.json, ./toolkit/scripts/toolchain/cgmanifest.json, .github/workflows/cgmanifest.json)
  • LICENSE-MAP files are up-to-date (./LICENSES-AND-NOTICES/SPECS/data/licenses.json, ./LICENSES-AND-NOTICES/SPECS/LICENSES-MAP.md, ./LICENSES-AND-NOTICES/SPECS/LICENSE-EXCEPTIONS.PHOTON)
  • All source files have up-to-date hashes in the *.signatures.json files
  • sudo make go-tidy-all and sudo make go-test-coverage pass
  • Documentation has been updated to match any changes to the build system
  • Ready to merge

Current-series validation is local x86_64 and ARM64; earlier pipeline results are scoped below. Regenerated Secure Boot signing/repackaging still requires the distribution pipeline. Go checks, toolchain rebuilding, dependent static-library release bumps, and build-system documentation changes are not applicable; none of those components changed.


Summary

AKS VMSS routine live migration occasionally leaves eth0 in systemd-networkd's failed state, stopping IPv6 router discovery. Existing routes continue forwarding traffic, masking the failure until the RA-derived IPv6 default route expires roughly 2.5 hours later. Although RAs are advertised every five seconds, the route is no longer refreshed. Manually running networkctl reconfigure eth0 restores RA processing and IPv6 connectivity.

This behavior is consistent with systemd/systemd#29246. Backport systemd/systemd#30936 to Azure Linux's existing systemd v255, adding rate-limited automatic reconfiguration rather than upgrading systemd.

Include the failed-state initialization and duplicate DHCP STOP fixes from systemd/systemd#34054, plus its reviewer-requested DHCP renew/release/decline guards. These ensure automatic recovery actually proceeds and handles inactive DHCP clients safely. Each upstream commit is a separate patch, preserving its original metadata and commit message; necessary v255 adaptations are documented after the patch separator.

Change Log
  • Bump systemd and its entangled systemd-boot-signed spec from 255-34.azl3 to 255-35.azl3, retaining upstream v255 and the existing downstream patch stack.
  • Register the following individual upstream patches:
Upstream commit Change v255 adaptation
c2eb775 Automatically reconfigure failed interfaces, limited to five attempts per ten seconds per interface. Byte-identical mainline patch.
6d1cea7 Add the upstream frequent-interface-flapping regression. Adjust older test context only; all upstream added/deleted lines preserved.
685fd0d Enter initialized in shared reconfiguration, so the GETLINK callback does not ignore a failed link. Omit only the hunk in link_reconfigure_on_bus_method_reload(), absent in v255. v255 reload already calls the patched shared link_reconfigure(); retained hunks unchanged.
e5b19cb Hold DHCP client lifetime and suppress duplicate STOP callbacks. Byte-identical mainline patch.
f8cdd37 Gracefully skip DHCP renew/release/decline when client state does not permit sending. One unchanged context line uses v255's attempt instead of discover_attempt; all upstream added/deleted lines preserved.
338ed5b Accept both OpenSSL issuer-whitespace formats in the ukify test, enabling checked builds. Byte-identical mainline patch; test-only prerequisite.
  • Keep locally authored failed-link/rate-limit, repeated-STOP, and inactive-client regression coverage in the separate test-networkd-failed-link-and-dhcp-client-recovery.patch, not mixed into upstream patches.
  • Exclude the unrelated CAN readiness cleanup and speculative NDisc changes. No CVEs are claimed.
Does this affect the toolchain?

NO. The runtime systemd package is not in the toolchain/worker manifests; systemd-bootstrap is unchanged.

Associated issues
Links to CVEs

None.

Test Methodology
  • Final series, x86_64: local validation used mcr.microsoft.com/azurelinux/base/core:3.0, with LINUX_IMMUTABLE for the existing capability test. The spec's existing chroot exclusions were unchanged; no additional skips or --nocheck were introduced.
  • Full checked RPM build passed: RUN_CHECK=y rpmbuild -ba --noclean -D "_topdir /work/rpmbuild" -D "_smp_mflags -j8" -D "with_check 1" /work/rpmbuild/SPECS/systemd.spec. All unsigned RPMs and the SRPM were generated: 1,224 passed, 26 skipped, zero failures.
  • Network suite passed: meson test -C /work/rpmbuild/BUILD/systemd-stable-255/x86_64-azl-linux --no-rebuild --print-errorlogs --suite network reported 16 passed, zero failures.
  • Five negative controls detected missing fixes: individually removing the initialized-state transition made meson test -C /work/rpmbuild/BUILD/systemd-stable-255/x86_64-azl-linux --no-rebuild --print-errorlogs test-network fail its recovery assertion. Removing duplicate STOP suppression or each of the renew/release/decline guards made meson test -C /work/rpmbuild/BUILD/systemd-stable-255/x86_64-azl-linux --no-rebuild --print-errorlogs test-dhcp-client fail its corresponding assertion. Restoring the fixes made the tests pass.
  • Installed-RPM live IPv6 regression passed: installed the generated systemd, libs, PAM, udev, networkd, resolved, and RPM-macros packages, then ran NETWORKD_BINARY=/usr/lib/systemd/systemd-networkd NETWORKCTL_BINARY=/usr/bin/networkctl python3 /artifacts/test-network-flap.py. The harness exercises the patched upstream IPv6 test against real veth interfaces in an isolated Docker network namespace: three iterations, 120 down/up cycles, nine successful address/readiness checks.
  • Final series, native ARM64 (795172d87): the same checked RPM build command and base image generated unsigned RPMs and the SRPM: 1,231 passed, 19 skipped, zero failures. meson test -C /work/rpmbuild/BUILD/systemd-stable-255/aarch64-azl-linux --no-rebuild --print-errorlogs --suite network passed all 16 tests. All five negative controls detected their respective missing fixes; restored tests passed. The same installed-RPM harness passed: three iterations, 120 cycles and nine readiness checks.
  • Pipeline build 1233628, also before the revision: systemd's package checks passed on AMD64 (1,216 passed, 34 skipped) and ARM64 (1,217 passed, 33 skipped). The overall build failed in unrelated Open vSwitch and Trident tests; it is not evidence of a passing pipeline for the final series.
  • Spec guidelines passed: python3 toolkit/scripts/check_spec_guidelines.py --toolchain_specs "" --specs SPECS/systemd/systemd.spec SPECS-SIGNED/systemd-boot-signed/systemd-boot-signed.spec.
  • Entanglement check passed: python3 toolkit/scripts/check_entangled_specs.py ..
  • License-map check passed: python3 toolkit/scripts/license_map.py LICENSES-AND-NOTICES/SPECS/data/licenses.json LICENSES-AND-NOTICES/SPECS/LICENSES-MAP.md SPECS SPECS-EXTENDED SPECS-SIGNED.
  • Spec-cleaner remains blocked by baseline tooling issues: the checked-in linter patch fails plain git apply; after recovering its runtime rules, spec-cleaner -d --diff-prog="git --no-pager diff" /work/specs/systemd.spec /work/specs/systemd-boot-signed.spec fails parsing an existing RPM version containing ^. The untouched 255-34 systemd spec produces the same exception. No linter pass is claimed.
  • Source archive hashes were verified. The v255 component registration/source URL and existing hashed local sources are unchanged; patches do not require signature entries.
  • Actual AKS live migration and the production route-expiry interval were not reproduced. Boot testing, regenerated EFI signing/repackaging, and a passing distribution pipeline remain outstanding.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@microsoft-github-policy-service microsoft-github-policy-service Bot added Packaging 3.0-dev PRs Destined for AzureLinux 3.0 labels Oct 23, 2026
@acortelyou

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 19686 in repo microsoft/azurelinux

@acortelyou
Alex Cortelyou (acortelyou) marked this pull request as ready for review October 23, 2026 02:55
@acortelyou
Alex Cortelyou (acortelyou) requested a review from a team as a code owner October 23, 2026 02:55
@mfrw

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
1 pipeline(s) were filtered out due to trigger conditions.

Comment thread SPECS/systemd/systemd.spec Outdated
@acortelyou

Copy link
Copy Markdown
Member Author

The BuildAndTest pipeline failures are unrelated to the backport.
Systemd’s ADO checks passed: 1,216 tests on AMD64 and 1,217 on ARM64, zero failures.

Failing package Diagnosis
Open vSwitch, both architectures Isolated dummy-datapath tests hit PMD timeouts; AMD64 also has flow/packet-count mismatches. These tests bypass system networking.
Trident, AMD64 A unit test closes a descriptor and assumes it stays invalid; concurrent descriptor reuse makes its unwrap_err() panic.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your merging of multiple upstream commits into a single patch makes it significantly more difficult to review, as well as including patches unrelated to this issue. You additionally appear to be missing f8cdd37d0aed8423d6744dd6637955760346939a so this will definitely need significantly closer inspection, and isn't mergable in its current state.

@acortelyou
Alex Cortelyou (acortelyou) force-pushed the backport-systemd-networkd-auto-reconfigure branch from 37aa5dd to e33b0a4 Compare October 24, 2026 16:54
Backport systemd's rate-limited failed-interface recovery and its initialized-state and DHCP-client follow-ups, including f8cdd37d0aed8423d6744dd6637955760346939a.

Keep each mainline commit in its own patch with original metadata and minimal v255 adaptations. Include separate downstream regression coverage and the upstream OpenSSL test compatibility fix.

Synchronize the unsigned and signed boot package releases at 255-35.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.0-dev PRs Destined for AzureLinux 3.0 Packaging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants