Repository navigation
Upload kolla logs and container health of failed builds - #2977
Merged
Merged
Conversation
kolla services log to files under /var/log/kolla, not to the container's stdout, so the job output carries none of them. A failure inside a service -- nova-compute stalling a shelve, cinder-volume never finishing an attach, mariadb going unhealthy on one node -- is then only visible as the symptom the deploy or tempest step reports, and the evidence is torn down with the environment. Add a post-run playbook that copies /var/log/kolla from every host into the uploaded logs, under kolla/<host>/, mirroring the on-host layout. The directory is copied whole, rotated files included, so the next failing service is covered without having to name it first. Each file is gzipped on its own and keeps its name with ".gz" appended, e.g. kolla/testbed-node-0/nova/nova-compute.log.gz. A full tempest run with DEBUG logging writes 17-28 MB of nova-compute.log and about 56 MB of nova-api.log per node, and gzip shrinks these logs 8-25x. Single-file gzip, unlike one archive, stays readable with zless and zgrep. Files the job already uploads are not changed. The orchestrator reaches the nodes only through the manager, so the host list comes from "osism get hosts" with the manager's inventory as fallback, the same discovery collect-glance-logs.yml uses. Each host streams a tar.gz of the directory through the manager; tar is told not to warn about logs that grow while it reads them. Every ssh gets "-n" so the osism wrapper on the manager cannot drain stdin. The play is best-effort, never fails the job, and prints per host how many files it copied and their gzipped size. Only failed builds are collected. A green build needs none of this, and skipping it keeps the upload volume to the builds someone will read. Outside Zuul, where zuul_success is unset, nothing is collected: a local deployment is still up to be inspected directly. Against a live OSISM 11.0.1 testbed with seven hosts the copy takes about two minutes. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
When kolla-ansible's service check fails with "The following mariadb containers are unhealthy", the job output names the container and the host but not what the health probe saw, or since when. Docker keeps that record -- status, failing streak and the last five probe results with their output -- but only on the host, and only until the environment is torn down. Before the kolla logs are copied, read the health state of every container on every testbed host with "docker inspect" and upload it as container-health/<host>.json: one entry per container with its name, state, start time and docker's health record (null for containers without a health check). The unhealthy containers and their failing streak are also printed into the job output, so the symptom is visible without opening the files. The files are small and stay plain JSON. Like the log copy, this runs only for failed builds. Docker keeps only the last five probes, which at kolla's 30 second interval covers about two and a half minutes. A container that has recovered by post-run therefore shows healthy probes only; one that is still failing shows the probe output and, through the streak, how long it has been failing. To get as close to the failure as post-run allows, the playbook now runs first in post-run, and host discovery moves into a task of its own so the health capture and the log copy share one host list. The full "docker inspect" output is reduced on the orchestrator, which keeps the remote command free of nested template quoting. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
collect-glance-logs.yml printed the last 2000 lines of glance-api.log from every host into the job output. It was added to diagnose Octavia amphora images stuck in "queued", whose cause lives only in that log. collect-kolla-logs.yml now uploads all of /var/log/kolla from every host when a build fails, and that includes the whole glance-api.log. The tail is a strict subset of it, and a short one: with DEBUG logging 2000 lines cover only about 20 minutes, so the import error of a long build can already have scrolled out of it. Remove the playbook and its post-run entry. The note on the stdin drain of the osism wrapper, which collect-kolla-logs.yml pointed to there, now stands on its own in that playbook. Green builds no longer carry any Glance log. The import stalls this was written for end in a failed build, which is collected. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
marked this pull request as ready for review
October 19, 2026 12:06
ideaship
marked this pull request as draft
October 19, 2026 12:07
ideaship
force-pushed
the
collect-kolla-logs
branch
from
October 19, 2026 12:08
3f4be01 to
3137409
Compare
ideaship
marked this pull request as ready for review
October 19, 2026 12:28
This was referenced Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kolla services log to files under
/var/log/kolla, not to container stdout, so a failed testbed build carries none of them. A service failing — nova-compute stalling a shelve, cinder-volume never finishing an attach, mariadb going unhealthy on one node — shows only as the symptom the deploy or tempest step reports, and the evidence is torn down with the environment.This PR uploads that evidence when a build fails, and drops the narrower Glance capture it supersedes.
Changes
/var/log/kollafrom every host (newplaybooks/collect-kolla-logs.yml). The directory is copied whole, rotated files included, tokolla/<host>/…, mirroring the on-host layout. Every file is gzipped on its own (kolla/testbed-node-0/nova/nova-compute.log.gz), so it stays readable withzless/zgrep. Files the job already uploads are unchanged.docker inspectof every container on every host goes tocontainer-health/<host>.json: name, state, start time and docker's health record (status, failing streak, last five probes with their output). Unhealthy containers are also printed into the job output. The playbook runs first in post-run to stay as close to the failure as possible.collect-glance-logs.yml. Its 2000-line tail ofglance-api.logis a strict subset of the uploaded file, and at DEBUG level it covers only about 20 minutes, so a long build's import error could already have scrolled out of it.Sizes
Measured on a test cluster running the full tempest suite with DEBUG logging: per node, nova-compute.log grows by 17–28 MB and nova-api.log by about 56 MB during the run; gzip shrinks the service logs 8–25×. A CI build holds a single deploy, so the gzipped upload should be in the tens of MB.
Testing
The top commit, "DNM: collect kolla logs on green builds too", removes the failure-only condition so that a green
zuullabel run exercises the playbook. It is dropped before this leaves draft.🤖 Generated with Claude Code