Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/gc-cron.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4.3.1
with:
role-to-assume: arn:aws:iam::${{ vars.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-to-assume: arn:aws:iam::${{ secrets.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-session-name: GitHubActions-${{ github.run_id }}
aws-region: eu-north-1

Expand Down Expand Up @@ -63,7 +63,7 @@ jobs:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4.3.1
with:
role-to-assume: arn:aws:iam::${{ vars.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-to-assume: arn:aws:iam::${{ secrets.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-session-name: GitHubActions-${{ github.run_id }}
aws-region: eu-north-1

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/platform-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ jobs:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4.3.1
with:
role-to-assume: arn:aws:iam::${{ vars.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-to-assume: arn:aws:iam::${{ secrets.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-session-name: GitHubActions-${{ github.run_id }}
aws-region: eu-north-1

Expand Down Expand Up @@ -87,7 +87,7 @@ jobs:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4.3.1
with:
role-to-assume: arn:aws:iam::${{ vars.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-to-assume: arn:aws:iam::${{ secrets.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-session-name: GitHubActions-${{ github.run_id }}
aws-region: eu-north-1

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/snapshot-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4.3.1
with:
role-to-assume: arn:aws:iam::${{ vars.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-to-assume: arn:aws:iam::${{ secrets.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-session-name: GitHubActions-${{ github.run_id }}
aws-region: eu-north-1

Expand Down Expand Up @@ -141,7 +141,7 @@ jobs:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4.3.1
with:
role-to-assume: arn:aws:iam::${{ vars.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-to-assume: arn:aws:iam::${{ secrets.AWS_MAIN_ACCOUNT_ID }}:role/scality-github-e2e-infrastructure
role-session-name: GitHubActions-${{ github.run_id }}
aws-region: eu-north-1

Expand Down
14 changes: 12 additions & 2 deletions action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,16 @@ runs:
)
echo "STACKS_LIST=$stack_list" >> "$GITHUB_OUTPUT"

# NOTE: Written here rather than left to whoever called us. A list is
# the whole answer of this action, and a job that only hands it on has
# nothing of its own to show for the run.
{
echo "### Stacks older than ${AGE}h"
echo
jq -r 'if length == 0 then "None." else .[] | "- `\(.)`" end' <<< "$stack_list"
echo
} >> "$GITHUB_STEP_SUMMARY"


# Spawn {{{
# Prepare stack config
Expand Down Expand Up @@ -268,7 +278,7 @@ runs:
exit 0
fi
echo "Spawn failed [$attempt/$MAX_ATTEMPTS]"
uv run --group snapshot tools/clean_orphan_ports.py || true
uv run --group snapshot tools/clean_orphans.py || true
done
exit 1

Expand Down Expand Up @@ -432,7 +442,7 @@ runs:
exit 0
fi
echo "Destroy failed [$attempt/$MAX_ATTEMPTS]"
uv run --group snapshot tools/clean_orphan_ports.py || true
uv run --group snapshot tools/clean_orphans.py || true
done
exit 1
- name: Remove the stack
Expand Down
5 changes: 3 additions & 2 deletions docs/GITHUB_ACTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,9 @@ to be given to both, or left out of both to take the name of the repository.
|-------|----------|------------|
| `age` | yes | An age in hours |

It answers with `stacks_list`, the stacks older than that age. This is what a
garbage collection runs on.
It answers with `stacks_list`, the stacks older than that age, and writes the
same names to the summary of the job. This is what a garbage collection runs
on.

## Details

Expand Down
33 changes: 21 additions & 12 deletions docs/TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,32 @@ touched, so this works just as well on a platform someone else spawned.
A platform running on a key of your own is reported as such. That key is yours
to point at.

## A failed spawn left ports behind
## A failed spawn left something behind

A cloud that goes down mid-spawn can answer the call that creates a port and
then not the one that waits for it. The port exists and the stack never
learned of it. Every address here is fixed, so that port holds the one the
next attempt needs: spawning again answers `IpAddressAlreadyAllocated`, and
destroying answers that the subnet still has an allocation, which leaves the
platform running and unremovable.
A cloud that goes down mid-spawn can answer the call that does the work and
then not the one that waits for it. What was asked for exists, the stack never
learned of it, and the next attempt walks straight into it.

```bash
uv run --group snapshot tools/clean_orphan_ports.py
uv run --group snapshot tools/clean_orphans.py
```

Only a port of ours, on a network of ours, that nothing is using and that the
stack does not know about. Whatever a router, the DHCP agent or an instance
holds is left alone. The action runs it between two attempts, so a spawn or a
destroy that trips over one recovers on its own.
Two kinds of leftover get a platform stuck for good:

- A port holds a fixed address, and every address here is fixed, so it holds
the one the next attempt needs. Spawning again answers
`IpAddressAlreadyAllocated`, and destroying answers that the subnet still
has an allocation.
- A volume attachment holds its volume. Attaching again answers that the
volume is already attached, and destroying cannot delete a volume that is in
use. The volume is detached rather than taken into the stack, so the next
attempt makes the attachment itself.

Only what the stack owns and does not know about is touched: a port on a
network of ours, a volume of ours the stack holds no attachment for. Whatever
the cloud itself holds, its routers and its DHCP agents, is left alone. The
action runs it between two attempts, so a spawn or a destroy that trips over
one recovers on its own.

## Work against another state backend

Expand Down
119 changes: 0 additions & 119 deletions tools/clean_orphan_ports.py

This file was deleted.

Loading
Loading