Skip to content

[Bug Bash] Blocked egress looks like a DNS failure #3228

Open

Description

The guest shows wget: bad address 'api.github.com', with no sign that a policy blocked it.

This is part of the user flow from: https://docs.google.com/document/d/1yvnITgzq0qlFc1M8s9dLqzysFZOQkybbb9zrYJuWD_E/edit?resourcekey=0-4OfZZh7hv6ITygAQfbU_tg&tab=t.eha9gghaxj1r

Seen in step 4 of the ate-env flow (ate-env alice shell 'wget -q -T 20 -O /data/zen.txt https://api.github.com/zen'), with no egress policy on the actor. That flow pins Substrate ddb46651. main (ffed855) has the same code path.

This is the designed behavior. The sandbox DNS relay answers a hostname that no egress rule allows with a synthesized NXDOMAIN (internal/atunnel/dns/dnshandler.go, "A disallowed name results in an NXDOMAIN"). docs/egress-traffic.md documents it. The problem is that a user can't tell a policy denial from a typo or a real DNS outage:

  • The guest sees an ordinary resolver failure.
  • I found no log or metric for DNS policy denials. The DNS handler only logs at debug level, when it drops a query because too many are in flight.
  • kubectl ate logs actor --source lifecycle and the egress gateway logs don't mention the denied name.

Possible fixes, smallest first:

  • Log or count denied DNS names per actor so an operator can see them, for example as a lifecycle event or an atenet-egress metric with the atespace attribute.
  • Add an Extended DNS Error (RFC 8914, info code 15 "Blocked" or 18 "Prohibited") to the synthesized NXDOMAIN. Resolvers that show EDE can then report a policy block. Plain NXDOMAIN behavior is unchanged.
  • Say in the ate-env tab and in docs/egress-traffic.md that "bad address" or "could not resolve host" for an external name usually means the actor has no egress rule for it.

Found during the GKE bug bash, with help from an AI agent.

Activity

  1. bowei commented on Oct 27, 2026

    @bowei
    Collaborator

    Looks like this should be:

    • P1 before GA: docs change
    • Add metrics can be post-GA
    • Use of RFC 8914 will be post-GA
  2. masavik commented on Nov 9, 2026

    @masavik

    Another use case for the per-actor part of this, which #3395's counter (no actor or atespace label) does not cover.

    We run untrusted, user-written code in actors. Each call gets a fresh actor whose EgressPolicy lists only the hosts
    that code declared. Our current sandbox fails the call whenever the code tried an undeclared host, even if the code
    catches the network error. On Substrate the access is blocked, which is the part that matters most. But the code can
    catch the NXDOMAIN and finish "successfully", and nothing records which actor looked up which name. The gateway's 403
    path already logs at WARN with the actor reference (cmd/atenet/internal/router/egress/egress.go); the DNS path is
    the only denial with no per-actor trace.

    What would close it for us, in order of preference:

    1. A rate-limited log line or lifecycle event per denied query, carrying atespace, actor, QNAME and QTYPE (the
      follow-up Count actor DNS queries by egress policy decision #3395 mentions).
    2. Any per-actor signal a controller can read after the actor is deleted.

    Seen at 94e0a26 on kind with gVisor. Repro: a policy with one tlsPassthrough rule for example.com:443, then
    fetch("https://example.org/") from inside the actor → NXDOMAIN, and no log line in atelet, ateworker or
    atenet-egress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions