Skip to content

ate-setup: default the apiserver's read/write pool to 20 connections - #2493

Open
Davanum Srinivas (dims) wants to merge 1 commit into
agent-substrate:mainfrom
dims:ateapi-pool-default
Open

Davanum Srinivas (dims) wants to merge 1 commit into
agent-substrate:mainfrom
dims:ateapi-pool-default

Conversation

@dims

@dims Davanum Srinivas (dims) commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

On a large host ateapi's read/write pool defaulted to one connection per CPU (pgxpool's max(4, NumCPU)), so a few replicas exhausted PostgreSQL's 100 default slots and requests failed with remaining connection slots are reserved.

The installer now sets ATE_API_POSTGRES_POOL_MAX_CONNS to 20 per replica, and to 64 on --cluster-size size10 with the bundled PostgreSQL, which used to pin that through its DSN and is provisioned for it. An external database keeps 20, and an adopted Cloud SQL install keeps its recorded limit. A supplied value wins over the size profile; 0 is refused at install time. ateapi's own flag keeps its unset default, so a direct run still follows the DSN or pgxpool.

Docs: docs/guides/postgres.md, docs/guides/operator-install.md, docs/guides/benchmarking.md and docs/dev/cloud-sql-scaling-guide.md.

  • Tests pass: go test -race ./cmd/ate-setup/... (new TestLoadPostgresPoolMaxConns, including size10 with an external database, and an adopted install without a recorded limit in TestCreateAPIServerEnvVarsAdoptsPostgresIdentity) and the new TestReadWritePoolConfigMaxConns in atepg, which pins that a set limit replaces the DSN's pool_max_conns and an unset one keeps it; go vet, golangci-lint.
  • Appropriate changes to documentation are included in the PR

@dims
Davanum Srinivas (dims) marked this pull request as ready for review October 12, 2026 13:51
@dims
Davanum Srinivas (dims) force-pushed the ateapi-pool-default branch 4 times, most recently from d122ae5 to e8ab6c6 Compare October 14, 2026 00:52
@dims
Davanum Srinivas (dims) force-pushed the ateapi-pool-default branch 2 times, most recently from f277ee5 to d986670 Compare October 16, 2026 11:31
@iplay88keys

Copy link
Copy Markdown
Collaborator

I think the reason for this fix is good, but the new default causes two problems:

🤖 AI review finding

  1. Size10 clusters get fewer connections. For size10 clusters, ate-setup puts pool_max_conns=64 in the read/write connection string (Size10PostgresPoolParams). atepg.Connect replaces the connection string value when PoolMaxConns > 0. With a default of 20, this is always true, so size10 clusters drop from 64 to 20 connections. A pool_max_conns value that a user sets in their own connection string is also replaced.
  2. The value 0 does not work for installed clusters. The flag help says that 0 keeps the connection string or pgx default. But the installer sets the limit through ATE_API_POSTGRES_POOL_MAX_CONNS, and loadFlagsFromEnv rejects values of 0 or less. Only a user who runs ateapi directly with the flag can use 0.

Possible fixes:

  • Use 20 only when the connection string does not set pool_max_conns.
  • Or set the default in ate-setup. ate-setup already sets the size10 limit, so all pool sizes would then be in one place.

Separately, two docs still show the old default of max(4, NumCPU): hack/install-ate.sh and tools/setup-gcp/cloud-sql.md.

Please also add a test for the new default and for a connection string that already sets pool_max_conns. That test would find the size10 problem.

@dims

Copy link
Copy Markdown
Collaborator Author

Jeremy Alvis (@iplay88keys) thank you for the review. i will explore the second option. will take care of the other things you pointed out as well

@dims Davanum Srinivas (dims) changed the title ateapi: default the read/write pool to 20 connections ate-setup: default the apiserver's read/write pool to 20 connections Oct 16, 2026
@dims

Copy link
Copy Markdown
Collaborator Author

Jeremy Alvis (@iplay88keys) Done and ready. PTAL

@dims
Davanum Srinivas (dims) force-pushed the ateapi-pool-default branch 3 times, most recently from 7c3b191 to 4bbcc70 Compare October 18, 2026 21:48
@iplay88keys

Copy link
Copy Markdown
Collaborator

I'm not sure my approval carries mergeability weight, so you'll want a review by one of the maintainers.

That being said, here's a follow-up review:

🤖 AI review finding
One new behavior change needs a decision:

The size10 limit now also applies to external PostgreSQL. Before this change, the limit of 64 was only in the bundled connection string. Now buildConfig sets 64 for every size10 install (load.go:182). A size10 cluster on Cloud SQL or another external database now opens up to 64 connections per replica. The external database might not accept that many. Is this the intended behavior? If not, use the size10 limit only with bundled PostgreSQL. If it is, please add it to tools/setup-gcp/cloud-sql.md and docs/dev/cloud-sql-scaling-guide.md.

Nits:

  • atepg.go: readWritePoolConfig now sits between poolConfig and its doc comment. Move the poolConfig comment down to its function.
  • load.go: the comment "see the roles" is not clear. Say what the flag is for, for example "lets an adopted install keep its recorded limit".
  • docs/operator-install.md: the default column shows 20. Add that size10 uses 64.

@dims

Copy link
Copy Markdown
Collaborator Author

I'm not sure my approval carries mergeability weight, so you'll want a review by one of the maintainers.

You are all i've got :)

@dims
Davanum Srinivas (dims) force-pushed the ateapi-pool-default branch 5 times, most recently from d79833e to 97edb07 Compare October 31, 2026 17:05
@dims
Davanum Srinivas (dims) force-pushed the ateapi-pool-default branch 2 times, most recently from 1d05771 to 8229e48 Compare November 9, 2026 19:24
pgxpool's default of max(4, NumCPU) opens one connection per CPU of the
node, which on a large host exhausts PostgreSQL's 100 default slots. The
installer now sets ATE_API_POSTGRES_POOL_MAX_CONNS to 20 per replica, and
to 64 on size10 with the bundled PostgreSQL, which used to pin that
through its DSN and is provisioned for it; an external database keeps 20,
and an adopted Cloud SQL install keeps its recorded limit. A supplied
value wins over the size profile. ateapi's own flag keeps its unset
default, so a direct run still follows the DSN or pgxpool. Zero is
refused by the installer.

Signed-off-by: Davanum Srinivas <davanum@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants