fix regex tenant federation to work in single binary mode

[SungJin1212]

Fixes regex tenant federation (-tenant-federation.regex-matcher-enabled=true):

• In single binary mode, regex queries returned empty results. The query-frontend replaced the process-wide default tenant resolver with RegexValidator, so the in-process querier queried the regex as a literal tenant ID. The write path also used RegexValidator, which rejected tenant IDs that are not valid regexes (e.g. a(b and stored a|b as a single tenant).
• A federated query failed with multiple org IDs present when a matched tenant ID contained regex metacharacters and also matched other tenants as a regex (e.g. team.* matching team.a and teamXa), because each tenant was matched again as a regex below the federation layer.

Other changes:

• Since matched tenants are no longer resolved as regexes again, the regex resolver cache only holds regex entries. Updated the -tenant-federation.regex-cache-size description accordingly.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
• docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

[CharlieTLe]

Thanks for tracking this down — the diagnosis is right: in single binary mode the query-frontend replaces the process-wide default resolver with RegexValidator, and the in-process querier then treats the regex as a literal tenant ID.

I think the fix introduces a problem on the write path, though. users.WithDefaultResolver is process-wide, so skipping the RegexValidator swap leaves RegexResolver (set in initRegexResolverService) as the default resolver for every module in the process, not only the querier. In single binary mode that includes the distributor (distributor.go:762), the ingester (ingester.go:1361), the ruler and the alertmanager.

RegexResolver treats every org ID as a regex matched against the tenants discovered in the bucket, and tenant IDs are allowed to contain ., *, ( and ). I checked this with a throwaway test on this branch that installs RegexResolver as the default with known users fooXbar, team-a and team-b, then resolves the tenant the way the push path does:

push X-Scope-OrgID="foo.bar" -> TenantID="fooXbar"   err=<nil>
push X-Scope-OrgID="team-a"  -> TenantID="team-a"    err=<nil>
push X-Scope-OrgID="team-."  -> TenantID=""          err=multiple org IDs present
push X-Scope-OrgID="a(b"     -> TenantID=""          err=invalid regex present

So with this change, in single binary mode with regex federation enabled, a write for tenant foo.bar would be stored under fooXbar, and rule group / alertmanager config uploads would be affected the same way. Microservices deployments are not affected, since distributor-only processes never initialize the regex resolver and keep the MultiResolver from cortex.go:443.

Suggested direction: avoid relying on the process-wide default for regex federation. Pass t.RegexResolver explicitly to the tenant-federation merge queryable (the frontend's results cache already gets it explicitly via tenantResolverFn), and pass RegexValidator explicitly to the query-frontend / query-scheduler handlers. The default resolver used by the write path then stays the MultiResolver. As a side benefit this would also fix a pre-existing issue: today, single binary with regex federation rejects writes for tenant IDs that are not valid regexes (e.g. a(b), because RegexValidator becomes the process default.

Smaller points:

• The new integration test only covers the query path. It would be worth also pushing to a regex-like tenant ID (e.g. foo.bar alongside an existing fooXbar) and asserting the data stays in foo.bar.
• Nit: TestRegexResolver_SingleBinary doesn't follow the file's Test_TenantFederationRegexResolver_... naming.
• The PR description still has Fixes #<issue number>.

[SungJin1212]

@CharlieTLe
Thanks, good catch on the write path. I reworked it along the lines you suggested.

It also fixes a federated query failing with multiple org IDs present when a matched tenant ID contains regex metacharacters, e.g. team.* matching team.a and teamXa: team.a was matched again as a regex below the federation layer. I added Test_TenantFederationRegexResolver_TenantIDWithRegexMetacharacters, which fails on master.