Repository navigation
BED-9346 - Managed OpenHound Implementation #116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
6e6064d
feat(config): add managed mode foundation
codemonkey2841 6caeb2c
docs(config): document managed mode default
codemonkey2841 4e43066
feat(logging): default managed mode to structured logs
codemonkey2841 605bcf4
BED-9509: add aws secrets manager service (#83)
StranDutton f514f16
refactor(managed): generalize transient request detection
codemonkey2841 a61364b
feat(managed): add managed job completion
codemonkey2841 d7882cf
test(managed): cover managed job completion
codemonkey2841 f7185d3
fix(client): narrow managed job retry exceptions
codemonkey2841 08e6606
test(client): cover request exception retry policies
codemonkey2841 6555c64
Check available jobs (managed OH)- BED-9512 (#86)
StranDutton 214a5d4
fix: corrected 'end job' endpoint
codemonkey2841 a801f16
BED-9510 - Adding BHE Auth path (#88)
jplarose 6f59b04
Claim a Job (managed OH) - BED-9515 (#94)
StranDutton 1897469
feat: add managed collector start and heartbeat requests
codemonkey2841 d602658
test: isolate integration collection output
codemonkey2841 b85b594
feat: run claimed managed jobs with five-minute heartbeats
codemonkey2841 373e73d
feat: Review fixes
codemonkey2841 60b9e2a
fix: ensure abort process can safely clean up jobs
codemonkey2841 69b9717
feat: start claimed managed jobs
codemonkey2841 cb084f0
fix: heartbeat active managed jobs
codemonkey2841 e88926d
fix: report managed job completion
codemonkey2841 9528b65
refactor: centralize managed job ID normalization
codemonkey2841 e7fc4e5
fix: retain managed claims until job start is confirmed
codemonkey2841 6ace0fc
chore(managed): Disable heartbeats for managed mode
codemonkey2841 672d95c
BED-10083: Improve managed-mode AWS failure diagnostics (#100)
StranDutton 0ae0fc3
fix(managed): request jobs in priority order (BED-10130) (#101)
StranDutton debf3ca
fix(managed): use 'collie' job queue API paths (BED-10135) (#103)
StranDutton 4e2c645
BED-9517 - Ingest path for Managed OpenHound (#102)
jplarose 5f50edf
fix(managed): align profile and schedule IDs with BHE UUIDs - BED-101…
StranDutton a9d36ce
chore: cleanup some unused code and stop OH from making calls that wo…
jplarose 2a79790
feat(managed): resolve nested job configuration - BED-10080 (#105)
StranDutton 13ab19a
feat(managed): add job-scoped AWS profile adapter - BED-10231 (#106)
StranDutton 5ee3388
feat(managed): support AWS role profiles - BED-10272 (#111)
StranDutton 4865ab5
fix(managed): fail AWS jobs on initial auth failure (BED-10368) (#114)
StranDutton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Repository guidance | ||
|
|
||
| ## Managed collector job queue API | ||
|
|
||
| Managed collector job queue endpoints must use `/api/v2/collie-job-queue` in | ||
| public URL paths. Do not use `collector` in this endpoint path: ad blockers may | ||
| block URLs containing that term. This applies to all managed queue operations. | ||
| This URL naming requirement does not change internal terminology. Keep names | ||
| such as `CollectorJob`, collector-related Python identifiers, job keys, models, | ||
| and database concepts unchanged unless a separate requirement calls for it. | ||
|
|
||
| ## BloodHound API contracts | ||
|
|
||
| When adding or changing a BloodHound or BloodHound Enterprise (BHE) API endpoint, | ||
| verify its path, HTTP method, parameters, request body, and response against the | ||
| applicable API contract. Refer to the [BloodHound repository](https://github.com/SpecterOps/BloodHound) | ||
| and [BloodHound Enterprise repository](https://github.com/SpecterOps/bloodhound-enterprise) | ||
| for their contracts and implementation details. If an endpoint is being | ||
| developed concurrently and is not yet in the published contract, check the | ||
| relevant repository's implementation or mock for the agreed behavior and | ||
| coordinate any contract changes with that work. Update existing OpenHound API | ||
| specifications or endpoint documentation when present; do not invent a new | ||
| specification if the repository has none. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| """OpenHound managed-mode configuration.""" | ||
|
|
||
| from dataclasses import dataclass | ||
|
|
||
| import dlt | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class ModeDefaults: | ||
| """Configuration defaults that differ between OpenHound modes.""" | ||
|
|
||
| log_format: str | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class ManagedBHEConfig: | ||
| """Managed-mode settings for BHE credentials and scheduler polling.""" | ||
|
|
||
| secret_name: str | ||
| poll_interval: int | ||
|
|
||
|
|
||
| _UNMANAGED_DEFAULTS = ModeDefaults(log_format="text") | ||
| _MANAGED_DEFAULTS = ModeDefaults(log_format="json") | ||
| DEFAULT_MANAGED_BHE_POLL_INTERVAL = 30 | ||
| MAX_MANAGED_BHE_POLL_INTERVAL = 599 | ||
|
|
||
|
|
||
| def is_managed() -> bool: | ||
| """Return whether OpenHound is configured to run in managed mode. | ||
|
|
||
| The value is resolved on every call so all platform code observes the current | ||
| DLT configuration. Missing configuration defaults to unmanaged mode. Invalid | ||
| boolean values raise DLT's configuration coercion error. | ||
| """ | ||
|
|
||
| configured = dlt.config.get("openhound.managed", bool) | ||
| return configured if configured is not None else False | ||
|
|
||
|
|
||
| def get_mode_defaults() -> ModeDefaults: | ||
| """Return the defaults profile for the configured mode.""" | ||
|
|
||
| return _MANAGED_DEFAULTS if is_managed() else _UNMANAGED_DEFAULTS | ||
|
|
||
|
|
||
| def get_managed_bhe_config() -> ManagedBHEConfig: | ||
| """Return AWS secret and scheduler settings used by managed BHE clients. | ||
|
|
||
| ``openhound.aws_secrets_manager.secret_name`` is deliberately required in | ||
| managed mode: there is no safe default credential source. The scheduler | ||
| cadence defaults to 30 seconds, matching the historical fixed interval. | ||
| """ | ||
|
|
||
| secret_name = dlt.config.get("openhound.aws_secrets_manager.secret_name", str) | ||
| if not secret_name: | ||
| raise ValueError( | ||
| "Managed mode requires " | ||
| "openhound.aws_secrets_manager.secret_name to be configured" | ||
| ) | ||
|
|
||
| poll_interval = dlt.config.get( | ||
| "openhound.aws_secrets_manager.poll_interval", int | ||
| ) | ||
| if poll_interval is None: | ||
| poll_interval = DEFAULT_MANAGED_BHE_POLL_INTERVAL | ||
| if not 0 < poll_interval <= MAX_MANAGED_BHE_POLL_INTERVAL: | ||
| raise ValueError( | ||
| "openhound.aws_secrets_manager.poll_interval must be between 1 and " | ||
| f"{MAX_MANAGED_BHE_POLL_INTERVAL} seconds" | ||
| ) | ||
|
|
||
| return ManagedBHEConfig( | ||
| secret_name=secret_name, | ||
| poll_interval=poll_interval, | ||
| ) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: SpecterOps/OpenHound
Length of output: 3980
Script executed:
Repository: SpecterOps/OpenHound
Length of output: 1932
Add the other new test modules to CI.
The workflow runs explicitly listed test files and has no catch-all
pyteststep. This PR adds five test modules that no workflow step runs. Add them to CI.Proposed addition
- name: Run managed configuration and AWS profile adapter tests run: | .venv/bin/pytest tests/test_managed_configuration.py tests/test_aws_profile_adapter.py -v + + - name: Run managed client, credential, and dataflow tests + run: | + .venv/bin/pytest tests/test_collector_job_ingest.py tests/test_scheduler_dataflow.py tests/test_bhe_credentials.py tests/test_scheduler_start.py tests/test_config.py -vCommittable suggestion