Repository navigation
fix(cli): use the resolved token and the real API in pysus dadosgov download - #355
Merged
Merged
Conversation
…download`
Three defects made the command fail on every invocation.
The resolved token was thrown away:
_get_token(token) # resolves --token or DADOSGOV_TOKEN
...
await dg.connect(token=token) # still the raw option, usually None
`connect()` falls back to a previously stored token and raises
`AuthenticationError` when there is none, so the documented
`export DADOSGOV_TOKEN=...` workflow died with "A token is required to
connect to DadosGov" even though the variable was set. Verified: with
DADOSGOV_TOKEN=env_token the old code called `connect(token=None)`.
`Dataset.get_files()` does not exist. `Dataset` is a
`BaseRemoteDataset`, whose files are reached through the
`content`/`search` contract that recurses into groups; `get_files`
appears nowhere in the codebase. Replaced with `await target.search()`.
`download()` was handed the output *directory*:
await dg.download(f, out_dir)
The client opens its second argument with `open(output, "wb")`, so this
only fails once a file is actually transferred (`IsADirectoryError`).
The path is now `out_dir / basename`, taken from the URL with
`PurePosixPath` since dados.gov.br paths always use forward slashes.
Filters also compare against `str(f.path)`, matching the FTP CLI.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #355 +/- ##
=======================================
Coverage ? 97.21%
=======================================
Files ? 180
Lines ? 22760
Branches ? 0
=======================================
Hits ? 22127
Misses ? 633
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
🎉 This PR is included in version 2.11.6 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pysus dadosgov downloadfailed on every invocation, for three separate reasons.1. The resolved token was discarded
connect()falls back to a previously stored token and raisesAuthenticationErrorwhen there is none. The documented workflow therefore died withA token is required to connect to DadosGoveven though the variable was set.Verified directly against the old code — with
DADOSGOV_TOKEN=env_tokenit calledconnect(token=None).2.
Dataset.get_files()does not existDatasetis aBaseRemoteDataset(pysus/api/dadosgov/models.py:359); its files are reached through thecontent/searchcontract that recurses into groups.get_filesappears nowhere in the codebase, so the command died withAttributeError: 'SINAN' object has no attribute 'get_files'. Replaced withawait target.search().3.
download()was handed the output directoryThe client opens its second argument as a file, so this raises
IsADirectoryErroras soon as a real transfer starts. The path is nowout_dir / basename, taken withPurePosixPathsince dados.gov.br paths always use forward slashes.Filters also compare against
str(f.path), matching the FTP CLI.Impact
The command is documented in the CLI help and the README but has never worked: it either raises
AuthenticationError,AttributeError, or — once both are fixed —IsADirectoryError.Tests
The existing
test_download_bad_slugset the env var but asserted only on the slug lookup, so execution never reached the client. Added 5 tests with a fake async client:DADOSGOV_TOKENis forwarded toconnect()(token='env_token')--tokentakes precedence over the env varout_dir, not the directory itself--yearfilter reaches the download stageAll 5 fail against the previous code. Confirmed the token defect independently by driving the old command with a stubbed client:
connect(token=None)before,connect(token='env_token')after.1738 passed, 6 skippedblackandisortclean