[Irozuku]

Summary

DocumentJob is now built from atomic units, like the dataset, converter and explainer jobs. The source file can come from an upload or from a completed hub download, so files downloaded through the Datafile Hub can be imported as documents. The hub import page lists document extractors next to data loaders, grouped by category the same way as the Upload data page.

Based on feat/rag-vectorizer-converters.

---

Type of Change

Check all that apply like this [x]:

• Backend change
• Frontend change
• CI / Workflow change
• Build / Packaging change
• Bug fix
• Documentation

---

Changes (by file)

Backend

• DashAI/back/units/store_uploaded_document_source_unit.py: new unit that copies an upload into the document's source/ folder and publishes source_path and file_hash. upload_path and file_name are runtime params.
• DashAI/back/units/store_datafile_document_source_unit.py: new unit that does the same for a file inside a completed hub download (datafile_id, optional selected_file). The copy keeps the document independent of the download.
• DashAI/back/units/parse_document_unit.py: new unit that parses source_path with a document loader and publishes document. Its validate rejects a missing loader or a dataset loader before anything is copied.
• DashAI/back/units/save_document_to_path_unit.py: new unit that writes document to a given directory, the document counterpart of SaveDatasetToPathUnit.
• DashAI/back/units/document_source.py: helpers shared by the two source units (copy plus sha256, shared path schema field).
• DashAI/back/units/datafile_source.py: hub download lookup and file resolution moved out of LoadDatafileDatasetUnit so both datafile units share it. Error messages are unchanged.
• DashAI/back/units/load_datafile_dataset_unit.py: uses the shared helper.
• DashAI/back/job/document_job.py: run now composes the units. With datafile_id in kwargs the source comes from the hub download, otherwise from the upload. Status hooks, cancel cleanup and row bookkeeping are unchanged.
• DashAI/back/initial_components.py: registers the four new units.

Frontend

• DashAI/front/src/api/documents.ts: enqueueHubDocumentJob posts a DocumentJob to /v1/job/ with datafile_id and selected_file and no file.
• DashAI/front/src/components/hub/HubImportPanel.jsx: lists dataset loaders and document extractors grouped by category. Picking an extractor replaces the preview step with a name field, creates the document and enqueues the hub job. Shows an error and blocks import when the extractor does not support the selected file, and reloads the file list when the page is opened past the file step.
• DashAI/front/src/pages/hub/HubImportPage.jsx: no longer filters out document loaders, shows DocumentLoaderConfigBar in the right panel for extractors, and opens the new document after import.
• DashAI/front/src/utils/i18n/locales/*/hub.json: documentName, importDocument, stepDocumentTitle, stepDocumentSubtitle in en, es, pt, de and zh.

Tests

• tests/back/units/test_document_ingest_units.py: contract tests for the four units.
• tests/back/api/test_document_job.py: hub import, download not ready, and enqueueing through /api/v1/job/ without a file.
• tests/back/units/test_unit_contracts.py, tests/back/api/test_units_api.py: new units, schema fields and runtime params added to the audits.

---

Testing (optional)

1. Download a hub dataset that contains a text or PDF file, open it from the Datafiles list and pick the file.
2. On the loader step, check that "Document Extractor" appears as its own group.
3. Pick an extractor, set a name and import. The document should open and finish parsing.
4. Pick an extractor that does not support the selected file and check that import is blocked with the accepted extensions.
5. Upload a document from Upload data to check the upload path still works.

---

Notes (optional)

• Hub document imports reuse the generic job endpoint instead of adding one under /dataset-source, since only the datafile is needed, not the source.