[Irozuku]

Summary

Adds translation converters that wrap pretrained Opus MT models from Hugging Face, one converter per model. Each converter has its own download and works on Text columns of datasets and on Text items of documents. Table items are translated cell by cell.

Each converter has an optional Translate by parameter, so a document item that holds a lot of text (for example a whole file loaded as a single item) can be translated in parts:

• Whole value (default): each value is translated as one piece. Use it for text that is already split by its structure.
• Paragraphs: the text is split at blank lines.
• Lines: each line is translated on its own.
• Sentences: each sentence is translated on its own, including sentences that end in Chinese punctuation (。！？).

Line breaks, spacing and lines without letters (such as ----- underlines) are kept exactly as they were. Values that are missing, empty or have no letters never reach the model.

In every mode, a part longer than 128 tokens is cut further: first by lines, then by sentences (each translated on its own), and a single sentence that is still too long is cut between words. Without this, Opus MT silently skipped sentences of long inputs while testing (about 250 tokens).

---

Type of Change

Check all that apply like this [x]:

• Backend change
• Frontend change
• CI / Workflow change
• Build / Packaging change
• Bug fix
• Documentation

---

Changes (by file)

• DashAI/back/converters/hugging_face/translation/base.py: TranslationConverter base. It loads the tokenizer and model on first use, has the split_by, batch_size and device options, and does the splitting, batching (by length, to reduce padding) and reassembly. Generated length is capped relative to the input so a degenerate output cannot run away. Subclasses can override source_prefix, prepare_tokenizer and generate_options for models that need language tokens.

• DashAI/back/converters/hugging_face/translation/opus_mt.py: the 10 Opus MT converters:

  • English to Spanish, Spanish to English
  • English to German, German to English
  • English to French, French to English (skips the duplicate .bin weights, the repo also ships safetensors)
  • Chinese to English
  • Romance languages to English
  • English to Romance languages, with a target language dropdown (Portuguese, Spanish, French, Italian, Romanian, Catalan, Galician)
  • English to Chinese, with a target language dropdown (Simplified, Traditional, Cantonese)

  The last two put the model's >>id<< target token in front of every input. Display names and descriptions are translated to all UI languages.

• DashAI/back/converters/hugging_face/translation/__init__.py: exports the converters and TRANSLATION_CONVERTERS.

• DashAI/back/initial_components.py: registers TRANSLATION_CONVERTERS.

• tests/back/converters/test_translation_converters.py: tests for the splitting modes, exact reassembly in every mode, the fallback cuts, missing and letterless values, documents (including a table item), target tokens and download metadata.

• tests/back/test_component_metadata_contract.py: adds the translators to the list of tools offered on documents.

• tests/back/preprocessing/test_structure_matches_runtime.py: adds the translators to the converters that need a download.

---

Testing (optional)

1. Open a document notebook (or a dataset notebook with a Text column).
2. Add Translator English to Spanish (Opus MT) and download its model.
3. Run it on a long multi line text item with Translate by set to Lines or Sentences. Line breaks should be kept and no sentence should be missing.

---

Notes (optional)

• In Whole value mode, a value that has several lines is sent as one piece, and Opus MT drops line breaks inside a piece. A line without letters inside that piece can also make the model repeat characters and lose text. Lines or Sentences avoid this.