[Irozuku]

Summary

RAG chunks now use the same structure as every other document, and the vectors a retriever uses now come from regular converters.

• Chunks are DashAIDocuments. A chunker takes the parsed DashAIDocument and returns a chunked one, with one "chunk" item per chunk. The item records the pages and source items it covers. Chunk sets are saved as Arrow files under <RAG_PATH>/chunk_sets/<chunk_set>/<document>, and the SQL chunk table, which stored a copy of the text for every chunk set, is dropped. Chunkers are RAG components, not converters, so they do not appear in notebooks.
• Dense embeddings are converters. There are 14 text embedding converters, one per pretrained model, so each one downloads on its own. They replace both models/RAG/embeddings/ and the old Embedding converter. They are built on sentence_transformers, which applies each model's own pooling, dense layers, token limit and query prompt. This fixes wrong embeddings in the previous RAG code: LaBSE and Qwen3 pooling, the 128 token limit of the paraphrase models, and the missing Qwen3 and harrier query prompts. They show in dataset notebooks only, never in document notebooks.
• Sparse vectorizers are converters. TFIDFConverter and BagOfWordsConverter gain the options the RAG vectorizers had. BM25 and TF-IDF retrievers now take them through a vectorizer component field, and TFIDFVectorizerModel, BM25VectorizerModel and BaseVectorizer are removed.

---

Type of Change

Check all that apply like this [x]:

• Backend change
• Frontend change
• CI / Workflow change
• Build / Packaging change
• Bug fix
• Documentation

---

Changes (by file)

Chunking and storage

• DashAI/back/dataloaders/classes/document_elements.py: new "chunk" element kind with pages and sources attributes.
• DashAI/back/models/RAG/chunking_models/base_chunking_model.py: chunk(document) -> DashAIDocument replaces the old documents kwarg and chunk_document(s) API. Image items are ignored, and pages come from the items each chunk covers.
• DashAI/back/models/RAG/chunking_models/chunking_model_factory.py, DashAI/back/models/RAG/RAG_models_factory.py: build a chunker without documents.
• DashAI/back/services/RAG/chunking_service.py:
  • create() writes one chunked document per source with save_document, and skips documents already stored.
  • read_chunks() rebuilds the Chunk objects. Chunk.id is now the ordinal of the chunk inside its chunk set.
• DashAI/back/models/RAG/documents/source_document.py (new) replaces rag_document.py (removed). DocumentService.load returns SourceDocument(id, file_name, content).
• DashAI/back/dependencies/database/models.py: Chunk model removed. RAGChunkSetDocument gains path and n_chunks.
• DashAI/alembic/versions/d8f3a1c6e2b9_store_rag_chunks_as_documents.py: drops chunk, adds the two columns and detaches existing chunk sets from their session. The existing startup cleanup then deletes them together with their retrievers and files. The migration does no filesystem I/O.
• DashAI/back/services/RAG/cleanup_service.py, index_status_service.py, setup_service.py, DashAI/back/models/RAG/RAG_pipeline.py: use the chunk folders and n_chunks.

Dense embeddings

• DashAI/back/converters/hugging_face/text_embedding/ (new):
  • TextEmbeddingConverter base with encode(texts, role="passage" | "query"). Notebook transform produces emb_{column}_{i} columns.
  • Families: Sentence Transformers, E5 (query: / passage: prefixes) and Instructor (instruction field).
  • The 14 models: all-MiniLM-L6-v2, all-mpnet-base-v2, multi-qa MiniLM and mpnet, paraphrase-multilingual MiniLM and mpnet, LaBSE, multilingual-e5 small, base and large, Qwen3-Embedding-0.6B, harrier-oss-v1 270m and 0.6b, instructor-large.
• DashAI/back/converters/hugging_face/embedding.py and DashAI/back/models/RAG/embeddings/ removed.
• DashAI/back/models/RAG/retrievers/dense/dense_retriever.py, dense_embedding_retriever.py: embedding_model is component_field(parent="TextEmbeddingConverter"). Passages are encoded with the passage role and queries with the query role. Documents without chunks are skipped. huggingface_dense_retriever.py removed.
• DashAI/back/dependencies/registry/component_registry.py: a class whose own name contains Base (for example AllMpnetBaseV2Embedding) is no longer counted as its own base type.
• DashAI/back/services/RAG/retriever_presets.py: the semantic preset uses the paraphrase-multilingual-MiniLM-L12-v2 converter.

Sparse vectorizers

• DashAI/back/converters/scikit_learn/text_vectorizer.py (new):
  • Shared schema with max_features (empty means no limit), strip_accents, analyzer, stop_words plus custom_stop_words, n-gram bounds, and min_df / max_df with an ordering rule.
  • SparseTextVectorizer base with fit_matrix / transform_matrix.
• DashAI/back/converters/scikit_learn/tf_idf.py, bag_of_words.py: built on that base. TF-IDF adds norm, use_idf, smooth_idf and sublinear_tf.
• DashAI/back/models/RAG/retrievers/sparse/bm25_retriever.py, tfidf_retriever.py:
  • They take BagOfWordsConverter / TFIDFConverter.
  • Their saved files hold a row to chunk id map, not the chunks, so chunk text is no longer duplicated in the pickles.
  • The BM25 weighting is unchanged.
• DashAI/back/models/RAG/retrievers/sparse/base_vectorizer.py removed.

API, jobs and frontend

• DashAI/back/api/api_v1/endpoints/rag.py, DashAI/back/job/RAG_index_job.py: a session whose stored retriever uses removed components gets a 422 asking to choose a retriever again, instead of a generic indexing error.
• DashAI/back/api/api_v1/endpoints/generative_session.py: updating RAG session parameters now checks that the selected components are downloaded, as session creation already did, and returns a 409 otherwise.
• DashAI/front/src/pages/generative/RAGSession/advanced/RetrieverConfigurationStep.jsx, CompositeRetrieverBuilder.jsx, retrieverKinds.js: list embeddings with parent TextEmbeddingConverter.
• DashAI/front/src/utils/schema.js: nested component forms apply the placeholder params of the parent field. For example, a BM25 vectorizer starts with no max_features limit.

Tests

• New: test_document_chunking.py, test_chunk_storage.py, test_dense_retriever_converters.py, test_sparse_retriever_converters.py, test_RAG_update_download_gate.py, tests/back/converters/test_text_embedding_converters.py, test_text_vectorizers.py and a fake model in text_embedding_stub.py, so no test downloads a model.
• Removed with the code they covered: test_rag_document.py, test_chunk_pages.py, test_chunking_service.py, test_chunking_query_count.py, test_chunking_cache_integration.py.
• Existing RAG, registry, contract and session API tests updated to the new component names.

---

Testing (optional)

1. Start the app on a database with RAG sessions indexed before this branch. Re-index one keyword session and one semantic session: both should ask to choose a retriever again.
2. Create a session with the semantic preset while its embedding is not downloaded: you should get the download prompt.
3. Download and use instructor-large and one harrier model, index, and ask a question that cites a page.
4. Build a BM25 retriever in the advanced step: max_features starts empty.
5. Dataset notebook with a text column: the 14 embeddings, TF-IDF and Bag of Words are listed and produce columns. Document notebook: no embedding converter is listed.

---

Notes (optional)

• No data migration. Old indexes are deleted at startup, and old sessions must pick their retriever again. Stored retriever params still use the old component names.
• The old Embedding converter is removed. Notebook history that used it cannot be replayed.
• Embeddings change: LaBSE, Qwen3 and the paraphrase models now produce embeddings that match their official definitions, so the vectors differ from the ones the previous RAG code produced.
• Null text in TF-IDF and Bag of Words: a null value becomes an empty string. Before, it became the token none.