Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,42 @@ poetry run python manage.py runserver 0.0.0.0:8000

That last path will not start Celery/Redis for you.

#### Verified: run the training backend locally without AWS (2026-10-25)

The Django + Celery + Redis stack **boots without AWS credentials**. AWS is only
reached at two runtime edges: Firebase auth via Secrets Manager (`authenticator.py`
→ `cli.py init_firebase()`) and saving results to S3 (`worker.py`). Booting and the
unauthenticated surface need neither.

```sh
mkdir -p ~/.aws # lets the $HOME/.aws bind mount succeed even with no AWS
cd training
docker compose up --build
```

Verify it is up:

```sh
curl http://localhost:8000/api/test # -> {"result": "200 Backend surface test successful"}
docker compose exec redis redis-cli ping # -> PONG
docker compose exec web mamba run -n dlp poetry run pytest tests
```

Expected: `/api/test` returns 200, Redis replies `PONG`, and pytest reports
`17 passed, 1 failed`. The single failure (`test_columns_endpoint[CALIFORNIA_HOUSING]`)
is an environment gap, not a pipeline bug: `/home/app` is root-owned while the
container runs as user `app`, so `fetch_california_housing` cannot write its download
cache. Run `pytest tests -k "not CALIFORNIA_HOUSING"` for an all-green run.

Two fixes on a fresh checkout were needed to reach this state (both in this branch):
the unpinned `pytorch`/`torchvision` in `environment.yml` resolve to a build that is
incompatible with Python 3.9 (`import torch` → `ImportError: cannot import name
'TypeIs'`), so they are pinned to `pytorch=2.5.1` / `torchvision=0.20.1`; and the
training `Dockerfile`s needed `README.md` available before `poetry install`.

A full end-to-end training job (`POST /api/tabular`) still requires AWS for Firebase
auth and the S3 result save, so it cannot complete on a no-AWS setup.

### 3. Flask API (`backend/`)

```sh
Expand Down
4 changes: 2 additions & 2 deletions training/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ WORKDIR /usr/src/build
# install dependencies
RUN mamba create --name dlp -y

COPY environment.yml pyproject.toml poetry.lock ./
COPY environment.yml pyproject.toml poetry.lock README.md ./
RUN mamba run --live-stream -n dlp mamba env update --file environment.yml --prune

# create directory for the app user
Expand All @@ -25,7 +25,7 @@ RUN mkdir $APP_HOME
FROM base AS development
ENV ENVIRONMENT development

RUN mamba run --live-stream -n dlp poetry install
RUN mamba run --live-stream -n dlp poetry install --no-root


# we previously mounted training/ to APP_HOME
Expand Down
4 changes: 2 additions & 2 deletions training/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ channels:
dependencies:
- python=3.9
- poetry
- pytorch
- torchvision
- pytorch=2.5.1

@karkir0003 karkir0003 Oct 27, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought for a followup issue (might not be the most urgent thing):

when i originally created DLP, I used an environment.yml setup to make compatible for mamba, but now that uv has been around for a while and the package manager is faster + simpler to use, have we considered taking a followup to use uv for the environment config

https://docs.astral.sh/uv/ @Comment-XD

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for the late response. Yes, this is something we can implement to make package installation faster.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good. @Comment-XD can you create a followup issue for that

- torchvision=0.20.1
7 changes: 4 additions & 3 deletions training/training/core/celery/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ WORKDIR /usr/src/build
# install dependencies
RUN mamba create --name dlp -y

COPY environment.yml pyproject.toml poetry.lock ./
COPY environment.yml pyproject.toml poetry.lock README.md ./
# This is to just see if the readme file exists

RUN mamba run --live-stream -n dlp mamba env update --file environment.yml --prune

# create directory for the app user
Expand All @@ -20,12 +22,11 @@ ARG APP_HOME=/home/app/training
RUN mkdir "$APP_HOME"



######################## development target ########################
FROM base AS development
ENV ENVIRONMENT development

RUN mamba run --live-stream -n dlp poetry install
RUN mamba run --live-stream -n dlp poetry install --no-root


# we previously mounted training/ to APP_HOME
Expand Down
Loading