Shuai Xiang, James Burridge, Shouyang Liu, Hao Lu, Tokihiro Fukatsu, Yinqiang Zheng, and Wei Guo
Paper | Pretrained Weights | Pretraining | MCD-2.6M Dataset | Citation
SPROUT is a diffusion foundation model for multi-crop plant phenotyping. Pretrained on 2.6 million unlabeled open-field images, it supports plant organ segmentation, crop–weed parsing, monocular depth estimation, and yield-organ counting.
This repository provides SPROUT-S/B/L checkpoints, pretraining code, and downstream training and inference code. The MCD-2.6M pretraining dataset is available on Hugging Face.
SPROUT supports four plant phenotyping tasks:
| Task | Evaluated crops or settings |
|---|---|
| Plant organ segmentation | Apple, peach, pear, grape, wheat, and rice |
| Crop–weed parsing | Ten crop-field datasets |
| Monocular depth estimation | Sugar beet |
| Yield-organ counting | Wheat spikes and soybean pods |
MCD-2.6M contains 2,604,450 open-field images spanning crops, growth stages, sites, and field conditions. It is the image corpus used to pretrain SPROUT.
Download the images from Hugging Face. The dataset card lists the sources, licenses, and attribution requirements.
Use Python 3.8 for pretraining. For CUDA 11.6, install PyTorch and torchvision first:
pip install torch==1.13.0+cu116 torchvision==0.14.0+cu116 --extra-index-url https://download.pytorch.org/whl/cu116Install the pretraining dependencies:
pip install -r requirements-pretraining.txtFor downstream tasks, follow the DODA environment setup. Run commands from the repository root.
Download a checkpoint and place it in pretrained_models/. Choose the matching S/B/L configuration when training a downstream model.
| Model | Params | Iterations | Download |
|---|---|---|---|
| SPROUT-S | 51M | 300k | Google Drive |
| SPROUT-B | 112M | 450k | Google Drive |
| SPROUT-L | 361M | 700k | Google Drive |
your_dataset/
├── train/
│ ├── images/ # RGB images (e.g., 001.png, 002.png, ...)
│ └── masks/ # Segmentation masks (same filenames as images)
├── val/
│ ├── images/
│ └── masks/
└── test/
├── images/
└── masks/
- Images and masks must share identical filenames (e.g.,
001.pngin bothimages/andmasks/). - Training images must be square (height == width).
- Masks should contain integer class IDs starting from 0. Use
255as the ignore label.
your_dataset/
├── train/
│ ├── images/ # RGB images
│ └── depth/ # Depth maps (same filenames as images)
└── val/
├── images/
└── depth/
- Images and depth maps must share identical filenames.
- Depth maps should be 16-bit PNG files with values in millimeters (the code divides by 1000 to convert to meters).
- Depth values outside the valid range (
min_depthtomax_depth, in meters) are treated as invalid.
your_dataset/
├── train/
│ ├── images/ # RGB images
│ └── train.txt # Annotation file
├── val/
│ ├── images/
│ └── val.txt
└── test/
├── images/
└── test.txt
List each image filename and its numeric target on one line, separated by a space:
image001.png 42
image002.png 15
image003.png 87
Create a UTF-8 text file with one image path per line:
wheat/001.png
maize/002.jpg
Paths are absolute or relative to --data-root; spaces in filenames are supported. Pretraining uses RGB images without annotations.
Train SPROUT-S with a fresh output directory:
python pretrain.py --config config/sprout-S-pretrain.yaml --image-list datasets/train.txt --data-root datasets/images --output-dir logs/pretrain-S --gpus 1Choose sprout-B-pretrain.yaml or sprout-L-pretrain.yaml for the other model sizes. The default input size is 256 × 256, with batch size 16 per GPU. Set training parameters in the YAML, or pass --batch-size, --max-steps, and --accumulate-grad-batches.
Effective batch size = per-GPU batch size × GPU count × accumulation.
For Linux multi-GPU training with FP16:
python pretrain.py --config config/sprout-L-pretrain.yaml --image-list datasets/train.txt --data-root datasets/images --output-dir logs/pretrain-L --gpus 4 --precision 16The output directory contains:
| File | Contents |
|---|---|
pretraining.yaml / arguments.yaml |
Model and run settings |
metrics/ |
CSV training metrics |
SPROUT-step=*.ckpt |
Checkpoints every 25,000 optimizer steps |
last.ckpt |
Final training checkpoint |
Checkpoints save model weights, EMA, diffusion buffers, the image normalization scale, optimizer, scheduler, and mixed-precision state when enabled.
Resume a run with its original config and output directory:
python pretrain.py --config config/sprout-S-pretrain.yaml --image-list datasets/train.txt --data-root datasets/images --output-dir logs/pretrain-S --resume logs/pretrain-S/last.ckpt --gpus 1--max-steps sets the total target, including completed steps. Lightning 1.4 resumes at an epoch boundary and starts a new data order. Use checkpoints from pretrain.py for full-state resume; use the released pretrained weights to initialize downstream models.
To fine-tune your pretrained model, set weight to its checkpoint and select the matching S/B/L downstream config. Set the downstream script's scale to the value saved in your checkpoint:
checkpoint = torch.load("logs/pretrain-S/last.ckpt", map_location="cpu")
scale = checkpoint["state_dict"]["scale_factor"].item()Set the dataset, checkpoint, model configuration, and training parameters in each script's __main__ block:
| Variable | Description |
|---|---|
dataset_path |
Path to your dataset root directory |
num_classes |
Number of segmentation classes (segmentation only) |
max_number |
Maximum regression target value (regression only) |
min_depth / max_depth |
Valid depth range in meters (depth only) |
weight |
Path to pretrained checkpoint |
configs |
Path to model config YAML |
input_shape |
Input resolution (default: 256 for seg/depth, 384 for regression) |
time_step |
Diffusion timestep (default: 50 for seg/depth, 10 for regression) |
lr |
Learning rate |
batch_size |
Batch size |
total_iters |
Total training iterations |
Set dataset_path, num_classes, weight, and configs in train_segmentation.py, then run:
python train_segmentation.py- Loss: Focal Loss + Lovász Loss
- LR schedule: Cosine with warmup
- Validation: Sliding-window mIoU evaluation
- Checkpoints: Saved to
logs/segmentation/
Set the parameters in train_depth_estimation.py, then run:
python train_depth_estimation.py- Loss: SiLog Loss
- Validation: RMSE and other depth metrics with horizontal flip TTA
- Checkpoints: Saved to
logs/depth/
Set the parameters in train_regression.py, then run:
python train_regression.py- Loss: MSE Loss
- Output mapping:
sigmoid(output) * max_number - Validation: R², MAE, MSE, RMSE, MAPE
- Checkpoints: Saved to
logs/regression/
Set dataset_path, weight, configs, and num_classes in segmentation_ms_inference.py, then run:
python segmentation_ms_inference.py- Uses multi-scale inference at scales
[0.75, 1.0, 1.25]with sliding window (window_size=256,step_size=128). - Applies horizontal flip TTA (test-time augmentation).
- Reports mIoU on the test set.
Set dataset_path, weight, configs, and max_number in regression_inference.py, then run:
python regression_inference.py- Applies horizontal flip TTA.
- Reports R², MAE, MSE, and RMSE on the test set.
Set weight, min_depth, and max_depth in visiualize_depth.py, then run:
python visiualize_depth.py- Place input images in
visualization/depth/input/. - Colorized depth maps (rainbow colormap) are saved to
visualization/depth/output/{model_name}/.
Cite SPROUT when using the models, code, or MCD-2.6M dataset:
@article{xiang2026sprout,
title={SPROUT: A Scalable Diffusion Foundation Model for Multi-Crop Plant Phenotyping},
author={Xiang, Shuai and Burridge, James and Liu, Shouyang and Lu, Hao and Fukatsu, Tokihiro and Zheng, Yinqiang and Guo, Wei},
journal={arXiv preprint arXiv:2603.27519},
year={2026},
doi={10.48550/arXiv.2603.27519},
url={https://arxiv.org/abs/2603.27519}
}