Course project · Deep learning for computer vision
Contents
Contents
Turn the pages with the arrows below or the ← → keys. M picks up the magnifying glass; + / − zoom.
A sketchbook of ripening fruit
Grading ripeness, in order
Ripeness is a scale, not a set of unrelated labels. We train a CNN with an ordinal-regression loss (CORAL) and compare it with a standard cross-entropy classifier on identical splits.
Plate I — four grades of tomato and banana, from unripe to overripe.
Plate II
The problem with plain classes
A grader that calls an unripe tomato overripe has made a far worse mistake than one that calls it turning. Ordinary multi-class training cannot see that: cross-entropy only asks “was the right box ticked?”, so every wrong answer costs the same.
Ripeness grades have a natural order — Unripe < Turning < Ripe < Overripe — and an ordinal model uses it. It learns how far along the fruit is, so its mistakes tend to land next to the truth.
Objectives
- Build a 4-grade ripeness dataset for banana and tomato from Fruits-360, extended with a real-world banana subset and a workflow for self-collected photos.
- Train the same ResNet-18 twice: with a cross-entropy head and with a CORAL ordinal head.
- Compare them on identical splits with accuracy and MAE, plus severe-error rate and quadratic-weighted kappa.
- Ship a working demo that grades a photo with both models.
What a mistake costs
Cross-entropy
Every error costs 1.
Ordinal
Cost grows with distance |true − predicted|.
Two numbers to judge a grader
Accuracy = share of images graded exactly right
MAE = mean | true grade − predicted grade |
MAE is measured in grades: 0.10 means the average prediction is a tenth of a grade off. Two models can have the same accuracy and very different MAE.
Plate III
The specimens
| Source | What it adds |
|---|---|
| Fruits-360 | Studio photos of whole fruit rotating on a white ground. Folders mapped to a grade by colour and condition (e.g. Tomato 11 → Unripe, Tomato 9 → Overripe). Cultivar colours (yellow, maroon tomatoes) are excluded. |
| Real-world bananas | 240 phone photos with captions (“peel partially green”, “rotten”) mapped to grades by keyword rules; 12 ambiguous captions dropped. |
| Self-collected | Not in these results yet (0 photos). Plate IX prepares labelled photos for data/self_collected/; the next run splits them 70/15/15. |
Three test sets, three questions
- Seen varieties — new frames of cultivars seen in training.
- Unseen varieties — four cultivars held out entirely (Tomato 5, 8, 10, Banana 2).
- Real-world bananas — hand-held photos, cluttered backgrounds.
Fruits-360 has no green bananas — the reason a self-collected subset is part of the brief.
Images per split
A tray of test specimens
A tray of test specimens
Click a specimen to grade it on the bench (Plate V). Pick up the lens to look closer.
Plate IV
One backbone, two heads
Both models share everything except the last layer and the loss: an ImageNet-pretrained ResNet-18, 112 × 112 inputs, AdamW (lr 3·10⁻⁴, 10× for the new head, cosine schedule), 8 epochs, batch 32, grade-balanced sampling, and the same augmentations, seeds and splits.
Augmentation, with one rule
Random crops, flips, ±20° rotation, brightness and contrast jitter — but no hue jitter. Colour is the ripeness signal; shifting the hue would turn a green tomato red and keep the label “Unripe”.
Baseline · cross-entropy
4 logits → softmax → pick the largest.
L = − log py
Grades are treated as unrelated categories.
Ordinal · CORAL
Three yes/no questions: is the grade > 0? > 1? > 2? All three share one weight vector w and differ only in a bias bk:
P(y > k) = σ( w·x + bk )
A “Ripe” label (grade 2) becomes the targets [1, 1, 0]. The loss is the sum of three binary cross-entropies, so predicting Overripe for a Turning fruit gets two questions wrong, not one.
ŷ = Σk 1[ P(y > k) > 0.5 ]
Sharing w keeps the answers rank-consistent: P(y>0) ≥ P(y>1) ≥ P(y>2), so the model can never say “not past Turning, but past Ripe”.
Plate V
The grading bench
Drop a banana or tomato here
or click to choose a photo · or paste an image
…or take one from the tray
Both models grade the same photo. The ripeness score is the expected grade, Σ k · P(k), a continuous 0–3 reading of how far along the fruit is.
No specimen on the bench yet.
UnripeTurningRipeOverripe
Plate VI
What the tests say
Results appear after bash run_experiments.sh.
Mean ± std over seeds, per test set
Cross-entropyCORAL ordinal
Plate VII
Where the errors land
Rows are the true grade, columns the prediction; the outlined diagonal is “exactly right”. For an ordinal task the question is not only how many errors, but how far from the diagonal they fall.
Cells far from the diagonal — Unripe called Ripe, Turning called Overripe — are the costly mistakes. Hover a cell for counts.
Error distance
Cross-entropy
CORAL ordinal
All three test sets pooled.
Plate VIII
Training notes
Validation MAE per epoch. The checkpoint with the lowest validation MAE is kept for both heads, so neither gets a selection advantage. Hover the chart to read an epoch.
Cross-entropyCORAL ordinalbold: seed 42 · faint: other seeds
Where the two graders disagree
Seed 42: unseen-variety and real-world images where the heads disagree, one per variety. Both models get some of these wrong.
Plate IX
Field notes
Limitations
- Fruits-360 frames come from a few physical fruits each; “seen variety” scores are optimistic.
- Grades come from folder and caption rules, not from a sugar or firmness measurement.
- No green banana exists in either public source, so these models have never seen an unripe banana and no self-collected photos are included yet.
- Validation frames come from the same fruits as training, so checkpoint selection cannot detect poor generalisation to new cultivars.
- The two sources label spotted bananas differently (some “Ripe” phone photos look like “Overripe” studio ones).
Next steps
- Collect 50+ photos per fruit and grade under mixed light; re-run run_experiments.sh.
- Try CORN (conditional ordinal) and a fruit-type head; calibrate the score against days-to-ripe.
Add to the sketchbook
Self-collected subset
Photograph a fruit, pick its grade, and save it. Photos land in data/self_collected/<fruit>/<grade>/ and join the next training run (70% train, 15% val, 15% real-world test).
Labelling guide — Unripe: no yellow/red at all. Turning: colour change has started but green remains. Ripe: full colour, firm. Overripe: dark patches, wrinkles, rot or many brown spots.
Image sources. Fruits-360 by Mihai Oltean (CC BY-SA 4.0), github.com/fruits-360. Real-world bananas: josedonoso/bananas-dataset-v1 on Hugging Face (no licence stated). Used here for a course project.