Selected work · 13
Mars
A controlled crater-detection benchmark that keeps the failures, data lineage, and model-selection boundary visible.
Explore the engineering ↓
- My role
- Evaluation pipeline · dataset audits · controlled model experiment
- Status
- Locked Faster R-CNN baseline
- Focus
- Geospatial data · ML · reproducibility
Result & scope
Locked test: 6 true positives, 10 false positives, and 8 missed craters at score 0.8 and IoU 0.5. AP50 = 0.263; this is a small-data baseline, not a production catalog.
- 01
Prepare
Project catalog labels into 1024 px CTX tiles.
- 02
Separate
Audit geographic overlap and crater identities across splits.
- 03
Select
Choose the model and threshold using validation only.
- 04
Lock & test
Freeze the decision, evaluate once, and retain error cases.
Evidence from the project
Inside the work.
Catalog-to-tile labels provide the evaluation target; scale comes from raster metadata.
Source-produced pipeline graphic showing how imagery becomes evaluated detections.
Training histories are evidence for validation selection, not an invitation to tune on test results.
Validation-only sweep used to lock the operating threshold.
Matched detections from the locked evaluation.
Circular-looking terrain and other false alarms remain part of the result.
Ground-truth examples that the locked model did not recover.
Source counts: small (<150 m), 3/3 matched; medium (150–300 m), 3/10; large (>300 m), 0/1. Sparse bins limit interpretation.
A single overview of the dataset, pipeline, evaluation, and representative errors.
The supplied project plate introduces the controlled crater-detection experiment.
Model choice is shown as part of the controlled baseline rather than a claim of production readiness.
The held-out result preserves the modest precision and recall instead of hiding negative evidence.
The source imagery establishes the visual conditions presented to the detector.
Annotations show how predicted candidates align with the source scene.
The supplied interface connects model output to human inspection.
A controlled comparison keeps evaluation conditions fixed across outputs.
| Measure | Result |
|---|---|
| True positives / false positives / misses | 6 / 10 / 8 |
| Precision / recall | 0.375 / 0.429 |
| F1 / AP50 | 0.400 / 0.263 |
| Median matched IoU | 0.663 |
My technical contribution
I built the evaluation and evidence layer around preserved research notebooks: catalog audits, affine label generation, geographic splits, map-level deduplication, threshold sweeps, and traceable run manifests. I designed and executed the isolated S2 Experiment 3 comparison with validation/test separation.
Dataset and leakage control
The broader benchmark contains 2,809 generated tiles and 1,683 catalog craters. CTX imagery is divided into 1024 × 1024 tiles with 128 px overlap and 896 px step; catalog diameters are converted using raster scale, not screenshot measurements.
The geographic audit records zero cross-split overlapping tile pairs and zero crater IDs shared across splits. The broad benchmark and the smaller controlled Experiment 3 test are different evaluation scopes; their counts must not be combined.
Why the first baseline failed usefully
The one-epoch YOLO11n smoke test exercised the complete pipeline but returned 0 true positives, 53,878 post-deduplication false positives, and 143 false negatives at IoU 0.5. It established execution, not scientific usefulness. That result motivated a smaller controlled experiment.
Model and validation-only selection
The selected model is torchvision Faster R-CNN ResNet-50 FPN v2, initialized from COCO weights with background/crater classes. Training uses SGD (learning rate 0.002, momentum 0.9), balanced batches of two positive and two negative tiles, flips, right-angle rotations, and contrast/gamma jitter.
Candidate A used default resize and anchors; Candidate B used fixed 1024 px input and smaller anchors. A’s best validation AP50 was 0.292 at epoch 11. The checkpoint and score threshold 0.8 were locked before final test inference.
Results and representative errors
At IoU 0.5, the locked test gives precision 0.375, recall 0.429, F1 0.400, AP50 0.263, and median matched IoU 0.663. Seven false positives occur on negative tiles and three on positive tiles.
The gallery includes actual matches, false alarms, and missed ground truth. Low-contrast or degraded craters and circular terrain remain difficult. The small sample does not establish generalization to all Martian terrain.
Reproducibility and next experiment
The archive includes configuration, frozen split and selection records, checkpoint identity, training history, metric tables, and a 14-test pipeline result. The large source GeoTIFF is referenced by provenance/hash rather than duplicated.
The next experiment should expand geographically separated labels and hard negatives, keep this result frozen, and evaluate on a new untouched test set. Reusing the existing test set for tuning would weaken the evidence.
Code & methodology
Read the source
Technical records.
About the evidence
Locked model card; Mars CTX case study; Experiment 3 research brief; saved metrics, training history, and pipeline-test record. The broad benchmark and controlled test have different scopes.
Continue exploring.
← Back to Selected Builds