1 · Headline results (out-of-fold)
| Model | ROC-AUC [95 % CI] | PR-AUC [95 % CI] | TSS [95 % CI] | Balanced acc. | Kappa | Brier [95 % CI] | ECE | Fold AUC mean±SD |
|---|---|---|---|---|---|---|---|---|
| RF | 0.956 [0.935–0.971] | 0.989 [0.980–0.996] | 0.749 [0.697–0.801] | 0.875 | 0.711 | 0.064 [0.044–0.083] | 0.052 | 0.958 ± 0.026 |
| XGBoost | 0.956 [0.936–0.974] | 0.990 [0.980–0.997] | 0.771 [0.718–0.824] | 0.886 | 0.713 | 0.065 [0.044–0.086] | 0.051 | 0.957 ± 0.028 |
Pooled out-of-fold predictions: every record is predicted by a model that never saw it (or, for spatial CV, its neighbourhood) during fitting. Confidence intervals come from a cluster bootstrap over spatial groups. Thresholded metrics use probability ≥ 0.50.
Warnings and notes from this run
- Predictor 'distance_to_river1': 7 missing value(s) (median-imputed inside every fold).
- Predictor 'distance_to_road1': 7 missing value(s) (median-imputed inside every fold).
- Predictor 'elevation_srtm_30m_clipped': 3 missing value(s) (median-imputed inside every fold).
- Predictor 'slope_degree_30m_clipped': 3 missing value(s) (median-imputed inside every fold).
- RF: residuals are still spatially clustered (Moran I = 0.35, p = 0.002); important spatial structure may be missing from the predictors.
- XGBoost: residuals are still spatially clustered (Moran I = 0.32, p = 0.002); important spatial structure may be missing from the predictors.
2 · Study design
- Cross-validation
- spatial (5 folds)
- Validation design
- Every outer fold fits NEW preprocessing (imputation, scaling, one-hot) and a NEW model on its training rows only; the deployment model is refitted on all rows and never evaluated. Hyper-parameters were tuned by inner CV inside each outer training set (nested CV).
- Block size
- 1,073 m
- Class weighting
- balanced
- Predictors
- distance_to_river1, distance_to_road1, elevation_srtm_30m_clipped, ndvi_s2_10m, slope_degree_30m_clipped
- Normalisation
- distance_to_river1: reverse, distance_to_road1: none, elevation_srtm_30m_clipped: reverse, ndvi_s2_10m: reverse, slope_degree_30m_clipped: reverse
- Block-size advice
- 1,073 m – median autocorrelation range of 4 variable(s); capped at 1,073 to leave enough blocks for 5 folds
| Fold | Training records | Validation records | Validation flood |
|---|---|---|---|
| 1 | 3157 | 751 | 629 |
| 2 | 3113 | 795 | 668 |
| 3 | 3127 | 781 | 631 |
| 4 | 3211 | 697 | 577 |
| 5 | 3024 | 884 | 732 |
3.1 · RF
Interpretation and spatial diagnostics
Per-fold metrics
fold,n_train,n_validation,roc_auc,pr_auc,balanced_accuracy,tss,brier 1,3157,751,0.9837,0.9968,0.9089,0.8177,0.0400 2,3113,795,0.9178,0.9800,0.8405,0.6811,0.0771 3,3127,781,0.9671,0.9922,0.8850,0.7699,0.0585 4,3211,697,0.9723,0.9942,0.8709,0.7417,0.0510 5,3024,884,0.9506,0.9893,0.8679,0.7358,0.0861
Held-out permutation importance (table)
feature,importance_mean,importance_sd,n_folds ndvi_s2_10m,0.13248,0.02626,5 distance_to_river1,0.11959,0.02165,5 elevation_srtm_30m_clipped,0.02628,0.01938,5 distance_to_road1,0.00415,0.00380,5 slope_degree_30m_clipped,0.00294,0.00172,5
3.2 · XGBoost
Interpretation and spatial diagnostics
Per-fold metrics
fold,n_train,n_validation,roc_auc,pr_auc,balanced_accuracy,tss,brier 1,3157,751,0.9790,0.9958,0.9051,0.8103,0.0520 2,3113,795,0.9082,0.9774,0.8323,0.6646,0.0847 3,3127,781,0.9634,0.9914,0.8902,0.7804,0.0647 4,3211,697,0.9637,0.9921,0.8754,0.7508,0.0603 5,3024,884,0.9695,0.9937,0.9190,0.8380,0.0620
Held-out permutation importance (table)
feature,importance_mean,importance_sd,n_folds ndvi_s2_10m,0.12852,0.04151,5 distance_to_river1,0.09737,0.02361,5 elevation_srtm_30m_clipped,0.02960,0.02397,5 distance_to_road1,0.00414,0.00465,5 slope_degree_30m_clipped,0.00180,0.00130,5
4 · Spatial autocorrelation of residuals
| model | moran_i | expected | z | permutation_p | neighbors |
|---|---|---|---|---|---|
| RF | 0.3549 | -0.0003 | 43.8308 | 0.0020 | 8 |
| XGBoost | 0.3235 | -0.0003 | 40.6798 | 0.0020 | 8 |
Moran's I of out-of-fold residuals (k = 8 nearest neighbours, permutation test). Values clearly above zero suggest spatial structure that the predictors do not capture.
5 · Reproducibility
- python
- 3.12.13
- platform
- Windows-11-10.0.26300-SP0
- floodml_geo
- 0.3.1
- numpy
- 2.4.6
- pandas
- 3.0.3
- scipy
- 1.18.0
- sklearn
- 1.9.1
- joblib
- 1.6.0
- matplotlib
- 3.11.0
- xgboost
- 3.4.1
- lightgbm
- 4.7.0
- shap
- 0.54.1
- gdal
- 3.13.2
Full configuration (JSON)
{
"features": [
"distance_to_river1",
"distance_to_road1",
"elevation_srtm_30m_clipped",
"ndvi_s2_10m",
"slope_degree_30m_clipped"
],
"target": "flood",
"positive": [],
"reverse": [
"distance_to_river1",
"elevation_srtm_30m_clipped",
"ndvi_s2_10m",
"slope_degree_30m_clipped"
],
"categorical": [],
"models": [
"RF",
"XGBoost"
],
"cv_method": "spatial",
"n_splits": 5,
"block_m": 0.0,
"buffer_m": 0.0,
"xcol": "x",
"ycol": "y",
"groupcol": "",
"seed": 42,
"class_weight": "balanced",
"tune": true,
"tune_iter": 15,
"tune_folds": 3,
"n_jobs": -1,
"threshold": 0.5,
"permutation_importance": true,
"shap": true,
"shap_max_samples": 500,
"moran": true,
"bootstrap": 300,
"figures": true,
"fig_dpi": 300,
"fig_formats": [
"png"
],
"report": true,
"training_support": {
"geometry": "polygon",
"unit": "m",
"median_cell_side_m": 100.0,
"min_cell_side_m": 1.4790806174259223,
"max_cell_side_m": 100.0
},
"title": "FloodML-Geo \u2013 model comparison",
"block_m_used": 1073.4417200668477
}6 · Interpretation caveats
- Spatial / temporal proxy leakage cannot be detected automatically: verify that every predictor was known BEFORE the flood event and was not derived from flood outcomes.
- Out-of-fold metrics describe the chosen CV design. Choosing the best model from them and then quoting that score as unbiased is optimistic: confirm with an independent spatial / temporal test (tool 05).
- Probabilities reflect the sampling ratio of the flood / non-flood records; they are susceptibility scores, not annual exceedance probabilities, depths or risk.
- Predictor direction (positive / reverse) changes the numeric scaling only; tree models do not enforce a monotonic relationship.
- SHAP and permutation importance describe the fitted model, not causal effects; correlated predictors share importance.
- Extrapolation beyond the training range of the predictors is not penalised: check the extrapolation raster.