FloodML-Geo – model comparison

Spatially validated flood susceptibility modelling • 3,908 observations (3,237 flood / 671 non-flood)

1 · Headline results (out-of-fold)

ModelROC-AUC [95 % CI]PR-AUC [95 % CI]TSS [95 % CI]Balanced acc.KappaBrier [95 % CI]ECEFold AUC mean±SD
RF0.956 [0.935–0.971]0.989 [0.980–0.996]0.749 [0.697–0.801]0.8750.7110.064 [0.044–0.083]0.0520.958 ± 0.026
XGBoost0.956 [0.936–0.974]0.990 [0.980–0.997]0.771 [0.718–0.824]0.8860.7130.065 [0.044–0.086]0.0510.957 ± 0.028

Pooled out-of-fold predictions: every record is predicted by a model that never saw it (or, for spatial CV, its neighbourhood) during fitting. Confidence intervals come from a cluster bootstrap over spatial groups. Thresholded metrics use probability ≥ 0.50.

Model comparison: pooled ROC and precision–recall curves, and fold-to-fold AUC variability.
Model comparison: pooled ROC and precision–recall curves, and fold-to-fold AUC variability.
Warnings and notes from this run
  • Predictor 'distance_to_river1': 7 missing value(s) (median-imputed inside every fold).
  • Predictor 'distance_to_road1': 7 missing value(s) (median-imputed inside every fold).
  • Predictor 'elevation_srtm_30m_clipped': 3 missing value(s) (median-imputed inside every fold).
  • Predictor 'slope_degree_30m_clipped': 3 missing value(s) (median-imputed inside every fold).
  • RF: residuals are still spatially clustered (Moran I = 0.35, p = 0.002); important spatial structure may be missing from the predictors.
  • XGBoost: residuals are still spatially clustered (Moran I = 0.32, p = 0.002); important spatial structure may be missing from the predictors.

2 · Study design

Cross-validation
spatial (5 folds)
Validation design
Every outer fold fits NEW preprocessing (imputation, scaling, one-hot) and a NEW model on its training rows only; the deployment model is refitted on all rows and never evaluated. Hyper-parameters were tuned by inner CV inside each outer training set (nested CV).
Block size
1,073 m
Class weighting
balanced
Predictors
distance_to_river1, distance_to_road1, elevation_srtm_30m_clipped, ndvi_s2_10m, slope_degree_30m_clipped
Normalisation
distance_to_river1: reverse, distance_to_road1: none, elevation_srtm_30m_clipped: reverse, ndvi_s2_10m: reverse, slope_degree_30m_clipped: reverse
Block-size advice
1,073 m – median autocorrelation range of 4 variable(s); capped at 1,073 to leave enough blocks for 5 folds
FoldTraining recordsValidation recordsValidation flood
13157751629
23113795668
33127781631
43211697577
53024884732
Spatial layout of the cross-validation folds.
Spatial layout of the cross-validation folds.
Autocorrelation range of predictors and label (block-size advisor).
Autocorrelation range of predictors and label (block-size advisor).

3.1 · RF

Receiver operating characteristic: pooled out-of-fold curve and individual folds.
Receiver operating characteristic: pooled out-of-fold curve and individual folds.
Precision–recall curve (useful when classes are imbalanced).
Precision–recall curve (useful when classes are imbalanced).
Out-of-fold confusion matrix.
Out-of-fold confusion matrix.
Reliability diagram with 95 % Wilson intervals and score histogram.
Reliability diagram with 95 % Wilson intervals and score histogram.
Score distribution per class and threshold sensitivity.
Score distribution per class and threshold sensitivity.
Metric variability across the spatial folds.
Metric variability across the spatial folds.

Interpretation and spatial diagnostics

Permutation importance on held-out folds (and SHAP when enabled).
Permutation importance on held-out folds (and SHAP when enabled).
SHAP summary of the deployment model.
SHAP summary of the deployment model.
SHAP dependence on the original predictor values.
SHAP dependence on the original predictor values.
Spatial structure of the out-of-fold residuals.
Spatial structure of the out-of-fold residuals.
Per-fold metrics
fold,n_train,n_validation,roc_auc,pr_auc,balanced_accuracy,tss,brier

1,3157,751,0.9837,0.9968,0.9089,0.8177,0.0400

2,3113,795,0.9178,0.9800,0.8405,0.6811,0.0771

3,3127,781,0.9671,0.9922,0.8850,0.7699,0.0585

4,3211,697,0.9723,0.9942,0.8709,0.7417,0.0510

5,3024,884,0.9506,0.9893,0.8679,0.7358,0.0861

Held-out permutation importance (table)
feature,importance_mean,importance_sd,n_folds

ndvi_s2_10m,0.13248,0.02626,5

distance_to_river1,0.11959,0.02165,5

elevation_srtm_30m_clipped,0.02628,0.01938,5

distance_to_road1,0.00415,0.00380,5

slope_degree_30m_clipped,0.00294,0.00172,5

3.2 · XGBoost

Receiver operating characteristic: pooled out-of-fold curve and individual folds.
Receiver operating characteristic: pooled out-of-fold curve and individual folds.
Precision–recall curve (useful when classes are imbalanced).
Precision–recall curve (useful when classes are imbalanced).
Out-of-fold confusion matrix.
Out-of-fold confusion matrix.
Reliability diagram with 95 % Wilson intervals and score histogram.
Reliability diagram with 95 % Wilson intervals and score histogram.
Score distribution per class and threshold sensitivity.
Score distribution per class and threshold sensitivity.
Metric variability across the spatial folds.
Metric variability across the spatial folds.

Interpretation and spatial diagnostics

Permutation importance on held-out folds (and SHAP when enabled).
Permutation importance on held-out folds (and SHAP when enabled).
SHAP summary of the deployment model.
SHAP summary of the deployment model.
SHAP dependence on the original predictor values.
SHAP dependence on the original predictor values.
Spatial structure of the out-of-fold residuals.
Spatial structure of the out-of-fold residuals.
Per-fold metrics
fold,n_train,n_validation,roc_auc,pr_auc,balanced_accuracy,tss,brier

1,3157,751,0.9790,0.9958,0.9051,0.8103,0.0520

2,3113,795,0.9082,0.9774,0.8323,0.6646,0.0847

3,3127,781,0.9634,0.9914,0.8902,0.7804,0.0647

4,3211,697,0.9637,0.9921,0.8754,0.7508,0.0603

5,3024,884,0.9695,0.9937,0.9190,0.8380,0.0620

Held-out permutation importance (table)
feature,importance_mean,importance_sd,n_folds

ndvi_s2_10m,0.12852,0.04151,5

distance_to_river1,0.09737,0.02361,5

elevation_srtm_30m_clipped,0.02960,0.02397,5

distance_to_road1,0.00414,0.00465,5

slope_degree_30m_clipped,0.00180,0.00130,5

4 · Spatial autocorrelation of residuals

modelmoran_iexpectedzpermutation_pneighbors
RF0.3549-0.000343.83080.00208
XGBoost0.3235-0.000340.67980.00208

Moran's I of out-of-fold residuals (k = 8 nearest neighbours, permutation test). Values clearly above zero suggest spatial structure that the predictors do not capture.

5 · Reproducibility

python
3.12.13
platform
Windows-11-10.0.26300-SP0
floodml_geo
0.3.1
numpy
2.4.6
pandas
3.0.3
scipy
1.18.0
sklearn
1.9.1
joblib
1.6.0
matplotlib
3.11.0
xgboost
3.4.1
lightgbm
4.7.0
shap
0.54.1
gdal
3.13.2
Full configuration (JSON)
{
  "features": [
    "distance_to_river1",
    "distance_to_road1",
    "elevation_srtm_30m_clipped",
    "ndvi_s2_10m",
    "slope_degree_30m_clipped"
  ],
  "target": "flood",
  "positive": [],
  "reverse": [
    "distance_to_river1",
    "elevation_srtm_30m_clipped",
    "ndvi_s2_10m",
    "slope_degree_30m_clipped"
  ],
  "categorical": [],
  "models": [
    "RF",
    "XGBoost"
  ],
  "cv_method": "spatial",
  "n_splits": 5,
  "block_m": 0.0,
  "buffer_m": 0.0,
  "xcol": "x",
  "ycol": "y",
  "groupcol": "",
  "seed": 42,
  "class_weight": "balanced",
  "tune": true,
  "tune_iter": 15,
  "tune_folds": 3,
  "n_jobs": -1,
  "threshold": 0.5,
  "permutation_importance": true,
  "shap": true,
  "shap_max_samples": 500,
  "moran": true,
  "bootstrap": 300,
  "figures": true,
  "fig_dpi": 300,
  "fig_formats": [
    "png"
  ],
  "report": true,
  "training_support": {
    "geometry": "polygon",
    "unit": "m",
    "median_cell_side_m": 100.0,
    "min_cell_side_m": 1.4790806174259223,
    "max_cell_side_m": 100.0
  },
  "title": "FloodML-Geo \u2013 model comparison",
  "block_m_used": 1073.4417200668477
}

6 · Interpretation caveats

  • Spatial / temporal proxy leakage cannot be detected automatically: verify that every predictor was known BEFORE the flood event and was not derived from flood outcomes.
  • Out-of-fold metrics describe the chosen CV design. Choosing the best model from them and then quoting that score as unbiased is optimistic: confirm with an independent spatial / temporal test (tool 05).
  • Probabilities reflect the sampling ratio of the flood / non-flood records; they are susceptibility scores, not annual exceedance probabilities, depths or risk.
  • Predictor direction (positive / reverse) changes the numeric scaling only; tree models do not enforce a monotonic relationship.
  • SHAP and permutation importance describe the fitted model, not causal effects; correlated predictors share importance.
  • Extrapolation beyond the training range of the predictors is not penalised: check the extrapolation raster.