Independent test

3,908 test records (3,237 flood) · threshold 0.5

Result is NOT valid

NOT INDEPENDENT
  • 100 % of the test records are the same records the models were trained on. These scores are not an independent test. Use data from another area, event or year.

Results

ModelROC-AUC [95 % CI]PR-AUC [95 % CI]TSSBalanced acc.KappaBrierECECV ROC-AUC
XGBoost0.997 [0.996–0.998]0.999 [0.999–1.000]0.9540.9770.8850.0260.0430.956
RF0.990 [0.988–0.992]0.998 [0.997–0.998]0.8950.9470.8180.0410.0620.956

Confidence intervals: cluster bootstrap over individual records (no group or block size given; intervals may be too narrow if records are spatially clustered). No model was refitted and the threshold was not tuned on these data.

Notes
  • Bootstrap resamples individual records: confidence intervals are too narrow when test records are spatially clustered. Provide a block size or a group column.
  • NOT INDEPENDENT: 100 % of the test records are identical to training records of XGBoost (same coordinates). The scores below measure how well the model remembers its training data, not how it generalises. Do not report them. Build the test table from another area, event or year.
  • NOT INDEPENDENT: 100 % of the test records are identical to training records of RF (same coordinates). The scores below measure how well the model remembers its training data, not how it generalises. Do not report them. Build the test table from another area, event or year.

XGBoost

ROC curve
ROC curve
Precision–recall
Precision–recall
Confusion matrix
Confusion matrix
Calibration
Calibration
Score distribution
Score distribution

RF

ROC curve
ROC curve
Precision–recall
Precision–recall
Confusion matrix
Confusion matrix
Calibration
Calibration
Score distribution
Score distribution