Skip to content

Return NaN from nse and r2 for pandas input with NaN model values - #721

Closed
syahra712 wants to merge 1 commit into
DHI:mainfrom
syahra712:fix-720-nse-nan-series
Closed

syahra712 wants to merge 1 commit into
DHI:mainfrom
syahra712:fix-720-nse-nan-series

Conversation

@syahra712

Copy link
Copy Markdown

Fixes #720

nash_sutcliffe_efficiency and r2 called np.sum directly on their inputs. On a pandas Series that skips NaN, so a model with missing values (or an all-NaN model) scored 1.0. NumPy input gave NaN. Both functions now convert obs and model with np.asarray before computing, as the issue suggests, so Series and arrays give the same result.

Tests: I added test_nse_r2_nan_in_model_series, parametrized over nse/r2 and over partly-NaN and all-NaN model values. It checks that Series input returns NaN, the same as ndarray input. It fails on main and passes with this change. The existing metric tests, the metrics.py doctests, ruff check/ruff format --check and mypy on metrics.py all pass locally.

Scope note: I kept this to the two metrics in the issue. When I compared every metric on Series against ndarray with the same input, a few others also differ when model values are NaN: bias, max_error, mape, explained_variance, kge, scatter_index/scatter_index2 and willmott. Happy to follow up with the same conversion for those if you'd like them handled consistently, in this PR or a separate one.

np.sum on a pandas Series skips NaN, so nse and r2 returned 1.0 when
model values were missing, while NumPy input returned NaN. Convert the
inputs with np.asarray first.

Fixes DHI#720
@ecomodeller

Copy link
Copy Markdown
Member

Thanks for the PR. Please describe in what real modelling context this is a problem, and how this PR solves it.

@syahra712

Copy link
Copy Markdown
Author

Thanks for taking a look. To be precise about where this shows up:

Where it doesn't: ms.match() drops time steps where any model or observation value is NaN (matching.py, the dropna(dim="time", subset=mo_cols) near the end of _match_space_time), so comparers built through match() never pass NaN to the metrics.

Where it does:

  1. ms.from_matched() with pre-matched data. from_matched rejects NaN in the observation column but accepts it in the model column. One example is a dfs0 or CSV with model results already extracted at a station. When the model run stopped part-way, or when the model writes delete values (NaN after reading) for dry periods at an intertidal station, cmp.skill() reports the full n. rmse, bias and cc come out NaN, but nse is computed with a numerator over the wet/valid steps only and a denominator over all observations, so it is inflated. If the model column is entirely NaN, NSE is 1.0. In a multi-station skill table that run looks like the best station, not a broken one.
  2. Calling the metrics directly on DataFrame columns, e.g. ms.metrics.nse(df["obs"], df["mod"]). That gives a different answer than ms.metrics.nse(df["obs"].values, df["mod"].values) for the same data.

What the PR changes: nse and r2 convert their inputs with np.asarray first, so missing model values propagate to NaN, the same as rmse/bias/cc and the ndarray path. A failed or partial run then shows up as NaN in the skill table instead of a perfect or inflated score.

If you'd rather handle it at the source, an alternative is to have from_matched drop NaN model rows the way match() does. That would make n and every metric consistent for pre-matched input. I'm happy to switch the PR to that, or do both, whichever fits the design better.

@ecomodeller

Copy link
Copy Markdown
Member

Thanks for taking the time to look into this. We're handling this issue internally, so I'm closing this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NSE and R2 return 1.0 when model values are NaN

2 participants