Scores of count forecasts

Scores Poisson-lognormal predictive distributions of counts, those of latent Gaussian-process models with a poisson_likelihood(), with proper scores, a dispersion check, interval coverage, and randomized probability integral transform (PIT) values. gp_holdout_scores() returns these scores for latent Poisson models.

Usage

gp_count_scores(
  observed,
  latent_mean,
  latent_variance,
  exposure = NULL,
  interval_levels = c(0.5, 0.8, 0.9, 0.95),
  seed = NULL,
  quadrature_order = 40L
)

Arguments

observed: Non-negative integer counts.
latent_mean, latent_variance: Mean and variance of the latent log rate \(f\) at each observation, as returned by predict_latent_gp() (mean and latent_variance).
exposure: Optional positive exposure: one value or one per count.
interval_levels: Levels of the equal-tailed prediction intervals whose coverage is reported.
seed: Optional seed for the randomization of the PIT values. The caller's random state is restored afterwards.
quadrature_order: Number of Gauss-Hermite nodes.

Value

An object of class gaussianprocesses_count_scores with

summary: Named numeric vector: n, log_score, dawid_sebastiani, dispersion, pit_mean, pit_variance, and pit_ks_p_value.
calibration: Data frame with one row per interval level: the level, empirical coverage, and mean width (upper minus lower count).
pointwise: Data frame with observed, predictive_mean, predictive_variance, log_score, dawid_sebastiani, pearson_residual, and pit.

Details

Given \(f \sim N(\mu, \sigma^2)\), a count \(y \sim \mathrm{Poisson}(E e^f)\) has the Poisson-lognormal predictive distribution, with mean \(E e^{\mu + \sigma^2 / 2}\) and probabilities, distribution function, and quantiles computed by Gauss-Hermite quadrature.

log score: The mean log predictive probability of the counts. Proper; higher is better.
Dawid-Sebastiani score: \((y - m)^2 / s^2 + 2 \log s\) with the predictive mean \(m\) and standard deviation \(s\); proper for the first two moments and recommended for counts by Czado, Gneiting, and Held (2009). Lower is better.
dispersion: The mean squared Pearson residual \((y - m)^2 / s^2\). It is about 1 when the predictive variance is right; clearly larger values mean that the counts are overdispersed relative to the model.
coverage: Of the equal-tailed prediction intervals at each level. Counts are discrete, so the intervals cover at least their level: their coverage is conservative, by up to the probability of their end points.
randomized PIT: \(u = F(y - 1) + v\, p(y)\) with \(v \sim U(0, 1)\): uniform on \([0, 1]\) when the forecasts are calibrated, unlike the plain PIT \(F(y)\) of a discrete distribution. The summary reports their mean (1/2), variance (1/12), and the p-value of a Kolmogorov-Smirnov test of uniformity.

Stability

Experimental: this interface may change in a minor release, and every change is listed in NEWS. See gaussianprocesses-package for the policy.

References

Czado, C., Gneiting, T., and Held, L. (2009). Predictive model assessment for count data. Biometrics, 65(4), 1254–1261.

Examples

set.seed(1)
x <- seq(0, 10, length.out = 60)
counts <- rpois(60, exp(1 + sin(x)))
training <- seq(1, 60, by = 2)

model <- fit_latent_gp(x[training], counts[training], rbf_kernel(0.5, 1.5),
  poisson_likelihood(), mean = constant_mean(1))
gp_holdout_scores(model, x[-training], counts[-training], seed = 1)
#> Count scores for 30 observations
#>   log score (mean): -1.993
#>   Dawid-Sebastiani score (mean): 2.342
#>   dispersion: 0.8775 (about 1 if the predictive variance is right)
#>   randomized PIT mean and variance: 0.4525, 0.08191 (0.5 and 0.0833 if calibrated)
#>   coverage: 50% 63.3%, 80% 90.0%, 90% 96.7%, 95% 100.0%