remove normalization of likelihood on cl-test

[bayonato89]

No description provided.

[wsavran]

i think this comment somehow made it to the commit on accident.

[wsavran]

these lines should be indented

[bayonato89]

Indented, Bill.

[bayonato89]

The lines are properly idented now :)

[wsavran]

fixes #78

[wsavran]

the ci build/test failed because of some python tabs vs. spaces issue in the poisson_evaluations.py file. @bayonato89 i fetched the branch from your forked copied and i fixed the spacing issues in the commit. the build and tests run without error.

i believe the original implementation of this pull request didn't produce the correct normalization / conditioning for the cl test. i went through the _poisson_likelihood_test(...) function and greatly simplified the logic for choosing the number of events to simulate and whether we want to normalize the likelihoods. before i merge these changes, @bayonato89 @mjw98 and @khawajasim do you all mind reviewing the changes to make sure that we have the correct form implemented?

desired behavior for normalizing and conditioning

TEST NAME	normalize_likelihood	use_observed_counts
M-test	True	True
S-test	True	True
CL-test	False	True
L-test	False	False

implementation from previous commit

# used for conditional-likelihood, magnitude, and spatial tests to normalize the rate-component of the forecasts.
if use_observed_counts:
    scale = n_obs / n_fore
    if normalize_likelihood:
        expected_forecast_count = numpy.sum(forecast_data)
        log_bin_expectations = numpy.log(forecast_data.ravel())
    else:    
        expected_forecast_count = int(n_obs)
        log_bin_expectations = numpy.log(forecast_data.ravel() * scale)
else:
    expected_forecast_count = numpy.sum(forecast_data)
    log_bin_expectations = numpy.log(forecast_data.ravel())

# ... code omitted ...

for idx in range(num_simulations):
    if use_observed_counts:
        if normalize_likelihood:
            num_events_to_simulate = int(n_obs)
        else:    
            num_events_to_simulate = expected_forecast_count
    else:
        num_events_to_simulate = int(numpy.random.poisson(expected_forecast_count))

the error appears to be in the if normalize_likelihood clause, because if we are normalizing the likelihood we want to scale the rates to the observed rates. additionally, the m-test and s-test are called with normalize_likelihood=True and l-test and cl-test are called with normalize_likelihood=False. i propose a fix to this along with some simplification of the logic below.

simplified implementation

expected_forecast_count = numpy.sum(forecast_data)
log_bin_expectations = numpy.log(forecast_data.ravel())
# used for conditional-likelihood, magnitude, and spatial tests to normalize the rate-component of the forecasts
if use_observed_counts and normalize_likelihood:
    scale = n_obs / n_fore
    expected_forecast_count = int(n_obs)
    log_bin_expectations = numpy.log(forecast_data.ravel() * scale)

# ... code omitted ...

for idx in range(num_simulations):
    if use_observed_counts:
        num_events_to_simulate = int(n_obs)
    else:
        num_events_to_simulate = int(numpy.random.poisson(expected_forecast_count))

in this implementation, we scale the rates when normalizing the likelihood and simulating using the observed event counts, else the rates aren't scaled. we now either simulate using the number of observed events or numbers drawn from a poisson distribution. the simplified version matches our desired behavior.

[codecov-commenter]

Codecov Report

Merging #117 (88d171d) into master (415ebfc) will decrease coverage by 0.04%.
The diff coverage is 76.92%.

@@            Coverage Diff             @@
##           master     #117      +/-   ##
==========================================
- Coverage   57.23%   57.19%   -0.05%     
==========================================
  Files          19       19              
  Lines        3136     3133       -3     
  Branches      452      452              
==========================================
- Hits         1795     1792       -3     
  Misses       1231     1231              
  Partials      110      110              

Impacted Files	Coverage Δ
csep/core/poisson_evaluations.py	42.46% <76.92%> (-0.78%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 415ebfc...88d171d. Read the comment docs.

[wsavran]

@bayonato89 im merging this into master. nice job on finding this issue!