[R-package] v4.0.0 CRAN submission issues

[jameslamb]

Description

The recent submission of {lightgbm} v4.0.0 to CRAN was rejected because of the following issues detected by R CMD check.

* checking examples ... [8s/4s] NOTE
Examples with CPU time > 2.5 times elapsed time
                    user system elapsed ratio
lgb.restore_handle 1.819  0.128   0.165  11.8

* checking S3 generic/method consistency ... NOTE
Mismatches for apparent methods not registered:
merge:
  function(x, y, ...)
merge.eval.string:
  function(env)

format:
  function(x, ...)
format.eval.string:
  function(eval_res, eval_err)
See section 'Registering S3 methods' in the 'Writing R Extensions'
manual.

These need to be fixed for CRAN to accept our submission.

Reproducible example

These check failures happened during the CRAN incoming checks.

Environment info

LightGBM version or commit hash: v4.0.0.

Additional Comments

I will put up a pull request shortly to fix these things. We can figure out later how to catch them in CI so they don't surprise us on CRAN submissions in the future.

The examples timing issue is because I did not complete #5102 as part of this release, sorry 😞 . I said I would almost a year ago (#5102 (comment)), just forgot about it.

[jameslamb]

Re-opening this. Sadly our v4.0.0.1 submission still did not resolve the issue about using more than 2 threads in examples and tests, and was not accepted to CRAN.

[mayer79]

Regarding the timing: in the worst case, pack the firing examples in a dontrun block:

@examples
#' \dontrun{
#' do some lgb magic
#' }

[jameslamb]

Thanks for the suggestion @mayer79 ! But I don't think that's the fix we should pursue.

I don't think we can safely assume it's only the one example that happened to be reported in CRAN's initial checks. The timings are unpredictable and depend on, for example, what else happens to be running on the CRAN check machines at the same time as they're checking {lightgbm}.

In addition...I no longer trust \dontrun or \donttest mechanisms as a way to avoid issues in the CRAN checks.

We used to wrap all examples in \dontrun (see #3270)... until CRAN manual review by a human told us we had to drop that in favor of \donttest (#3338).

... and then with R 4.0, R CMD check started running all the \donttest{} examples anyway 🙃

See https://cran.r-project.org/doc/manuals/r-devel/NEWS.html

*R CMD check --as-cran now runs \donttest examples

We have to find a more permanent solution to ensure that everything multi-threaded in this project uses at most 2 threads in tests and examples. It's frustrating that v4.0.0 of the project is being blocked from CRAN because of an example that takes less than 2 seconds to run but... we can only work on the things we can control.

[mayer79]

Ha, darn. I recently had the same problem: Setting all num_threads in all examples in https://github.com/ModelOriented/shapviz would not bring down those example runtimes. Then I just packed them into {dontrun}.

The issue about merge.* is already solved, thx!

[david-cortes]

Regarding the issue with the threads on lgb.restore_handle - given that bug #4705 hasn't been addressed, I guess it could be solved by passing num_threads=0, or by a few calls to RhpcBLASctl (which is already under Suggests).

I also don't think that'd be grounds for rejection (the one about S3 mismatch would though) since, when checks fail due to large thread usage, it typically manifests as a segmentation fault on the solaris checks (or at least it does when passing the numer of threads on individual pragmas, which this library doesn't).

[jameslamb]

I also don't think that'd be grounds for rejection

That is not correct. Our second attempted submission was rejected by the automatic CRAN checks, and the only issue was that NOTE about the example timings.

[jameslamb]

There has been significant discussion about this recently on the R-pkg-devel mailing list. Many package authors have been facing CRAN's newly-strict insistence on projects only using 2 threads in all places.

• (June 14 - present) Best practice for using max 2 cores on CRAN? Rdatatable/data.table#5658
• (July 16) "Issue with CPU time > 2.5 times elapsed time" (link)
• (Aug 24) "Re-building vignettes had CPU time 9.2 times elapsed time" (link)
• (Sep 1) "CRAN Detected Using More Than 2 Cores By Default" (link)

I'm still reading through all of them, so won't attempt to summarize yet... just linking these to make it clear this is not a LightGBM-specific issue.