[REVIEW]: Optim: An optimization package for Julia

[whedon]

Submitting author: @anriseth (Asbjørn Nilsen Riseth)
Repository: https://github.com/JuliaNLSolvers/Optim.jl
Version: v0.14.0
Editor: @labarba
Reviewer: @ahwillia
Archive: 10.5281/zenodo.1211551

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/ccd4ac7b58bbb2c577866594733968f2"><img src="http://joss.theoj.org/papers/ccd4ac7b58bbb2c577866594733968f2/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/ccd4ac7b58bbb2c577866594733968f2/status.svg)](http://joss.theoj.org/papers/ccd4ac7b58bbb2c577866594733968f2)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ahwillia, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @labarba know.

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (v0.14.0)?
• Authorship: Has the submitting author (@anriseth) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @ahwillia it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

--> Check article proof 📄 <--

[labarba]

Reviewer 2 instructions & questions

@abelsiqueira, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @labarba know.

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: Does the release version given match the GitHub release (v0.14.0)?
• Authorship: Has the submitting author (@anriseth) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[labarba]

@abelsiqueira — Our journal management system hasn't been upgraded yet to assign two reviewers, so we manually create a Review Checklist for you. Check It out ☝️

[labarba]

@anriseth — In your paper, I do miss some brief note about existing software providing similar functionality (in the same or other programming language). We don't have a novelty requirement, but we do want to see other software mentioned or cited, as appropriate.

[pkofod]

In your paper, I do miss some brief note about existing software providing similar functionality (in the same or other programming language). We don't have a novelty requirement, but we do want to see other software mentioned or cited, as appropriate.

Does a brief note include a discussion / comparison or is it just mentions?

[labarba]

Mention other software that has same/similar functionality, briefly comment on differences and equivalence, but no need to discuss or make comparisons requiring you to run the other software. It's a matter of giving acknowledgement to the existence of the other software, not claim originality or superiority.

[abelsiqueira]

Review

The Optim package for Julia is well known in the community, and it's now considered an essential part of it. The package follows the good practices expected by JOSS, and I have only a few minor questions, corrections, and suggestions to make:

• Clarify that optimization means mathematical optimization. Possibly in the title.
• Shouldn’t John Myles White be an author? I’m not familiar with the package history, nor with ownership “rules”, but since he’s made major contributions, it warrants the question. Answering this will probably clarify Tim Holy's situation as well.
• What’s the difference between “Newton with trust region” and “Hessian-vector with trust region”?
• ParticleSwarm doesn’t appear to be documented.
• There are no clear community guidelines, as requested by JOSS. In my opinion, indicating the issues tab for reporting issues, problems and discussing contributions and indicating the gitter or discourse channel for help should be enough.
• At least one reference needs DOI.

Best,

[pkofod]

Shouldn’t John Myles White be an author? I’m not familiar with the package history, nor with ownership “rules”, but since he’s made major contributions, it warrants the question. Answering this will probably clarify Tim Holy's situation as well.

Maybe we should bring in @labarba to clarify the rules. The situation is that John Myles White's current employment has very strict rules regarding review of work and publication done outside of his place of work. What we could do is finish these reviews, and then when they "good to go" we can try to see if what is there in the end fits their legal requirements. For good measure, I think I'll ping @johnmyleswhite to show that we're not trying to cheat anyone here.

Regarding @timholy the story is relevant as well, as he initially made a lot of contributions to the package, and is a relevant author as well. It would be 👍 to add him as well, but we were not comfortable adding an author that we have not been able to get explicit consent from.

The two potential co-authors were the two founding persons of Optim, but neither have been active contributors or maintainers for quite a while, and both are aware of this submission. However, @anriseth and I are perfectly happy to include one or both as authors as long as the legality and explicit consent can be established.

Clarify that optimization means mathematical optimization. Possibly in the title.

Fair point

What’s the difference between “Newton with trust region” and “Hessian-vector with trust region”?

The difference is that the former expects a function that calculates the Hessian, but the latter expects a function that calculates the Hessian-vector products directly. Do note that the Hessian-vector code is experimental, not exported, and for that reason not documented either. It's not ready for public consumption, but it is part of the code base to allow people to try it out and give feedback.

ParticleSwarm doesn’t appear to be documented.

Correct, we will add an entry.

There are no clear community guidelines, as requested by JOSS. In my opinion, indicating the issues tab for reporting issues, problems and discussing contributions and indicating the gitter or discourse channel for help should be enough.

We can add a CONTRIBUTING.md or add the things you mention to the README.md. I think we would need a JOSS representative to help with the best thing to do here.

At least one reference needs DOI.

Fair.

[pkofod]

Review

The Optim package for Julia is well known in the community, and it's now considered an essential part of it. The package follows the good practices expected by JOSS, and I have only a few minor questions, corrections, and suggestions to make:

Clarify that optimization means mathematical optimization. Possibly in the title.
Shouldn’t John Myles White be an author? I’m not familiar with the package history, nor with ownership “rules”, but since he’s made major contributions, it warrants the question. Answering this will probably clarify Tim Holy's situation as well.
What’s the difference between “Newton with trust region” and “Hessian-vector with trust region”?
ParticleSwarm doesn’t appear to be documented.
There are no clear community guidelines, as requested by JOSS. In my opinion, indicating the issues tab for reporting issues, problems and discussing contributions and indicating the gitter or discourse channel for help should be enough.
At least one reference needs DOI.

Best,

@abelsiqueira Thanks again for your feedback. We've tried to address your concerns in the two latest PRs: JuliaNLSolvers/Optim.jl#546 JuliaNLSolvers/Optim.jl#548 . We would appreciate if you could verify that the changes cover your concerns. The authorship concerns will depend partially upon what @labarba has to say on the matter based on my earlier comment.

[labarba]

Normally, we expect all persons who've made substantial contributions to the software to be in the list of authors. If both @johnmyleswhite and @timholy have been inactive in the project for a long time, and they approve of the JOSS submission not listing them as authors, then we can accept that. But their names should still be listed in an original copyright notice, as per the license conditions. (I didn't check if this is already the case.)

[pkofod]

Normally, we expect all persons who've made substantial contributions to the software to be in the list of authors. If both @johnmyleswhite and @timholy have been inactive in the project for a long time, and they approve of the JOSS submission not listing them as authors, then we can accept that. But their names should still be listed in an original copyright notice, as per the license conditions. (I didn't check if this is already the case.)

The LICENSE.md file mentions "John Myles White and other contributors"

and they approve of the JOSS submission not listing them as authors, then we can accept that

Yes, but let me repeat that this is not due to an explicit wish to exclude them. I think we will try to go the route of waiting for the review of @ahwillia , and then when everything is good to go besides the authorship, we will ship the draft off to facebook's legal department and see what they say. In Tim's case it's not so much that we've been looking for an explicit approval to exclude him, we've rather been unable to receive consent to publish what you see in the draft with his name attached to it. I figure it's a busy time in general, and especially so for Julia core contributors, so maybe we'll hear back before this review is over.

[ahwillia]

Does the year of the LICENSE matter? It is listed as 2012 and John Myles White is the only name listed. I think it would make sense to add a few more names there. Perhaps "John Myles White, JuliaNLSolvers organization, and other contributors" ?

I might be wrong, but I seem to remember the package being nearly entirely re-written by @pkofod at some point. So to me it seems fine that John doesn't necessarily need to be on the author list.

Overall the package is extremely mature and extensively used in the Julia community. So it is a no-brainer that this should be accepted and published. I struggle to come up with suggestions and feedback. Here a couple of small comments:

• There is a checkbox for me at the top of the page which says that all references need a DOI? If this is a requirement, it seems like some of the references in the paper are missing them.

• I think this was mentioned before, but I think it would be nice to have a short section in the documentation and/or the paper to discuss what other libraries are available to users. It seems like you have recently added support for manifold optimization (really cool!!) and to my knowledge this isn't available in scipy.optimize (but see pymanopt). There is a nice discussion about the advantages of Julia in the documentation, but it might still be useful to mention these other libraries by name in the paper?

• Similar to above, it might be nice to have a citation to Convex.jl and JuMP.jl somewhere in the paper. I know Optim.jl is quite different than those packages, but it would be nice to mention them and briefly explain to new users how they are different.

• This is really minor and not necessary - but in addition to grid_search (a zeroth order multivariate solver) you might consider adding rand_search, see Bergstra & Bengio (2012). I expect this would be trivial to implement and it could be useful.

• Out of curiosity, is there a nice API to do a nested optimization? Let's say I wanted to use rand_search to tune a few hyperparameters of a complex model. For each set of hyperparameters I used a more sophisticated method (e.g. L-BFGS). That would be a super rad use case of this package!

I am happy to sign off on the package as is. Great work putting this together!

[labarba]

We do it like this, reflecting major release years, with author lists changing according to level of contribution:

(c) 2012 Founding Author, Second Founder
(c) 2014 Founding Author, Second Founder, New Contributor
(c) 2018 Main Contributor, New Contributor, Another Contributor, Founding Author, Second Founder

You just keep adding (c) lines below, and over time some authors may drop.

Then the license text comes below.

[pkofod]

Thanks for the feedback @ahwillia

There is a checkbox for me at the top of the page which says that all references need a DOI? If this is a requirement, it seems like some of the references in the paper are missing them.

I think some of these have already been adressed. Is it possible to generate a new paper pdf @labarba ?

For the rest,

I think this was mentioned before, but I think it would be nice to have a short section in the documentation and/or the paper to discuss what other libraries are available to users. It seems like you have recently added support for manifold optimization (really cool!!) and to my knowledge this isn't available in scipy.optimize (but see pymanopt). There is a nice discussion about the advantages of Julia in the documentation, but it might still be useful to mention these other libraries by name in the paper?

I am totally fine mentioning scipy.optimize (and JuMP and Convex), but it is not totally clear to me where to draw the line. Should R's optim also be mentioned then?

[labarba]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

--> Check article proof 📄 <--

[ahwillia]

I am totally fine mentioning scipy.optimize (and JuMP and Convex), but it is not totally clear to me where to draw the line. Should R's optim also be mentioned then?

It wouldn't hurt, I suppose. Is the biggest different between R's optim, scipy.optim, and Julia's Optim the language? Or are there other differences that are worth mentioning?

Those are the questions I could see a new user wondering.

[pkofod]

It wouldn't hurt, I suppose. Is the biggest different between R's optim, scipy.optim, and Julia's Optim the language? Or are there other differences that are worth mentioning?

Those are the questions I could see a new user wondering.

Gotcha. I'll talk to @anriseth and come up with some changes to address your comments (the other ones as well). Thanks.

[ahwillia]

Yes I recommend acceptance. On Mar 19, 2018 12:39 PM, "Lorena A. Barba" <notifications@github.com> wrote: I see that both reviewers have checked off all items in the review checklist. Can I have confirmation that the reviewers recommend acceptance? @abelsiqueira <https://github.com/abelsiqueira> @ahwillia <https://github.com/ahwillia> ? Also, can I have confirmation from the authors that they've finished working on it? @anriseth <https://github.com/anriseth> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#615 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAm20UaRny6sX_6jQKkoo03WG8Xqg0TIks5tgAlVgaJpZM4Sm3fC> .

[pkofod]

Also, can I have confirmation from the authors that they've finished working on it?
@anriseth

I know Asbjørn submitted it, but please allow me to reach out to the other potential authors one last time, to get their final comments on wether they want to be credited as co-authors of if the acknowledgement is sufficient.

[abelsiqueira]

I recommend acceptance

[labarba]

OK, everyone. Thank you sincerely to @ahwillia and @abelsiqueira for their review.

@arfon -- We're waiting for the authors to do a final effort of adjusting their author list with contributors currently not named, due to their employment restrictions. After they give us the flag, the paper is ready for publication.

[pkofod]

JMW has responded that the acknowledgement is sufficient, but I've yet to hear back from Tim Holy.

[pkofod]

I think the constructive way forward is to publish as is with the acknowledgement of John Myles White and Tim Holy's contributions early in Optim.jl's life. As mentioned above, this was suggested by JWM in private correspondence, but we have still not heard back from Tim since our very early communication about making a submission to JOSS in January through e-mail, github pings, or otherwise. @anriseth @labarba @arfon

[labarba]

Looks good — small typo in the Acknowledgements:

We would also like to thank everyone who has contributed…

[pkofod]

JuliaNLSolvers/Optim.jl#565 the typo is fixed on master

[labarba]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

--> Check article proof 📄 <--

[labarba]

@arfon : This submission is ready for acceptance.

@anriseth : We now need you to make a new release, upload the revised software to your DOI-granting data/software repository, and post the DOI here.

[pkofod]

Alright, but then the v0.14 self-reference in the paper needs to be changed as well, right?

Edit: I mean, you're asking us to create a new tag such that the JOSS article is archived in the zenodo repo, right?

http://doi.org/10.5281/zenodo.1211551

[pkofod]

Sorry, since I edited, I guess it doesn't create a notification. Does that doi look alright @labarba ?

[labarba]

@arfon --- ball on your court now to complete acceptance.

[arfon]

@whedon set 10.5281/zenodo.1211551 as archive

[whedon]

OK. 10.5281/zenodo.1211551 is the archive.

[arfon]

@abelsiqueira, @ahwillia - many thanks for your reviews here and to @labarba for editing this submission ✨

@anriseth - your paper is now accepted into JOSS and your DOI is https://doi.org/10.21105/joss.00615 ⚡ 🚀 💥

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippet:

[![DOI](http://joss.theoj.org/papers/10.21105/joss.00615/status.svg)](https://doi.org/10.21105/joss.00615)

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://www.flipcause.com/secure/cause_pdetails/Mjk3Nzk=

[anriseth]

Great to hear, thank you to everyone who gave us feedback and has contributed 😃