Only generate Javadocs for released versions of Pulsar

[michaeljmarshall]

Main issues: apache/pulsar#14838 apache/pulsar#13916 apache/pulsar#16094

Motivation

The current Javadocs are generated based on unreleased versions of Pulsar. As a result, the website links have -SNAPSHOT in them. This is confusing, and can also lead to problems where the docs do not align with specific versions. This PR is a step towards cleaning up the -SNAPSHOT references (there are several more locations, but I am just fixing the Javadoc case in this PR).

Changes

• Update Javadoc generation script so that it downloads the released source code for pulsar and then builds the Javadocs from that source. (This change is in the first commit.)
• Run the script against versions 2.7.0 through 2.10.1. (These were all added in the second commit.)
• The admin api docs for 2.7.x are generated using mvn -pl pulsar-client-admin javadoc:javadoc.
• The admin api docs for 2.8 and above are generated using mvn -pl pulsar-client-admin-api javadoc:javadoc.
• Stop generating the pulsar-broker Javadocs (discussed on mailing list)

Note

This change does not make these docs referenced on the website. There is additional work that I plan to complete to make the replace.js script reference these docs. I will contribute once I fix all of the generated docs with -SNAPSHOT.

[michaeljmarshall]

@urfreespace - I did not realize just how big these Javadoc files were until opening this PR. This PR literally adds 8 million lines of generated Javadocs. Should we add the script to the main branch and only commit the generated docs to the asf-site-next branch?

[michaeljmarshall]

Looks like the generated docs from the broker for all versions are about 304 MB.

[michaeljmarshall]

Interestingly, the following returns no results when run on the asf-site-next branch for this repo:

grep -R "/api/pulsar-broker" content/docs/

That indicates to me that the website doesn't actually point to the broker's Javadocs. Why do we generate and host them if they are not referenced? I wonder if we can remove them and save some space.

Either way, we still need to decide the right workflow for adding generated docs. I think it would make sense to keep them in the main branch, but I am open to opinions.

[urfreespace]

Interestingly, the following returns no results when run on the asf-site-next branch for this repo:

grep -R "/api/pulsar-broker" content/docs/

That indicates to me that the website doesn't actually point to the broker's Javadocs. Why do we generate and host them if they are not referenced? I wonder if we can remove them and save some space.

Either way, we still need to decide the right workflow for adding generated docs. I think it would make sense to keep them in the main branch, but I am open to opinions.

I totally agree, thanks @michaeljmarshall, @Anonymitaet how do you think?

[Anonymitaet]

@michaeljmarshall thanks for your great contribution! I have several questions:

Release frequency

I see you've generated API docs for major and minor versions, so for future releases, we need to generate API docs for each version (align with the Pulsar release), correct?

Release owner

Currently, API docs are automatically generated.

In this PR, you've changed the automatic workflow to manual work, so who will be responsible for generating API docs?
Release managers? If so, would you like to add this required step to the Pulsar release process - wiki?

[Anonymitaet]

FYI
@D-2-Ed @DaveDuggins

@SignorMercurio @momo-jun: this PR removes SNAPSHOT

[michaeljmarshall]

I see you've generated API docs for major and minor versions, so for future releases, we need to generate API docs for each version (align with the Pulsar release), correct?

Yes, that is correct. We currently expect the release manager to generate the swagger docs and the python client docs. I think we could make it so the release manager only has to run a single command to generate all docs for the release.

Alternatively, we could probably make some github action that runs when tags are released, but it's probably not worth the effort, and it might lead to minor errors.

Release owner

Currently, API docs are automatically generated.

In this PR, you've changed the automatic workflow to manual work, so who will be responsible for generating API docs? Release managers? If so, would you like to add this required step to the Pulsar release process - wiki?

The current automated process only updates the latest docs based on the apache/pulsar repo's master branch. The docs have -SNAPSHOT on the end because they are not generated based on the correct tag of the Pulsar code base. While it'd be nice to have a completely automated solution, I think it'll be easier to add a manual solution that relies on running a single script. Once we have the script completely implemented, it might be easier to automate the process.

Additionally, I will update the release process wiki once we have a new process.

For now, I think my primary question is if we can get rid of the broker's Javadocs.

[michaeljmarshall]

I updated the PR to remove the pulsar-broker doc generation. I started a discussion on the dev@ mailing list to propose removing these docs from the website permanently.

[michaeljmarshall]

I just discovered that the Pulsar Admin API Javadocs changed their generation mechanism starting with 2.8.0 because of this change: apache/pulsar#9246. In my two most recent commits, I fix the Javadoc generation script. This fix takes care of a primary issue where the docs reference links like https://pulsar.apache.org/api/admin/2.9.0-SNAPSHOT/org/apache/pulsar/client/admin/Sink.html#deleteSink-java.lang.String-java.lang.String-java.lang.String-, but the links do not work unless they are https://pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/Sink.html#deleteSink-java.lang.String-java.lang.String-java.lang.String-.

By generating the docs based on the interfaces, not the Impl classes, we'll get the right docs and the links should be restored.

[michaeljmarshall]

Note: no docs reference pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/internal/, but plenty reference pulsar.apache.org/api/admin/org/apache/pulsar/client/admin/, so there shouldn't be any issues replacing the existing pulsar implementation docs. These are all in the io-use.md docs files. Note that in the 2.10 docs, these links are fixed to look like /api/admin/org/apache/pulsar/client/admin/Sink.html#createSink-SinkConfig-java.lang.String- (they rely on the replace.js logic in the pulsar-site build scripts). However, the links still don't work for versioned links because we don't generate the right docs.

[momo-jun]

Hi @michaeljmarshall did your PR fix the reported three issues mentioned above? They are still in an open state. Not sure whether you will have follow-up fixes or not.

[michaeljmarshall]

@momo-jun - this PR only fixed the -SNAPSHOT issue for Javadocs. The tools and the cpp client still have -SNAPSHOT docs.