Skip to content

developing-poly.adoc

Latest commit

 

History

History
679 lines (521 loc) · 26 KB

developing-poly.adoc

File metadata and controls

679 lines (521 loc) · 26 KB
Raw

Developing poly

Table of Contents

The curious are most welcome, but we wrote this document for folks who are (or want to) contribute to the development poly itself.

Contributing

We very much appreciate contributions from the community.

Issues First Please

If you have an idea or a fix, please do raise a GitHub issue before investing in any coding effort. That way, we can discuss first. Writing code is the easy part; maintaining it forever is the hard part!

That said, if you notice a simple typo, a PR without an issue is fine.

Tip
 We are also happy to chat about your idea or fix on the Clojurians polylith channel.

Include Tests and Docs

We expect you to include/update tests for your change.

You’ve probably gotten the sense that providing good docs for our users is important to us. When your change affects our docs or needs to be described for our users, we expect you to do so.

Add a brief line summarizing your change to next-release.md. The poly core team moves the content in this file to the changelog for each stable release.

Submitting a Pull Request

Please never force push on your PR, as this makes reviewing incremental changes impossible for us. When we merge your PR, we’ll usually squash it, so that will clean up any rambling work-in-progress.

Special poly Arguments

Some poly arguments are special-case or poly developer-focused. For this reason, the shell will not include these arguments in its autocomplete suggestions by default.

Most special arguments tweak the output of the various poly commands.

all - shell

By default, the shell does not suggest special poly arguments because they are irrelevant and inappropriate for typical users of poly. If you are developing poly itself, suggesting special poly arguments can be appropriate and meaningful. To start a shell with autocomplete suggestions enabled for all poly arguments:

poly :all

changed-files

Have poly react as if specified files have changed. We use this to support testing poly itself.

poly info changed-files:components/file/src/polylith/clj/core/file/core.clj

More than one file can be faked as changed by separating them with colons.

Although multiple files are supported, the shell autocomplete support works only for the first file.

Specify a brick or project path with a trailing forward slash, and poly will treat the brick or project as changed:

poly info changed-files:bases/mybase/:components/mycomponent/:projects/myproject/

fail-if-nothing-to-test

Forces the test command to return a non-zero code (1) if there are no tests to run.

poly test :fail-if-nothing-to-test

fake-sha

Fake the Git sha output by the info command. We use this to generate consistent info output and screenshots for our documentation.

poly info fake-sha:c91fdad

fake-tag

Fake the Git tag for output by the info command. We sometimes use this to generate consistent info output and screenshots for our documentation.

poly info fake-tag:stable-lisa

If you want no Git tag to be shown in info output:

poly info fake-tag:

fake-poly

Fake that you are using a stable production release of poly when you might actually be using a SNAPSHOT release, polyx, or both.

poly help :fake-poly # (1)
polyx :fake-poly # (2)

  1. Excludes -SNAPSHOT from the version at the top of the help output

  2. Shows poly instead of polyx and excludes -SNAPSHOT from the Polylith ASCII art logo

This can be handy when capturing output for docs.

hide-lib-size

If :hide-lib-size is given to the libs command, values in the KB column appear as -:

poly libs :hide-lib-size # (1)

This is used in tests to guarantee that the output will look the same, regardless if a library has been downloaded or not.

latest-sha

Populate the latest-sha from your Git repository in your workspace structure (by default, poly does not populate this value).

poly ws get:settings:vcs:polylith:latest-sha :latest-sha

no-changes

Fake that there are no changes to your workspace since the last stable point in time; can be useful when generating output for docs.

# Returns no rows:
poly diff :no-changes
# Gets rid of the * characters:
poly info :no-changes
# ...has the same effect as:
poly info changed-files:

no-exit

By default, poly explicitly exits via System/exit . If executing poly from a REPL, this will also exit the REPL, which probably isn’t what you want. To avoid that, specify :no-exit.

See poly-cli source comment block for plenty of examples.

Outside of REPL support, :no-exit has little use. If you run, e.g.:

poly info :no-exit

You’ll have to press kbd:[Ctrl+C] to exit!

replace

Manipulate output from the ws command.

You can tell the ws command to search for strings (using regular expressions) and replace the occurrences with another string.

poly ws get:settings:user-home

On Joakim’s system, output is:

"/Users/joakimtengstrand"

But maybe we want some repeatable output for test or documentation purposes, regardless of which user is running the ws. Assuming that your HOME environment variable is set appropriately and that you are running from your OS shell (not poly’s shell):

An example:

poly ws get:settings:user-home replace:$HOME:MY-HOME

Outputs a user-generic:

"MY-HOME"

Another example:

poly ws get:settings:user-config-filename replace:$HOME:MY-HOME

Outputs a user-generic:

"MY-HOME/.polylith/config.edn"

An example with multiple replacements:

poly ws get:settings:user-config-filename replace:$HOME:MY-HOME:config.edn:USER-CONFIG

Outputs:

"MY-HOME/.polylith/USER-CONFIG"

Sometimes you need to deal with your OS shell escaping rules and add, for example, "".

poly ws get:settings:vcs:stable-since:sha replace:"[0-9]+":"*"

The output will vary because git shas are unique, but on Joakim’s system once:

"*e*d*b*cee*fb*e*ff*fafcf"

set

The poly tool starts by reading the workspace into a representation in memory. Before it continues with the next step, to provide that data structure to a command, we get the chance to set a value.

If we want to change the behaviour of the libs command, we can update configuration read from workspace.edn:

 ...
 :validations {:inconsistent-lib-versions {:type :warning
                                           :exclude []}}

Like this:

// Set :type to :error
poly libs set:configs:workspace:validations:inconsistent-lib-versions:type value:error type:keyword

// Set :exclude to ["clj-time/clj-time"]
poly libs set:configs:workspace:validations:inconsistent-lib-versions:exclude value:clj-time/clj-time type:strings

If a value already exists in the workspace structure, it will be converted to the type of that value. If it’s a non-existing value, the type will be a string if not explicitly set. Use type to explicitly give a type.

type string value converted to

boolean

true

the boolean value true

long

123

the long value 123

keyword

warning

the keyword :warning

symbol

hi

the symbol hi

booleans

true:false

A vector [true false]

longs

123

A vector [123]

strings

hi:again

A vector ["hi" "again"]

keywords

hi:again

A vector [:hi :again]

symbols

hi:again

A vector [hi again]

Documentation

We strive to write great documentation for our Polylith users.

AsciiDoc is thoughtfully designed for writing articles; we like that, so all of our articles are written in AsciiDoc.

Some conventions we have found helpful are:

  • Write one sentence per line. Among other things, this helps us to avoid run-on sentences.

  • These particular docs cover an implementation of a Polylith tool, so use the concrete poly instead of the abstract "Polylith tool".

  • For command-line poly create workspace top-ns:my-ns branch:my-branch :commit refer to:

    • create workspace as a command

    • top-ns:my-ns, branch:my-branch and :commit as arguments

  • Link to referenced commands and concepts to make it easier for readers to explore and dig into details.

  • Use the AsciiDoc keyboard macro for keyboard shortcuts. For an example usage, see Shell page.

  • Use "we" when talking from the Polylith core team and "you" when directing the reader through some steps or a tutorial.

  • Try to maintain a casual and friendly tone.

Diagrams

We create our diagrams with the ever-so-awesome Inkscape. Inkscape’s native format is .svg. We export to .png for use in our documentation. We keep the source .svg beside the exported .png in our GitHub repo.

We store doc images under ./doc/images/, for example for the overview diagram in ./doc/doc.adoc doc:

Generated images (currently from bb create-example via polyx) are grouped with their related images but are prefixed with an output dir. This strategy helps us distinguish hand-crafted images from generated ones and reminds us not to edit images in the output dirs.

Sometimes, we’ll annotate a generated image. We do so via an Inkscape SVG file and link (rather than embed) the generated image. As of this writing, if any of these images change, the .svg will have to be manually re-exported to its .png.

Tips:

  • Use a transparent background.

  • We host our docs on cljdoc, which uses a light theme. Folks viewing our docs on GitHub might have opted for a dark theme, so choose colors that will also be dark-theme friendly.

  • Choose cross-platform-friendly fonts. Courier New and Arial seem to be what existing diagrams use. If you are a Linux user, you can install these as part of Microsoft’s fonts.

Cljdoc

We host our docs on cljdoc, which generates docs for our:

  • API by analyzing our source code from our release jar

  • pages (which cljdoc calls "articles") which cljdoc finds under doc, and whose layout is defined by doc/cljdoc.edn.

We have cljdoc build our docs each time we release to clojars. This includes SNAPSHOT releases, which we create for every successful push to master. See Polylith CI Setup for more details.

Previewing Cljdoc Locally

When making changes to docs, you can preview what they will look like on cljdoc. If you have not already done so, clone the polylith repository:

git clone git@github.com:polyfy/polylith.git

Take a look at the help for our babashka doc-preview task:

$ cd polylith
$ bb doc-preview help

Commands:
 start   Start docker containers supporting cljdoc preview
 ingest  Locally publishes poly for cljdoc preview
 view    Opens cljdoc preview in your default browser
 stop    Stops docker containers supporting cljdoc preview
 status  Status of docker containers supporting cljdoc preview
 help    Show this help

Start the server and publish the documentation locally (the first start will take some time to download the cljdoc docker image, and ingest is on the slow side):

$ bb doc-preview start
$ bb doc-preview ingest
Tip
 Commit and push all your changes before you ingest.

Bring the docs up in your web browser with:

bb doc-preview view

Poly Shell and Local Cljdoc

If you are in the polylith workspace root directory (which is also the git repo root dir), you can start a shell specifying the :local argument to tell the doc command to open pages on https://localhost:8000 instead of https://cljdoc.org.

clojure -M:poly :local

Now docs open in your browser against the locally running cljdoc:

polylith$ doc page:component

Refreshing Local Cljdoc Preview

From time to time, there is a new release of cljdoc. To pick it up:

bb doc-preview stop
bb doc-preview start

When you’ve changed the polylith codebase, you need to run ingest again (no need to stop and start):

bb doc-preview ingest

Examples of when you might want to ingest are when you have:

  • Switched to a different git branch.

  • Pulled changes down with git pull.

  • Pushed commits with git push. You must commit and push your changes for ingest to function; doing so from a branch is fine.

Open Page Source in GitHub

Fire up a shell:

clojure -M:poly :local

In addition to opening pages locally instead of on cljdoc.org, the :local argument also enables, for convenience, shell autocompletion for the :github argument. Feel free to also specify :all.

Specify the :github argument to bring up the page in your web browser on GitHub:

polylith$ doc page:component :github

If you started your shell from the polylith repository (not a fork), then pages will automatically be opened in your current working branch; otherwise, pages will open on the master branch.

To choose a specific branch:

polylith$ doc page:component branch:issue-318 :github

If you aren’t running cljdoc locally, you may want to always open pages on GitHub instead of cljdoc; specify the :github argument when starting a shell:

clojure -M:poly :github
polylith$ doc page:component

Referencing a Local poly

So long as you’ve cloned the polylith workspace locally, you can access it from other workspaces. This can be convenient when testing another workspace but wanting to use a local polylith that has some work in progress.

Let’s say you have cloned the Realworld example app as a sibling to the polylith repo in your directory tree.

You could then replace the :poly alias in clojure-polylith-realworld-example-app/deps.edn:

    ...
    :poly {:main-opts ["-m" "polylith.clj.core.poly-cli.core"]
           :extra-deps {polylith/clj-poly {:local/root "../polylith/projects/poly"}}} ;; (1)

  1. Notice the .. relative path to poly

And now, from the clojure-polylith-realworld-example-app dir, you can launch local poly via, e.g.:

clojure -M:poly

Number of downloads on Clojars

The number of downloads per version can be checked by executing:

bb scripts/clojars-stats.bb
Version       | Downloads
-------------------------
0.3.32        |    10,243
0.3.31        |    18,236
0.3.30        |     2,902
0.3.0         |       665
0.2.22        |     4,899
0.2.21        |    18,890
0.2.20        |     4,428
0.2.19        |    17,509
0.2.18        |    25,032
0.2.17-alpha  |   111,701
0.2.16-alpha  |        72
0.2.15-alpha  |       715
0.2.14-alpha  |     2,180
0.2.13-alpha  |     2,084
0.2.12-alpha  |       440
0.2.0-alpha11 |       113
0.2.0-alpha10 |        96
0.1.0-alpha9  |       127
0.1.0-alpha8  |        89
0.1.0-alpha7  |        90
0.1.0-alpha6  |        89
0.1.0-alpha5  |        87
0.1.0-alpha4  |        90
0.1.0-alpha3  |        93
0.1.0-alpha2  |        94
0.1.0-alpha1  |        88

SNAPSHOT Releases & Caching

Clojure tools.deps only checks for updates once per day by default.

This caching behavior means users referencing a specific SNAPSHOT will be more likely to temporarily be on an older release if you frequently release under the same SNAPSHOT version.

Users can -Sforce a check for an update, but most probably won’t think to do that.

Create a SNAPSHOT release

Make a 0.2.20-SNAPSHOT 34 release:

  1. Run all tests: poly test :all

  2. Update the version component and ensure that:

  3. Ensure the versions are correctly specified in readme.adoc:

    • :snapshot-number: is set to 34.

    • :snapshot-version: is set to 0.2.20.

    • :stable-version: is set to 0.2.19.

  4. Run the script scripts/clojars-stats.bb, and update the section on this page.

  5. When adding a new doc page, update the Doc updates section in next-release.md.

  6. Whenever the workspace structure changes:

    • update the Versions table

    • update the code that migrates exported files in the ws-file-reader component.

    • update next-release.md

  7. If a new attribute is added to workspace.edn

    • update all workspace.edn template files under scripts/sections

    • update the content of workspace.edn after "The workspace.edn file looks like:" in workspace.adoc.

  8. Check that :template-data in workspace.edn uses the latest version for :clojure-ver and :shadow-cljs-ver, by executing:

    • clojure -X:deps find-versions :lib org.clojure/clojure | grep -Eo '"\.[0-9]\.[0-9]+"' | tail -1

    • npm view shadow-cljs version

  9. Run bb check to check for reflection warnings and such.

    • If new bricks, add the source directory to the :check alias in ./deps.edn.

  10. Execute bb gen-all

    • Compare the output with what’s in git, and commit the changes if everything looks good.

  11. Verify that we have included everything in next-release.md.

  12. Commit and push your changes.

  13. If the next release is a stable release, build a local version of Cljdoc:

    • Check that new content looks correct.

    • Verify that we have included everything in next-release.md.

    • Check that the badges at the top and the API at the bottom left are displayed correctly (most namespaces should begin with a ^:no-doc tag, to exclude them from the API doc).

  14. Go to the polylith repo and prepare to merge the snapshot-022034 branch into master, by clicking the "Compare & pull requests" button at the top. If the circlci pipeline is green, merge it into the master branch.

  15. When built, find newly built release under releases

    • Edit and update it with "0.2.20-SNAPSHOT 34"

    • List included issues and other changes.

  16. Execute git switch master followed by git pull to get the latest from master.

  17. If the next release is a snapshot release, execute git switch -c snapshot-022035, otherwise follow the instructions for creating a stable release.

Create a stable release

In this example, we want to create a 0.2.20 release out of a 0.2.20-SNAPSHOT 34 release.

Run all the tests:

poly test :all

Announce in good time in the polylith channel in Slack, that people should start testing current SNAPSHOT version. Also mention that there will only be minor changes until release, such as updating the documentation. Wait a few days before you make the new release.

Make a 0.2.20 release:

  1. Execute git switch master followed by git pull to get the latest.

  2. Create the release branch by executing git switch -c release-0220.

  3. Update the version component and ensure that:

  4. Ensure the versions are correctly specified in readme.adoc:

    • :snapshot-number: is set to 0.

    • :snapshot-version: is set to 0.2.20.

    • :stable-version: is set to 0.2.20.

  5. Run the script scripts/clojars-stats.bb, and update the section on this page.

  6. Whenever the workspace structure changes, remember to update the Versions table, and also update the code that migrates exported files in the ws-file-reader component.

  7. Ensure that readme.adoc lists the correct sponsors by cross-checking it against the actual sponsors.

  8. Update the doc and its navigation:

    • If a page has been added to the doc directory, also add it to cljdoc.edn and execute this function to generate the doc navigation.

    • If a blog post, video or high-level doc page has been added, then update more.clj.

    • If any doc content has been added, make sure to update corresponding examples in the doc, e.g. what more outputs.

  9. Search for occurrences of :poly-version: constants, and set the version to 0.2.20 wherever it appears (at the top of .adoc files).

  10. Commit.

  11. Execute bb gen-all

    • Compare the output with what’s in git

    • Commit and push the changes if everything looks correct.

  12. Go to the polylith repo and prepare to merge the release-0220 branch into master, by clicking the "Compare & pull requests" button at the top.

    • Add a comment indicating that this is release 0.2.20.

    • If the circlci pipeline is green, merge it into the master branch.

    • List all issues and PRs with hashes in the commit to ensure they are properly linked.

    • Ensure that the poly-clj documentation has been built.

  13. Update the release notes for the 0.2.20 release:

    • Summarize the release in the introduction.

    • Copy the content from next-release.md into the release notes.

  14. Update the Polylith Homebrew Tap.

  15. Execute git switch master followed by git pull to get the latest from master.

  16. Execute git switch -c snapshot-022100 to create a first snapshot branch for the next release (here we imagine that the next release is 0.2.21).

  17. Update the version component:

    • Update version to 0.2.21 SNAPSHOT #0 (revision to SNAPSHOT, patch to 21, snapshot to 0)

  18. Update readme.adoc:

    • leave :snapshot-number: as 0

    • leave :stable-version: as 0.2.20

    • set :snapshot-version: to 0.2.21

  19. Update next-release.md:

    • Change the heading to "Included since 0.2.20"

    • Clean everything (remove everything under each header)

  20. Create a 0.2.21-SNAPSHOT 0 release, see make a snapshot release for instructions.

    • Add comment: 0.2.21-SNAPSHOT 0 (same as 0.2.20).

    • Check what is included under the tab "File Changed"

  21. Delete old 0.2.20-SNAPSHOT releases.

  22. Announce in #releases if a minor release, or in #announcements if a big release (in clojurians in Slack) by summarizing the most important changes + the included issues and PRs + add a link to the #polylith channel.

  23. Announce on social media