[nix] Use niv and poetry2nix to more easily manage dependencies

[purcell]

Description

Here are some changes that improve the Python side of the nix environment and allow us to have recent packages without explicitly overriding anything: we add a Poetry manifest, and then use poetry2nix to create our python env from that. Additionally, the popular niv tool is introduced and used to manage the nixpkgs version.

In their basic form, these changes probably appear barely worth the effort: bear in mind, though, that the python packages are constantly evolving in PyPi while they are only intermittently frozen in nixpkgs, which means that if the qmk python dependencies need to be updated, then nix overrides or nixpkgs pull requests can be necessary in order to make those python packages available. The poetry2nix route instead allows the latest pypi packages to be pulled in, and this can be done independently of nixpkgs updates.

Types of Changes

• Core
• Bugfix
• New feature
• Enhancement/optimization
• Keyboard (addition or update)
• Keymap/layout/userspace (addition or update)
• Documentation

Issues Fixed or Closed by This PR

N/A

Checklist

• My code follows the code style of this project: C, Python
• I have read the PR Checklist document and have made the appropriate changes.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• I have tested the changes and verified that they work and don't break anything (as well as I can manage).

[fauxpark]

#pragma weak

[skullydazed]

Is it possible to either update this more often or not pin versions? MILC is up to version 1.0.12 and we'll be releasing a new version that pulls in deps develop will need soon. I'm worried about this falling behind even further as qmk progresses.

[purcell]

Is it possible to either update this more often or not pin versions? MILC is up to version 1.0.12 and we'll be releasing a new version that pulls in deps develop will need soon. I'm worried about this falling behind even further as qmk progresses

If versions weren't pinned, how would the newer MILC get picked up? Are you thinking you would just get it updated in nixpkgs?

In general pinning is exactly the right thing to do to make this work reliably, but of course manually bumping hashes etc. is not super convenient. If the project were to have a poetry manifest for the python deps then it would be pretty easy to automate this with poetry2nix, which can grab arbitrary pypi package versions (instead of being restricted to those registered in nixpkgs), but probably using poetry is a larger change that would need socialisation.

[skullydazed]

For better or worse most platforms get "whatever the latest" is at install time. Our story for keeping everyone up to date is still being written. I recognize that this does not guarantee that it's always installable, but in practice we haven't run into an issue yet. This isn't a permanent situation, but it is the current situation.

Homebrew users have pinned versions due to the nature of how homebrew works, but we have people within qmk motivated to keep that up to date. I'd like to make sure nix users aren't left behind if they're to be pinned as well.

[purcell]

Homebrew users have pinned versions due to the nature of how homebrew works, but we have people within qmk motivated to keep that up to date. I'd like to make sure nix users aren't left behind if they're to be pinned as well.

There's probably a few of us who'd be willing to make sure updates get into nixpkgs too.

[skullydazed]

Could we start with this PR? Both hjson and MILC have updates.

[tzarc]

@purcell @skullydazed if I read this correctly, we were waiting on updates to python packages? What's the status?

[purcell]

I've pushed some changes here that improve the Python side of the nix environment and allow us to have recent packages without explicitly overriding anything: we add a Poetry manifest, and then use poetry2nix to create our python env from that. I need to do a little more testing locally, but this should be much easier to maintain.

[purcell]

Okay, this has now been rebased against master, includes the latest python dependencies listed there, and with this code I can use the bin/qmk compile command to successfully compile firmwares inside a nix shell. 🎉

[skullydazed]

This is pretty clever, but unfortunately we won't be able to accept poetry.lock and pyproject.toml at the root dir due to potential for confusion. Can you put those files into a nix specific directory, maybe util/nix?

[purcell]

Yeah, that should be possible. A nix subdir at the project top level would be my suggestion, fitting general nix best practice.

[purcell]

I've made that change (using "nix" as the subdir) and I've also switched to "niv" for managing the pinned nixpkgs version, for easy future updates (e.g. "niv update nixpkgs").

[purcell]

I updated this to a newer nixpkgs snapshot too, just to keep things more current. I verified that I can still build a randomly-selected firmware.

[con-f-use]

#12295 can be merged NOW, while this more advanced PR here is being discussed along with the general approach to python dependencies in qmk.

[purcell]

I don't think this one should be too controversial tbh. I made the changes suggested by @skullydazed, and @fauxpark was in favour.

[con-f-use]

As said in the other PR, I'm very much in favor of the poetry to nix approach here. Just saying the other one is a quick no-brainer. This one should be merged eventually, too (or now if people are happy).

[purcell]

Got it, thanks for the support.

[purcell]

Ugh, must have closed this by accident, apologies. I'll resolve the conflict now.

[purcell]

There we go, just building locally to confirm. I'll update the PR description, though, since an interim update to shell.nix will technically have resolved the Big Sur compatibility issue.

[purcell]

Confirmed working.

[purcell]

Thanks for the merge, @skullydazed.