Use neat-interpolation instead of interpolate

[sjakobi]

In addition to interpolate, this also removes the following
transitive dependencies:

• haskell-src-exts
• haskell-src-meta
• safe
• th-expand-syns
• th-lift
• th-lift-instances
• th-orphans
• th-reify-many

Since neat-interpolation's text quasiquoter trims more whitespace
than interpolate's i, the expected output of some tests for
unsafeGetAttrPos had to be updated.

This also improves the test failure output of the constantEqual
helper.

Fixes #634.

[sorki]

Looks like nixpkgs spots version = "0.3.2.6"; - should be good if you relax the lower bound I guess.

[sjakobi]

Thanks @sorki. I'm giving this a try. I originally went with >= 0.4 because there seems to have been a little behaviour change that might be relevant (I'm not sure):

http://hackage.haskell.org/package/neat-interpolation-0.5.1/changelog

If this doesn't work out, I'd try using neat_interpolation_0_5_1.

[sjakobi]

Test failures: https://travis-ci.org/github/haskell-nix/hnix/jobs/700029072#L7031-L7081

The parser test failures are probably due to the additional newline with n-i < 0.4.

The other ones unfortunately don't say what's wrong. I'll try changing that.

[sjakobi]

How can I reproduce the issue of the incompatible n-i version as in https://travis-ci.org/github/haskell-nix/hnix/builds/699920672 locally?

When I follow the instructions from the readme (https://github.com/haskell-nix/hnix#getting-started), I end up building neat-interpolation-0.5.1 with the cabal v2-build step, but that doesn't seem to happen in CI.

[sjakobi]

@Anton-Latukha Do you have any advice for me regarding my reproducibility problem above? Should I just switch to nix-build for this? (I have limited disk space though…)

My goal is to figure out how to modify the Nix setup so it uses neat_interpolation_0_5_1 from nixpkgs.

[Anton-Latukha]

Yes - there is a definite difference between nix-build and nix-shell -> cabal v2-build. I - do know there is a difference. Thou, I do not know its definition. I never read about cabal v2-* internal processes and what is exposed by nix-shell and picked-up by cabal. Cabal definitely sees that it is running in nix-shell environment, and so loads the Haskell packages as Nix packages (from Nixpkgs into /nix/store). Cabal states that build and deps management is "Nix like" - I do not know what they mean by that.

Probably if Nix was better - the Cabal would've reused and integrated with Nix more closely.

---

Yes, I know there is a difference between CI and README instructions. Since nix-build is definite reproducibility - it is used in CI, and we see here that cabal v2- does not copy its reproducibility.

1. John was from the start pointing to nix-shell -> cabal v2-* - that is more casually known way for people, they feel like they not getting separated from Cabal.

2. It is strange for people to come into HNix project and we direct them to use Nix to build the project, our project constantly faces the chicken and egg problem.

3. But maybe to direct them to Cabal is as strange as to Nix when HNix should be a own packge manager.

For me - I would point the people to nix-build for reproducibility, especially now when default.nix has so many features. We should create feature presets for people. And refactor the README, fix a couple of commands there, extend it with nix-build.

---

If to reproduce the error:

nix-build

If you want to reproduce CI build almost fully:

nix-build --quiet --quiet --argstr compiler ghc8101 --arg allowInconsistentDependencies false --arg doJailbreak false --arg doCheck true --arg sdistTarball false --arg buildFromSdist true --arg failOnAllWarnings false --arg buildStrictly false --arg enableDeadCodeElimination true --arg disableOptimization false --arg linkWithGold true --arg enableLibraryProfiling false --arg enableExecutableProfiling false --arg doTracing false --arg enableDWARFDebugging false --arg doStrip false --arg doHyperlinkSource false --arg enableSharedLibraries true --arg enableStaticLibraries false --arg enableSharedExecutables false --arg justStaticExecutables false --arg checkUnusedPackages false --arg doCoverage false --arg doHaddock true --arg doBenchmark true --arg generateOptparseApplicativeCompletions true --arg executableNamesToShellComplete '[ "hnix" ]' --arg withHoogle false

Also trough time the revision of the channel/repository can/changes.

For one build - the disc space is relatively similar to classic build. But on further builds to free space - you would need to garbage-collect the store so it removes old builds.

---

Regarding the package.

It is also a caveat of Nixpkgs community, that people like to dodge to be by default with the current upstream, and instead of following the stream of progress - just stall the progress inside the store and form side-packages for some time, to replace default package afterwards. It is just perpetual laziness, more over it is not complete (effective) laziness. Nixpkgs functions should be responsible for total package migrations, it is not hard to arrange and it would be easier for everyone, but nobody dragged it through RFC review (and so community rust to old way) so far.

---

In Nixpkgs there is haskellPackages.neat-interpolation_0_5_1, notice the haskellPackages namespace, and notice that the package has other symbols (-,_) from which you mentioned.

The changes like haskellPackages.neat-interpolation to haskellPackages.neat-interpolation_0_5_1 are always - an overrides (overshadowing, overloads, rewriting paradigm) of values.

In developPackage it is a bit trickier, since it is more rarely used/searchable function, and also because it imports from Cabal descriptions. So remove ineterpolate from there and put neat-interpolation there.

So, you opne the developPackage in both local sources, and its definition (developPackage =) inside Nixpkgs to see its source/docs.

Nixpkgs: https://github.com/NixOS/nixpkgs/blob/a9422ba768f5c1a18866c778015610a1ca8c09eb/pkgs/development/haskell-modules/make-package-set.nix#L218-L244

And in the source. It should become:

package = haskellPackages.developPackage {
...

  overrides = self: super: {
    neat_interpolation = pkgs.haskellPackages.neat-interpolation_0_5_1;

    # OR Also possible:
    neat_interpolation = pkgs.haskellPackages.callHackage "neat_interpolation" "0.5.1" {};

    # OR even:
    neat_interpolation = hack (and (override (as (soul (desires (pkgs.haskellPackages.neat-interpolation))))));

  };


...
};

I am not sure of haskellPackages.developPackage.overrides scoping, maybe haskellPackages.neat-interpolation = haskellPackages.neat-interpolation_0_5_1.

The self: super: is a lambda \ self super ->. I write \ n1 ... with space because it is beautiful and GHC parser loves me more because of this.

Also do not forget that nix-shell always remembers... the environment that was initially loaded.

---

nix-build - is always reproducible. If you preserve the input information (Nixpkgs/channel revision and loaded Nix configuration). If there is one thing Nix is and stays - Nix is always reproducible.

If pure build is set (today almost everywhere it is pure build), and even if the environment is unpure in 98% of cases it does not change the needed reproducibility.

Many times/days it can be seen and feel like it is its falt and that it does not reproduce. If there is one thing about Nix - Nix is always reproducible, it is the referentially transparent purely functional language as Haskell.

[sjakobi]

Many thanks for the explanation, @Anton-Latukha, and especially for the hint regarding - vs. _! ;)

[Anton-Latukha]

Yes, mentioned since that could've been the whole cause of it.

Also when writing so big complex texts it seems almost mandatory to have typos in it, so sorry for those.

[sjakobi]

BTW, for some reason cabal's --test-options option doesn't seem to work right with this testsuite, so I'm using cabal run instead of cabal test:

$ cabal run hnix-tests -- -p unsafegetattr
Up to date
hnix
  EvalTests
    unsafegetattrpos1: FAIL
      tests/EvalTests.hs:457:
    unsafegetattrpos2: FAIL
      tests/EvalTests.hs:457:
    unsafegetattrpos3: FAIL
      tests/EvalTests.hs:457:
    unsafegetattrpos4: FAIL
      tests/EvalTests.hs:457:

4 out of 4 tests failed (0.02s)

[Anton-Latukha]

Maybe this is related: Cabal has some differences in options between v1 and v2 --keys, their forms and their availability, mostly they are the same, but there is a difference.

[Anton-Latukha]

As you know Cabal currently does switcharoo with v1 and v2 and default, so I put the explicit v2 into Readme, so those instructions are explicit and independent from the Cabal version people have.

[sjakobi]

These are the remaining failing tests:

    unsafegetattrpos1:                                               FAIL
      tests/EvalTests.hs:457:
    unsafegetattrpos2:                                               FAIL
      tests/EvalTests.hs:457:
    unsafegetattrpos3:                                               FAIL
      tests/EvalTests.hs:457:
    unsafegetattrpos4:                                               FAIL (0.01s)
      tests/EvalTests.hs:457:

Here's one of the test cases:

hnix/tests/EvalTests.hs

Lines 199 to 212 in e14cbf4

	case_unsafegetattrpos1 =
	constantEqualText "[ 6 20 ]" [text|
	let e = 1;
	f = 1;
	t = {};
	s = {
	inherit t e f;
	a = 1;
	"b" = 2;
	c.d = 3;
	};
	p = builtins.unsafeGetAttrPos "e" s; in
	[ p.line p.column ]
	|]

and the helpers that we use

hnix/tests/EvalTests.hs

Lines 448 to 470 in e14cbf4

	constantEqual :: NExprLoc -> NExprLoc -> Assertion
	constantEqual a b = do
	time <- getCurrentTime
	let opts = defaultOptions time
	-- putStrLn =<< lint (stripAnnotation a)
	res <- runWithBasicEffectsIO opts $ do
	a' <- normalForm =<< nixEvalExprLoc Nothing a
	b' <- normalForm =<< nixEvalExprLoc Nothing b
	valueEqM a' b'
	assertBool "" res
	constantEqualText' :: Text -> Text -> Assertion
	constantEqualText' a b = do
	let Success a' = parseNixTextLoc a
	Success b' = parseNixTextLoc b
	constantEqual a' b'
	constantEqualText :: Text -> Text -> Assertion
	constantEqualText a b = do
	constantEqualText' a b
	mres <- liftIO $ lookupEnv "ALL_TESTS" <|> lookupEnv "MATCHING_TESTS"
	when (isJust mres) $
	assertEvalMatchesNix b

So apparently the quasi-quoted expressions still parse fine but their normal forms are inequal to the expected values according to valueEqM:

hnix/src/Nix/Value/Equal.hs

Lines 155 to 172 in e14cbf4

	valueEqM
	:: forall t f m
	. (MonadThunk t m (NValue t f m), Comonad f)
	=> NValue t f m
	-> NValue t f m
	-> m Bool
	valueEqM ( Pure x) ( Pure y) = thunkEqM x y
	valueEqM ( Pure x) y@(Free _) = thunkEqM x =<< thunk (pure y)
	valueEqM x@(Free _) ( Pure y) = thunkEqM ?? y =<< thunk (pure x)
	valueEqM (Free (NValue (extract -> x))) (Free (NValue (extract -> y))) =
	valueFEqM (compareAttrSetsM f valueEqM) valueEqM x y
	where
	f (Pure t) = force t $ \case
	NVStr s -> pure $ Just s
	_ -> pure Nothing
	f (Free v) = case v of
	NVStr' s -> pure $ Just s
	_ -> pure Nothing

[sjakobi]

The test output I added looks absolutely bonkers to me:

  EvalTests
    unsafegetattrpos1: FAIL
      tests/EvalTests.hs:462:
      Inequal normal forms:
      Expected: [ 6 20 ]
      
      Actual:   [ 5 14 ]
    unsafegetattrpos2: FAIL
      tests/EvalTests.hs:462:
      Inequal normal forms:
      Expected: [ 6 20 ]
      
      Actual:   [ 5 14 ]
    unsafegetattrpos3: FAIL
      tests/EvalTests.hs:462:
      Inequal normal forms:
      Expected: [ 7 13 ]
      
      Actual:   [ 6 7 ]
    unsafegetattrpos4: FAIL
      tests/EvalTests.hs:462:
      Inequal normal forms:
      Expected: [ 8 13 ]
      
      Actual:   [ 7 7 ]

How could changing the text interpolation result in such different expressions?!

[sjakobi]

Oh, now that I've taken a closer look at the actual test cases, it makes more sense:

We're determining the position of attributes in the interpolated text, and these positions change consistently: The line number changes by -1, the column by -6.

[sjakobi]

Any idea why the jobs with GHC 8.4.4, 8.6.5 and 8.10.1 would still fail with

Setup: Encountered missing dependencies:
neat-interpolation >=0.4 && <0.6

?

[sorki]

builder for '/nix/store/vyd67qgmgmnqgayl00xsxfgbs60bp03f-hnix-source-0.9.0.drv' failed with exit code 1 suggests that its failing during cabal sdist phase but I'm not sure why..

[sjakobi]

@sorki Which job is that from? The most recent failures in https://travis-ci.org/github/haskell-nix/hnix/builds/701250357 all have

Configuring hnix-0.9.0...
<snip>
Setup: Encountered missing dependencies:
neat-interpolation >=0.4 && <0.6

[sorki]

From https://travis-ci.org/github/haskell-nix/hnix/jobs/701250361

It's deceptive as you can only see it after it fails as there are two failing jobs listed:

builder for '/nix/store/vyd67qgmgmnqgayl00xsxfgbs60bp03f-hnix-source-0.9.0.drv' failed with exit code 1

cannot build derivation '/nix/store/drggygz5bdc4gym2k0bs762x6zbbmzhi-hnix-0.9.0.drv': 1 dependencies couldn't be built

The latter is actual hnix package (according to name) and hnix-source is the cabal sdist attempt.

[sorki]

I can reproduce - looks like it needs overlay instead of local override.

.. no dice. Looks bit more complicated.

Btw to reproduce you can run nix-build --arg sdistTarball true

[sjakobi]

Thanks for looking into it @sorki!

Curiously

nix-build --arg sdistTarball true

succeeds for me. Are we using the same tree? I'm at 2601c58, which is the current branch HEAD.

[sorki]

Weird, yes I've fetched the branch and its the same commit.

My nixpkgs are at 2fd146f though (master from like two days ago) but with forced GHC8.10 package set instead of the default GHC8.8.

[sjakobi]

My nixpkgs are at 2fd146f though (master from like two days ago) but with forced GHC8.10 package set instead of the default GHC8.8.

Oh! How do I update my local nixpkgs?

It would be nice if builds were more reproducible BTW, but I guess that's a separate discussion…

[sorki]

Clone https://github.com/NixOS/nixpkgs/ and checkout the rev, then you can do NIX_PATH=nixpkgs=/path/to/nixpkgs_checkout nix-build .. - this will download a bunch of more recent stuff from cache so expect loosing a chunk of free disk space.

The reason why it's not pinned by default is to be able to roll with nixpkgs in CI and me not fond of this default as the pin is typically behind nixpkgs and pulls bunch of older things with older GHC.

Pinning is preserved though and can be enabled with --arg useRev true.

[sjakobi]

Thanks @sorki! It does look a bit tedious though.

How about using the pinned revision by default to simplify debugging situations like this one?

[sjakobi]

The reason why it's not pinned by default is to be able to roll with nixpkgs in CI and me not fond of this default as the pin is typically behind nixpkgs and pulls bunch of older things with older GHC.

Ah, but you wouldn't want that because it makes local builds slower and eats more disk space?!

I'm really confused by all this. In dhall-haskell it's pretty simple to make the nix build use a specific package version. You simply add the output of cabal2nix to https://github.com/dhall-lang/dhall-haskell/tree/master/nix/packages.

[sorki]

The reason why it's not pinned by default is to be able to roll with nixpkgs in CI and me not fond of this default as the pin is typically behind nixpkgs and pulls bunch of older things with older GHC.

Ah, but you wouldn't want that because it makes local builds slower and eats more disk space?!

Its fine but IMHO not a good default since you already have some GHC and Haskell packages present on your system and pinned nixpkgs forces another one. Like I don't have disk space issues anymore, sure it slows things down but it also forces me to either disable the pin or pass nixpkgs as arguments when I want nix-build to use my systems pkgs.

We can resolve this in some CI related issue if you find it a bad default :) Also https://github.com/expipiplus1/update-nix-fetchgit/ is related as it can be used to automatically update the pins which could be used instead current rolling nixpgks URL for CI.

I'm really confused by all this. In dhall-haskell it's pretty simple to make the nix build use a specific package version. You simply add the output of cabal2nix to https://github.com/dhall-lang/dhall-haskell/tree/master/nix/packages.

Yes, I think that's just a bit of overlays magic. I've tried to move your override just after https://github.com/haskell-nix/hnix/blob/master/default.nix#L102 but that's not enough it seems. Might be better (or even required) to pass overlays directly to import <nixpkgs> { overlays = [ ... ]; } but I haven't tried that yet. I've tried tracing why it's not respected when used with sdistTarball but it's not clear to me yet.

[Anton-Latukha]

Regarding #635 (comment)

They are reproducible as much as possible. Otherwise, we would need to statically pin Nixpkgs and move it around all the time, which is hardly a good solution. It is analog to drop-in the current library version and statically link it.

One of points to not to pin - is that we are in sync with all Haskell ecosystem and Nixpkgs store - is to track how we live in the ecosystem and are we a good citizen, who is our neighbors and what their news in life and that we must adjust our relationships to keep them great, we should be aware of the world around us, and that Nixpkgs in line with us. Otherwise, developers are tempted to just diverge and use some two-five-year old library or even the whole software store revision in case of Nixpkgs pinning, and drag all things around with them, because the project rusted into it, compiles in it.

And yes, the pinned revision would get cleaned-up from Hydra CI cache at some point, and project builds/developers would need to compile the whole software stack, instead of downloading it ready from CDN. Just like recently, I tried to compile in NixOS unsupported HIE with older GHC and everything, and it needs a powerful machine and 10 hours to compile the whole project stack good if starting only from GHC.

And for HNix to work with the Nixpkgs seems only logical, we must work and build relationships with those for which we do the project.

---

The case of reproducibility here is very simple. Nix (HNix) during the build should resolve and emit Nixpkgs rev (commit) it uses. Since it is referentially transparent builds - it is trivially logical that it should list its inputs. Then we would be able to get it and pass into nix-build --arg useRev true --arg rev <hash> and referentially transparent receive the very same build. Moreover tooling should be able just emit the inputs information, so it can be saved and passed into the tooling later, it is essentially a drv in Nix.

Because Nix does not report its inputs - currently it is tricky to get such useful simple info, like Nixpkgs rev. Right now you basically need to get the time of the build, well, if you even have that info provided, then get the state of related local channel, then look what Hydra channel build is that, then look what Nixpkgs rev Hydra channel build is that. Instead of tolling just stating its referential transparent input, isn't that the point of the tool?

[sjakobi]

Many thanks for the extensive explanations, @sorki and @Anton-Latukha! :)

I'm still a bit unconvinced that the current unpinned default is so good, but you make good points and you have more experience with the nix ecosystem.

Regarding the synchronisation with the rest of the Haskell ecosystem, I think the inclusion of hnix in Stackage should mostly take care of that.

In the meantime, I've tried to help update the neat-interpolation version in nixpkgs via commercialhaskell/stackage#5476, but I don't expect it to work out very soon.

I'm still curious why the override I added isn't effective with the other GHC versions.

This PR isn't very urgent though and I don't expect it to acquire many merge conflicts soon, so maybe it can just wait a bit.

[sjakobi]

Note that this change will result in less leading whitespace in the interpolated text. I don't think it hurts in this case though.

[sjakobi]

So 8a75b5e seems to have fixed the CI job with GHC 8.10, but now those with 8.4 and 8.6 are broken:

https://travis-ci.org/github/haskell-nix/hnix/jobs/702322002:

Setup: Encountered missing dependencies:
containers ==0.6.*, stm ==2.5.*, time ==1.9.*

https://travis-ci.org/github/haskell-nix/hnix/jobs/702322003:

Setup: Encountered missing dependencies:
time ==1.9.*

I don't understand how 8a75b5e could have caused this. Any ideas?

[sorki]

Heh, that's rebase package related and probably due to the fact that we're forcing newer neat-interpolation for older package sets that have neat-interpolation ==0.3.2.6 forced. Seems like the new one triggers rebase and for example for nix-shell -p 'haskell.packages.ghc865.ghcWithPackages (ps: [ time ])' I get time-1.8.0.2.

Not sure what's the best approach here yet.

[sjakobi]

@sorki I don't understand how rebase comes into play here. It doesn't seem to be a transitive dependency of hnix.

[sorki]

@sorki I don't understand how rebase comes into play here. It doesn't seem to be a transitive dependency of hnix.

Looks like rerebase is part of the neat-interpolation testsuite https://github.com/nikita-volkov/neat-interpolation/blob/master/neat-interpolation.cabal#L51 which in turn pulls in rebase. You can try adding pkgs.haskell.lib.dontCheck for the neat-interpolation override to see if it helps - it should stop pulling test deps.

[sjakobi]

You're a Nix wizard, @sorki! :)

[sorki]

@sjakobi you'll soon become one too! ;)

[sorki]

LGTM, thanks!

[sjakobi]

Thanks again for all the help with this, @Anton-Latukha and @sorki! :)

[Anton-Latukha]

Great. An additional point to HNix.

Now in next release, we would see closure size drop.