`cabal init --tests` does not create any tests if not using --lib{andexe}

[kindaro]

Describe the bug

cabal init --tests does not create any tests.

To Reproduce

% cabal init --tests
Warning: The package list for 'hackage.haskell.org' is 23 days old.
Run 'cabal update' to get the latest list of available packages.

Guessing dependencies...

Generating LICENSE...
Warning: unknown license type, you must put a copy in LICENSE yourself.
Generating CHANGELOG.md...
Generating app/Main.hs...
Generating example.cabal...

Warning: no synopsis given. You should edit the .cabal file and add one.
You may want to edit the .cabal file and add a Description field.
% tree
.
├── app
│   └── Main.hs
├── CHANGELOG.md
└── example.cabal

1 directory, 3 files

Expected behavior

A directory is created with some tests. The tests are wired into example.cabal as appropriate.

System information

% cabal --version
cabal-install version 3.4.0.0
compiled using version 3.4.0.0 of the Cabal library 
% uname
Linux

[fgaz]

This issue is happening because by default cabal initializes a package with only an executable, and there is no way to wire tests directly to an executable without a library in the middle, as far as I can see.

Pheraps --tests without --lib or --libandexe should error out and point to those flags, or it should imply them. Or it could generate a non-wired testsuite, though that's probably less useful.

As a workaround, cabal init --libandexe --tests does generate a test.

If on the other hand you meant to initialize tests in an already existing package, this is not currently possible, though there are plans to allow editing .cabal files with cabal commands: #7337.

[kindaro]

This issue is happening because by default cabal initializes a package with only an executable, and there is no way to wire tests directly to an executable without a library in the middle, as far as I can see.

However, observe that you can have a test suite without either a library or an executable, and the default test suite being generated by cabal init does not depend in any way on either. Simply run cabal init --lib --tests and edit out the library stanza — the test suite still builds and runs!

That is to say, the test suite is not and does not need to be wired to any other stanza.

[fgaz]

That is true, but what would such a testsuite test? Answering that myself: other packages in a project, maybe in combination. Or there could be an overlap in the source directories, though I think that's discouraged.

So maybe it would be best to generate the testsuite but also point out in a warning/notice that the user may want to use --libandexe to test the code in the same package (without rebuilding stuff due to overlapping src-dirs).

/cc @emilypi @ptkato

[fgaz]

and the default test suite being generated by cabal init does not depend in any way on either

this may be another issue. when generating a lib+exe, the exe is made to depend on the lib since that's what one usually wants, so maybe the same should apply to lib+test

[kindaro]

My case is such that I needed a package with exactly one test-suite stanza. I was having an issue with property checks in a big project, and I wanted to write some small examples separately. That is to say, a feature of generating a standalone test suite would not be without use.

[emilypi]

Hmmm. First off, I would not call this a bug @fgaz, since it's explicitly the behavior of the init script to disallow creating test suite stanzas without an associated library. Primarily because init is for initializing new projects in the sense of libraries or executables. Second, I can see this behavior being legitimate, but it is very non-standard. We'd have to add support for standalone test-suites, since --tests, despite its historical naming convention, only provides a flag for initializing test suites. It doesn't tell the script that it wants to create a test stanza as a project type. To get this behavior, we probably need to add a new flag.

Ideally, something like this:

λ P temp → λ git master* → ../dist-newstyle/build/x86_64-osx/ghc-8.10.4/cabal-install-3.5.0.0/x/cabal/noopt/build/cabal/cabal init --simple --test-suite --minimal

Using cabal specification: 3.0

Warning: unknown license type, you must put a copy in LICENSE yourself.
Creating CHANGELOG.md...
Creating file CHANGELOG.md...
Creating directory ./test...
Creating file test/Main.hs...
Creating file temp.cabal...

Warning: no synopsis given. You should edit the .cabal file and add one.
You may want to edit the .cabal file and add a Description field.
λ P temp → λ git master* → tree
.
├── CHANGELOG.md
├── cabal.tix
├── temp.cabal
└── test
    └── Main.hs

1 directory, 4 files
λ P temp → λ git master* → rm cabal.tix 
λ P temp → λ git master* → tree        
.
├── CHANGELOG.md
├── temp.cabal
└── test
    └── Main.hs

1 directory, 3 files
λ P temp → λ git master* → cat temp.cabal 
cabal-version:   3.0
name:            temp
version:         0.1.0.0
license:         NONE
build-type:      Simple
extra-doc-files: CHANGELOG.md

test-suite temp-test
    default-language: Haskell2010
    type:             exitcode-stdio-1.0
    hs-source-dirs:   test
    main-is:          Main.hs
λ P temp → λ git master* → 

Is this what you're going for @kindaro?

[kindaro]

Please understand: I use Cabal as a tool to get the job done. Not a guru developer, not an occasional user either. So:

• It is not mine to have opinions as to how command line flags should be interpreted. I cannot account for backwards compatibility, difficulties of implementation, and so on. I only ever used maybe a tenth of all the variety of flags Cabal provides overall in its plethora of sub-commands.
• I can see when a given flag does not make sense — it makes me spend time adding other flags and reading help messages. This waste of time can be justified, say by demands of backwards compatibility, but it cannot be said to be good. It might be a wontfix, but it is still a bug. It being documented, briefly or extensively, does not make it any less a fault.

If you show me the results of «hallway testing» where people figure out to invoke cabal init --simple --test-suite --minimal sooner than cabal init --tests when they want Cabal to initialize a test suite, I shall be surprized, but I shall accept my experience to be an exception. (Or re-do the study.) But so far I am your «hallway».

I do not mean this as saying that I am somehow special and entitled. If you have other means of measuring the goodness of user experience, please go for it. I am fine with any behaviour as long as it is clear and consistent. All I can say is that, when I type cabal init --tests and it gives me an executable, it is neither clear nor consistent. (Compare with cabal init --lib which initializes a library!)

Cabal already has a myriad of flags that work or do not work depending on their exact combination. Especially since cabal init is a convenience feature, it better be convenient!

[emilypi]

Ah no I think maybe I wasn't clear - I provided a minimal example as a proof of concept for what this workflow might look like if we added --test-suite as a flag. It doesn't exist currently, and I'm suggesting we can add it to handle your use case 😄

The --minimal flag is just to cut down on comments in the .cabal, which makes it easier to read, and --simple was used to shortcut the prompting process.

Just for background, what i mean when i say "initializing a test suite", I mean that the --tests flag is a boolean value which tells the script to generate a test suite if and only if a library is created. It is a confusing flag which doesn't quite make sense, since it's dependent upon the other package type flags, which tell cabal what kind of package to generate. The solution here would be to introduce a new package type called "Test Suite" into the list of available package types, and then to provide a flag which tells cabal to generate a package of that type. This is what --lib is doing, for example, and this is what I mean when i say "introduce a --test-suite flag", which would act like --lib and --exe.

I agree that what we have now is poor, ergonomically. I'm agreeing with you that this is could be improved!

[kindaro]

Is this what you're going for @kindaro?

… I provided a minimal example as a proof of concept for what this workflow might look like if we added --test-suite as a flag. It doesn't exist currently, and I'm suggesting we can add it to handle your use case 😄

Sure, the result looks right! If there is a way to put it into Cabal in such a way that it does not increase confusion, that would be good.