[REG2.065.0] stdin.byLine.each!(map!somefunc) compiles, fails to link with ld

[dlangBugzillaToGithub]

Simon Naarmann (@SimonN) reported this on 2015-12-17T02:58:39Z

Transferred from https://issues.dlang.org/show_bug.cgi?id=15459

CC List

• ag0aep6g
• Walter Bright (@WalterBright)
• FeepingCreature (@FeepingCreature)
• Vladimir Panteleev (@CyberShadow)
• John Colvin
• ZombineDev
• Siarhei Siamashka

Description

http://dpaste.dzfl.pl/ce501c212d32

This code calls byLine, each, map!somefunc. It compiles fine, but fails to link. Tested on my Linux machine with DMD 2.069.2 and on dpaste with DMD 2.069.1.

Expected: Either an error during compilation, or a well-linked executable.

Code copied here for convenience:

    #!/usr/bin/rdmd

    import std.stdio;
    import std.algorithm;

    char somefunc(in char c)
    {
        return c;
    }

    void main()
    {
        stdin
            .byLine
            .each!(map!somefunc);
    }

dmd outputs:

/d773/f942.o: In function `_D3std9algorithm9iteration32__T3mapS21_D4f9428somefuncFxaZaZ11__T3mapTAaZ3mapFNaNbNiNfAaZS3std9algorithm9iteration41__T9MapResultS21_D4f9428somefuncFxaZaTAaZ9MapResult': /d773/f942.d:
(.text._D3std9algorithm9iteration32__T3mapS21_D4f9428somefuncFxaZaZ11__T3mapTAaZ3mapFNaNbNiNfAaZS3std9algorithm9iteration41__T9MapResultS21_D4f9428somefuncFxaZaTAaZ9MapResult+0x30):
undefined reference to 
`_D3std9algorithm9iteration41__T9MapResultS21_D4f9428somefuncFxaZaTAaZ9MapResult6__ctorMFNaNbNcNiNfAaZS3std9algorithm9iteration41__T9MapResultS21_D4f9428somefuncFxaZaTAaZ9MapResult' 
collect2: error: ld returned 1 exit status 
--- errorlevel 1

[dlangBugzillaToGithub]

petar.p.kirov commented on 2015-12-18T17:47:43Z

When you're iterating any kind of string (char, wchar or dchar based) the element type is always dchar. This special case for narrow strings (UTF-8 and UTF-16) is also known as UTF autodecoding. You can see this for yourself if you change `somefunc` into a template and add a pragma statement to make the compiler emit an information message about the type of `c` for each instantiation of the function:

auto somefunc(C)(C c)
{
	pragma (msg, C);
	return c;
}

See also: http://dlang.org/phobos/std_range_primitives.html#.ElementType,
http://dlang.org/phobos/std_range_primitives.html#.front and
http://dlang.org/glossary.html#narrow strings

This is still a valid bug report, because the compiler should refuse to compile this code.

[dlangBugzillaToGithub]

s.naarmann commented on 2015-12-19T04:23:42Z

Thanks for the thorough explanation!

[dlangBugzillaToGithub]

dlang-bugzilla (@CyberShadow) commented on 2017-06-25T12:44:51Z

The example presented is invalid, as the parameter of each is not expected to return anything, and map is lazy and by itself does not mutate global state (is pure if its predicate is). It also doesn't compile any more in the latest version of D.

I'm closing this for now, but please reopen if you can provide a working practical example.

[dlangBugzillaToGithub]

petar.p.kirov commented on 2017-06-25T12:55:05Z

Vladimir, shouldn't this bug be closed as "WORKSFORME", as the bug was that the compiler accepted the invalid example in the first place?

[dlangBugzillaToGithub]

dlang-bugzilla (@CyberShadow) commented on 2017-06-25T13:08:16Z

(In reply to ZombineDev from comment #4)
> Vladimir, shouldn't this bug be closed as "WORKSFORME", as the bug was that
> the compiler accepted the invalid example in the first place?

Well, the code wasn't technically invalid, just nonsensical. It stopped working after this PR:

https://github.com/dlang/phobos/pull/5148

I'm not sure if that behavior change was intentional either, hence my comment there.

There is also no practical difference between INVALID and WORKSFORME in practice, I think.

[dlangBugzillaToGithub]

ag0aep6g commented on 2017-06-25T13:41:11Z

(In reply to Vladimir Panteleev from comment #3)
> I'm closing this for now, but please reopen if you can provide a working
> practical example.

Reduced accepts-invalid:

----
enum e = is(typeof(map()));

struct MapResult()
{
    this(char[] a) {}
    
    char front()
    {
        dchar d;
        return d; /* should fail compilation */
    }
}

void map()()
{
    auto mr = MapResult!()([]);
}

void main()
{
    map();
}

[dlangBugzillaToGithub]

dlang-bugzilla (@CyberShadow) commented on 2017-06-25T14:38:42Z

(In reply to ag0aep6g from comment #6)
> (In reply to Vladimir Panteleev from comment #3)
> > I'm closing this for now, but please reopen if you can provide a working
> > practical example.
> 
> Reduced accepts-invalid:

This is not a valid reduction because .front is never called. The compiler seems to be skipping semantic analysis of uncalled functions here to avoid forward reference issues: https://github.com/dlang/dmd/pull/2851

[dlangBugzillaToGithub]

ag0aep6g commented on 2017-06-25T15:39:18Z

(In reply to Vladimir Panteleev from comment #7)
> This is not a valid reduction because .front is never called.

front is never called in the original code either.

> The compiler
> seems to be skipping semantic analysis of uncalled functions here to avoid
> forward reference issues: https://github.com/dlang/dmd/pull/2851

So the invalid front is supposed to be accepted (if it's never called)? Then I suppose the accepts-invalid aspect of this issue is WONTFIX.

But it should link then, no?

[dlangBugzillaToGithub]

dlang-bugzilla (@CyberShadow) commented on 2017-06-25T15:50:41Z

(In reply to ag0aep6g from comment #8)
> So the invalid front is supposed to be accepted (if it's never called)? Then
> I suppose the accepts-invalid aspect of this issue is WONTFIX.

Forward references and generally order of semantic analysis is a tricky subject and I'm not qualified to answer with authority whether the behaviour here can actually be called a bug or not, however I do know that we generally want the compiler to be as lazy as possible/reasonable for performance reasons. For example, currently unit tests are not even parsed without unittest, so you can put any series of tokens in an unittest block as long as the curly braces are balanced. So, even if there is a bug here, fixing it would not align with long-term goals.
 
> But it should link then, no?

Oh right, I missed that the test case you posted also exhibits the link failure.

[dlangBugzillaToGithub]

dlang-bugzilla (@CyberShadow) commented on 2017-06-25T16:02:42Z

Regression (changing the compilation error into a link error + possible accepts-invalid) introduced in https://github.com/dlang/dmd/pull/2851

[dlangBugzillaToGithub]

bugzilla (@WalterBright) commented on 2018-12-14T06:29:28Z

(In reply to Simon Naarmann from comment #0)
> http://dpaste.dzfl.pl/ce501c212d32
> 
> This code calls byLine, each, map!somefunc. It compiles fine, but fails to
> link. Tested on my Linux machine with DMD 2.069.2 and on dpaste with DMD
> 2.069.1.
> 
> Expected: Either an error during compilation, or a well-linked executable.
> 
> Code copied here for convenience:
> 
>     #!/usr/bin/rdmd
> 
>     import std.stdio;
>     import std.algorithm;
> 
>     char somefunc(in char c)
>     {
>         return c;
>     }
> 
>     void main()
>     {
>         stdin
>             .byLine
>             .each!(map!somefunc);
>     }
> 
> dmd outputs:

I get:

test.d(13): Error: template `std.algorithm.iteration.each!(map).each` cannot deduce function from argument types `!()(ByLine!(char, char))`, candidates are:
c:\cbx\mars\phobos\std\algorithm\iteration.d(890):        `std.algorithm.iteration.each!(map).each(Range)(Range r) if (!isForeachIterable!Range && (isRangeIterable!Range || __traits(compiles, typeof(r.front).length)))`
c:\cbx\mars\phobos\std\algorithm\iteration.d(925):        `std.algorithm.iteration.each!(map).each(Iterable)(auto ref Iterable r) if (isForeachIterable!Iterable || __traits(compiles, Parameters!(Parameters!(r.opApply))))`

So it works for me.

[dlangBugzillaToGithub]

s.naarmann commented on 2018-12-16T03:57:32Z

(In reply to Walter Bright from comment #11)
> So it works for me.

The code from the opening post now fails to compile in DMD v2.083.1 and the linker won't be called. That's good.

But ag0aep6g's reduced case from Comment 6 -- https://issues.dlang.org/show_bug.cgi?id=15459#c6 -- still compiles with DMD v2.083.1 on 64-bit Linux, then fails to link. Linker error below. Expected instead: That code should fail compilation, as annotated via /* should fail compilation */.

    /usr/bin/ld: a.o: in function `_D1a__T3mapZQfFNaNbNiNfZv':
    a.d:(.text._D1a__T3mapZQfFNaNbNiNfZv[_D1a__T3mapZQfFNaNbNiNfZv]+0x27): undefined reference to `_D1a__T9MapResultZQl6__ctorMFNaNbNcNiNfAaZSQBp__TQBqZQBu'
    collect2: error: ld returned 1 exit status
    Error: linker exited with status 1

Or should we file a separate bug for that?

[dlangBugzillaToGithub]

bugzilla (@WalterBright) commented on 2020-02-12T04:01:34Z

The Comment 6 example still fails to compile. It looks like the speculative compilation in the `is(typeof(map))` wound up hiding the error in somefunc().

[dlangBugzillaToGithub]

dlang-bot commented on 2021-11-25T15:36:43Z

@RazvanN7 created dlang/dmd pull request #13359 "Fix Issue 15459 - [REG2.065.0] stdin.byLine.each!(map!somefunc) compiles, fails to link with ld" fixing this issue:

- Fix Issue 15459 - [REG2.065.0] stdin.byLine.each\!(map\!somefunc) compiles, fails to link with ld

https://github.com/dlang/dmd/pull/13359

[dlangBugzillaToGithub]

razvan.nitu1305 commented on 2022-11-10T14:24:02Z

*** Issue 16481 has been marked as a duplicate of this issue. ***