Delete all ddmd. from modules

[RazvanN7]

When the DMD compiler still had portions of C++ and D code, the ddmd. prefix was used to differentiate between the former and the latter. Since that is no longer the case, I erased all ddmd. prefixes from modules and imports.

[UplinkCoder]

It is also so the back-end can be used independently from ddmd.
And to neatly septate the front-end.

[Geod24]

Why would you want to do that ?

[andralex]

Awesome. @MartinNowak @WalterBright ready for pulling the switch? All extant PRs will need to rebase.

[dnadlinger]

Can't we just move the source instead? Having a package for something that is slowly moving towards being a reusable library (at least as per your agenda, and indeed it is already used e.g. by LDC) is hardly a revolutionary idea. This grabs a bunch of top-level namespace entries for no good reason.

[andralex]

@klickverbot "ddmd" is an appendix since the days the C++ and the D code coexisted, now unnecessary. Let's remove it to clean the slate prior to reorganizing the compiler code.

[dnadlinger]

@andralex: I can't agree – it's a prefix, "unnecessary" implies the mismatch was necessary back then, and I don't see why fixing this would require touching everything first to establish a "clean slate", only to then again break things. But go ahead if you are interested in a mostly harmless round of bouncing the rubble.

[andralex]

@klickverbot it was necessary to avoid name clashes IIRC. Anyhow, the basic idea here is to have the module names map 1:1 with the file names. Are you saying it would be better to move all code temporarily from src/ to src/ddmd/?

[dnadlinger]

Anyhow, the basic idea here is to have the module names map 1:1 with the file names. Are you saying it would be better to move all code temporarily from src/ to src/ddmd/?

I agree that having the mismatch is a silly state to be in, just as with the .c files actually being C++. However, we've already had a ddmd -> src symlink in the root directory for a couple of months in an effort to solve the issue (although, if I recall, that came not without issues either). This now seems like yet another – incompatible – quick-shot attempt at the issue which isn't even a clear-cut improvement.

The couple dozen changes to LDC required after this change is merged are certainly trivial to do, but I'd rather not meander back and forth a couple of times only to then end up in the same place because it then turns out that Walter's editor doesn't support working across multiple directories or something silly like that.

[andralex]

@klickverbot the question is what steps do we take to move things forward? @RazvanN7 is working on converting the compiler into a library (or collection of libraries). In all likelihood that will involve things like creating new packages off of src/ for lexer, parser, semantic checking, glue, etc. As a first step it seemed natural to remove the ddmd. cruft throughout. If that seems like a mistaken first step, please give detail on a better one. We can't keep on doing nothing just because everything starts awkwardly. Thanks.

[dnadlinger]

If that seems like a mistaken first step, please give detail on a better one.

In my book, keep things like they are (or move source to ddmd/ matching the package if you prefer), then propose a structure under ddmd.* (or whatever other top-level namespace) which we'll then move to in a concerted effort. The genuine work towards feasibility as a library is not in shuffling files around, but in disentangling DMD's datastructures with their god-object aspirations and lack of clearly defined interfaces/invariants. The actual file-level reorganisation is something all of us can do given 20 minutes and a sed implementation of choice (well, okay, maybe an extra 10 to fix up the Windows makefiles).

However, as I mentioned, if you want to merge this to create the illusion of progress – by all means, feel free to. It might not even be the worst idea, given our historical inability to move forward in such matters at all. Please just make sure the symlink is also cleaned up, etc.

[timotheecour]

I dont see whats wrong with nesting every future ddmd package under ddmd, eg ddmd.ast.definitions Seems cleaner, and less opportunities for name clashes.

…

On Jan 26, 2017 8:58 AM, "David Nadlinger" ***@***.***> wrote: If that seems like a mistaken first step, please give detail on a better one. In my book, keep things like they are (or move source to ddmd/ matching the package if you prefer), then propose a structure under ddmd.* (or whatever other top-level namespace) which we'll then move to in a concerted effort. The genuine work towards feasibility as a library is not in shuffling files around, but in disentangling DMD's datastructures with their god-object aspirations and lack of clearly defined interfaces/invariants. The actual file-level reorganisation is something all of us can do given 20 minutes and a sed implementation of choice (well, okay, maybe an extra 10 to fix up the Windows makefiles). However, as I mentioned, if you want to merge this to create the illusion of progress – by all means, feel free to. It might not even be the worst idea, given our historical inability to move forward in such matters at all. Please just make sure the symlink is also cleaned up, etc. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6489 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACF9YBug0_gSVanqaXNukx9gsvdrZLP_ks5rWNC9gaJpZM4LtYbC> .

[andralex]

if you want to merge this to create the illusion of progres

That's not the purpose here. Thanks for your feedback.

I dont see whats wrong with nesting every future ddmd package under ddmd,
eg ddmd.ast.definitions
Seems cleaner, and less opportunities for name clashes.

An additional package level is not of any use.

[timotheecour]

An additional package level is not of any use.

Why not?
Suppose I want to use ddmd as a library:

module myproject;
import ddmd.ast.definitions; // no name clash
import ddmd.semantic.parsing; // no name clash

// versus this: much higher likelihood of name clash; can't be used (without care) from within a project which starts ast, semantic, etc as top-level

import ast.definitions;
import semantic.parsing;

If every project was like this (no top-level namespace), it'd be hard to mix several projects

[andralex]

@timotheecour sensible enough. OK, let's do this then:

• Rename src/ to dmd/
• Rename ddmd. to dmd.

This will set up the top-level package name for future users to dmd. @WalterBright @MartinNowak @klickverbot @ibuclaw please weigh in. We gotta get the cart moving.

[dnadlinger]

Sounds good to me, although I have a slight preference for ddmd:

• Mindless conservatism. ;)
• It disambiguates nicely between ddmd, the (collection of) libraries for processing D source, and dmd, the compiler.

There is the question of whether to enact any change now or to wait until there is more to be gained, but I have no opinion on that.

[ibuclaw]

This pretty much only affects users who compile sources separately, right? So this will force ldc and gdc to use a specific directory structure underneath their own trees.

Have no battle plan yet for how things will end up looking like, but I use the brand-free directory name dfrontend to store all sources.

[dnadlinger]

This pretty much only affects users who compile sources separately, right? So this will force ldc and gdc to use a specific directory structure underneath their own trees.

The modules currently are prefixed with ddmd, so for separate compilation (and tooling, …) you already need "a specific directory structure". And there is nothing wrong with that.

[RazvanN7]

@andralex @klickverbot @MartinNowak @WalterBright @ibuclaw Is this ok?

[RazvanN7]

@andralex renaming src to dmd forces changes on the druntime and phobos makefiles (src -> dmd). Should I make separate PRs for those as well? That will force everyone to rebase all 3 components. What do you think is the best approach?

[Geod24]

@RazvanN7 : I still don't understand why you can't mv src/*.d src/ddmd/ ?
That won't change the Makefile locations but make the directory structure match with the declarations.

[PetarKirov]

I agree with @Geod24. Many prominent D projects use this directory structure. For example:
https://github.com/dlang/dub/tree/master/source
https://github.com/sociomantic-tsunami/ocean/tree/v2.x.x/src/
https://github.com/vibe-d/vibe-core/tree/master/source
https://github.com/higgsjs/Higgs/tree/master/source
https://github.com/libmir/dcv/tree/master/source/
https://github.com/libmir/mir-cpuid/tree/master/source/
https://github.com/libmir/mir-glas/tree/master/source/
https://github.com/stonemaster/dlang-tour/tree/master/source
https://github.com/Hackerpilot/libdparse/tree/master/src
https://github.com/DlangScience/scid/tree/master/source/
https://github.com/economicmodeling/containers/tree/master/src
https://github.com/atilaneves/unit-threaded/tree/master/source
https://github.com/nomad-software/dunit/tree/master/source
https://github.com/atilaneves/reggae/tree/master/src
https://github.com/Abscissa/SDLang-D/tree/master/src
https://github.com/atilaneves/cerealed/tree/master/src
https://github.com/DerelictOrg/DerelictUtil/tree/master/source
https://github.com/DerelictOrg/DerelictGL3/tree/master/source
https://github.com/buggins/dlangui/tree/master/src
https://github.com/Pure-D/workspace-d/tree/master/source
https://github.com/gtkd-developers/GtkD/tree/master/src
https://github.com/DmitryOlshansky/pry/tree/master/src
https://github.com/tsbockman/checkedint/tree/master/source
https://github.com/msgpack/msgpack-d/tree/master/src
https://github.com/huntlabs/collie/tree/master/source
https://github.com/rejectedsoftware/diet-ng/tree/master/source
https://github.com/caraus-ecms/tanya/tree/master/source
https://github.com/BlackEdder/ggplotd/tree/master/source
https://github.com/etcimon/libasync/tree/master/source
https://github.com/etcimon/botan/tree/master/source
https://github.com/Abscissa/scriptlike/tree/master/src

Some repositories are collection of subprojects:
https://github.com/rejectedsoftware/vibe.d
https://github.com/rejectedsoftware/vibe.d/blob/v0.8.0-beta.2/core/vibe/core/core.d#L11
https://github.com/d-gamedev-team/gfm
https://github.com/d-gamedev-team/gfm/blob/v6.2.3/sdl2/gfm/sdl2/audio.d#L1
https://github.com/AuburnSounds/dplug
https://github.com/AuburnSounds/dplug/blob/v4.0.27/graphics/dplug/graphics/color.d#L15
https://github.com/eBay/tsv-utils-dlang

And there are some that are in between:
https://github.com/adamdruppe/arsd/
https://github.com/adamdruppe/arsd/blob/v1.1.0/audio.d#L1
https://github.com/gecko0307/dlib/
https://github.com/gecko0307/dlib/blob/v0.10.0/dlib/async/package.d#L34
https://github.com/CyberShadow/ae
https://github.com/CyberShadow/ae/blob/v0.0.1869/utils/async.d#L14
https://github.com/PhilippeSigaud/Pegged/
https://github.com/PhilippeSigaud/Pegged/blob/master/pegged/peg.d#L11

But the common thing is that almost all of them use a namespace prefix. This is important because otherwise you would not be able to import modules from different projects, like would be the case with
https://github.com/gecko0307/dlib/blob/v0.10.0/dlib/async/package.d#L34
https://github.com/CyberShadow/ae/blob/v0.0.1869/utils/async.d#L14
if they didn't have namespace prefix to disambiguate between them.

E.g. if you drop ddmd. from https://github.com/dlang/dmd/blob/v2.073.0/src/lexer.d#L11, you won't be able to import it in the same project with https://github.com/Hackerpilot/libdparse/blob/v0.7.0-beta.3/src/dparse/lexer.d#L1 if libdparse didn't use the dparse package prefix.

[MartinNowak]

First step, there shouldn't be much disagreement with moving sources to src/d?dmd/, git also recognizes renames and only required rebasing after lots of edits following up the rename (renames are detected by source similarity).
Whether to call the top module dmd or ddmd, I don't care.

[andralex]

@MartinNowak excellent. @RazvanN7 let's move sources to src/dmd/ and take it from there. Thanks!

[RazvanN7]

@andralex The last commit moves all .d sources to src/dmd and renames all modules to dmd.$module_name. I kept the same tree hierarchy as in src so, all c files and headers are in src/ , src/backend , src/root etc. and all d files are in dmd/, dmd/backend/, dmd/root/ etc. I also modified the makefile to generate .o .deps and dmd files in a separate directory called generated in the root directory (on the same level with src).

[andralex]

Folks, things like the following were fine as long as we only distributed the compiler as libraries:

• lack of correspondence between the directory structure and the module names, requiring tricks such as slinks or command-line extras
• binaries generated in the same place as sources, requiring things like .gitignore. Most projects set up a separate directory where generated artifacts are produced
• legacy/unexplainable names that require an understanding of project history rather than project status

We need to fix these as a prerequisite to deploying the compiler as a library. For now we can reasonably assume that the third party tools authors can be contacted to update their chains. Let's go with /src/dmd, and place generated stuff in /generated/. Stuff still to do:

• Change the windows makefiles as well
• Use generated/OS/MODEL like in phobos

[mathias-lang-sociomantic]

Speaking of tools author, CC @CyberShadow

[timotheecour]

Why not use ddmd instead of dmd as others have suggested? I see no disadvantage. Using dmd will introduce needless confusion (eg try regex '\bdmd\b' to search for use of binary dmd in code, which will be made worse when its used as a library and ppl import dmd.something)

…

On Feb 22, 2017 7:27 AM, "Mathias Lang" ***@***.***> wrote: Speaking of tools author, CC @CyberShadow <https://github.com/CyberShadow> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6489 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACF9YJtqsGAaS4q89ePIM3ugaYCTAc8Rks5rfFPtgaJpZM4LtYbC> .

[andralex]

@timotheecour "legacy/unexplainable names that require an understanding of project history rather than project status"

[timotheecour]

I disagree; this issue concerns project status, not history; once this is all done, dmd will refer to both the dmd binary basename, as well as the package name for the dmd compiler. Source of gratuitous confusion for anyone relying on grep-like tools, searching in google, etc. Again, no disadvantage for using ddmd (or please explain which ones you see).

[andralex]

@timotheecour understood and noted, thanks. Decision stays.

[CyberShadow]

binaries generated in the same place as sources, requiring things like .gitignore. Most projects set up a separate directory where generated artifacts are produced

You will need a .gitignore either way.

We need to fix these as a prerequisite to deploying the compiler as a library. For now we can reasonably assume that the third party tools authors can be contacted to update their chains. Let's go with /src/dmd, and place generated stuff in /generated/.

I don't see the motivation for this churn.

A simple solution that solves all problems (that I've understood) has been proposed: move source files to src/ddmd/, and leave the binary generated in the old place (src/dmd). This fixes the discrepancy between file and module names, and doesn't break any tools or scripts. What's wrong with this again?

[andralex]

@CyberShadow ddmd is not and will not be the name of the project. Please let's not argue anymore, thanks.

To ease transition, @RazvanN7 will do it in three steps.

1. Generate the binary elsewhere, keep a (link to) it in the current location. This way existing tools won't be broken.
2. Update tools that assume the dmd is generated in src/ to use the correct location.
3. Proceed with moving the code as per this PR.

[CyberShadow]

When I complained about setExt and setExtension being two functions with slightly different meanings, you said (paraphrasing) "What's the big deal? I type it and it works."

When we proposed moving the language spec to its own subdirectory, so that it could be searchable properly, you were against the change because it was too much churn for too little benefit.

In light of the above, how do you expect me to interpret your message?

If my word still has any weight: This is not your decision to make, or mine. Please leave it to the people who are actually working on the compiler code, which is neither you or me. (I would add tool authors to that category, but that would include me and I am obviously biased.)

I have no motivation to make changes forced upon me by inconsistent inarguable authoritarian decisions, sorry. I answer to the community, not BDFLs, so please spend some time explaining your arguments. I don't know if you meant it or not, but "Please let's not argue anymore, thanks" is very demotivating to read when you are on the receiving end and it forces work upon you and your tools' users - especially for such a whimsical issue as a naming decision.

More to the point:

1. Generate the binary elsewhere, keep a (link to) it in the current location. This way existing tools won't be broken.
2. Proceed with moving the code as per this PR.

These two are mutually exclusive, because you can't have both a src/dmd directory and a src/dmd file/symlink.

[andralex]

This is not your decision to make, or mine. Please leave it to the people who are actually working on the compiler code, which is neither you or me.

@CyberShadow Walter is behind this. Generally, no need to worry - we're not experiencing with dictatorship here. If you have any further concerns I have reached out on IRC and email with my phone number and skype so we can talk. Thanks!

[andralex]

I'll be out for a couple hours and just noticed my phone battery died, so before I leave let me frame the problem in terms of an imaginary future discussion:

"So what's the name of the compiler project under github?"

"dmd under the dlang organization."

"Cool, I see it in github. Nice! I expect the source to be in src/dmd, awesome. Wait! It's actually src/ddmd. How come?"

"Well ddmd was an intermediary name because initially dmd was written in C++ and we had to avoid name clashes for a while."

"So now are you done with the C++ conversion?"

"Yes."

"Why did it stay ddmd?"

"Well it also generates code in src/. I know this doesn't make sense, but that what it was years ago and we couldn't change it."

"So you couldn't have a dir and a binary with the same name. Got it. Well was this something public? Did a bunch of folks depend on that nonsensical oddity? Generated code has no business in src/, literally."

"Well there were only a handful of projects depending on that location of the binary."

"I see a bunch of other projects on github and elsewhere that follow the convention projectname on github, src/projectname or source/projectname is where the source code is. Not src/xprojectname. Isn't it bothersome that you have an obsolete and surprising deviation from the usual straight in the middle of things? Why didn't those handful of projects change their paths instead?"

"Well one core member just refused to do so so we couldn't change anything."

Clearly different weights can be put on one of another argument, and ten people have eleven ideas on what's best to do. But we can't have one solution that satisfies everyone simultaneously. @CyberShadow, clearly you and I need to have consensus, which is why it would be great if we could talk live. Thanks.

[CyberShadow]

Walter is behind this.

Good, but I don't see him responding to the community's arguments in this thread. No cabals, please.

If you have any further concerns I have reached out on IRC and email with my phone number and skype so we can talk.

Appreciated, but I think we would benefit from more discussions happening in the open rather than in private.

let me frame the problem in terms of an imaginary future discussion

Except this discussion will never happen.

To put things into perspective, we are talking about internal file / package names, not even something that will be visible to D users. There are many more stranger naming decisions that we can't do anything about:

• Why is the standard library called Phobos?
• Why is part of the standard library in a separate repository called Druntime? (Does anyone even use D with Druntime but not Phobos today?)
• Why is the reference D compiler called dmd, and not dc or dlangc OSLT?
• Why is there a "D-Programming-Deimos" GitHub organization?
• Why are standard modules spread over 3 root-level packages (core, std and etc)?
• What exactly is Digital Mars, and what is its affiliation with the D Language foundation?
• Why is the standard library documentation generated with something called DDox, which is never mentioned anywhere in the language spec?
• What exactly is a tuple, anyway?

Well one core member

BTW I think we would benefit from fewer labels that divide the community into us vs. them. Can I just be the guy who maintains the forum and some tools, like the documentation tester? :)

These two are mutually exclusive

Still waiting for an answer to this. All else doesn't matter if the goals you argue for are literally impossible.

[mathias-lang-sociomantic]

Why is part of the standard library in a separate repository called Druntime? (Does anyone even use D with Druntime but not Phobos today?)

Yes.

Let me add to your bullet list: why are druntime's sources living in src but Phobos' are in top-level ?

[dejlek]

If I was working on these projects I would have them in src/d/. Examples: src/d/phobos src/d/dmd src/d/druntime etc. To some this may be like structure inherited from Maven, but actually to me it makes lots of sense especially in projects with mixed languages (C, C++, D code all in one project).

[andralex]

@CyberShadow please let's stay on topic. (To wit, we're already at what's wrong with druntime and phobos.) Just because we don't (yet!) improve everything doesn't mean we should improve nothing.

I understand how annoying it is to have a seemingly arbitrary decision imposed on you. On the other hand, we cannot afford to make the smallest decisions by means of debate - it's just ineffective. Nor can we afford to be a Polish democracy in which things move along only if there's unanimity - though it does feel like that sometimes :). Also, no organization makes all decisions by means of referendum, and that doesn't make them cabals.

The short of it is we can't afford to justify every little thing to smithereens, and especially not matters in which reasonable people may disagree. More often than not flamewars end in blockage. Worse yet, they tend to spiral into negativity as it happened just now. Look at us - a few good folks all animated by the best of intentions, and before long we get dangerously close to obeying Godwin's Law. We need to shed this pattern. This metapoint, and not this particular PR, is what I'm looking forward to discussing with you.

I noticed some arguments have been made but a few remained somewhat implicit. So let me restate them:

• Up until now, building and distributing dmd has been a binary of relatively narrow interest, so a variety of unpleasant little things were acceptable. Among them:
  • package and module names do not align with the directory layout
  • project only builds with special flags or by passing all module names in one command line (this is amazing!)
  • a symlink would help moving the cheat along
  • binaries were generated in the same place as sources (like, literally in the src/ directory where by definition they don't belong)
  • we had ddmd as a package name although nobody ever named the project by that name
• Now we're moving to distributing the compiler as libraries, so it would be a good opportunity to clean house given that the set of folks working with the compiler will likely increase
• In the process of doing so, we're trying to simply adhere to simple and popular conventions

That's pretty much it. Now I understand there may be amendments and qualifications to each of these arguments, to which there are further amendments and qualifications, but the point is the argument exists; it's not an arbitrary decision that comes out of thin air. That's why I said "let's not argue about this" - meaning being we could easily do it forever, to no conclusion. The positive thing to do is do one thing together, instead of everybody arguing some slightly different thing is the only right thing to do.

Thanks.

[DmitryOlshansky]

Speaking of 'dmd' name - for the whole compiler

[DmitryOlshansky]

Pardon, typing on mobile has its downsides.

I was wondering if we should call the D frontend a 'dmd' which is Digital Mars D. Customers in the near future using the frontend will keep asking what this company has to do with and if its truly open source. It's the same kind of historical baggage as 'ddmd' is. Something like 'dfront' or 'dlang' might make more sense.

[andralex]

@DmitryOlshansky yes, if we want to rebrand the compiler this would be the right time. There'd be many aspects involved. I'll discuss with @WalterBright, Thanks!

[CyberShadow]

OK, here is a proposed course of action:

• Name the compiler source code package and the directory containing the source files dlang, and leave it in src..
• Put generated files in /generated/ or anywhere you wish.
• Leave a symlink in src/dmd for compatibility.

This resolves all issues, yes? And it differs from the earlier proposal only by replacing ddmd with dlang, and is actually implementable.

[CyberShadow]

The short of it is we can't afford to justify every little thing to smithereens, and especially not matters in which reasonable people may disagree.

I will need to disagree with that. We are humans, we make mistakes, arguments help iron out those mistakes and arrive to an optimal solution (as I hope we're doing right now). I would not want to participate in a project where unargumented decisions are forced upon others, since I don't know how much that decision makes sense and which arguments have been considered while arriving to it. Poignantly, I am "glad" that by "holding a piece of critical infrastructure hostage" I've forced a discussion to happen that will achieve an improved solution :)

To clarify, this is not cool: "I've discussed this with Walter, Walter says he's OK with it, you need to go do this now. End of discussion, let's please not waste time on any more debates." Instead, it would be much better if there would be a list of the arguments discussed, the points in favor, the points against the decision and whey they are outweighted by the pros, etc. For one thing, I'm skeptical that among the arguments you discussed in private were whether a variation of the plan consisting of one little 'd' would result in either immediate massive breakage of everything building DMD from source, or be literally unimplementable.

[CyberShadow]

If you have any further concerns I have reached out on IRC and email with my phone number and skype so we can talk.

Phone call summary:

• There was a miscommunication regarding point 2/3 from Andrei's plan - the idea was to have a grace period for tool authors and users to update their tools/setups and use the new DMD location. A duration of one week was mentioned.
• I personally think that calling the package/directory exactly dmd is not worth the tool breakage, but Walter/Andrei disagree. We've discussed the pertinent arguments, but since this is a subjective disagreement I've agreed to update Digger and the other tools I maintain to use a new DMD binary location.
• We've found a workaround for a DMD bug that was previously blocking a new Digger release, so a new release should happen soon.
• Making decision-making more transparent is something we're looking forward to improving.

The next step is for someone (@RazvanN7?) to move forward with step 1 from Andrei's plan:

1. Generate the binary elsewhere, keep a (link to) it in the current location. This way existing tools won't be broken.

[RazvanN7]

@CyberShadow I already submitted a new PR which tackles phase 1 : #6562 . I already did the modifications for the posix makefile and I am working on modifying the windows one. I suggest we move the discussion to that PR. Also, I do not have a windows machine so I am currently spaming the auto-tester on dlang.org. If someone with windows developing experience wishes to aid me we can co-op the PR.

Thanks