Skip to content

Conversation

@dkgroot
Copy link
Contributor

@dkgroot dkgroot commented Dec 18, 2017

Bootstrapped via v2.067.1: See

All tests / unittests were completed successfully both on Linux as well as DragonFlyBSD after the port. (This was prior to rebasing, where 37d37a6 is currently failing)

Note: This port only supports 64-bit, because DragonFlyBSD is currently x86_64 only (i386 support removed a while back). This is also the reason why the dmd option '-m32' is being suppressed when the target is dragonflybsd.

Links:

@dlang-bot
Copy link
Contributor

dlang-bot commented Dec 18, 2017

Thanks for your pull request, @dkgroot! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.

Some tips to help speed things up:

  • smaller, focused PRs are easier to review than big ones

  • try not to mix up refactoring or style changes with bug fixes or feature enhancements

  • provide helpful commit messages explaining the rationale behind each change

Bear in mind that large or tricky changes may require multiple rounds of review and revision.

Please see CONTRIBUTING.md for more information.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

@dkgroot
Copy link
Contributor Author

dkgroot commented Dec 18, 2017

Question Regarding Bootstrapping on DragonFlyBSD
I settled on (after quite a bit of experimenting and back and forth See):

  • dmd v2.067.1
  • druntime dmd-cxx (port based on v2.067.1 also available)
  • phobos dmd-cxx (port based on v2.067.1 also available)

But i am not sure any of these branches are 'still' open for Pull Requests / Updates ?
If not, I could just provide the zip file with the dmd compiler, if requested.

Notes:

  • I could not get dmd-cxx to compile cleanly on linux (because of -fPIC / hardening issues on ubuntu). Fixing this takes quite a large number of changes and i did not want to mix PR's
  • I could not get dmd-cxx to compile the current master (2.077.1) on dragonflybsd (Lots of linking errors)
  • v2.067.1 worked flawlessly out of the box.

Can one of you guide me what would be preferred ? Should we fix dmd-cxx first and then rebase my changes on top of it / just go with v2.067.1 for dragonflybsd changes.

Copy link
Contributor

@wilzbach wilzbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow. First of all. Thanks a lot for your work. I start with first, quick review & tiny nit and answering your questions.

test/d_do_test.d Outdated
(!testArgs.requiredArgs.empty ? " " : ""),
testArgs.permuteArgs);

version (DragonFlyBSD) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Please add a comment that DragonFlyBSD is only x86_64 here.

@wilzbach
Copy link
Contributor

But i am not sure any of these branches are 'still' open for Pull Requests / Updates ?

Of course. As explained on the mailing list, dmd-cxx has been created a couple of months ago & @ibuclaw has been working on backporting fixes to it since, e.g. his last PR: #7227

Can one of you guide me what would be preferred ? Should we fix dmd-cxx first and then rebase my changes on top of it / just go with v2.067.1 for dragonflybsd changes.

tl:dr: how you bootstrap DMD is imho completely your own choice. As far as I can judge this doesn't affect "upstream" much, except you can / want to setup up a CI for DragonFlyBSD (which would be really cool). I would just go with the easiest way which seems to be v2.067.1 for now.
I don't know how package management works on DragonFlyBSD - will users be able to install DMD as binary package directly?

FYI: @WalterBright is pushing to convert the backend to D too (see e.g. #6907) and for this a couple of new features like -betterC would be required, which means there might be another bootstrapping waypoint.

The long-term plan was to maintain dmd-cxx, s.t. the life of porters is simplified and the number of bootstrapping points is reduced.
However, maintaining the backend and glue code is a lot of work and as no one has volunteered for it, so I doubt that that much work will happen here in the near future. dmd-cxx is based on 2.068.2, contains all commits until the latest cdmd tag + Ian's backfixes. It would be cool to be able to compile master directly from dmd-cxx, but to be honest with the fast compilation time of DMD, it's a lot easier just use one more bootstrapping point and save all the hard energy required to add fixes and backports to dmd-cxx and invest it instead in building more cool things in D.
Also nowadays LDC is almost in sync with the latest DMD release and I presume it's a lot easier to use the LLVM's backend which supports cross-compilation for porting than bootstrapping through a couple of releases.

@dkgroot
Copy link
Contributor Author

dkgroot commented Dec 18, 2017

But i am not sure any of these branches are 'still' open for Pull Requests / Updates ?

Of course. As explained on the mailing list, dmd-cxx has been created a couple of months ago & @ibuclaw has been working on backporting fixes to it since, e.g. his last PR: #7227

I read the mailinglist and forum post (Referenced in the PR).

tl:dr: how you bootstrap DMD is imho completely your own choice. As far as I can judge this doesn't affect "upstream" much, except you can / want to setup up a CI for DragonFlyBSD (which would be really cool).

I tried doing this on Travis (See, which worked, but was to slow, so it timed out during unittesting (mostly because running qemu-system on top of travis is really slow and i had to create a dragonflybsd installation (from iso) during testing). The process could have been split up in several stages (By storing the intermediate results somewhere). For now i gave up on this one.
I then tried using a x86_86-pc-dragonflybsd-gcc cross compiler (built for linux), but the dmd build system is not expecting a cross compiler (i think).
If you guys have a hosted environment (Like the auto tester uses), i would be happy to help setting it up.

I would just go with the easiest way which seems to be v2.067.1 for now.

Ok once we conclude this PR for master I can push the PR for dragonflybsd support on v2.067.1. Seperating them might be a good idea, to keep focus. If you like me to push the PR right away, just let me know.

I don't know how package management works on DragonFlyBSD - will users be able to install DMD as binary package directly?

DragonFlyBSD uses the 'Ports' from FreeBSD and does provide binary packages as well. Once we have all the PR's committed,I will update the FreeBSD ports Makefile to use the latest git revision. When FreeBSD accepts the updates, the results will automatically get picked up on DragonFlyBSD. It's a bit of a stack to work through, but we will get there.

FYI: @WalterBright is pushing to convert the backend to D too (see e.g. #6907) and for this a couple of new features like -betterC would be required, which means there might be another bootstrapping waypoint.

No problem. Now that everything is working/running I already know where attention is needed.

The long-term plan was to maintain dmd-cxx, s.t. the life of porters is simplified and the number of bootstrapping points is reduced.
However, maintaining the backend and glue code is a lot of work and as no one has volunteered for it, so I doubt that that much work will happen here in the near future. dmd-cxx is based on 2.068.2, contains all commits until the latest cdmd tag + Ian's backfixes. It would be cool to be able to compile master directly from dmd-cxx, but to be honest with the fast compilation time of DMD, it's a lot easier just use one more bootstrapping point and save all the hard energy required to add fixes and backports to dmd-cxx and invest it instead in building more cool things in D.
Also nowadays LDC is almost in sync with the latest DMD release and I presume it's a lot easier to use the LLVM's backend which supports cross-compilation for porting than bootstrapping through a couple of releases.

I do agree. The biggest problem i had was figuring out the porting process and figuring out which revision worked for me. My first attempt was using dmd-cxx, but it was not able to compile the current master. The funny thing i did find out was that i could use dmd-cxx to compile 2.068.2, and then use it to compile master. Something must have gone wrong in the dmd-cxx backporting process. I would have loved to fix dmd-cxx first, but when focussing on a port you cannot also fix the compiler at the same time (See the forum posts). When trying to fix dmd-cxx first, i ran into the -fPIC/hardening issue ubuntu. To much stuff at the same time.

Ideas:

  • might be a good idea to have a wiki page about the porting process
  • backport the massive number of -fPIC PR's to dmd-cxx
  • figure out what prevents the current dmd-cxx version from compiling master
  • Add bootstrap stage using dmd-cxx to all CI's, so that everyone is forced to keep it up date.

@dkgroot dkgroot changed the title Dragonflybsd port Add DragonFlyBSD support for dmd Dec 18, 2017
test/d_do_test.d Outdated
testArgs.permuteArgs);

// DragonFlyBSD is x86_64 only, instead of adding DISABLED to a lot of tests, just exclude them from running
version (DragonFlyBSD) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curly braces on their own lines.

@wilzbach
Copy link
Contributor

I tried doing this on Travis (See, which worked, but was to slow, so it timed out during unittesting (mostly because running qemu-system on top of travis is really slow

Hmm you could give CircleCi (2 vCPUs, 4GB) or Semaphore a try. They both are a lot faster.
On CircleCi you can roll your own docker containers, maybe that already helps?
Alternatively, GDC for example, uses buildbot.

If you guys have a hosted environment (Like the auto tester uses), i would be happy to help setting it up.

CC @braddr (the person in charge of auto-tester).

It might also be possible to add more machines to the Jenkins CI (CC @MartinNowak).

might be a good idea to have a wiki page about the porting process

Agreed and based on your recent experience you are an excellent candidate to start such a page ;-)

Add bootstrap stage using dmd-cxx to all CI's, so that everyone is forced to keep it up date.

Currently the following CIs test bootstraping:

  • auto-tester (uses v2.068.2 - last cdmd release)
  • Travis (tests bootstrapping with gdc, ldc and dmd)

CircleCi, DAutotTest and Jenkins use a more recent, but fixed, DMD version.

Keeping dmd-cxx up to date is a good proposal, but as I said before, it needs an volunteer(s).

backport the massive number of -fPIC PR's to dmd-cxx

I can help with that (it's just adding -fPIC flags to a lot of places), but FWIW we still can't run parts of the test suite on a PIE hardened OS (though that's the last missing bit, see e.g. #7420)

@dkgroot
Copy link
Contributor Author

dkgroot commented Dec 18, 2017

CC @braddr (the person in charge of auto-tester).
It might also be possible to add more machines to the Jenkins CI (CC @MartinNowak).

I think, we should move this to a seperate PR when we get working on this. Thanks for the links though.

Add bootstrap stage using dmd-cxx to all CI's, so that everyone is forced to keep it up date.

I was not expecting anything else :-)

Currently the following CIs test bootstraping:

  • auto-tester (uses v2.068.2 - last cdmd release)
  • Travis (tests bootstrapping with gdc, ldc and dmd)
    Auto Tester should not be a problem
    Travis will be a bit of an issue, because of the speed. Running unittests in qemu is quite slow. Building bootstrap + master + druntime + phobos + all the unittests, is certainly going to time out.

CircleCi, DAutotTest and Jenkins use a more recent, but fixed, DMD version.

Keeping dmd-cxx up to date is a good proposal, but as I said before, it needs an volunteer(s).

I can help with that (it's just adding -fPIC flags to a lot of places), but FWIW we still can't run parts of the test suite on a PIE hardened OS (though that's the last missing bit, see e.g. #7420)

Once the dust settles on this (has only been a couple days) I will give it another shot.

@joakim-noah
Copy link
Contributor

Nice work. As for bootstrapping, the problem is that there's currently no strategy on how dmd wants to handle that for new platforms. ldc, on the other hand, puts out regular releases of its C++ ltsmaster branch, based on the dmd 2.068.2 frontend, which is continually tested on CI, so it's easy to do there.

If I were you, I'd focus on getting ldc ported first, then you could use your ldc package to build the latest dmd easily. There is little value in porting the last C++ dmd, especially given the confusion about how dmd currently wants to deal with such ports.

Ideally, I'd like to see the dmd-cxx frontend updated as far as Iain got it, and then we could work to make sure it's a viable branch on all existing platforms, that could be used to bootstrap new x64 platforms. But ldc ltsmaster is already here and it works.

@dkgroot
Copy link
Contributor Author

dkgroot commented Dec 19, 2017

@joakim-noah
I full intend to get ldc working on DragonFly as well. Dave MacFarlane previously tried committing his DragonFly changes to ldc, but ran into problems (linking and needing upstream druntime/phobos changes committed).
I already have the dmd bootstrap working and all unittests are able to pass, so no real issue there. So the path taken is a little different but the end result will be the same.

@joakim-noah
Copy link
Contributor

I understand that you already have it working, I'm only talking about with respect to getting stuff merged in the official D git repos and adding D packages to DragonFly ports, which you presumably would like to create without a bunch of local patches.

There is currently nowhere to merge changes to a C++ dmd, but there is for ldc, the ltsmaster branch. In terms of getting these pulls merged into the master branch, there's no problem.

@dkgroot
Copy link
Contributor Author

dkgroot commented Dec 19, 2017

@joakim-noah
Ok that sound like a good solution. I think porting the dragonfly changes to ldc:ltsmaster should be pretty straight forward. Already got an initial version compiling cleanly (using llvm38 for now. I did try llvm50 first, but that gave me some 'DIExpression' trouble, which i did not fully understand). Still some work to do though :-).

@joakim-noah
Copy link
Contributor

ldc has forked the 2.068.2 branches of druntime/phobos as ldc-ltsmaster. You may have a few merge conflicts, particularly if you are merging from a later release tag, but likely not many.

@dkgroot
Copy link
Contributor Author

dkgroot commented Dec 20, 2017

@joakim-noah
Regarding initial DragonFly port on ldc:ltsmaster. Everything compiled cleanly.

99% tests passed, 8 tests failed out of 663
Total Test time (real) =  59.37 sec
The following tests FAILED:
	227 - std.math (Failed)
	232 - std.numeric (Failed)
	304 - std.internal.math.gammafunction (Failed)
	552 - std.math-debug (Failed)
	557 - std.numeric-debug (Failed)
	629 - std.internal.math.gammafunction-debug (Failed)
	660 - dmd-testsuite-debug (Failed)
	662 - dmd-testsuite (Failed)

Not to bad i would say. There is something something funky with std/math.d, causing multiple unittests to fail, but have not been able to figure out what is causing this one yet :-)

@joakim-noah
Copy link
Contributor

Look to all be math-related, my guess is that DragonFly has the same long double issues as NetBSD, ie is long double 64-bit and DragonFly doesn't provide 80-bit math functions?

@dkgroot
Copy link
Contributor Author

dkgroot commented Dec 21, 2017

@joakim-noah
Thanks for the links / reading material. Apparently this was not the issue. The mapping in core/stdc/math.d i used was wrong. Fixed the mappings.

Will have to copy these math changes to the druntime PR :-)

BTW: We are getting a little offtopic here, regarding LDC. Can we maybe move this LDC related topid somewhere else ?

@joakim-noah
Copy link
Contributor

else
{
private enum targetOS = TarggetOS.all;
private enum targetOS = TargetOS.all;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This spelling fix is unrelated to this PR

Option("m32",
"generate 32 bit code"
"generate 32 bit code",
(TargetOS.all & ~TargetOS.dragonFlyBSD) // available on all OS'es except DragonFly, which does not support 32-bit binaries
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this was an intended use ?
Not sure if it merits an explanation / comment ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that's how it was intended to be used.
BTW -m32 is also mostly defunct on ArchLinux too as they dropped native support for 32-bit too, but I'm not sure whether it makes sense to complicate the code with such subtleties.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using -m32 with dmd on dragonfly currently does fail (not implemented -> assertion failure). I guess this is not the same on archlinux. They might just lack the 32-bit libraries to compile against.
I can remove it if you want (Or wait for another reviewer).

Copy link
Contributor

@wilzbach wilzbach Jan 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you can still compile gcc-multilib yourself from the AUR (arch user repository).
I don't mind this being used like you did as I said before, that was intended ;-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK issue closed

macOS = 4,
freeBSD = 8,
solaris = 16,
dragonFlyBSD = 32,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These questions/comments are not really related to this PR specifically:
What about the other OS's like NetBSD / OpenBSD etc ?
Would adding posix make sense ?
Do we really need yet another spelling / enum for the target OS ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the problem is that dlang.org needs to be able to set the target OS and the current TARGET_ are constants. I'm not really happy with this approach.
@ZombineDev wanted to do a more general revamp and introduce target triples. In any case, I think currently there are only differences between Windows and Posix, so it might make sense to just use those two.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wilzbach : I did not see the original PR, or i would have made these comments there.

The scattered nature and their repetition of these OS specific definitions is what throws me off a little to be honest. From a porting perspective it would be nice to see all OS specific stuff in one single file / directory, but that has been discussed before (over and over/ad nausea-um), so dropping that.

Except i just added one more 'exception' in line 215 (see above) so now there is three :-)

@dkgroot
Copy link
Contributor Author

dkgroot commented Jan 15, 2018

Hi guys, would it help if this PR were be split by file, so that it would make a little more progress ?

@joakim-noah
Copy link
Contributor

No, that would make things worse. It just needs to be reviewed, I'll do it.

@dkgroot
Copy link
Contributor Author

dkgroot commented Jan 16, 2018

@joakim-noah Ok thanks for letting me know. I just noticed the speed with which the musl PR's went through. Good that I asked before splitting it 😃

@dkgroot
Copy link
Contributor Author

dkgroot commented Jan 18, 2018

Rebased to include latest changes. Builds cleanly on DFly: dmd_dragonfly_ci

@dkgroot dkgroot force-pushed the dragonflybsd-master branch from 1f160ea to 254dda3 Compare January 19, 2018 14:40
@dkgroot dkgroot closed this Jan 19, 2018
@dkgroot dkgroot reopened this Jan 19, 2018
@dkgroot dkgroot force-pushed the dragonflybsd-master branch 4 times, most recently from 386bb18 to 4a269fc Compare January 21, 2018 22:22
Copy link
Contributor

@joakim-noah joakim-noah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Four small nits, otherwise fine.

}
}
else if (!global.params.is64bit && global.params.isFreeBSD && t.sym.fields.dim == 1 &&
else if (!global.params.is64bit && (global.params.isFreeBSD || global.params.isDragonFlyBSD) && t.sym.fields.dim == 1 &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't even support 32-bit, why bother with this?


/* DragonFlyBSD Version
* -------------
* There are two main issues: hosting the compiler on OpenBSD,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong BSD. 😃

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed
Also fixed a faulty reference to "TARGET_OPENBSD" in the OpenBSD entry above (in a seperate commit).

#include <string.h>

#if __linux__ || __APPLE__ || __FreeBSD__ || __OpenBSD__ || __sun
#if __linux__ || __APPLE__ || __FreeBSD__ || __OpenBSD__ || __DragonFly__ || __DragonFly__ || __DragonFly__ || __sun
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regex failure? Two more below.

}
version (DragonFlyBSD)
{
browse("http://dlang.org/dmd-dragonflybsd.html");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't exist, would rather you removed the nonexistent OpenBSD link above, as we don't put out official releases for these platforms.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
Also remarked out the openbsd entry (in a separate commit)

@dkgroot
Copy link
Contributor Author

dkgroot commented Jan 22, 2018

@joakim-noah Corrected as per your request/review. Thanks !
Will rebase later/when necessary

Copy link
Contributor

@joakim-noah joakim-noah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@dkgroot dkgroot force-pushed the dragonflybsd-master branch from e7f4233 to 81e8633 Compare January 22, 2018 22:39
DFLAGS=-I%@P%/../../src/phobos -I%@P%/../../src/druntime/import -L-L%@P%/../lib32 -L--export-dynamic

[Environment64]
DFLAGS=-I%@P%/../../src/phobos -I%@P%/../../src/druntime/import -L-L%@P%/../lib64 -L--export-dynamic
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Linux (and other 64-bit OSes, IIRC) we add -fPIC by default.

Copy link
Contributor Author

@dkgroot dkgroot Jan 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ZombineDev I would add -fPIC, even though it is only actually required on (most) Linux versions, the PR already got merged though.
FYI: checked the other platforms' ini-files and they also do not have -fPIC added by default.

@dlang-bot dlang-bot merged commit 62e615b into dlang:master Jan 23, 2018
@PetarKirov
Copy link
Member

Nice work, @dkgroot! Congratulations for getting through all the necessary steps!

@dkgroot
Copy link
Contributor Author

dkgroot commented Jan 23, 2018

Thanks everyone for your reviews, hints and time !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants