Skip to content

Commit bfb2675

Browse files
glathoudglathoud
committed
#7 proof-of-concept D dynamic compilation (dmd & ldc)
1 parent d62a279 commit bfb2675

File tree

8 files changed

+373
-0
lines changed

8 files changed

+373
-0
lines changed

‎.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
11
.hg
22
*~
33
*.bin
4+
*.map
45
*.o
56
*.s
7+
*.so
68
DIRECTIONS.org
79
*.outdir
810
*.pyc

‎misc/d_dynamic_loading/a_research.org

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
# -*- coding: utf-8 -*-
2+
3+
https://github.com/glathoud/flatorize/issues/7
4+
5+
points in the reverse chronological order
6+
7+
* sketching out practicalities: implementation
8+
9+
Local cache directory where we write out generated code and compiled
10+
libraries.
11+
12+
In fact, since this process is not specific to flatorize, I'd rather
13+
write a separate tool.
14+
15+
But but but wait, at the top of main.d there is `extern (C) int
16+
dll();` which means we'd still have to write that declaration, which
17+
is a no-go for flatorize.
18+
19+
Does it work without that declaration?
20+
21+
YES, because the local declaration:
22+
23+
int function() fn = cast(int function())dlsym(lh, "dll");
24+
25+
is enough. Notice how the string "dll" can be determined at runtime.
26+
27+
* sketching out practicalities: user
28+
29+
We'll need the user to configure:
30+
* fullpath to the dmd or ldmd2 binary
31+
* fullpath to the Phobos shared library
32+
33+
In case ldmd2 is used, and Phobos is missing, gently point out to
34+
step-by-step information to build LDC2 with shared libraries.
35+
36+
That should pretty much be all that is needed from the user's point
37+
of view :)
38+
39+
* try dlopen with ldc2 (works)
40+
41+
42+
** build LDC with shared libraries (esp. Phobos)
43+
Use LDC's ldmd2, a wrapper script for ldc2 which converts argument
44+
formats from dmd style over to ldc style.
45+
46+
Requirement: LDC compiled with shared libs, especially Phobos
47+
48+
Example:
49+
50+
https://forum.dlang.org/thread/lbrfycmutwrrghtzazin@forum.dlang.org?page=3
51+
52+
This is a fresh build on Ubuntu 14.04 with cmake called via:
53+
54+
cmake -DCMAKE_INSTALL_PREFIX=/opt/ldc -DBUILD_SHARED_LIBS=ON ..
55+
56+
.
57+
58+
so here we go
59+
60+
https://wiki.dlang.org/Building_LDC_from_source
61+
62+
#+BEGIN_SRC sh
63+
64+
# tools
65+
66+
sudo apt install cmake
67+
68+
#https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages
69+
sudo apt install ninja-build
70+
71+
# llvm (slightly tweaked by the LDC guys)
72+
73+
cd ~/d/glathoud/software
74+
mkdir ldc-llvm5-tweak
75+
cd ldc-llvm5-tweak
76+
77+
curl -L -O https://github.com/ldc-developers/llvm/releases/download/ldc-v5.0.1/llvm-5.0.1.src.tar.xz
78+
tar xf llvm-5.0.1.src.tar.xz
79+
cd llvm-5.0.1.src/
80+
mkdir build && cd build/
81+
82+
cmake -GNinja .. -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM;PowerPC;NVPTX" -DLLVM_BUILD_TOOLS=OFF -DLLVM_BUILD_UTILS=OFF # remove -GNinja to use Make instead
83+
84+
cd ../../../
85+
86+
# now LDC itself
87+
#
88+
# looked there: https://github.com/ldc-developers/ldc/releases
89+
# for the latest stable release
90+
91+
mkdir ldc-1.10.0 && cd ldc-1.10.0
92+
curl -L -O https://github.com/ldc-developers/ldc/releases/download/v1.10.0/ldc-1.10.0-src.tar.gz
93+
tar xf ldc-1.10.0-src.tar.gz
94+
95+
cd ldc-1.10.0-src/
96+
mkdir /media/ssd2/software 2>>/dev/null
97+
mkdir build && cd build
98+
cmake -G Ninja -DLLVM_CONFIG=../../../ldc-llvm5-tweak/llvm-5.0.1.src/build/bin/llvm-config -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=/media/ssd2/software/ldc ..
99+
100+
# Build and install LDC. Use -j<n> to limit parallelism if running out of memory.
101+
ninja
102+
sudo ninja install
103+
#+END_SRC
104+
105+
** run it
106+
107+
file:./main_ldc2.sh
108+
109+
* try dlopen with ldc2 (old notes: til I got it working)
110+
111+
file:./try_00/main_ldc2.sh
112+
113+
https://forum.dlang.org/group/digitalmars.D.ldc
114+
115+
maybe try with ldmd2
116+
117+
https://stackoverflow.com/questions/35515138/ldc2-vs-ldmd2-whats-the-difference
118+
119+
ldmd2 is just a wrapper script for ldc2 which converts argument formats from dmd style over to ldc style.
120+
121+
but there should be phobos as shared lib
122+
https://wiki.dlang.org/Compilers
123+
Phobos as a shared library
124+
125+
https://wiki.dlang.org/Building_LDC_runtime_libraries
126+
127+
https://github.com/ldc-developers/ldc/blob/master/runtime/ldc-build-runtime.d.in
128+
129+
https://packages.ubuntu.com/artful/libphobos2-ldc74
130+
131+
https://packages.ubuntu.com/artful/libphobos2-ldc-dev
132+
133+
should work by building it
134+
135+
https://forum.dlang.org/thread/lbrfycmutwrrghtzazin@forum.dlang.org?page=3
136+
137+
This is a fresh build on Ubuntu 14.04 with cmake called via:
138+
139+
cmake -DCMAKE_INSTALL_PREFIX=/opt/ldc -DBUILD_SHARED_LIBS=ON ..
140+
141+
.
142+
143+
so here we go
144+
145+
https://wiki.dlang.org/Building_LDC_from_source
146+
147+
#+BEGIN_SRC sh
148+
149+
# tools
150+
151+
sudo apt install cmake
152+
153+
#https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages
154+
sudo apt install ninja-build
155+
156+
# llvm (slightly tweaked by the LDC guys)
157+
158+
cd ~/d/glathoud/software
159+
mkdir ldc-llvm5-tweak
160+
cd ldc-llvm5-tweak
161+
162+
curl -L -O https://github.com/ldc-developers/llvm/releases/download/ldc-v5.0.1/llvm-5.0.1.src.tar.xz
163+
tar xf llvm-5.0.1.src.tar.xz
164+
cd llvm-5.0.1.src/
165+
mkdir build && cd build/
166+
167+
cmake -GNinja .. -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM;PowerPC;NVPTX" -DLLVM_BUILD_TOOLS=OFF -DLLVM_BUILD_UTILS=OFF # remove -GNinja to use Make instead
168+
169+
cd ../../../
170+
171+
# now LDC itself
172+
#
173+
# looked there: https://github.com/ldc-developers/ldc/releases
174+
# for the latest stable release
175+
176+
mkdir ldc-1.10.0 && cd ldc-1.10.0
177+
curl -L -O https://github.com/ldc-developers/ldc/releases/download/v1.10.0/ldc-1.10.0-src.tar.gz
178+
tar xf ldc-1.10.0-src.tar.gz
179+
180+
cd ldc-1.10.0-src/
181+
mkdir /media/ssd2/software 2>>/dev/null
182+
mkdir build && cd build
183+
cmake -G Ninja -DLLVM_CONFIG=../../../ldc-llvm5-tweak/llvm-5.0.1.src/build/bin/llvm-config -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=/media/ssd2/software/ldc ..
184+
185+
# Build and install LDC. Use -j<n> to limit parallelism if running out of memory.
186+
ninja
187+
sudo ninja install
188+
#+END_SRC
189+
190+
* try dlopen with dmd (works)
191+
192+
file:./try_00/main_dmd.sh
193+
194+
* about dlopen
195+
196+
file:./
197+
198+
https://dlang.org/articles/dll-linux.html
199+
200+
201+
dll.d
202+
203+
https://dlang.org/articles/dll-linux.html#dso9
204+
205+
main
206+
207+
https://dlang.org/articles/dll-linux.html#dso10
208+
209+
210+
some experience:
211+
212+
https://forum.dlang.org/post/rmdtpuhcpttktxdjvhnl@forum.dlang.org
213+
214+
* wiki
215+
216+
https://wiki.dlang.org/Win32_DLLs_in_D
217+
218+
* initial research
219+
220+
https://forum.dlang.org/thread/rronfcwusoixzqbyhvdf@forum.dlang.org
221+
222+
I've written a program where user input drives the generation of a
223+
code snippet that then gets passed to an invocation of dmd,
224+
compiled, linked and the loaded as a shared library via dlopen(), et
225+
al. It works fairly well, and dmd generally is fast enough that the
226+
pause is not very noticeable. (This was before dmd-as-a-library was
227+
available; I imagine it would be even faster today now that you
228+
don't have to spawn a separate dmd process.)
229+
230+
The generated snippet does import std.math, and it seems to work
231+
fine. But then the generated snippets tend to be fairly small, and
232+
only use a limited subset of the language, so there may be gotchas
233+
that I'm not aware of.
234+
235+
.
236+
237+
https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library/
238+
239+
Recently there has been some progress in making the D compiler (DMD)
240+
available as a library. Razvan Nitu has been working on it as part
241+
of his D Foundation scholarship at the University Politechnica of
242+
Bucharest. He gave a presentation at DConf 2017 (a video of the talk
243+
is available, as well as examples in the DMD repository). So I had
244+
the idea that as part of the DConf 2017 hackathon I could create a
245+
simple DUB package for DMD to make only the lexer and the parser
246+
available as a library, something his work has made possible.
247+
248+
http://code.dlang.org/packages/dmd
249+
250+
.
251+
252+
But if I want fast code... I'd rather invoke ldc2

‎misc/d_dynamic_loading/try_00/dll.d

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
import core.stdc.stdio;
2+
3+
extern (C) int dll()
4+
{
5+
printf("dll()\n");
6+
return 0;
7+
}
8+
9+
shared static this()
10+
{
11+
printf("libdll.so shared static this\n");
12+
}
13+
14+
shared static ~this()
15+
{
16+
printf("libdll.so shared static ~this\n");
17+
}

‎misc/d_dynamic_loading/try_00/main.d

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
import core.stdc.stdio;
2+
import core.stdc.stdlib;
3+
import core.sys.posix.dlfcn;
4+
5+
// not needed: extern (C) int dll();
6+
7+
int main()
8+
{
9+
printf("+main()\n");
10+
11+
void* lh = dlopen("libdll.so", RTLD_LAZY);
12+
if (!lh)
13+
{
14+
fprintf(stderr, "dlopen error: %s\n", dlerror());
15+
exit(1);
16+
}
17+
printf("libdll.so is loaded\n");
18+
19+
int function() fn = cast(int function())dlsym(lh, "dll");
20+
char* error = dlerror();
21+
if (error)
22+
{
23+
fprintf(stderr, "dlsym error: %s\n", error);
24+
exit(1);
25+
}
26+
printf("dll() function is found\n");
27+
28+
fn();
29+
30+
printf("unloading libdll.so\n");
31+
dlclose(lh);
32+
33+
printf("-main()\n");
34+
return 0;
35+
}
36+
37+
shared static this() { printf("main shared static this\n"); }
38+
39+
shared static ~this() { printf("main shared static ~this\n"); }

‎misc/d_dynamic_loading/try_00/main_dmd.sh

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/usr/bin/env sh
2+
3+
COMPILO="dmd"
4+
PATH_WHERE_SHARED_LIB_IS="."
5+
6+
rm *.map *.o *.so *.bin 2>>/dev/null
7+
8+
cd "$PATH_WHERE_SHARED_LIB_IS"
9+
rm *.map *.o *.so *.bin 2>>/dev/null
10+
11+
cd -
12+
13+
$COMPILO -c dll.d -fPIC
14+
$COMPILO -oflibdll.so dll.o -shared -defaultlib=libphobos2.so -L-rpath="$PATH_WHERE_SHARED_LIB_IS"
15+
16+
$COMPILO -c main.d
17+
$COMPILO -ofmain.bin main.o -L-ldl -defaultlib=libphobos2.so -L-rpath=".:$PATH_WHERE_SHARED_LIB_IS" -map
18+
19+
./main.bin

‎misc/d_dynamic_loading/try_00/main_ldc2.sh

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/usr/bin/env sh
2+
3+
LDC_DIR="/media/ssd2/software/ldc"
4+
COMPILO="$LDC_DIR/bin/ldmd2"
5+
FULLNAME_SHARED_LIB="$LDC_DIR/lib/libphobos2-ldc-shared.so.2.0.80"
6+
7+
rm *.map *.o *.so *.bin 2>>/dev/null
8+
9+
cd "$PATH_WHERE_SHARED_LIB_IS"
10+
rm *.map *.o *.so *.bin 2>>/dev/null
11+
12+
cd -
13+
14+
$COMPILO -c dll.d -fPIC
15+
$COMPILO -oflibdll.so dll.o -shared -L-rpath="." -L"$FULLNAME_SHARED_LIB"
16+
17+
$COMPILO -c main.d
18+
# $COMPILO -ofmain.bin main.o -L-ldl -L-rpath="." -L"$FULLNAME_SHARED_LIB" -map
19+
# Warning: command-line option '-map' not yet supported by LDC.
20+
$COMPILO -ofmain.bin main.o -L-ldl -L-rpath="." -L"$FULLNAME_SHARED_LIB"
21+
22+
./main.bin

‎misc/d_dynamic_loading/try_00/output_main_dmd.txt

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
/home/oa/d/glathoud/tmp/flatorize/misc/d_dynamic_loading/try_00
2+
main shared static this
3+
+main()
4+
libdll.so shared static this
5+
libdll.so is loaded
6+
dll() function is found
7+
dll()
8+
unloading libdll.so
9+
libdll.so shared static ~this
10+
-main()
11+
main shared static ~this

‎misc/d_dynamic_loading/try_00/output_main_ldc2.txt

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
/home/oa/d/glathoud/tmp/flatorize/misc/d_dynamic_loading/try_00
2+
main shared static this
3+
+main()
4+
libdll.so shared static this
5+
libdll.so is loaded
6+
dll() function is found
7+
dll()
8+
unloading libdll.so
9+
libdll.so shared static ~this
10+
-main()
11+
main shared static ~this

0 commit comments

Comments
 (0)