Support Windows as a first class platform (note: not WSL or emulation)

[xtagon]

Hi,

Crystal is an exciting project with lots of potential. I am interested in the possibility of a Crystal compiler for Windows. How feasible is that idea, and how much work would it be?

Specifically, I would like to be able to compile a Crystal program into a Windows .DLL file with exported C functions. I don't care if I run the actual Crystal compiler on a Linux or Windows machine, I just want to end up with a DLL library that will run on Windows.

I have knowledge of Ruby and some C, and I'm willing to dive into learning about LLVM if it means I can contribute here. I would be pleased if you could point me in the right direction.

Regards,
Justin

[asterite]

Hi,

Thank you for your interest in Crystal!

Yes, it would be awesome to have Crystal running on Windows.

I don't think there is much to do on LLVM side to make it work, except having the necessary libraries to link to.

I read this the other day, it might be useful: http://www.phoronix.com/scan.php?page=news_item&px=MTQ1NjI

If you want to contribute, I think the path is the following:

• Checkout the project and try to compile a simple program. A very simple program is just this: "1" (just an integer literal). But Crystal automatically includes a "prelude" file that gives you Array, Hash, etc. So you should comment these two lines (https://github.com/manastech/crystal/blob/master/lib/crystal/compiler.rb#L120-L121) so it will be easier to test. This point involves making LLVM work on Windows, specifically the ruby-llvm gem.
• Once the previous point is working, you will need to make other programs work, mostly programs that depend on libc. For example you could try opening and writing a file with File.open("C:/foo.txt") { |file| file.write "Hello" }. If the API for opening and writing a file is different on Windows than on unix (and I'm sure it is), you will have to create separate files for Windows. For this there is a temporary solution: you can place an "if" after a require. Take a look at the first two lines of https://github.com/manastech/crystal/blob/master/std/time.cr

Any doubts you have about how the compiler works, we can answer and help you :-)

Regards,
Ary

[xtagon]

Thanks, Ary! I will give it a shot.
On Sep 10, 2013 5:09 AM, "Ary Borenszweig" notifications@github.com wrote:

Hi,

Thank you for your interest in Crystal!

Yes, it would be awesome to have Crystal running on Windows.

I don't think there is much to do on LLVM side to make it work, except
having the necessary libraries to link to.

I read this the other day, it might be useful:
http://www.phoronix.com/scan.php?page=news_item&px=MTQ1NjI

If you want to contribute, I think the path is the following:

• Checkout the project and try to compile a simple program. A very
  simple program is just this: "1" (just an integer literal). But Crystal
  automatically includes a "prelude" file that gives you Array, Hash, etc. So
  you should comment these two lines (
  https://github.com/manastech/crystal/blob/master/lib/crystal/compiler.rb#L120-L121)
  so it will be easier to test. This point involves making LLVM work on
  Windows, specifically the ruby-llvm gem.
• Once the previous point is working, you will need to make other
  programs work, mostly programs that depend on libc. For example you could
  try opening and writing a file with File.open("C:/foo.txt") { |file|
  file.write "Hello" }. If the API for opening and writing a file is
  different on Windows than on unix (and I'm sure it is), you will have to
  create separate files for Windows. For this there is a temporary solution:
  you can place an "if" after a require. Take a look at the first two lines
  of https://github.com/manastech/crystal/blob/master/std/time.cr

Any doubts you have about how the compiler works, we can answer and help
you :-)

Regards,
Ary

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-24154638
.

[asterite]

I forgot to mention there are thousands of specs, so if you manage to pass
them all you will be very close.

Lexer, parser, normalization and type inference specs, except those
involving macros, should pass right away. Codegen specs will be the tough
ones.
On Sep 10, 2013 9:29 AM, "Justin Workman" notifications@github.com wrote:

Thanks, Ary! I will give it a shot.
On Sep 10, 2013 5:09 AM, "Ary Borenszweig" notifications@github.com
wrote:

Hi,

Thank you for your interest in Crystal!

Yes, it would be awesome to have Crystal running on Windows.

I don't think there is much to do on LLVM side to make it work, except
having the necessary libraries to link to.

I read this the other day, it might be useful:
http://www.phoronix.com/scan.php?page=news_item&px=MTQ1NjI

If you want to contribute, I think the path is the following:

• Checkout the project and try to compile a simple program. A very
  simple program is just this: "1" (just an integer literal). But Crystal
  automatically includes a "prelude" file that gives you Array, Hash, etc.
  So
  you should comment these two lines (

https://github.com/manastech/crystal/blob/master/lib/crystal/compiler.rb#L120-L121)

so it will be easier to test. This point involves making LLVM work on
Windows, specifically the ruby-llvm gem.

• Once the previous point is working, you will need to make other
  programs work, mostly programs that depend on libc. For example you
  could
  try opening and writing a file with File.open("C:/foo.txt") { |file|
  file.write "Hello" }. If the API for opening and writing a file is
  different on Windows than on unix (and I'm sure it is), you will have to
  create separate files for Windows. For this there is a temporary
  solution:
  you can place an "if" after a require. Take a look at the first two
  lines
  of https://github.com/manastech/crystal/blob/master/std/time.cr

Any doubts you have about how the compiler works, we can answer and help
you :-)

Regards,
Ary

—
Reply to this email directly or view it on GitHub<
https://github.com/manastech/crystal/issues/26#issuecomment-24154638>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-24155615
.

[xtagon]

I apologize for forgetting to follow up on this. I was able to compile LLVM on Windows (that was a "fun" task in itself) but failed to get ruby-llvm working with it and eventually gave up (for now).

The website says that the Crystal compiler is now written in Crystal itself. Does that carry any implications as to how one might get it running on Windows? Do you think a better pursuit would be to make Crystal cross-compile (compile on Linux, run on Windows)?

[asterite]

Well, if you get cross-compilation to work that would indeed be awesome and very, very useful.

If you are on Mac or Linux you can download a preocmpiled compiler that will let you compile new versions of the compiler. Since such compiler doesn't exist for Windows, you will need to create one, but I'm not sure how. There are the options:

1. Checkout the tag "ruby" and make that compiler work (in Ruby). That would involve making 'ruby-llvm' work, I guess. It will also involve making the code written in Crystal to work in windows (currently the compiler relies on some commands like "uname" to determine the operating system, I'm not sure cygwin or the libc for windows that we'll use will have those).
2. Make cross compilation work. This has the advantage of not having to deal with 'ruby-llvm' (the LLVM wrapper in Crystal is much thinner and it's easier to make it work). I would try first to cross compiler from linux to Mac or from Mac to linux (if you have those available) to understand what is needed for cross compiling, and then try to cross compile to Windows.

Maybe you can tell us what problem you had with ruby-llvm on Windows. If you make that work, point 1 would be the easiest, I think.

Thanks!

[xtagon]

ruby-llvm says it requires LLVM built with shared library enabled. However, the LLVM CMake doc says "Shared libraries are not supported on Windows and not recommended in the other OSes."

I don't know whether "not supported" means it's not possible, so I'll try rebuilding LLVM and find out.

[xtagon]

Here is what happens when I try to install ruby-llvm:

C:\Users\Justin>gem install ruby-llvm --no-rdoc --no-ri --platform=ruby
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
ERROR:  Error installing ruby-llvm:
        ERROR: Failed to build gem native extension.

    "C:/Ruby200-x64/bin/ruby.exe" -rubygems C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/rake-0.9.6/bin/rake RUBYARCHDIR=C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby
-llvm-3.0.0/lib RUBYLIBDIR=C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby-llvm-3.0.0/lib
g++ -shared support.cpp -IC:/Users/Justin/beta/sandbox/crystal/llvm/include -IC:/Users/Justin/beta/sandbox/crystal/build/include  /DWIN32 /D_WINDOWS /W3 /GR /EH
sc   /MP -wd4275 -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_SCL_SECURE_NO_DEPRECATE -D_SCL_S
ECURE_NO_WARNINGS -wd4146 -wd4180 -wd4244 -wd4267 -wd4345 -wd4351 -wd4355 -wd4503 -wd4624 -wd4800 -wd4291 -w14062 -we4238 -D__STDC_CONSTANT_MACROS -D__STDC_FORM
AT_MACROS -D__STDC_LIMIT_MACROS -LC:/Users/Justin/beta/sandbox/crystal/build/lib/  /machine:X86   -lLLVMJIT -lLLVMOption -lLLVMIRReader -lLLVMAsmParser -lLLVMIn
terpreter -lLLVMInstrumentation -lLLVMDebugInfo -lLLVMMCJIT -lLLVMRuntimeDyld -lLLVMExecutionEngine -lLLVMTableGen -lLLVMLTO -lLLVMObjCARCOpts -lLLVMLinker -lLL
VMipo -lLLVMVectorize -lLLVMBitWriter -lLLVMBitReader -lLLVMXCoreDisassembler -lLLVMXCoreCodeGen -lLLVMXCoreDesc -lLLVMXCoreInfo -lLLVMXCoreAsmPrinter -lLLVMX86
Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMX86Desc -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMSystemZDisassembler -lLLVMSystemZCodeGen -
lLLVMSystemZAsmParser -lLLVMSystemZDesc -lLLVMSystemZInfo -lLLVMSystemZAsmPrinter -lLLVMSparcCodeGen -lLLVMSparcDesc -lLLVMSparcInfo -lLLVMR600CodeGen -lLLVMR60
0Desc -lLLVMR600Info -lLLVMR600AsmPrinter -lLLVMPowerPCCodeGen -lLLVMPowerPCAsmParser -lLLVMPowerPCDesc -lLLVMPowerPCInfo -lLLVMPowerPCAsmPrinter -lLLVMNVPTXCod
eGen -lLLVMNVPTXDesc -lLLVMNVPTXInfo -lLLVMNVPTXAsmPrinter -lLLVMMSP430CodeGen -lLLVMMSP430Desc -lLLVMMSP430Info -lLLVMMSP430AsmPrinter -lLLVMMipsDisassembler -
lLLVMMipsCodeGen -lLLVMMipsAsmParser -lLLVMMipsDesc -lLLVMMipsInfo -lLLVMMipsAsmPrinter -lLLVMHexagonCodeGen -lLLVMHexagonAsmPrinter -lLLVMHexagonDesc -lLLVMHex
agonInfo -lLLVMCppBackendCodeGen -lLLVMCppBackendInfo -lLLVMARMDisassembler -lLLVMARMCodeGen -lLLVMARMAsmParser -lLLVMARMDesc -lLLVMARMInfo -lLLVMARMAsmPrinter
-lLLVMAArch64Disassembler -lLLVMAArch64CodeGen -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMTransformUtils -lLLVMi
pa -lLLVMAnalysis -lLLVMTarget -lLLVMCore -lLLVMAArch64AsmParser -lLLVMMCParser -lLLVMAArch64Desc -lLLVMAArch64Info -lLLVMAArch64AsmPrinter -lLLVMAArch64Utils -
lLLVMMCDisassembler -lLLVMMC -lLLVMObject -lgtest_main -lgtest -lLLVMSupport  -o ../../lib/RubyLLVMSupport-3.0.0.dll
g++: error: /DWIN32: No such file or directory
g++: error: /D_WINDOWS: No such file or directory
g++: error: /W3: No such file or directory
g++: error: /GR: No such file or directory
g++: error: /EHsc: No such file or directory
g++: error: /MP: No such file or directory
g++: error: unrecognized command line option '-wd4275'
g++: error: unrecognized command line option '-wd4146'
g++: error: unrecognized command line option '-wd4180'
g++: error: unrecognized command line option '-wd4244'
g++: error: unrecognized command line option '-wd4267'
g++: error: unrecognized command line option '-wd4345'
g++: error: unrecognized command line option '-wd4351'
g++: error: unrecognized command line option '-wd4355'
g++: error: unrecognized command line option '-wd4503'
g++: error: unrecognized command line option '-wd4624'
g++: error: unrecognized command line option '-wd4800'
g++: error: unrecognized command line option '-wd4291'
g++: error: unrecognized command line option '-w14062'
g++: error: unrecognized command line option '-we4238'
g++: error: /machine:X86: No such file or directory
rake aborted!
Command failed with status (1): [g++ -shared support.cpp -IC:/Users/Justin/...]

Tasks: TOP => default => build => RubyLLVMSupport-3.0.0.dll
(See full trace by running task with --trace)


Gem files will remain installed in C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby-llvm-3.0.0 for inspection.
Results logged to C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby-llvm-3.0.0/ext/ruby-llvm-support/gem_make.out

Any ideas?

Windows 7 64-bit
Ruby 2.0.0p353 (with DevKit)
Rubygems 2.0.14

[Kelet]

Hello,

I've been watching the development of Crystal for a while now, and I've become interested in possibly contributing to the ecosystem.

I'm posting in this issue because Windows support is integral to my adoption of the language. I work on both Windows and Linux and need all of my tools to work on both.

I'll be busy for the next couple of weeks, but I should have some free time toward the end of the month. I'd like to try and get Crystal compiling on Windows if possible. Is this information up to date? Judging by some of the recent blog posts, it seems out of date.

[asterite]

Hi @Kelet,

First, thanks for wanting to help in the development of Crystal!

My last comment is still up-to-date, I think. But now I think the best thing to do is to cross-compile from mac/linux to Windows. We can start by making a very simple "Hello World" and try to run it on Windows. This will be the program:

# file: hello.cr
lib C
  fun puts(str : UInt8*) : Int32
end

class String
  def cstr
    pointerof(@c)
  end
end

C.puts "Hello World!"

We will compile it like this:

bin/crystal foo.cr --prelude=empty --single-module

The "--prelude=empty" tells the compiler to not include the file prelude.cr and instead use empty.cr. This just defines the main function, which will contain whatever is in your program.

The "--single-module" just makes it generate a single LLVM module, otherwise it generates one per class/module. So this way is simpler.

Then we bind to the C function puts, which I don't know if it exists in Windows. If not, it should be replaced by another one provided by Windows that does that (I don't know much about Windows, it seems it does exist)

Then the String#cstr method just gets a pointer to the first character in the String. This method is used to implicitly convert a String to UInt8* by the compiler (I guess that method should be built-in so we don't have to do this...). But it's not a big deal anyway.

Then if you run it like this:

DUMP=1 bin/crystal foo.cr --prelude=empty --single-module

you will see the generated LLVM code:

; ModuleID = 'main_module'

%String = type { i32, i32, i8 }

@symbol_table = global [0 x %String*] zeroinitializer
@str = private constant { i32, i32, [13 x i8] } { i32 1, i32 12, [13 x i8] c"Hello World!\00" }

define i32 @__crystal_main(i32 %argc, i8** %argv) {
alloca:
  br label %const

const:                                            ; preds = %alloca
  br label %entry

entry:                                            ; preds = %const
  %0 = call i8* @"*String#cstr<String>:UInt8*"(%String* bitcast ({ i32, i32, [13 x i8] }* @str to %String*))
  %1 = call i32 @puts(i8* %0)
  ret i32 %1
}

define i32 @main(i32 %argc, i8** %argv) {
alloca:
  %argc1 = alloca i32
  %argv2 = alloca i8**
  br label %entry

entry:                                            ; preds = %alloca
  store i32 %argc, i32* %argc1
  store i8** %argv, i8*** %argv2
  %0 = load i32* %argc1
  %1 = load i8*** %argv2
  %2 = call i32 @__crystal_main(i32 %0, i8** %1)
  ret i32 0
}

define internal i8* @"*String#cstr<String>:UInt8*"(%String* %self) {
alloca:
  br label %entry

entry:                                            ; preds = %alloca
  %0 = getelementptr %String* %self, i32 0, i32 2
  ret i8* %0
}

declare i32 @puts(i8*)

We can get that LLVM code, put it in a file "hello.ll" in Windows and try to compile it:

llc hello.ll
clang hello.s -o hello
./hello
# prints: Hello World!

Of course these last commands are in linux/mac, for Windows it will be similar. We are using LLVM 3.3 so the commands might be named like llc-3.3, clang-3.3 and so on.

I don't know if in Windows the main function is just "int main(int argc, char** argv)" or you always need to use WinMain (I think that's for graphical stuff).

There's a simple way to do all of the above: we use the --cross-compile flag:

bin/crystal hello.cr --prelude=empty --single-module --cross-compile "windows"

This will genreate a hello.bc file (similar to ll file, but binary). The last command will also print this:

llc hello.bc  -o hello.s && clang hello.s -o hello

So you copy the "hello.bc" file to Windows and executing that command should generate an executable.

The "windows" part passed to the --cross-compile flag just passes "windows" as a flag to the compiler so when you use ifdef it will be true. In linux/mac these flags are obtained by executing uname -m -s, lowercased and split by spaces. In fact, right now you can pass "" to the --cross-compile flag, but once we start with more complex program it would be nice to separte all windows stuff with ifdef windows.

If you manage to compile a Crystal program in linux/mac, copy the ll/bc file to Windows, compile it in Windows, execute it and get "Hello World!" in the conosle, we'll be a bit closer to having a compiler in Windows :-)

The next steps would be trying to compile bigger programs until we reach the program that is the compiler itself :-P

If you make some progress with the above, please let us know and we can continue trying to make it work.

Again, thanks for your help!

[Kelet]

Excellent, I'll try this process when I find some time and report back to you!

[Kelet]

I tried it out real quick:

$ llc hello.bc -o hello.s && clang -v hello.s -o hello
clang version 3.4 (198054)
Target: i686-pc-mingw32
Thread model: posix
Selected GCC installation:
 "c:\TDM-GCC-32\bin\gcc.exe" -v -c -m32 -o "C:\\Users\\Kyle\\AppData\\Local\\Temp\\hello-d405d2.o" -x assembler hello.s
Using built-in specs.
COLLECT_GCC=c:\TDM-GCC-32\bin\gcc.exe
Target: mingw32
Configured with: ../../../src/gcc-4.8.1/configure --build=mingw32 --enable-languages=ada,c,c++,fortran,lto,objc,obj-c++ --enable-libgomp --enable-lto --enable-graphite --enable-libstdcxx-debug --enable-threads=posix --enable-version-specific-runtime-libs --enable-fully-dynamic-string --enable-libstdcxx-threads --enable-libstdcxx-time --with-gnu-ld --disable-werror --disable-nls --disable-win32-registry --disable-symvers --enable-cxx-flags='-fno-function-sections -fno-data-sections -DWINPTHREAD_STATIC' --prefix=/mingw32tdm --with-local-prefix=/mingw32tdm --with-pkgversion=tdm-2 --enable-sjlj-exceptions --with-bugurl=http://tdm-gcc.tdragon.net/bugs
Thread model: posix
gcc version 4.8.1 (tdm-2)
COLLECT_GCC_OPTIONS='-v' '-c' '-m32' '-o' 'C:\Users\Kyle\AppData\Local\Temp\hello-d405d2.o' '-mtune=generic' '-march=pentiumpro'
 c:/tdm-gcc-32/bin/../lib/gcc/mingw32/4.8.1/../../../../mingw32/bin/as.exe -v -o C:\Users\Kyle\AppData\Local\Temp\hello-d405d2.o hello.s
GNU assembler version 2.23.1 (mingw32) using BFD version (GNU Binutils) 2.23.1
hello.s: Assembler messages:
hello.s:18: Error: invalid character '"' before operand 1
hello.s:45: Error: junk at end of line, first unrecognized character is `"'
hello.s:50: Error: junk at end of line, first unrecognized character is `"'
clang.exe: error: assembler (via gcc) command failed with exit code 1 (use -v to see invocation)

hello.s

    .def     @feat.00;
    .scl    3;
    .type   0;
    .endef
    .globl  @feat.00
@feat.00 = 1
    .def     ___crystal_main;
    .scl    2;
    .type   32;
    .endef
    .text
    .globl  ___crystal_main
    .align  16, 0x90
___crystal_main:                        # @__crystal_main
# BB#0:                                 # %entry
    pushl   %eax
    movl    $L_str, (%esp)
    calll   "_*String#cstr<String>:UInt8*"
    movl    %eax, (%esp)
    calll   _puts
    popl    %edx
    ret

    .def     _main;
    .scl    2;
    .type   32;
    .endef
    .globl  _main
    .align  16, 0x90
_main:                                  # @main
# BB#0:                                 # %alloca
    subl    $16, %esp
    movl    24(%esp), %eax
    movl    20(%esp), %ecx
    movl    %ecx, 12(%esp)
    movl    %eax, 8(%esp)
    movl    12(%esp), %ecx
    movl    %eax, 4(%esp)
    movl    %ecx, (%esp)
    calll   ___crystal_main
    xorl    %eax, %eax
    addl    $16, %esp
    ret

    .def     "_*String#cstr<String>:UInt8*";
    .scl    3;
    .type   32;
    .endef
    .align  16, 0x90
"_*String#cstr<String>:UInt8*":         # @"*String#cstr<String>:UInt8*"
# BB#0:                                 # %alloca
    movl    4(%esp), %eax
    addl    $8, %eax
    ret

    .bss
    .globl  _symbol_table           # @symbol_table
    .align  4
_symbol_table:

    .section    .rdata,"r"
    .align  16                      # @str
L_str:
    .long   1                       # 0x1
    .long   12                      # 0xc
    .asciz  "Hello World!"
    .zero   3

[asterite]

Cool! That's progress.

It seems clang for windows doesn't like names like "_String#cstr:UInt8" for labels.

What happens if you change:

"_*String#cstr<String>:UInt8*"

to this:

String_cstr

in the hello.s file and try to execute clang -v hello.s -o hello again?

[asterite]

Or maybe to:

_String_cstr

(I see all .def in the file start with _... so just in case...)

[Kelet]

Both String_cstr and _String_cstr work like a charm and print out "Hello World!"

[asterite]

Awesome!! :-)

So our first step is to have a different mangling for functions, at least for Windows. I'll think of something. I don't think replacing all non-alphanumeric characters to underscore will work, but it need to be thought a bit to avoid conflicts between definitions.

I'll let you know when I do this. In the meantime you can try to do it. The method that mangles a name is here, just in case you wonder or want to try to change it too.