Native windows support

[eiennohito]

It should be possible to compile Juman++ on windows.
We need to port memory mapping and memory management utils to native Windows API for that.

[DoumanAsh]

@eiennohito Does jumanpp continue to use boost? (I think first version used it)
It turns out there is mmap in boost http://www.boost.org/doc/libs/1_66_0/doc/html/boost/interprocess/mapped_region.html

[eiennohito]

Nope, no boost, please. The Windows impl should be only around 60-70 lines of code with perfect error handling.

[DoumanAsh]

Considering #20 I assume you would like to avoid strict dependency on C++17 std::aligned_alloc in place of posix_memalign?

[eiennohito]

Yes, no C++17 for us (yet).
A lot of people use CentOS 7 or even 6 here.

Windows has https://msdn.microsoft.com/en-us/library/8z34s9c6.aspx, which should solve the problem.

[eiennohito]

Even better idea would be to use https://msdn.microsoft.com/en-us/library/windows/desktop/aa366887(v=vs.85).aspx with actual large pages support, but I don't mind at least somehow working implementation. It's much better than nothing already.

[DoumanAsh]

@eiennohito If we would go VirtualAlloc route, then wouldn't we also use linux's valloc? Not sure about its availability as it is part of glibc, not posix.

P.s. You cannot though specify alignment with VirtualAlloc, if it is needed.

[eiennohito]

If you try to dig the history of memory.cpp, then you will find out that it was using mmap before! But the thing is, if you want to be a page to use huge tlb on Linux without defragmentation, you must align it on 2M boundary, and the best API to do that is posix_memalign.

Memory allocator itself should be ok with 4096/2M alignment which the VirtualAlloc will return.
It will handle smaller granularity by itself.

You should use _aligned_malloc for MallocEalloc (a bad name) nevertheless though.

[DoumanAsh]

Now I hit quite strange problem with soa.h: https://github.com/ku-nlp/jumanpp/blob/master/src/util/soa.h#L50

It uses ManagedVector which is vector with your own custom allocator https://github.com/ku-nlp/jumanpp/blob/master/src/util/memory.hpp#L166-L192

But it seems this allocator misses custom copy constructor? To be honest MSVC is really bad with this error(I tried clang-cl though and it is not really helpful too)

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include\vector(554): error C2664: 'jumanpp::util::memory::StlManagedAlloc<_Newfirst>::StlManagedAlloc(jumanpp::util::memory::StlManagedAlloc<_Newfirst> &&)': cannot convert argument 1 from 'jumanpp::util::memory::StlManagedAlloc<jumanpp::util::memory::SOAField *>' to 'jumanpp::util::memory::PoolAlloc *'
        with
        [
            _Newfirst=std::_Container_proxy
        ]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include\vector(554): note: No user-defined-conversion operator available that can perform this conversion, or the operator
cannot be called
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include\vector(553): note: while compiling class template member function 'void std::_Vector_alloc<std::_Vec_base_types<_Ty,_Alloc>>::_Free_proxy(void)'
        with
        [
            _Ty=jumanpp::util::memory::SOAField *,
            _Alloc=jumanpp::util::memory::StlManagedAlloc<jumanpp::util::memory::SOAField *>
        ]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include\vector(505): note: see reference to function template instantiation 'void std::_Vector_alloc<std::_Vec_base_types<_Ty,_Alloc>>::_Free_proxy(void)' being compiled
        with
        [
            _Ty=jumanpp::util::memory::SOAField *,
            _Alloc=jumanpp::util::memory::StlManagedAlloc<jumanpp::util::memory::SOAField *>
        ]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include\vector(659): note: see reference to class template instantiation 'std::_Vector_alloc<std::_Vec_base_types<_Ty,_Alloc>>' being compiled
        with
        [
            _Ty=jumanpp::util::memory::SOAField *,
            _Alloc=jumanpp::util::memory::StlManagedAlloc<jumanpp::util::memory::SOAField *>
        ]
d:\repos\cpp-code\jumanpp\src\util\soa.h(50): note: see reference to class template instantiation 'std::vector<jumanpp::util::memory::SOAField *,jumanpp::util::memory::StlManagedAlloc<jumanpp::util::memory::SOAField *>>' being compiled

I'll try to poke around how custom allocators are supposed to be implemented, might take some time.

[eiennohito]

Actually, there shouldn't be any use of soa.h inside the project now, it is an artifact of the past. If there is I will fix it.

[DoumanAsh]

Ah yeah, it is not used anywhere. I guess soa.h and soa_test.cc can be removed