Port WAIL to Linux

[machawk1]

The question remains: where is the most appropriate location that is system independent where WAIL could reside on a Linux system? C:\WAIL is used for Windows, /Applications/WAIL on MacOS X.

[machawk1]

Per http://docs.python-guide.org/en/latest/shipping/freezing/

On Linux, an alternative to freezing is to create a Linux distro package[1](e.g. .deb files for Debian or Ubuntu, or .rpm files for Red Hat and SuSE.)

[1] http://docs.python-guide.org/en/latest/shipping/packaging/#packaging-for-linux-distributions-ref

[kanihal]

/opt/ should be location for system indp. installations.
BTW when is the expected release date of WAIL for linux?

[machawk1]

@kanihal Thanks for the suggestion, /opt/ is likely the most appropriate location.

Regarding your question, we did a revamp of WAIL in Electron that is available for macOS, Windows, and Linux. It uses a different UI model (collection-based) and contains some different tools (e.g., it interfaces with Headless Chrome for crawling) but might fit your need. Please give it a try, as I do not currently have an estimate for the release of the Linux version of WAIL (this repo).

[kanihal]

@machawk1 WAIL in electron works on ubuntu & is easy to use. Nice work.

[machawk1]

There are a couple of outstanding issues that need to be resolved that are preventing the generation of a build for Linux. In using Ubuntu 18.10, pip install wxPython produces an error:

machawk1@ubuntu:~$ pip install wxPython
Collecting wxPython
  Downloading https://files.pythonhosted.org/packages/17/74/7c3ced03c3c76b9f98e4a0edae1801755a7599ebf481c04d9f77dfff17e3/wxPython-4.0.4.tar.gz (68.8MB)
    100% |████████████████████████████████| 68.8MB 25.7MB/s 
Collecting Pillow (from wxPython)
  Downloading https://files.pythonhosted.org/packages/0d/f3/421598450cb9503f4565d936860763b5af413a61009d87a5ab1e34139672/Pillow-5.4.1-cp27-cp27mu-manylinux1_x86_64.whl (2.0MB)
    100% |████████████████████████████████| 2.0MB 45.2MB/s 
Collecting six (from wxPython)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: Pillow, six, wxPython
  Running setup.py install for wxPython ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-IL7i2Y/wxPython/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-Pz5Enm-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    running install
    running build
    WARNING: Building this way assumes that all generated files have been
    generated already.  If that is not the case then use build.py directly
    to generate the source and perform the build stage.  You can use
    --skip-build with the bdist_* or install commands to avoid this
    message and the wxWidgets and Phoenix build steps in the future.
    
    "/usr/bin/python" -u build.py build
    Will build using: "/usr/bin/python"
    2.7.15+ (default, Oct  2 2018, 22:12:08)
    [GCC 8.2.0]
    Python's architecture is 64bit
    cfg.VERSION: 4.0.4
    
    Running command: build
    Running command: build_wx
    wxWidgets build options: ['--wxpython', '--unicode', '--gtk3']
    Configure options: ['--enable-unicode', '--with-gtk=3', '--with-opengl', '--enable-sound', '--enable-graphics_ctx', '--enable-mediactrl', '--enable-display', '--enable-geometry', '--enable-debug_flag', '--enable-optimise', '--disable-debugreport', '--enable-uiactionsim', '--enable-autoidman', '--with-sdl']
    /tmp/pip-build-IL7i2Y/wxPython/ext/wxWidgets/configure --enable-unicode --with-gtk=3 --with-opengl --enable-sound --enable-graphics_ctx --enable-mediactrl --enable-display --enable-geometry --enable-debug_flag --enable-optimise --disable-debugreport --enable-uiactionsim --enable-autoidman --with-sdl
    checking build system type... x86_64-pc-linux-gnu
    checking host system type... x86_64-pc-linux-gnu
    checking for --disable-gui... no
    checking for --enable-monolithic... no
    checking for --enable-plugins... no
    checking for --without-subdirs... no
    checking for --enable-official_build... no
    checking for --disable-all-features... no
    checking for --enable-universal... no
    checking for --enable-nanox... no
    checking for --enable-gpe... no
    checking for toolkit... gtk
    checking for --with-libpng... yes
    checking for --with-libjpeg... yes
    checking for --with-libtiff... yes
    checking for --without-libjbig... no
    checking for --without-liblzma... no
    checking for --with-libxpm... yes
    checking for --with-libiconv... yes
    checking for --with-libmspack... no
    checking for --without-gtkprint... no
    checking for --with-gnomevfs... no
    checking for --with-libnotify... yes
    checking for --with-hildon... no
    checking for --with-opengl... yes
    checking for --with-dmalloc... no
    checking for --with-sdl... yes
    checking for --with-regex... yes
    checking for --with-zlib... yes
    checking for --with-expat... yes
    checking for --with-macosx-sdk...
    checking for --with-macosx-version-min...
    checking for --enable-debug... default
    checking for --disable-debug_flag... no
    checking for --enable-debug_info... no
    checking for --enable-debug_gdb... no
    checking for --enable-debug_cntxt... no
    checking for --enable-mem_tracing... no
    checking for --disable-shared... no
    checking for --enable-stl... no
    checking for --enable-std_containers... no
    checking for --enable-std_iostreams... yes
    checking for --enable-std_string... yes
    checking for --enable-std_string_conv_in_wxstring... no
    checking for --disable-unicode... no
    checking for --enable-mslu... no
    checking for --enable-utf8... no
    checking for --enable-utf8only... no
    checking for --enable-extended_rtti... no
    checking for --disable-optimise... no
    checking for --enable-profile... no
    checking for --enable-no_rtti... no
    checking for --enable-no_exceptions... no
    checking for --enable-permissive... no
    checking for --enable-no_deps... no
    checking for --disable-vararg_macros... no
    checking for --enable-universal_binary... no
    checking for --enable-macosx_arch... no
    checking for --enable-compat26... no
    checking for --disable-compat28... no
    checking for --disable-rpath... no
    checking for --enable-objc_uniquifying... no
    checking for --disable-visibility... no
    checking for --disable-tls... no
    checking for --enable-intl... yes
    checking for --enable-xlocale... yes
    checking for --enable-config... yes
    checking for --enable-protocols... yes
    checking for --enable-ftp... yes
    checking for --enable-http... yes
    checking for --enable-fileproto... yes
    checking for --enable-sockets... yes
    checking for --enable-ipv6... no
    checking for --enable-ole... yes
    checking for --enable-dataobj... yes
    checking for --enable-ipc... yes
    checking for --enable-baseevtloop... yes
    checking for --enable-epollloop... yes
    checking for --enable-selectloop... yes
    checking for --enable-any... yes
    checking for --enable-apple_ieee... yes
    checking for --enable-arcstream... yes
    checking for --enable-base64... yes
    checking for --enable-backtrace... yes
    checking for --enable-catch_segvs... yes
    checking for --enable-cmdline... yes
    checking for --enable-datetime... yes
    checking for --enable-debugreport... no
    checking for --enable-dialupman... yes
    checking for --enable-dynlib... yes
    checking for --enable-dynamicloader... yes
    checking for --enable-exceptions... yes
    checking for --enable-ffile... yes
    checking for --enable-file... yes
    checking for --enable-filehistory... yes
    checking for --enable-filesystem... yes
    checking for --enable-fontenum... yes
    checking for --enable-fontmap... yes
    checking for --enable-fs_archive... yes
    checking for --enable-fs_inet... yes
    checking for --enable-fs_zip... yes
    checking for --enable-fsvolume... yes
    checking for --enable-fswatcher... yes
    checking for --enable-geometry... yes
    checking for --enable-log... yes
    checking for --enable-longlong... yes
    checking for --enable-mimetype... yes
    checking for --enable-printfposparam... yes
    checking for --enable-snglinst... yes
    checking for --enable-sound... yes
    checking for --enable-stdpaths... yes
    checking for --enable-stopwatch... yes
    checking for --enable-streams... yes
    checking for --enable-sysoptions... yes
    checking for --enable-tarstream... yes
    checking for --enable-textbuf... yes
    checking for --enable-textfile... yes
    checking for --enable-timer... yes
    checking for --enable-variant... yes
    checking for --enable-zipstream... yes
    checking for --enable-url... yes
    checking for --enable-protocol... yes
    checking for --enable-protocol_http... yes
    checking for --enable-protocol_ftp... yes
    checking for --enable-protocol_file... yes
    checking for --enable-threads... yes
    checking for --enable-iniconf... no
    checking for --enable-regkey... yes
    checking for --enable-docview... yes
    checking for --enable-help... yes
    checking for --enable-mshtmlhelp... yes
    checking for --enable-html... yes
    checking for --enable-htmlhelp... yes
    checking for --enable-xrc... yes
    checking for --enable-aui... yes
    checking for --enable-propgrid... yes
    checking for --enable-ribbon... yes
    checking for --enable-stc... yes
    checking for --enable-constraints... yes
    checking for --enable-loggui... yes
    checking for --enable-logwin... yes
    checking for --enable-logdialog... yes
    checking for --enable-mdi... yes
    checking for --enable-mdidoc... yes
    checking for --enable-mediactrl... yes
    checking for --enable-gstreamer8... no
    checking for --enable-richtext... yes
    checking for --enable-postscript... yes
    checking for --enable-printarch... yes
    checking for --enable-svg... yes
    checking for --enable-webkit... yes
    checking for --enable-webview... yes
    checking for --enable-graphics_ctx... yes
    checking for --enable-clipboard... yes
    checking for --enable-dnd... yes
    checking for --disable-controls... no
    checking for --enable-markup... yes
    checking for --enable-accel... yes
    checking for --enable-animatectrl... yes
    checking for --enable-bannerwindow... yes
    checking for --enable-artstd... yes
    checking for --enable-arttango... auto
    checking for --enable-bmpbutton... yes
    checking for --enable-bmpcombobox... yes
    checking for --enable-button... yes
    checking for --enable-calendar... yes
    checking for --enable-caret... yes
    checking for --enable-checkbox... yes
    checking for --enable-checklst... yes
    checking for --enable-choice... yes
    checking for --enable-choicebook... yes
    checking for --enable-collpane... yes
    checking for --enable-colourpicker... yes
    checking for --enable-combobox... yes
    checking for --enable-comboctrl... yes
    checking for --enable-commandlinkbutton... yes
    checking for --enable-dataviewctrl... yes
    checking for --enable-datepick... yes
    checking for --enable-detect_sm... yes
    checking for --enable-dirpicker... yes
    checking for --enable-display... yes
    checking for --enable-editablebox... yes
    checking for --enable-filectrl... yes
    checking for --enable-filepicker... yes
    checking for --enable-fontpicker... yes
    checking for --enable-gauge... yes
    checking for --enable-grid... yes
    checking for --enable-headerctrl... yes
    checking for --enable-hyperlink... yes
    checking for --enable-imaglist... yes
    checking for --enable-infobar... yes
    checking for --enable-listbook... yes
    checking for --enable-listbox... yes
    checking for --enable-listctrl... yes
    checking for --enable-notebook... yes
    checking for --enable-notifmsg... yes
    checking for --enable-odcombobox... yes
    checking for --enable-popupwin... yes
    checking for --enable-prefseditor... yes
    checking for --enable-radiobox... yes
    checking for --enable-radiobtn... yes
    checking for --enable-richmsgdlg... yes
    checking for --enable-richtooltip... yes
    checking for --enable-rearrangectrl... yes
    checking for --enable-sash... yes
    checking for --enable-scrollbar... yes
    checking for --enable-searchctrl... yes
    checking for --enable-slider... yes
    checking for --enable-spinbtn... yes
    checking for --enable-spinctrl... yes
    checking for --enable-splitter... yes
    checking for --enable-statbmp... yes
    checking for --enable-statbox... yes
    checking for --enable-statline... yes
    checking for --enable-stattext... yes
    checking for --enable-statusbar... yes
    checking for --enable-taskbaricon... yes
    checking for --enable-tbarnative... yes
    checking for --enable-textctrl... yes
    checking for --enable-timepick... yes
    checking for --enable-tipwindow... yes
    checking for --enable-togglebtn... yes
    checking for --enable-toolbar... yes
    checking for --enable-toolbook... yes
    checking for --enable-treebook... yes
    checking for --enable-treectrl... yes
    checking for --enable-treelist... yes
    checking for --enable-commondlg... yes
    checking for --enable-aboutdlg... yes
    checking for --enable-choicedlg... yes
    checking for --enable-coldlg... yes
    checking for --enable-filedlg... yes
    checking for --enable-finddlg... yes
    checking for --enable-fontdlg... yes
    checking for --enable-dirdlg... yes
    checking for --enable-msgdlg... yes
    checking for --enable-numberdlg... yes
    checking for --enable-splash... yes
    checking for --enable-textdlg... yes
    checking for --enable-tipdlg... yes
    checking for --enable-progressdlg... yes
    checking for --enable-wizarddlg... yes
    checking for --enable-menus... yes
    checking for --enable-miniframe... yes
    checking for --enable-tooltips... yes
    checking for --enable-splines... yes
    checking for --enable-mousewheel... yes
    checking for --enable-validators... yes
    checking for --enable-busyinfo... yes
    checking for --enable-hotkey... auto
    checking for --enable-joystick... yes
    checking for --enable-metafile... auto
    checking for --enable-dragimage... yes
    checking for --enable-accessibility... no
    checking for --enable-uiactionsim... yes
    checking for --enable-dctransform... yes
    checking for --enable-webviewwebkit... yes
    checking for --enable-palette... yes
    checking for --enable-image... yes
    checking for --enable-gif... yes
    checking for --enable-pcx... yes
    checking for --enable-tga... yes
    checking for --enable-iff... yes
    checking for --enable-pnm... yes
    checking for --enable-xpm... yes
    checking for --enable-ico_cur... yes
    checking for --enable-dccache... yes
    checking for --enable-ps-in-msw... yes
    checking for --enable-ownerdrawn... yes
    checking for --enable-uxtheme... yes
    checking for --enable-wxdib... yes
    checking for --enable-webviewie... yes
    checking for --enable-autoidman... yes
    checking for gcc... gcc
    checking whether the C compiler works... yes
    checking for C compiler default output file name... a.out
    checking for suffix of executables...
    checking whether we are cross compiling... no
    checking for suffix of object files... o
    checking whether we are using the GNU C compiler... yes
    checking whether gcc accepts -g... yes
    checking for gcc option to accept ISO C89... none needed
    checking whether we are using the Intel C compiler... no
    checking how to run the C preprocessor... gcc -E
    checking for grep that handles long lines and -e... /bin/grep
    checking for egrep... /bin/grep -E
    checking whether gcc needs -traditional... no
    checking for g++... g++
    checking whether we are using the GNU C++ compiler... yes
    checking whether g++ accepts -g... yes
    checking whether we are using the Intel C++ compiler... no
    checking whether we are using the Sun C++ compiler... no
    checking for ar... ar
    checking for ANSI C header files... yes
    checking for sys/types.h... yes
    checking for sys/stat.h... yes
    checking for stdlib.h... yes
    checking for string.h... yes
    checking for memory.h... yes
    checking for strings.h... yes
    checking for inttypes.h... yes
    checking for stdint.h... yes
    checking for unistd.h... yes
    checking for langinfo.h... yes
    checking for wchar.h... yes
    checking for sys/select.h... yes
    checking for cxxabi.h... yes
    checking for an ANSI C-conforming const... yes
    checking for inline... inline
    checking size of short... 2
    checking size of void *... 8
    checking size of int... 4
    checking size of long... 8
    checking size of size_t... 8
    checking size of long long... 8
    checking size of wchar_t... 4
    checking for va_copy... yes
    checking whether the compiler supports variadic macros... yes
    checking for _FILE_OFFSET_BITS value needed for large files... 64
    checking if large file support is available... yes
    checking for _LARGEFILE_SOURCE value needed for large files... no
    checking whether byte ordering is bigendian... no
    checking for iostream... yes
    checking if C++ compiler supports the explicit keyword... yes
    checking for std::wstring in <string>... yes
    checking for std::istream... yes
    checking for std::ostream... yes
    checking how to run the C++ preprocessor... g++ -E
    checking type_traits usability... yes
    checking type_traits presence... yes
    checking for type_traits... yes
    checking for __sync_fetch_and_add and __sync_sub_and_fetch builtins... yes
    checking for libraries directories... /usr/lib/x86_64-linux-gnu /usr/lib
    checking for cos... no
    checking for floor... no
    checking if floating point functions link without -lm... no
    checking for sin... yes
    checking for ceil... yes
    checking if floating point functions link with -lm... yes
    checking for strtoull... yes
    checking for pkg-config... no
    configure: WARNING: Defaulting to the builtin regex library for Unicode build.
    checking for zlib.h >= 1.1.4... no
    checking for zlib.h... (cached) no
    configure: WARNING: zlib library not found or too old, will use built-in instead
    checking for png.h > 0.90... no
    checking for png.h... (cached) no
    configure: WARNING: system png library not found or too old, will use built-in instead
    checking for jpeglib.h... no
    configure: WARNING: system jpeg library not found, will use built-in instead
    checking for tiffio.h... no
    configure: WARNING: system tiff library not found, will use built-in instead
    checking for lzma_code in -llzma... no
    checking for jbg_dec_init in -ljbig... no
    checking for expat.h... yes
    checking if expat.h is valid C++ header... yes
    checking for XML_ParserCreate in -lexpat... yes
    checking for GTK+ version...
    checking for pkg-config... no
    checking for GTK+ - version >= 3.0.0... no
    *** A new enough version of pkg-config was not found.
    *** See http://pkgconfig.sourceforge.net
    configure: error:
    The development files for GTK+ were not found. For GTK+ 2, please
    ensure that pkg-config is in the path and that gtk+-2.0.pc is
    installed. For GTK+ 1.2 please check that gtk-config is in the path,
    and that the version is 1.2.3 or above. Also check that the
    libraries returned by 'pkg-config gtk+-2.0 --libs' or 'gtk-config
    --libs' are in the LD_LIBRARY_PATH or equivalent.
    
    Error running configure
    ERROR: failed building wxWidgets
    Traceback (most recent call last):
      File "build.py", line 1321, in cmd_build_wx
        wxbuild.main(wxDir(), build_options)
      File "/tmp/pip-build-IL7i2Y/wxPython/buildtools/build_wxwidgets.py", line 375, in main
        "Error running configure")
      File "/tmp/pip-build-IL7i2Y/wxPython/buildtools/build_wxwidgets.py", line 85, in exitIfError
        raise builder.BuildError(msg)
    BuildError
    Finished command: build_wx (6.142s)
    Finished command: build (6.142s)
    Command '"/usr/bin/python" -u build.py build' failed with exit code 1.
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-IL7i2Y/wxPython/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-Pz5Enm-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-IL7i2Y/wxPython/

Another issue that the MAKEFILE.sh script is currently tailored to macOS with regard to installation directories.

[machawk1]

After manually installing wxPython from source (using the package's python build.py dox --nodoc sip build, installing Pyinstaller via (and needing to manually reference the binary's path due to pip not doing this by default), then running
~/.local/bin/pyinstaller -p bundledApps bundledApps/WAIL.py --onefile --windowed --clean

produces a dist/WAIL binary. Running this produces an error:

$ ./dist/WAIL 
Traceback (most recent call last):
  File "bundledApps/WAIL.py", line 11, in <module>
    import wx
  File "/tmp/pip-build-Pw9LOp/PyInstaller/PyInstaller/loader/pyimod03_importers.py", line 395, in load_module
  File "wx/__init__.py", line 17, in <module>
  File "/tmp/pip-build-Pw9LOp/PyInstaller/PyInstaller/loader/pyimod03_importers.py", line 395, in load_module
  File "wx/core.py", line 12, in <module>
  File "/tmp/pip-build-Pw9LOp/PyInstaller/PyInstaller/loader/pyimod03_importers.py", line 684, in load_module
ImportError: libpng12.so.0: cannot open shared object file: No such file or directory
[23909] Failed to execute script WAIL

This makes we question whether pyinstaller is seeing the wxPython installation to import at the time of building WAIL.

[machawk1]

A more fundamental example that should probably be resolved first:
wxTest.py

import wx

print('hello')

...then build the py file with pyinstaller:
~/.local/bin/pyinstaller wxTest.py --onefile --clean
...then try running the produced binary:
./dist/wxTest
produces:

Traceback (most recent call last):
  File "wxTest.py", line 1, in <module>
    import wx
  File "/tmp/pip-build-Pw9LOp/PyInstaller/PyInstaller/loader/pyimod03_importers.py", line 395, in load_module
  File "wx/__init__.py", line 17, in <module>
  File "/tmp/pip-build-Pw9LOp/PyInstaller/PyInstaller/loader/pyimod03_importers.py", line 395, in load_module
  File "wx/core.py", line 12, in <module>
  File "/tmp/pip-build-Pw9LOp/PyInstaller/PyInstaller/loader/pyimod03_importers.py", line 684, in load_module
ImportError: libpng12.so.0: cannot open shared object file: No such file or directory
[25966] Failed to execute script wxTest

[machawk1]

https://groups.google.com/forum/#!searchin/wxpython-users/pip$20ubuntu%7Csort:date/wxpython-users/HeI3SK2kXws/489LCruzCwAJ may be a lead in resolving the shared library objects warning. This will require recompilation of wxPython.

EDIT: It appears that the --enabled-shared flag is for when the Python environment is compiled, not wxPython. Given I installed Python via apt, I am unsure if specifying this flag is allowed.

Also, this rabbit hole is kind of ridiculous, especially compared to compiling WAIL on macOS and Windows.

Also also, the blame on the object lib can be attributed to wxPython and looks to be independent on pyinstaller:

$ python wxTest.py 
Traceback (most recent call last):
  File "wxTest.py", line 1, in <module>
    import wx
  File "/home/machawk1/.local/lib/python2.7/site-packages/wx/__init__.py", line 17, in <module>
    from wx.core import *
  File "/home/machawk1/.local/lib/python2.7/site-packages/wx/core.py", line 12, in <module>
    from ._core import *
ImportError: libpng12.so.0: cannot open shared object file: No such file or directory

[machawk1]

Per https://wxpython.org/pages/downloads/, pip install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.10 wxPython should work on my setup but the wxPython installation process is dependent on which flavor of Linux it is being built. This will need to be included in the Linux-specific MAKEFILE or thoroughly documented.

EDIT: Err, there is no 18.10, trying again with 18.04. And then, I receive a Memory Error from pip.

EDIT: Bumping up the memory in the Ubuntu VM from the default to 4 GB and repeating the about with 18.04. This allowed the wxPython to complete without reporting an issue. Running python wxTest.py now prints "Hello" instead of crashing due to the import.

[machawk1]

After building wxPython , building WAIL, and running the binary, a shared object issue still arises at the time of binary execution.

$ ./dist/WAIL 
User likely has the binary in the wrong location.
Traceback (most recent call last):
  File "bundledApps/WAIL.py", line 48, in <module>
    import wx.adv
  File "/tmp/pip-build-Pw9LOp/PyInstaller/PyInstaller/loader/pyimod03_importers.py", line 395, in load_module
  File "wx/adv.py", line 13, in <module>
  File "/tmp/pip-build-Pw9LOp/PyInstaller/PyInstaller/loader/pyimod03_importers.py", line 684, in load_module
ImportError: libSDL-1.2.so.0: cannot open shared object file: No such file or directory
[3241] Failed to execute script WAIL

[machawk1]

sudo apt install libsdl2-2.0 installs some form of the library but pyinstaller seems to want libsdl-1.2, which is obsolete. I may need to circle back install wxPython for my current system instead of relying on the 18.04 wheel.

Delegating the Linux build process to a Docker container is seeming pretty appealing at this point, given the necessary duct tape.

EDIT: Proposed using Docker in #340.