install/macos: Multiple issues with Go 1.13 and module mode

[ray-harris]

Summary
ISSUE 1: Source path for copying the kernel is incorrect in README.md
ISSUE 2: gophernotes looks for gomacro in $GOPATH, not as a module.
ISSUE 3: imports in a notebook don't find the module
ISSUE 4: gomacro does not add an import for package unsafe from the standard library

Pre-install environment

$ go version
go version go1.13.4 darwin/amd64

$ jupyter --version
jupyter core     : 4.6.1
jupyter-notebook : 6.0.2
qtconsole        : 4.5.5
ipython          : 7.6.1
ipykernel        : 5.1.3
jupyter client   : 5.3.4
jupyter lab      : not installed
nbconvert        : 5.6.1
ipywidgets       : 7.5.1
nbformat         : 4.4.0
traitlets        : 4.3.2

$ brew list zmq
/usr/local/Cellar/zeromq/4.3.2/bin/curve_keygen
/usr/local/Cellar/zeromq/4.3.2/include/ (2 files)
/usr/local/Cellar/zeromq/4.3.2/lib/libzmq.5.dylib
/usr/local/Cellar/zeromq/4.3.2/lib/pkgconfig/libzmq.pc
/usr/local/Cellar/zeromq/4.3.2/lib/ (2 other files)
/usr/local/Cellar/zeromq/4.3.2/share/man/ (67 files)

Installation

ISSUE 1: Source path for copying the kernel is incorrect in README.md.

$ go get -u github.com/gopherdata/gophernotes
go: extracting github.com/gopherdata/gophernotes v0.5.0
go: extracting github.com/cosmos72/gomacro v0.0.0-20190929212108-d70dd8a6f170
go: finding github.com/cosmos72/gomacro latest
go: finding golang.org/x/sys latest
go: extracting github.com/cosmos72/gomacro v0.0.0-20191102204824-93b7995afaa9
go: downloading golang.org/x/sys v0.0.0-20191112214154-59a1497f0cea
go: extracting golang.org/x/sys v0.0.0-20191112214154-59a1497f0cea

$ mkdir -p ~/Library/Jupyter/kernels/gophernotes

$ cp $GOPATH/src/github.com/gopherdata/gophernotes/kernel/* ~/Library/Jupyter/kernels/gophernotes
cp: cannot stat '/Users/ray/go/src/github.com/gopherdata/gophernotes/kernel/*': No such file or directory

Workaround: Since the go get command put gophernotes in $GOPATH/pkg/mod/github.com/gopherdata/gophernotes@v0.5.0, I changed the source path:

$ cp $GOPATH/pkg/mod/github.com/gopherdata/gophernotes@v0.5.0/kernel/* ~/Library/Jupyter/kernels/gophernotes

Test Installation

ISSUE 2: gophernotes looks for gomacro in $GOPATH, not as a module.

$ gophernotes
// warning: could not find package "github.com/cosmos72/gomacro" in $GOPATH = "/Users/ray/go", assuming package is located in "/Users/ray/go/src/github.com/cosmos72/gomacro"
2019/11/12 17:45:31 Need a command line argument specifying the connection file.

Workaround: Install gomacro in gopath mode.

$ GO111MODULE=off go get github.com/cosmos72/gomacro

$ gophernotes
2019/11/12 17:48:08 Need a command line argument specifying the connection file.

Start a Go Jupyter Notebook

ISSUE 3: imports in a notebook don't find the module

$ jupyter notebook

Create a new notebook: New -> Go in Jupyter home page

Run this in a cell:

In [1]: import "github.com/pkg/errors"

error loading package "github.com/pkg/errors" metadata, maybe you need to download (go get), compile (go build) and install (go install) it? can't find import: "github.com/pkg/errors"

I have the repo as module:

$ ls $GOPATH/pkg/mod/github.com/pkg
errors@v0.8.1

Workaround:

• Get the repo in gopath mode
• Restart jupyter notebook in gopath mode

I tried to get the repo in gopath mode:

$ GO111MODULE=off go get github.com/pkg/errors

I then restarted the jupyter notebook and ran the cell in a new notebook:

In [ ]: import "github.com/pkg/errors"

Dialog Box:
Kernel Restarting
The kernel appears to have died. It will restart automatically.

Logs:
$ jupyter notebook
[I 18:03:54.984 NotebookApp] Serving notebooks from local directory: /Users/ray
[I 18:03:54.984 NotebookApp] The Jupyter Notebook is running at:
[I 18:03:54.984 NotebookApp] http://localhost:8888/?token=[secure]
[I 18:03:54.984 NotebookApp]  or http://127.0.0.1:8888/?token=[secure]
[I 18:03:54.984 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 18:03:54.998 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///Users/ray/Library/Jupyter/runtime/nbserver-89880-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=[secure]
     or http://127.0.0.1:8888/?token=[secure]
[I 18:04:01.077 NotebookApp] Creating new notebook in
[I 18:04:02.093 NotebookApp] Kernel started: 7b1a7b3b-5d47-4871-a4e1-644872c32292
[I 18:04:02.250 NotebookApp] Adapting from protocol version 5.0 (kernel 7b1a7b3b-5d47-4871-a4e1-644872c32292) to 5.3 (client).
2019/11/12 18:04:02 Unhandled shell message:  comm_info_request
// debug: created file "/Users/ray/go/src/gomacro_imports/github.com/pkg/errors/errors.go"...
// debug: compiling "/Users/ray/go/src/gomacro_imports/github.com/pkg/errors/errors.go" ...
go: cannot find main module; see 'go help modules'
2019/11/12 18:04:15 Error polling heartbeat channel: interrupted system call
[I 18:04:17.091 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
WARNING:root:kernel 7b1a7b3b-5d47-4871-a4e1-644872c32292 restarted

I shut down the jupyter notebook and then restarted it with GO111MODULE=off

$ GO111MODULE=off jupyter notebook

Notebook Cell:
In [1]: import "github.com/pkg/errors"

At this point I was able to use the package.

ISSUE 4: gomacro does not add an import for package unsafe from the standard library

I tried to use another package:

$ GO111MODULE=off go get gorgonia.org/tensor

$ GO111MODULE=off jupyter notebook

In [1]: import "gorgonia.org/tensor"

error executing "/Users/ray/.gimme/versions/go1.13.4.darwin.amd64/bin/go build -buildmode=plugin" in directory "/Users/ray/go/src/gomacro_imports/gorgonia.org/tensor/": exit status 2

Logs:
[I 18:27:35.337 NotebookApp] Serving notebooks from local directory: /Users/ray
[I 18:27:35.338 NotebookApp] The Jupyter Notebook is running at:
[I 18:27:35.338 NotebookApp] http://localhost:8888/?token=c436d77c84f18672e8fa5afdf8b18096a61927f29065f2fa
[I 18:27:35.338 NotebookApp]  or http://127.0.0.1:8888/?token=c436d77c84f18672e8fa5afdf8b18096a61927f29065f2fa
[I 18:27:35.338 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 18:27:35.354 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///Users/ray/Library/Jupyter/runtime/nbserver-91222-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=c436d77c84f18672e8fa5afdf8b18096a61927f29065f2fa
     or http://127.0.0.1:8888/?token=c436d77c84f18672e8fa5afdf8b18096a61927f29065f2fa
[I 18:28:31.247 NotebookApp] Creating new notebook in
[I 18:28:32.276 NotebookApp] Kernel started: c41486b9-e0d7-43e7-97ed-5eeb63633cfa
[I 18:28:32.435 NotebookApp] Adapting from protocol version 5.0 (kernel c41486b9-e0d7-43e7-97ed-5eeb63633cfa) to 5.3 (client).
2019/11/12 18:28:32 Unhandled shell message:  comm_info_request
// debug: created file "/Users/ray/go/src/gomacro_imports/gorgonia.org/tensor/tensor.go"...
// debug: compiling "/Users/ray/go/src/gomacro_imports/gorgonia.org/tensor/tensor.go" ...
# gomacro_imports/gorgonia.org/tensor
./tensor.go:1164:29: undefined: unsafe
./tensor.go:1170:30: undefined: unsafe

From $GOPATH/src/gomacro_imports/gorgonia.org/tensor/tensor.go:

// this file was generated by gomacro command: import "gorgonia.org/tensor"
// DO NOT EDIT! Any change will be lost when the file is re-generated

package main

import (
	. "reflect"
	blas_0 "gonum.org/v1/gonum/blas"
	tensor_ "gorgonia.org/tensor"
	reflect_0 "reflect"
)

<SNIP>

// --------------- proxy for gorgonia.org/tensor.Memory ---------------
type P_Memory struct {
    Object  interface{}
    MemSize_    func(interface{}) uintptr
    Pointer_    func(interface{}) unsafe.Pointer
    Uintptr_    func(interface{}) uintptr
}
func (P *P_Memory) MemSize() uintptr {
    return P.MemSize_(P.Object)
}
func (P *P_Memory) Pointer() unsafe.Pointer {
    return P.Pointer_(P.Object)
}

<SNIP>

Original code that is being proxied is here:

https://github.com/gorgonia/tensor/blob/master/engine.go#L16

I don't know how to work around this issue.

[cosmos72]

Thanks for the detailed report :)
It's a known limitation of the underlying Go interpreter, I am working to add modules support to it.

[cosmos72]

I added modules support to the underlying Go interpreter.
ASAP I will update gophernotes to use it.

[cosmos72]

working on it, see #189

[cosmos72]

modules support added.
I will check again the issues listed above and see if some are still open.

[cosmos72]

To my knowledge, issue 1 3 and 4 have been fixed.

Issue 2 is a warning (not an error) and now it happens only if you ask gophernotes to write import files inside gophernotes source directory with import _b "path/to/some/package"

Since $GOPATH/src is usually writeable but source code downloaded as module in $GOPATH/pkg/mod is not writeable, such warning is still useful.