Running info on vector memory inputs return UNIX timestamps instead of ISO datetimes

[weiji14]

Description of the problem

UNIX timestamps (e.g. 1577836800) are returned instead of ISO datetimes (e.g. 2020-01-01T00:00:00) when running info on virtual tables. Originally noticed at GenericMappingTools/pygmt#619 (comment), check also #3396 where datetime inputs were originally accepted into GMT_Put_Vector.

Full script that generated the error

Sorry that I can't write C code, but given an input like so:

z time
10 2020-01-01T00:00:00
13 2020-01-02T00:00:00
12 2020-01-03T00:00:00
15 2020-01-04T00:00:00
14 2020-01-05T00:00:00

and then running info from PyGMT from the branch at GenericMappingTools/pygmt#619 (each column is passed into C API using GMT_Put_Vector)

import pygmt

# create data array 'table' and pass it into `info`
pygmt.info(table=table, V="d")

Full error message

gmtinfo [INFORMATION]: Processing input table data
gmtinfo [DEBUG]: gmtapi_init_import: Passed family = Data Table and geometry = Point|Line|Poly
gmtinfo (gmtapi_init_import): tried to free unallocated memory
gmtinfo [DEBUG]: gmtapi_init_import: Added 1 new sources
gmtinfo [DEBUG]: GMT_Init_IO: Returned first Input object ID = 0
gmtinfo [DEBUG]: gmtapi_init_export: Passed family = Data Table and geometry = Point|Line|Poly
gmtinfo [DEBUG]: Object ID 1 : Registered Data Table File /tmp/pygmt-gykf5n3t.txt as an Output resource with geometry Point|Line|Poly [n_objects = 2]
gmtinfo [DEBUG]: gmtapi_init_export: Added 1 new destination
gmtinfo [DEBUG]: GMT_Init_IO: Returned first Output object ID = 1
gmtinfo [DEBUG]: GMT_Begin_IO: Mode value 1 not considered (ignored)
gmtinfo [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Input
gmtinfo [DEBUG]: gmtapi_next_io_source: Selected object 0
gmtinfo [INFORMATION]: Reading Data Table from Input memory location via vector
gmtinfo [DEBUG]: GMT_Begin_IO: Input resource access is now enabled [record-by-record]
gmtinfo [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Output
gmtinfo [DEBUG]: gmtapi_next_io_source: Selected object 1
gmtinfo [INFORMATION]: Writing Data Table to file /tmp/pygmt-gykf5n3t.txt
gmtinfo [DEBUG]: GMT_Begin_IO: Output resource access is now enabled [record-by-record]
gmtinfo [DEBUG]: Number of numerical output columns has been set to 0
gmtinfo [DEBUG]: GMT_End_IO: Input resource access is now disabled
gmtinfo [DEBUG]: GMT_End_IO: Output resource access is now disabled
gmtinfo (GMT_gmtinfo): tried to free unallocated memory
gmtinfo (GMT_gmtinfo): tried to free unallocated memory
gmtinfo [DEBUG]: gmtlib_unregister_io: Unregistering object no 1 [n_objects = 1]
gmtinfo (gmtlib_free_tmp_arrays): tried to free unallocated memory

Actual outcome

<vector memory>: N = 5 <10/15> <1577836800/1578182400>

Expected outcome

<vector memory>: N = 5 <10/15> <2020-01-01T00:00:00/2020-01-05T00:00:00>

System information

• Operating system: Linux
• GMT version (gmt --version): 6.1.1

[PaulWessel]

Hm, yes since what is returned is a text record I agree there is no good reason it should not be formatted as you expect here. If -C was used then you would get a numerical record and it would have UNIX time of course, but here it is text. I will try to have a look as to why it is not doing the formatting.

[PaulWessel]

I would like to add another section to the DEVELOPER RESOURCES sidebar on debugging, for how to debug pyGMT via Xcode on macOS. Been a month since last time and cannot find all the step that @seisman helped with. Since I am a complete python rookie there is not much mental memory on even the most basics, so I need that recipe, which also had some issues with setting shared-dir and possibly library paths for all to work. Maybe @seisman could resurrect his instructions and I can turn it into a section in the developer RST?

[PaulWessel]

And @joa-quim has given similar instructions for setting up GMT.jl so I can debug cases he finds directly in Xcode as well?

[weiji14]

I would like to add another section to the DEVELOPER RESOURCES sidebar on debugging, for how to debug pyGMT via Xcode on macOS. Been a month since last time and cannot find all the step that @seisman helped with. Since I am a complete python rookie there is not much mental memory on even the most basics, so I need that recipe, which also had some issues with setting shared-dir and possibly library paths for all to work. Maybe @seisman could resurrect his instructions and I can turn it into a section in the developer RST?

Issue is at #3778. I should probably start to learn C too at some point 😆

[seisman]

The steps to debug PyGMT via Xcode was discussed in #3829.

Here is a summary:

1. Install PyGMT following the official instructions

# add conda-forge channel
conda config --prepend channels conda-forge
# NOTE: the next step is different from the PyGMT official instructions, because we want to use the GMT dev version
conda create --name pygmt python=3.8 pip numpy pandas xarray netcdf4 packaging
# activate the pygmt envirionment
conda activate pygmt

# install pygmt in editable/development mode
cd pygmt
make install

2. Compile GMT using Xcode
3. Tell PyGMT where to find the GMT library by setting the environmental variable GMT_LIBRARY_PATH

export GMT_LIBRARY_PATH=~/Gits/gmt/gmt/build/xcode/src/Debug

4. Open Xcode
5. Run a python console, attach the process id in xcode, and run PyGMT codes in the Python console and xcode will stop at the breakpoint.

[joa-quim]

Issue is at #3778. I should probably start to learn C too at some point

Please, please join the Force.

Debugging from any of the environments (Matlab, Julia, Python) should all the same procedure.

1. Open the shell (ML IDE, Julia or Py REPL), type the command that will trigger the program to debug, but HOLD ON
2. Open the debugger an Attach to process (this is VS talk, Xcode does similar but on its own way). The process is either the Matlab running process or Julia, Py shells.
3. Open the source code of the program to be debugged ans set break points.
4. Go back to 1. and hit RETURN

[seisman]

@weiji14 Perhaps we should merge GenericMappingTools/pygmt#619 first (mark the tests as xfail) and provide a minimal Python script to reproduce the issue. Your example above is incomplete, and @PaulWessel will need to rewrite a working script before debugging.

[joa-quim]

Ofc, on Unix everything is more complicated 😈

[weiji14]

Please, please join the Force.

http://www.nooooooooooooooo.com/

Ofc, on Unix everything is more complicated 😈

Yeah, I'm on Linux which doesn't seem to have Xcode 🤷

@weiji14 Perhaps we should merge GenericMappingTools/pygmt#619 first (mark the tests as xfail) and provide a minimal Python script to reproduce the issue. Your example above is incomplete, and @PaulWessel will need to rewrite a working script before debugging.

Yep, give me a few minutes to finish up that PR.

[PaulWessel]

Yes, it would be great if a snipped of Python code demonstrating the problem is submitted since that is not something I will know how to do. Same with Julia of course and @joa-quim usually posts a Julia command or mex command. We just want to lower the threshold for all of us to do these things. Even C-novices should be able to step in the the C library and at least detect where a crash happens - often that is enough of a clue for the C-team to figure things out.

[joa-quim]

Good question for Linux. I guess that gcc should have an equivalent mechanism to Attach to process ID and from that on, the doing should be the same ... except that using gdb is an act of masochism.

[PaulWessel]

On Linux, I have done lots of debugging using ddd, and even on macOS before Apple switched to clang and it became a bit harder (and I have not paid too much attention). But I dont know if you can connect to the process in ddd the same way we do in Xcode or Visual Studio.

[PaulWessel]

ddd is just a graphical front-end to gdb which I assume can connect to other processes.

[PaulWessel]

@joa-quim, do you mind posting the usual setup for a newbie to install GMT.jl from the developer version, including installing julia. I know you have some of that on your site but it would help me put the docs together if you can (like @seisman) cut/paste in here while I am thinking about this.