Using qgis_process instead of **RQGIS3**

[jannes-m]

From now on, we discourage the usage of RQGIS3 since it is no longer actively maintained and due to various unresolvable issues. However, the good news is that there is a new standalone executable available from QGIS 3.14 onwards (thanks to @paleolimbot for pointing me to it several months ago). This means, one can now run QGIS processing algorithms from the command line with ease. The tool is easy to use, takes care of setting the environmental variables to run QGIS and the user no longer has to use Python. In short, this seems to be the perfect replacement for RQGIS3 . For a short description how to use qgis_process from within R, please have a look at https://github.com/r-spatial/RQGIS3/
Of course, we should wait a bit to see how and if qgis_process matures but if it does it would be relatively easy to write an R wrapper package around it. I and @pat-s will probably not have the time to do so (unfortunately) but we would support anybody in doing so as best as we can. So please let us know if anybody has the time and the passion to write a new RQGIS3 package. If not, one could still access all QGIS geoprocessing via system() calls. Of course, this is not as user-friendly as a dedicated R package but still relatively easily managable. I also plan to update the RQGIS3 section in Geocomputation with R by explaining a bit more in detail how to use qgis_process (@Robinlovelace, @Nowosad).

[paleolimbot]

Just a quick note that it might be worth looking at how WhiteboxTools approaches this...I believe it's also wrapping a command-line application and I imagine that a QGIS tools wrapper would benefit from aligning syntax. (I haven't used yet...just stumbled across the other day!)

Also a note that processx is probably the way to go (rather than system)...more options for status updates and cancelling a process that's taking too long.

[Robinlovelace]

Minor update on this: there is now what I think is a reproducible environment to test/develop the interface proposed above with Docker. Reproducible example:

docker run -e PASSWORD=pw --rm -ti geocompr/geocompr:qgis /bin/bash
qgis --version
qgis_process

Unfortunately that fails currently with the following message:

QGIS 3.14.16-Pi 'Pi' (a235a149f0)
root@0e56c3189778:/# qgis_process
qt.qpa.xcb: could not connect to display 
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, xcb.

Aborted (core dumped)

Any ideas how to overcome that issue (perhaps running the command with arguments will mean it doesn't need QT but I'm worried about QT issues after rstudio/rstudio#4606 ) @paleolimbot (good idea on processx btw, do you have examples - love your Zotero Addin btw!), @jannes-m or anyone?

I have just sent a new image with the nightly build to DockerHub which should be available in the next hour or so that should be able to be run as follows:

docker run -e PASSWORD=pw --rm -ti geocompr/geocompr:qgis-dev /bin/bash
qgis --version
qgis_process

[florisvdh]

@Robinlovelace I haven't tried, but maybe you'll have more luck with env QT_QPA_PLATFORM='offscreen' qgis_process, for headless execution (from qgis/QGIS#37989). Regarding the referred issue you may run into other problems, still.

[Robinlovelace]

Update, the qgis-dev version shows the same output:

docker run -e PASSWORD=pw --rm -ti geocompr/geocompr:qgis-dev /bin/bash
qgis --version
qgis_process
Unable to find image 'geocompr/geocompr:qgis-dev' locally
qgis-dev: Pulling from geocompr/geocompr
...
Digest: sha256:3089fd762ce04939d144b2d06acb537afde8ca7f2e1de9019044f102a2039f76
Status: Downloaded newer image for geocompr/geocompr:qgis-dev
root@b26e15d7c50e:/# qgis --version
../../src/app/main.cpp:510 : (main) [0ms] Starting qgis main
QGIS 3.15.0-Master 'Master' (ef7ecbf95b)
root@b26e15d7c50e:/# qgis_process
qt.qpa.xcb: could not connect to display 
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, xcb.

Great news @florisvdh I tried your suggestion and it worked 🎉

docker run -e PASSWORD=pw --rm -ti geocompr/geocompr:qgis-dev /bin/bash
# ...
env QT_QPA_PLATFORM='offscreen' qgis_process
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
../../src/core/qgsproviderregistry.cpp:245 : (init) [338ms] Loaded 22 providers (DB2;OAPIF;WFS;arcgisfeatureserver;arcgismapserver;delimitedtext;gdal;geonode;gpx;mdal;memory;mesh_memory;mssql;ogr;ows;postgres;postgresraster;spatialite;vectortile;virtual;wcs;wms) 
QGIS Processing Executor - 3.15.0-Master 'Master' (3.15.0-Master)
Usage: /usr/bin/qgis_process.bin [command] [algorithm id or path to model file] [parameters]

Available commands:
	plugins	list available and active plugins
	list	list all available processing algorithms
	help	show help for an algorithm. The algorithm id or a path to a model file must be specified.
	run	runs an algorithm. The algorithm id or a path to a model file and parameter values must be specified. Parameter values are specified via the --PARAMETER=VALUE syntax.
		If required, the ellipsoid to use for distance and area calculations can be specified via the "--ELLIPSOID=name" argument.
		If required, an existing QGIS project to use during the algorithm execution can be specified via the "--PROJECT_PATH=path" argument.

[Robinlovelace]

BTW see Dockerfiles defining the geocompr/geocompr:qgis images here: https://github.com/geocompr/docker

[florisvdh]

If you want to run scripts in the Docker container that are meant to be examples for users working in a GUI environment, the QT_QPA_PLATFORM setting will better be made applicable to the whole bash session, so that you don't need to (repeatedly) use env. I'm not enough familiar with Docker (but should), but in general it could be done in .bashrc for example (export QT_QPA_PLATFORM='offscreen'). Something similar could be done to avoid the XDG_RUNTIME_DIR not set warning.

[jannes-m]

It is possible to set env vars in the Dockerfile using the ENV command, and I am pretty sure that this would work, when running the container with /bin/bash. However, I doubt that the env var will be exported when running the RStudio server version. At least, I didn't achieve this when I tried to do so but I vaguely remember that this is a known issue with rocker.

[paleolimbot]

I imagine that the env var should be set with every call (e.g., using withr::envvar()) since it might result in an error otherwise (and possibly a confusing UI popup otherwise!). Thanks @florisvdh for the tip!

I'll work up something in the next week or so with what I had in mind for wrapping the calls with processx...basically, it gives you some options for cancelling a long-runninng process and capturing output on stdout/stderr that might be useful for communicating progress in a way that system2() just can't.

[paleolimbot]

Ok! My two cents are here: https://github.com/paleolimbot/qgisprocess

Feel free to rewrite, fork, modify, do whatever! I provided a skeleton but haven't really tested. Basic idea is

• Use processx::run(), because it responds to the user interrupt and allows streaming of stdout
• Have a as_qgis_argument(x, type) hook so that sf/raster/stars/terra objects can be passed as arguments (and a qgis_clean_argument() hook to remove the tempfiles that might have been written)
• Dockerfile setup for a dev environment + GitHub actions to test multiple platforms (only Ubuntu focal so far but I think there are mac and windows nightly builds as well)

Still don't know how to deal with the output values, because they're printed to a crazy format that probably aren't even parseable using regexes. If anybody has opened QGIS issues before, it would be really helpful to have a --json argument so that we don't have to do any guessing with respsect to what shows up in the output.

[jannes-m]

I have just tried your README example on Arch, and it is working perfectly! Excellent work! Pls let me know if I can be of any help!

[jannes-m]

I have tested successfully a raster example as well:

library("qgisprocess")
#> Using 'qgis_process' at 'qgis_process'.
#> Run `qgis_configure()` for details.
library("raster")
#> Loading required package: sp
data(dem, package = "RQGIS3")
projection(dem) = sf::st_crs(32717)$proj4string
pcurv = file.path(tempdir(), "pcurv.tif")
vcurv = file.path(tempdir(), "vcurv.tif")
info = qgis_run_algorithm(
  alg = "grass7:r.slope.aspect",
  elevation = dem, 
  pcurvature = pcurv,
  tcurvature = vcurv,
  .quiet = TRUE
  )
out = stack(c(pcurv, vcurv))
plot(out)

^{Created on 2020-10-08 by the reprex package (v0.3.0)}

Two things to note:

1. Depending on the chosen output, quite a few default values are chosen. This is not bad in itself, however, the user should know what he/she is doing, i.e., what the default values are (afaik the default value is always 0 if multiple options are available in a QGIS algorithm).
2. @Robinlovelace @Nowosad projection(dem) = sf::st_crs(32717)$proj4string is necessary, otw a WKT error is thrown (NO WKT AVAILABLE FOR PROJ >= 6). I suppose that we have to update the crs strings of all spData and spDataLarge objects that are using PROJ4 strings (see e.g., https://inbo.github.io/tutorials/tutorials/spatial_crs_coding/), which at least is most likely the case for all raster objects.

[paleolimbot]

Just a heads up that you should reinstall! (I just did some major improvements/bugfixes).

I'm not sure the default values are available with qgis_show_help()...if you can find where they are we might be able to add in a warning or something!

[jannes-m]

Ok, thanks for the heads-up! As I said, if multiple options are available, then the first option (coded as 0) is afaik always the default in QGIS. In RQGIS(3) we simply collected all arguments, and if more than one option was available and the user did not specify anything, we simply used the first option by setting it to 0. Additionally, we issued a message telling the user, which default options were chosen. Not sure, if the same approach can be used here, since it might be difficult to extract all the arguments and potential options from qgis_show_help().

[paleolimbot]

There's a bit of grepping involved with qgis_arguments() and qgis_outputs(), but I didn't get quite as far as the "Acceptable values" bit. It would probably work as an argument of as_qgis_argument() with a sentinel like NULL signifying a value that wasn't provided. I'm worried about being too magical by default, but enums seem straightfoward. There's probably some other argument types whose default values could be filled in (e.g., "sink" or "vectorDestination" with qgis_tmp_vector())...I made a list here: r-spatial/qgisprocess#13

Some reprex-like thoughts:

qgisprocess::qgis_arguments("native:simplifygeometries")
#> # A tibble: 4 x 3
#>   name      description           qgis_type
#>   <chr>     <chr>                 <chr>    
#> 1 INPUT     Input layer           source   
#> 2 METHOD    Simplification method enum     
#> 3 TOLERANCE Tolerance             distance 
#> 4 OUTPUT    Simplified            sink

qgisprocess::qgis_outputs("native:simplifygeometries")
#> # A tibble: 1 x 3
#>   name   description qgis_output_type
#>   <chr>  <chr>       <chr>           
#> 1 OUTPUT Simplified  outputVector

# not provided by user
qgisprocess::as_qgis_argument(rlang::zap(), "enum")
#> Error: Don't know how to convert object of type 'rlang_zap' to QGIS type 'enum'
# provided as number
qgisprocess::as_qgis_argument(1, "enum")
#> [1] "1"
# provided as string (should return the index of the value)
qgisprocess::as_qgis_argument("some valid value", "enum")
#> [1] "some valid value"

^{Created on 2020-10-08 by the reprex package (v0.3.0)}