apply URL encoding in RestClient

[PhilipCubix]

Since the current method of URL encoding is to use a User defined Java Expression (see #5201), I would propose to apply URL encoding in the RestCLient component.

This avoids the need to add an extra step just to do URL encoding, that currently forces a java expression to be written, and run on the janino run-time compiler.

It also prevents users from having to find out they need to do this themselves by running into an "Illegal character in path" error, see https://pentaho.home.blog/2019/05/18/create-url-for-rest-query/ .

[hansva]

This might cause double encoding for people already encoding strings, won't it?

[PhilipCubix]

hm, URL encoding of a path seems to be trickier than first thought:

• encoding the entire url without exclusion of the question mark before query parameters leads to trouble
• space can be encoded as '+' or '%20' and it can be required to use '%20' while java encodes with '+' by default

[PhilipCubix]

This might cause double encoding for people already encoding strings, won't it?

I've looked into it and '%' would be encoded as '%25' again, so that would indeed cause trouble for people already encoding urls

[PhilipCubix]

So FYI, I was able to fix my problem using the following User defined Java expression:
java.net.URLEncoder.encode(path, "UTF-8").replaceFirst("%3F", "?").replace("+", "%20")

What would need to be done on the Rest Client is to:

1. check if the url is valid, and does not contain 'illegal characters',
2. only in case the url contains illegal characters, apply url encoding,
3. add an option for encoding spaces as '%20' instead of '+',
4. take care of the question mark

So my pull request is not cutting this... :-(

[hansva]

I don't have a test setup, but is the error you are getting with a malformed URL helpful?
I see we could also add UriBuilder.fromUri(data.realUrl), which should return a helpful error message if it is a malformed URL.

The only real solution would be to not add Query parameters on the URL, but use Parameters tab which will handle the encoding properly

[hansva]

So I think you are trying to solve a non-issue. Either encode the parameters properly for the URL or use the parameters tab, which will handle it for you.

[PhilipCubix]

Hi again,

ok, yes, using parameters is the solution!

thanks!!!

[PhilipCubix]

Hi,

sorry, too soon: parameters are not used on GET requests .... :-(
why are they not supported on GET requests????

[hansva]

Get should support it as far as I can see

[hansva]

I did a quick check in our Integration testing framework. The test used post initially but quickly changed it to a get and the parameter is being added correctly.

[PhilipCubix]

well that should be fixed in documentation then :-)

[hansva]

good catch #5849