Update to storage-manager release #343
Conversation
luzzu-lab
commented
Jul 31, 2018
luzzu-lab
commented
- Fixes stack overflow issue
- Adds optional limits to queries and downloads
- Adds optional limit to the download API as query param
- Adds keep-alive mechanism for livy client
- Reinforces naive validations for query APIs + tests
- Fixes API documentation
| case (None, None) => dataframe | ||
| case (None, Some(value)) => dataframe.limit { value } | ||
| case (Some(value), None) => dataframe.limit { value } | ||
| case (Some(value), Some(default)) => dataframe.limit { math.min(value, default) } |
There was a problem hiding this comment.
Why take the min here? Wouldn't be a little misleading for the user to get a lesser number of rows respect to the parameter he submitted (if the defaultLimit is minor than limit)?
There was a problem hiding this comment.
The min is a defensive measure in that the optional default limit set in the configuration is used to set the absolute maximum number of rows that the quick APIs are allowed to return.
The problem is that it will not be clear in certain situations whether the number of rows returned is actually legit, or just the server’s maximum but i don’t think that there is another way to set a bound, and there is also no clear way to relay this information to the user.
Note that there’s always a check on the size of the data before we do the quick methods
so in theory this is a optional last stand on the server part so to say.
