Document the command line app. #51
Conversation
- Resolves #14 - Documents archivesunleashed/aut#431
|
Link to review new documentation: https://github.com/archivesunleashed/aut-docs/blob/issue-14/current/aut-spark-submit-app.md |
There was a problem hiding this comment.
@ruebot, this is looking really good! I like the flow you've given to all the documentation. I have a few surface level suggestions. My only major comment is related to how to start off with the scripts - hopefully I'm not overthinking things.
Look forward to working with this more! :)
| @@ -0,0 +1,128 @@ | |||
| # Using the Toolkit with spark-submit | |||
|
|
|||
There was a problem hiding this comment.
Do these configuration options need to be used with a specific launch of the toolkit (e.g. package, uberjar, etc.)? At first glance, I guess I'm a little unsure of where to start or in terms of workflow, when this script would be introduced (e.g. use within or outside of sparkshell?)
There was a problem hiding this comment.
Adding a link to this https://spark.apache.org/docs/latest/submitting-applications.html
current/aut-spark-submit-app.md
Outdated
|
|
||
| The extration jobs have a basic outline of: | ||
|
|
||
| `spark-submit --class io.archivesunleashed.app.CommandLinAppRunner PATH_TO_AUT_JAR --extractor EXTRACTOR --input INPUT DIRECTORY --output OUTPUT DIRECTORY` |
There was a problem hiding this comment.
is it possible to put in an example of how this script would look if one of us were to run it, below the basic outline? I find that when I have an example it's a bit easier to see what needs to be changed in line.
There was a problem hiding this comment.
That's what all the examples are below.
There was a problem hiding this comment.
Oh yeah, I realize the examples further down are more detailed - I just was going down a different line of thinking, so disregard my original comment
@lintool @ianmilligan1 @SamFritz here is a first crack at documenting the command line app. Please let me know how this approach works.