-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add: flux radiuss tutorial 2024 #31
Conversation
I am starting preliminary changes for RADIUSS 2024. Builds explicitly cancelled to not waste resources. I'll include some notes, and screenshots of work so far. For the container, I've updated the base and changed a few of the orderings of commands to be more logical (COPY directives should come later, for example). For the notebook, the launcher is fixed to load the tutorials via buttons. This means it opens and all content is front and center, with working buttons for Tutorials and resources (at the bottom). The launcher took a hot minute to get working again (it broke between this and last year). Also note that the nesting is removed - I think it made the tutorial more complex to require the user to navigate a few directories deep to find the content. Now everything is there for the tutorial user when the notebook opens, no need to navigate into many sub- directories. I am still hoping to do the following:
As stated above, I am moving "core" flux tutorial content separate from what we can consider external or plugins, like dyad. The reason is because these are experimental and change from year to year, and having it alongside core content makes a promise about consistency that I am not sure we can keep. This means we have three chapters, and then supplementary material (which can be extended, removed, added, and doesn't make a promise about consistently). For example, DLIO doesn't even have that name anymore. I̶ h̶a̶v̶e̶ n̶o̶t̶ d̶o̶n̶e̶ t̶h̶i̶s̶ y̶e̶t̶I (update: this is done) I (have) separated the files (images and notebooks) for dyad cleanly as well - they were all mixed together and this would make it challenging from an organizational standpoint to add / remove / re-organize entire supplementary sections (chapters). Now when you click the Dyad tutorial button, it's cleanly it's own thing (I like this better): For the terminal, we've also been requiring the user to navigate to File -> Open -> Terminal to do that. That's really hard and annoying. I figured out how to make a button instead: This opens a terminal immediately. Finally, I have started marginal content changes. This will be a multi-step process. I also expect the DLIO/DYAD content to be entirely different, and we will need a round to work on that. The build takes a long time, and I remember getting it working was challenging. I'll engage this team when I'm done re-organizing. There is also some bug with flux-tree and awk that needs debugging. Cheers. |
This adds the new directory for the Flux Riken tutorial, with the following additions: 1. flux-tree was removed from flux-sched and is added here 2. tutorial files were kept in rse-ops, are now moved here 3. New tutorial content: flux tree and hierarchy section/examples 4. New tutorial content: flux archive (previously flux filemap) 5. images that show a dummy example of job throughout 6. update of names in login page / directory to be more general 7. automated builds updated for riken Signed-off-by: vsoch <vsoch@users.noreply.github.com>
I am starting preliminary changes for RADIUSS 2024. So far: - the launcher is fixed to load the tutorials via buttons - I am moving "core" flux tutorial content separate from what we can consider external or plugins, like dyad. The reason is because these are experimental and change from year to year, and having it alongside core content makes a promise about consistency that I am not sure we can keep. I have not done this yet, but I am going to separate the files (images and notebooks) cleanly as well. - top level: everything is there for the tutorial user when the notebook opens, no need to navigate into many sub- directories. - terminal button: instead of "click this long list of annoying paths to open a terminal" I figured out how to make a button in the notebook directly. - marginal content changes: I am starting to tweak / update content, this will be a multi-step process. Signed-off-by: vsoch <vsoch@users.noreply.github.com>
…tary section Signed-off-by: vsoch <vsoch@users.noreply.github.com>
The notebook structure should primarily be organized by command, since this is what the new user will interact with. This change set better does that, and flattens / organizes things a bit better overall. Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Improvement on sections and table for flux commands, and addition of flux accounting to container (likely will not easily work). Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Hello! I am writing to provide some feedback for this version of the tutorial. Please note that I have just learned how to use both Flux and Slurm in the last three weeks, so I am much less experienced with the subject than your target audience. Overall ImpressionThis tutorial is easy to set up, informative, well-organized, and useful. I love the feel of running through the cells and trying things in the terminal. I learned a lot and I got a good sense of why Flux is useful. All of the visuals are helpful and cohesive, as is the style. This is something I would reference for flux help and related links. Finally, it was a little intimidating in its size/scope, especially within Chapter 1. LogisticsThere were some things that were broken or out of place: Chapter 1
Chapter 2:
Chapter 3:
DYAD/DLIO:
SuggestionsI thought some things could change: Chapter 1
Assorted Opinions
All of that being said, I think this tutorial will already be very successful in its current state given its wealth of information, fun delivery, and engaging structure. It's clear to me that a ton of effort went into writing and updating this, and I am just hoping that some of these comments are helpful in the development process. Jacob Tkeio (via @ trws and @ milroy) |
@jacobtkeio this is the most extensive feedback I've ever received! I will go through this carefully and address the points - it might take me some of the weekend, but I'll post an update here. Thank you so much for doing this! Your perspective as a new user is immensely valuable. |
Okay I've turned your feedback into a list of TODO, and am going to comment (and update) here I fix or change things. I will ping once more in a new comment when I'm done, and I'll have a new branch (to PR on top of this) to show changes (and a new container to quickly test). Completed
Added! And with a cute example too 🐻❄️
oh man, so this just got really good 😆
Awesome! 😎
I love the colors too - added them newly this year 🟢 🟣
Nice!
This is good feedback - I actually intentionally put into one notebook because of people's attention - I assumed they would have attention for one notebook and then drop off. That said, I see your point about length, and that we have sort of created internal chapters. I think for this year (since there are a lot of changes, and this is the first time trying this change set) we should try to do it with one notebook, and then perhaps ask for feedback about it. We can definitely split the notebook up if people seem to agree, which they definitely might. Ping @milroy - let's make sure we discuss some kind of feedback form (with a specific set of questions for this).
That's fantastic! We had a lot of work to do with the automation bit last year, and this year I wanted to focus a lot more energy on the tutorial. I am going to build you a new container and I'll have a new testing container to post here (and in slack soon). Still TODO
Thank you again for this feedback @jacobtkeio I have been at this a long time and have never recevied such a solid set of very good suggestions and fixed. |
@ilumsden would u be looking into this? If not let me know. |
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
feedback: jacob review items
@hariharan-devarajan I may be able to work on this a little, but not a ton. I learned today that I've got to prepare a substantial chunk of the results for a paper due in 2 weeks. Plus, I have to prepare my 2 page extended abstract for SC. And I've got to start working on end of internship stuff since I (somehow) only have 2 weeks left. Any help you can provide on this would be appreciated. |
@jacobtkeio the updates are done, the testing should be the same as before: docker pull vanessa/flux-tutorial:radiuss-aws-2024
docker network create jupyterhub
docker run --rm -it --entrypoint...-v /var/run/docker.sock:/var/run/docker.sock --net jupyterhub --name jupyterhub -p 8888:8888 vanessa/flux-tutorial:radiuss-aws-2024 And you can see full edits here: Have a good weekend! |
The main road block I have is the testing environment. I remember the notebook had some differences between AWS and my local. I don't have access to AWS yet and so I may not be able to address and test the changes. |
@hariharan-devarajan you can run locally with Docker. There should be instructions in the README in the folder for this version of the tutorial |
@vsoch what is the deadline for this. Maybe I can prioritize this over other things depending on the urgency. I would love to get all comments addressed so that we can keep this integrated. |
You won't need AWS! Just docker locally. It works really well now, no changing directory once you open the notebook! And @hariharan-devarajan I've posted the instructions above, and in slack, and to control +S from the container and save locally just do: docker run --rm -it --entrypoint /start.sh -v $PWD/tutorial:/home/jovyan -v /var/run/docker.sock:/var/run/docker.sock --net jupyterhub --name jupyterhub -p 8888:8888 flux-tutorial That particular instruction in the README here needs to be fixed (and I'll update it next time around, the volume path is off so it won't persist when you save from the container to the host). |
The deadline for changes is Friday, August 23rd, and the tutorial is the following link. Here is the slack post with all the details. |
Oh, is that the deadline? I can definitely help out then. I just won't be able to do a ton before I'm back in Knoxville on August 11. But, once I'm back, I could easily put this at the top of my to-do list |
Yes, there is absolutely time! I don't plan a lot of deadlines (I mostly just do things when it feels right) but when it's needed, I like to be early so I can myself feel relaxed, expect the unexpected, and (in the case of wanting feedback from many people) give them ample weeks to do that. Have a good trip to Knoxville! |
@vsoch, thank you! :) I have never received such a thoughtful response to my feedback, and I really appreciate it. I've ran through the updated tutorial and I love the all the new examples and explanations. Pizzaquack is my favorite <3 and the #flux start section is excellent. The explanation is stellar and I think it works perfectly where you put it.
seems to fit quite well as a replacement for the "!flux submit -N1 -n2 sleep inf" command. And, pizzaquack seems like a perfect ending for #flux bulksubmit. What do you think about this? I will add lastly that I was confused about what "the middle example" was and that I noticed that, in it's current state, the tutorial says that two example in a row are fun ones.
Again, thank you so much for the swift and thorough response. I hope you had a good weekend! |
conduit still does not compile, and a note was added about that. Signed-off-by: vsoch <vsoch@users.noreply.github.com>
@hariharan-devarajan and @ilumsden - the main tutorial and workflows are done so I'm going to merge into main, the main reason so we can build the images and start preparing the last two bullets. This should also make it easier (and more clean) for you to PR to the main branch, so let's pick up there (when @ilumsden is back from his trip). Have a good weekend both! |
This preserves the state of the original changes for flux RADIUSS 2024, mostly to calm my own anxiety.
This adds the new directory for the Flux Riken tutorial, with the following additions: