2015 12 17

• Agenda:
  • Development Status:
    • AGENDA: EC2 support
      • LSU use case:
        • launch VM/Cluster on EC2 via RP
        • cluster: set of VMs, SGE over it
        • cluster as virtual grouping
        • TODO AM/MT: AWS has cluster startup capabilities?
        • tracking of data for all users on all machines
        • GGI
        • 1: job->VM, 2: spark/yarn cluster startup via RP, 3: RP create/use VM
        • 1, 3: 3 can be hacked
        • TODO AM: get some hack done for 3
    • TODO MS, MT: sync of OSG Data Staging capabilities, hacks
      • needed: bulk submission, off-band staging -> depends on new client arch?
      • short term: no data related work
      • osgng branch is now merged into devel
      • 64 possible, want 2048
      • only OSG-Connect, but lets complete that step first
      • TODO AM: sync on testing of OSG/SAGA with MT
  • YARN:
    • TODO AGENDA: re-discuss
      • shared FS (needs specific staging component)
      • data locality
      • fault tolerance
  • SPARK:
    • communication with slaves works
    • no spark scheduler - use RP agent scheduler, executor (spark submit)
    • TODO AM: LRMS on which host is it running
  • Visits:
    • debriefing:
      • focus on BW, stack install
      • testing worked ok
      • TODO VB: follow up
  • Tickets:
    • VEs can remain broken (#911/#932)
      • multiple pilots break, this should be part of the release testing...
    • GPU experiments
    • DONE AM: follow up on Philip's mail (EC2)
  • AOB:
    • XSEDE allocation got approved!
    • TODO AM: give workload to Matteo
    • TODO AGENDA: RTD procedures
    • TODO AGENDA: where go user credentials? context vs. user pilot description.
    • next call: Tue Dec.21, noon EST