Skip to content

PhotonVision Meeting Notes

Jump to bottom
Matt edited this page Jun 2, 2024 · 8 revisions

Welcome to the PhotonVision Meeting Notes Page!

All meetings are hosted on our discord and are planned in the #meetings channel. Feel free to drop in!

2024

June 2, 2024

Happy post-champs!

  • V4l sucks
    • V4l property set every time you drag the exposure slider slow, leads to delay and sometimes camera drivers can freak out
    • UI -- need to debounce somehow
  • Double-rounding and cscore + photon quirk handling also sucks
    • Drive switch to raw units on v4l properties in whole web UI plz
  • Arducam cameras all having same USB VID + PID -- quirk system needs reworking
    • Pop-up to select instead of hidden on cameras tab? Something more visible (see 1300)
  • Camera calibration UX
    • Large JSON files are hard to work with
    • Not clear that you can click on a resolution in the table to get more info?
    • Better metric for how many photos to capture instead of just a number? Do we even need a number, since people may just take exactly 15 and call it a day. Completeness instead of count?
  • Settings import/export
    • Split between camera calibrations, all the different kinds of settings files. Not super obvious how that works?
    • Document process for setting up hot spare devices?
  • Stream resolution probably shouldn't default to max resolution these days -- open up current CSI camera only logic
    • Do we also wanna expose quality in the UI?
  • Be louder about the bandwidth limit and how to work around it
  • Gstreamer camera streaming should be a thing please
  • (nit) the log viewer being fixed line height leads to some rendering silliness -- is there a new Vue component we can use instead?
  • Camera matching continues to suck and the impact when it doesn't work
    • Matching as a "new" camera is currently GG your settings are gone
    • Confusing why renaming a camera in the web UI != renaming the camera in arducam != camera's name
    • Importing/exporting the whole set of camera settings at once would help with recovery (we currently don't allow this, only all settings (network, camera, etc))
    • Method to change binding of physical camera <--> camera configuration
    • Can we get our hands on the arducam protocol to change USB parameters?
    • Maybe use serial number only if we can
    • Avoid accidentally creating new camera configurations wherever possible
    • Proposal: never create new camera configurations without user input, popup to take action (is this a new camera, or an existing camera that i need to copy configuration from?)
    • Can we also warn if cameras are identical by everything except for USB port? That's the one thing we can't detect right now.

2023

Dec 27, 2023

  • 10 days to kickoff!
  • Aruco-nano exposed as a toggle switch? And hook in adaptive thresh/other tuning things from normal Aruco
  • Download/upload camera calibration coefficients per-resolution would be sweet
  • Publish protobuf alongside Packets. Re: Peter - quickbuf is gone?? Replaced with python?? Look into... but quickbuf works fine for now. Duplicate java sources bug actually fixed!
  • Chris - status LEDs. More thought on how this works with multi-cam, maybe hardware per camera? Only useful for LL/Glowo/Snakeye. As is, one global set of status LEDs and results are ORed together. Can be aggressive with merging (just double check Jackson is happy with serde)
  • NetworkManager (i hate networking): Add user input before going and installing NetworkManager? "I'm gunna go install XYZ, if you don't say yes I won't be able to do Y". Also add disable-networking flag to systemd based on user input in install script? (And a send it flag for headless installs)
  • photon-image-modifier -- generates LL/Pi images from known good base. Pretty great. Need to update CI to pull those in instead. Also need to figure out auto-expanding FS
  • pose_calib_native -- Probably won't ship this season. RT13's got that standalone thing. Might be worth mainlining and shipping a standalone JAR and shoving in mrcal if we can't get this working.
  • Anand's OPi5 guide -- can we copy/paste this into our docs? ours is pretty bare-bones rn
  • Calibration visualization/improvements -- cut in config file changes to start saving calibration observations, even if we don't show the data right now.

May 29, 2023 at 6:00 PM PT

  • How do we wanna deal with multi-tag and tag filtering?
    • We can let people upload java code and JIT it but I'm sad
    • People tend to wanna filter tags different custom ways pre-combining for multi-tag
    • Declan: can probably offer a bunch of options for intra-frame optimization? Longer-term, inter-frame (history of observations)?
    • Factor graphs are another thing we can do
    • Probably just do the naive thing for now? Add a global filtering/localization tab
      • RANSAC (google says Random sample consensus?)
      • distance from camera
      • high ambiguity
      • constrain within field (would need robot pose to be piped in)
      • use gyro heading to help reject tags? (but then you need a gyro)
    • How should we let people upload field layouts and configure them?
      • Select tags that are used
      • One time field layout upload in the UI, with a view showing tag poses somehow
  • Gib h264
    • Gstreamer RTC -- no web
    • WebRTC -- No docs, usable in JavaScript. But we need this if we want browser UDP
    • Manged to get a webrtc stream working in Python to a browser
    • Maybe use a pure Rust/native webRTC implementation and wrap with a JNI
    • You have to do your own encoding and then pass it to WebRTC
    • h264 over websockets?
    • Better to use a real transport like WebRTC over a custom DIY websockets approach
    • LibAV can do some degree of cross-platform hardware acceleration, but it's a PITA to write code for
    • Libcamera seems to have a way to encode frames? https://github.com/raspberrypi/libcamera-apps/blob/a6267d51949d0602eedf60f3ddf8c6685f652812/apps/libcamera_raw.cpp#L70
    • We would probably really want hardware acceleration to run real-time h264 encoding
  • Sim refactor PR
    • 3044 was pretty happy
    • Maybe just merge and document the features?
    • Make it a hard transition from the old sim classes
    • Yank example from AprilTagWorkbench
  • Settings robustness
    • Should we just move to SQL?
    • Should we export all the files as a db, or a JSON?
    • We should export/import JSON ZIPs, and save as SQL DB locally so people still have plain-text
  • Camera streaming is super slow on Pi
    • High frame divisor to get good performance
    • Should we have a switch to totally disable streams per-camera in Photon? Just don't draw on input/output or anything, and streaming is a huge drain on Pi resources
  • We should do more profiling on new SBCs (like orange pi) to check on bottlenecks
  • Status LED/camera LEDs shouldn't be tied to a single Camera anymore
    • Dumb
    • We should have a GUI to map physical hardware to cameras
  • Frontend Vue refactor PR (we should merge it)
  • Proper robotpy support issues are open (836, ownership transfer into monorepo would be cool?)
  • GPU accel
    • unionfind in GPU!
    • someone should mess with orange pi v4l
  • Poke FIRST about usage reporting again
  • Libcamera shader: oversaturated colors are blown out and wrap around
  • Camera calibration rework
    • mrcal has a c API, technically
    • Desktop app wrapping mrcal python seems easy
    • Can find points using whatever you want (for us, that's openCV)
    • List of points is all mrcal needs, but it'll annotate snapshots
    • Can we wasm mrcal? (Banks: lots of dependencies, fortran, ewww, no; stick with prebuilt python wheel)
    • charuco board support?? for real this time??

2022

September 11th, 2022 at 8:00 PM EST

Attendees: Banks, BDaroz, Chris Gerth, Jeremiah, JN25, Mohammad, Tyler (WPILib)

  • Address camera exposure questions

    • So far most settings have been optimized for retroreflective, no longer the case
    • Proposal: create a "retroreflective" or "normal" setting in the UI, tweak parameters based on this
    • Retrorefletive = today. Normal = nearly defaults - let "auto everything" make the best human-visible picture possible. Use Normal for Aurco and Driver mode pipelines

  • Libcamera for 2023?

    • Is GPU accel necessary for the 2023 season / would people care

  • Turning back on AWB in order to detect AprilTags better

  • 3D Visualization bug

    • HTML Canvas bug
    • Clean up 3D display code with Javascript (find someone who wants to do this) to make it stop going off of the screen

  • SolvePNP is non deterministic

  • Parallel path and prioritize non GPU accel for LibCamera

  • April Tag Pose API

    • Need to flesh out the API from the camera detecting a certain tag to get your current Pose on the field
    • Working on it and make sure that kids learn something (not a blackbox that just returns robot pose)
    • Need to figure out pose flipping (don't know if FIRST will have all unique things on the field or just duplicates)
    • Either way can figure it out
    • Chris said he can work on the JS side of the webview, Matt can do the JNI, Mohammad will do PhotonLib API

  • NetworkTables bug

    • No one has been able to reproduce anything, wait for network and boot and recommend static IP everywhere

2021

November 23rd, 2021 at 7:00 PM EST

Attendees: Matt, Chris, Banks, Mohammad, Jason

  • PhotonVision CD Post
    • Kickoff release things / updates about what’s new
      • Mid december
      • Colored Shape and Pose2D stuff
    • Bump dev post?
  • Versioning Issue on 2021.1.8 (fixed)
    • Name of the release .jar is wrong
    • UI shows 2021.1.7 when running 2021.1.8 on PC
    • CI Broken on macos
  • 2022 Season Goals
    • 3D Model for 2022 goal soon after kickoff
      • Similar to how we have a 2020 goal
    • Writing a 2022 game example code for whatever it requires (if its different enough from what we have already)
    • Getting testing / more user feedback
    • Relicensing
      • Most things in the code base can be relicensed sans the UI
  • Gen. PhotonVision
    • Add support for uploading new .jar
    • Add persistent gain settings
    • Dependabot PRs
      • WPI Format failing still
    • Server Mode / NT Things
      • Merge connected notification icon PR (Matt)
      • Make it more clear in documentation what Server Mode is and how all network tables things should work (mohammad)
  • Romi (Mohammad)
    • Can test before kickoff on romi hardware
    • Note FS must be set R/W before installing
    • Note FS must be set R/W for settings changes to persist
    • Notes about logging, settings, etc. needing to be in RW for those to save
  • Documentation
    • Document colored shape pipelines (Jason?)
    • Finish and merge panels PR (Mohammad)
    • Document vendor pins and overall things like that (would need to be someone who knows all this stuff)
  • Branding
    • Update colors (lighter shade of black is used in reality)
  • Post Season Goals
    • AruCo Tag Support
    • Champs vision talk?

August 25th, 2021 at 9:00 PM EDT

Attendees: Matt, Mohammad, Krish, Banks, Declan, Chris, Jason

2020

August 7th, 2020 at 3:00 PM EDT

Attendees: Matt, Banks, Kiwi, Prateek, Declan

  • The-list, Matt edition (camera calibration and a slew of misc bugs)
  • Release naming
    • Is this first release gonna be like a "beta" release - Prateek
  • Installation bugs
    • First priority is making this work on our pi image (this seems to be done.)
    • Next step is smoothing over some issues that arise from users running the script manually on supported distros.
    • Next step is maybe making things work on less-supported distros??
    • Also uninstall script.
  • Feature freeze next steps
    • We don’t want Gloworm beta testers (who aren’t us) to be doing “firsts”--this is a huge support burden and doesn’t look good for us.
    • We need to focus on the pi image, networking config, and the vendor JSON (i.e. we must be feature-complete.)
    • Docs need to be ok.
    • Beta testing:
      • Anyone who got a beta Gloworm.
      • People who fill out an application off of CD and have a “Fully supported” platform.
  • Support guarantees
    • Fully supported (we’ll help you Discord, all features work, we’ll keep these up to date)
      • Gloworm
      • Pi 3 and Pi 4 with the official Pi image and the Pi Cam, ps3eye, and the Microsoft LifeCam 3000
    • Compatible (no guarantee of support on Discord, major features work, we hope to keep things up to date)
      • Linux (aarch64, armv7, x86_64)
      • Windows (x86_64)
    • Unsupported (we will tell you to use something else, we don’t try to make these work or keep them up to date, no guarantees)

July 31st, 2020 at 3:00 PM EDT

  • Merged Hardware Manager PR (finally)
  • Merged The List (UI tweaks)

July 24th, 2020 at 3:00 PM EDT

  • Roadmap progress standup
    • Matt
      • Played with JNI and PiCam stuff to interface directly with mmal and OpenMax
      • Reworked camera connection in vision source manager to maybe work better
    • Banks
      • Pi-Gen now building and CI automated
    • Declan
      • UI things
  • PRs merged this week
    • 44 - MutatingPipe (Banks)
    • 45 - dev tag cleanup (Matt)
    • 46 - Date-based version (Matt)
    • 47 - WebSocket user list fix (Matt)
    • 48 - drawing color fix (Matt)
    • 49 - Test config folder fix (Matt)
    • 50 - Pipe profiling (Banks)
    • 52 - Better NT byte packing (Prateek)
    • 53 & 56 - Task Scheduling (Banks)
    • 54 - V4L device blacklisting (Banks)
    • 55 - Logging additions/cleanups (Banks)
  • In Progress PRs
    • 40 - Camera auto-connect (Matt)
    • 51 - Save buffering (Matt)
    • 12 - Hardware stuff (Neel)
  • Trello Triage
    • Few high priority backend stuff left

July 17th, 2020 at 3:00 PM EDT

Attendees: Banks, Declan, Kiwi, Matt, Mohammad, Prateek, Vatan, Xzibit

  • Roadmap progress standup
    • Prateek
      • Docs for PhotonLib
      • Add skew measurement to PhotonLib
      • Automatic pushing of Dev tag with latest dev release
    • Declan
      • Finished first pass at UI redesign
      • Still has a long list of UI things to fix:
        • Change orientation dropdown to buttons?
        • Highlight selected stream/processing mode icon in yellow
        • Maybe show two columns in "beginner mode"?
        • Make the stream display/processing mode buttons less hard to use.
        • Remove dividers in the 3D tab (they look weird).
        • Change driver mode to a little switch at the top right hand corner of the camera card.
        • Do settings page.
        • Dropdown reflow.
        • Make the camera rename button more consistent.
        • Make sliders change on mouse down instead of mouse up?
        • Fix FPS indicator positioning; also use a drop shadow instead of a stroke.
      • Very close to being done with H264 encoding and decoding. Need to make it so that we can request the H264 video mode from the picam. Then we’ll be able to use something like gstreamer to actually stream video (sorry, RTSP in WPILib isn’t getting done this summer.)
    • Kiwi
      • Website.
    • Matt
      • At least one reflective pipe will always exist
      • Both input and output streams are published
    • Vatan
      • Offline docs merged.
      • Will work on crosshair size change on according to resolution
      • Will try to document same javadocs in photonvision (if time permits)
  • PRs merged this week
    • UI redesign
    • Latest commits pushed to dev tag
    • NetworkTables works now
    • Less/No lag dragging sliders
    • FPS calculation is camera FPS instead of 1/processing time
  • Trello Triage
    • New High Priority tag
    • Remove low-priority items?

July 10th, 2020 at 3:00 PM EDT

Attendees:Banks, CTT, Declan, Knufire, Mohammad, Prateek, Roboteer5291, Neel, Vatan

  • Roadmap progress standup
    • Docs in UI
      • Buttons that allow users to navigate to a certain part of the docs
      • Where to have the docs specifically located, embed, new tab, etc.
    • UI redesign
      • Usability testing now or later?
      • https://demo.photonvision.org
        • Community feedback received, deciding to move forward with this
      • Merge after PRs 10-11
    • Bootup-sprint merged
      • Puts Photon in mostly functional state, connecting frontend and backend again
      • Changed data output system, accessed through vendor dep
      • Add different types of pipelines, ex. ColorShape for powercell tracking
    • Prateek: Vendor dependency progress
      • Tested Java and C++ interop
      • Added automatic unit tests
      • Needs more info from Matt to complete about driver mode etc.
      • Needs docs
    • Prateek: link to latest build
    • Add more unit testing, low code coverage stats
    • TODO stuff & check Trello
    • Beta testing on robots before full release
      • Testing checklist
      • Trying to break the software
    • Outstanding PRs
      • Hardware (#12)
        • Needs testing on various SBCs
      • ColoredShape (#11)
      • Camera calibration (#10)
    • Vendor integration
      • Begin work on vendor .json for their customization on their specific image
        • Branding, specific hardware integration (ex. GPIO pins, brightness controls,
        • Talk to franklin about specifics of what he has in mind
        • Metrics reporting
        • Find a better name
    • Gloworm CD release
      • Release timeline/feature freeze change. Need to ping Franklin when this starts.
        • Focus on bug fixing, getting stability/reliability, documentation writing etc.
        • August 3rd, 2020
    • Documentation

July 3rd, 2020 at 3:00 PM EDT

Attendees: Banks, Matt, Mohammad, Prateek, Declan, Ben + 7 others

  • Introduce project leads and contribution structure
    • Banks and Matt say hello
    • Currently we have 9 contributors; this role pretty much means that you’ve contributed code or docs.
    • One of our main goals is to increase the number of contributors from what we had with Chameleon and create a more open, accessible project for them.
  • Discuss backend/frontend framework, how things work, how to use framework
  • Roadmap progress standup
    • Banks
    • Matt
      • Frontend refactor to more closely match structure of ChameleonConfiguration is mostly working (core features send data to backend, 3d doesn’t yet work)
      • Photon can support exposure/brightness/resolution/orientation changes, HSV/erode/dilate, contour sorting and filtering, changing stream resolution, saving program state
      • Not yet done: Stability between frontend/backend comms, add/remove/rename pipelines from UI
    • Declan
      • Quick UI rebrand.
      • Continued work on H264; this is probably going to be done in the next week or two.
      • GPU-acceleration code is working but hasn’t been PRed yet (currently on hold until H264 gets done.)
    • Neel
      • Made waffles (w/ choco chips)
      • Finished calibration pipeline
      • Finished GPIO/Metrics backend
      • Added colouredshape pipeline benchmarks
      • Finished Metrics frontend (still need to plug it in with backend)
      • Gonna work on frontend for calibration
    • Mohammad
      • Finished branding standards
      • Published branding repo
      • Working on documentation outline/what we want to see
        • Need full docs before first major release
      • Took meeting notes
      • Website brainstorming
    • Claudius
      • Docs CSS
      • Code coverage
    • Prateek
      • Photonlib-c++ port
      • Automatic builds for photonlib on all platforms
      • Automatic builds for main photonvision jar
      • Began writing tests for photonlib
      • Github Actions for everything
    • Matt (2)
      • Made the logo and a lot of branding materials
      • Helped with the UI rebrand
    • Etc
  • Assign meeting attendees to non low-priority items on the Trello
    • Vatan
      • Offline-accessible docs packaged with the JAR
      • Javadocs (already done in my fork in gradle)
      • Build instructions in docs
    • roboteer5291
      • Filter out not useful video modes (i.e. Grey)
        • Did Banks do this?
      • I can try unit tests if nobody else is able, but I’ve never unit tested before
  • Hardware
    • Gloworm - Frank
      • Pi compute module based solution
      • Similar specs to Limelight
        • 400 lumen LEDs etc.
        • Much cheaper!
      • Prototype phase
      • Available for purchase in 1-2 months
      • Open source, licensed, etc.
      • Targeting Limelight audience
    • ??? - Copperforge (Ben)
      • Selling the Gloworm (see above)
      • Targeting the teams who want to DIY their own solution but don’t have the EE experience or time
  • Discuss more about how we want to organize documentation
    • Decide what we want to take from Chameleon Vision / LL Docs (info they included, stuff like that)
    • How we want to organize them, new content that we need, etc.
    • Full rewrite
      • Getting started with a .jar file
        • Should be pretty seamless
        • Video example?
      • Using the image if/when we get it
      • Basic pipeline tuning
      • Contributing guidelines/setup similar to frc-docs
      • Photonlib docs should exist
    • Similar structure to frc-docs, same style guide, etc.
    • Code examples similar to LL
  • Create image for PhotonVision for use on hardware

June 29, 2020 at 3:00 PM EDT

Attendees: Banks, Declan, Matt, other Matt, Prateek

  • Impromptu branding and planning meeting
  • New communication standard for websockets/HTTP
  • UI rewrite?
    • Need to redo communication anyways…
    • Is it really appropriate? Too much to byte off?
    • Maybe instead of restarting, rework the current framework into something more usable?
      • Easier to test?
  • Balsamiq sketch https://balsamiq.cloud/s6nqda8/pobr8jx/r6DEE
    • Bootstrap, almost complete rewrite
  • Vue-bootstrap/bootstrap-vue, can replicate bootstrap UI with more vue code
  • Bootstrap means manipulating HTML to react to changes
    • Currently more popular would be a reactive framework (React, Vue, Vueify)
  • Vuetify might be more popular? *
  • Vue means it’s easy to manipulate the UI through JS
  • Bootstrap code would need to be implemented in Vue
  • Styling code should be cleaned up
    • Set primary/secondary colors globally instead of per-element
  • Who’s writing it?
  • Options for going forwards:
    • Rewrite the communication part of the frontend and use the current UI for testing; redo the UI later.
    • Redo the UI now and rewrite the communication part as we go.
  • Will be making current backend work with current frontend

June 27th, 2020 at 5:00 PM EDT

Attendees: Banks, Declan, Matt, Neel, Mohammad, Claudius, Prateek

  • Updates
  • Timeline/Roadway/Progress
    • Initial Setup
      • Domain
      • Logo coming soon
      • Chief Delphi Post
        • PhotonVision CD account?
        • Consolidated communication from 1-2 people
        • Google form to get contributors
      • Trello
    • Programming
      • Try to get a Pi Image and/or a jar with the 3.0 branch
        • Does not need to be working perfectly, just to get momentum
      • Fix RTD building issues
        • Brand docs once logo and branding standards are created
      • Find front end developers (vue.js)
        • Probably on CD or Matt
      • Work through chameleon issue list
        • Keep one pipeline at a time (Matt)
      • Create repo for branding/logos
      • Fork 3.0 of chameleon code into our github - Done
        • Change package name, folder names
      • Develop versioning system
        • Ex. 2020.6.1-b28d0e0
      • Release schedule?
        • Kickoff, offseason, mid season, etc.?
        • Declan advocates for doing a feature freeze by the end of summer (e.g. August 25.)
      • Add readme as needed
        • Keep authors (maybe original Chameleon Vision authors?)
        • Keep acknowledgements
        • Keep license
      • Meeting structure
        • Weekly during feature development period (schedule during feature freeze TBD)
        • Agenda:
          • Standup where everyone does a quick summary on their progress for their roadmap items.
            • Keep Trello up to date with this standup.
          • People can add other items to the meeting notes.
    • Hardware
      • Teams may be more willing to just get a box, image it, and go, kind of like LL
        • Would need further support
Clone this wiki locally