RFC: Liquid Tracking in Robot Software

[b-cooper]

Overview

This is an RFC on our approach to introducing basic well liquid/volume tracking to the Python protocol api.

The Python protocol api currently only tracks volume as far as the pipette, so the hardware controller knows how much to move the pipette's plunger for any given atomic pipetting action.

The recently introduced Thermocycler Module needs to know the volumes of the samples in its loaded labware in order to precisely heat and cool its wells. Now is the right time to consider how to track the position and volume of liquids on the deck of the OT2 at runtime.

Requirements

A successful first implementation should address the following goals:

• track the total liquid volume in every well on the deck through a protocol run
• produce identical results to the Protocol Designer's underlying step-generation liquid tracking
• provide opt-in functionality for PAPIv2 users that gracefully degrades (e.g. if initial well volume not supplied)

The implementation should also consider the following longer-term feature additions:

• tracking liquid identity, metadata, and volume ratios per well (ala Protocol Designer)
• tracking liquid properties, such as viscosity, evaporation, cohesion, volatility (not currently in Protocol Designer)
• Run time checks, potentially associated with other sensors or feedback

Proposal

This RFC proposes that we implement basic well-level volume tracking in the protocol api.

In opentrons/protocol-designer/step-generation (currently only used by Protocol Designer), lives all of the logic to track the state of liquids on the deck of the OT2 across protocol commands. It is important that whatever Python implementation we use not functionally diverge from the Javascript implementation.
We can enforce this relationship in a number of ways:

1. Create a method-to-method carbon-copy of step-generation in Python. Write robust tests that consume the same artifacts and enforce identical outputs.

• Pros:
  • changes to either implementation would be straight-forward to link to necessary changes in the other
  • writing shared regression/unit tests would be straight-forward, as they too could match one to one.
• Cons:
  • cost of maintaining two different modules in two different languages that perform the exact same task
  • could add development friction to updating either module, as every change would require 2x implementation resources

2. Create a Python module that accomplishes the same goals in a style that is closer to the rest of our Python logic. Write robust tests that consume the same artifacts and enforce identical outputs.

• Pros:
  • each module can maintain it's own coding style and approach that makes the most sense in it's language and context
  • regression/unit tests could still consume the same artifacts
  • less tight coupling across the stack (compared with option 1)
• Cons:
  • cost of maintaining two different modules in two different languages that perform the exact same task
  • could add development friction to updating either module, as every change would require 2x implementation resources

3. From the existing Python code, execute the step-generation logic in a Javascript interpretter, probably in a different thread running Node.

• Pros:
  • truly one source of truth for liquid state simulation
• Cons:
  • messiness of depending on multiple programming languages running the robot
  • because PD and the Robot Software would both rely on the same code, we'd either have to split it out into it's own versioned dependency, or deal with the consequences of possible discrepancies between platform releases (which undermines the purpose of this whole approach)

Given these three rough approaches, I think Option 2 is the best fit for our needs at the moment and will serve us going forward.

Option 3 increases the testing area of all our robot software, and has some serious implications surrounding tight coupling and implicit co-versioning across, PD, Run App, and robot software. Option 1 on the other-hand is less tightly coupled, but still has a hazy contract of implementation details. Our JS and Python have enough conventional differences, that I suspect we would be fighting from both sides to compromise whenever possible.

Option 2 is only as strong as the test suite that ensures parity between the two implementations. This feels like a fine place to assert sameness, as each implementation may develop as its domain requirements grow. The tests can serve as the contract that protects against cross-stack breaking changes. This brings up another relevant point. How does this effect our various versioning mechanisms?

As the tests in Option 2 only assert behavioral uniformity, not necessarily structural similarity, each project should be able to append functionality without "breaking" that contract. Editing existing functionality is a little bit tougher. For instance: if the JS step-generation were to be updated in a way that effected multichannel access rules, we'd want to make sure that the Python implementation would mirror that update. We're then faced with the possibility of people creating protocols in PD that simulate liquid state in the UI (via step-generation) differently than the robot software's run/simulation time representation of liquid state.

Without adding a new version for the step-generation module to the mix, we may be able to use PD's existing versioning with some new UI in the Run App. In PD, a Major or Minor bump constitutes a change that may be breaking (currently at the PD import file migration level). We could establish the convention that a change to step-generation that gets included into a release of PD always requires at least a Minor version bump. On the Run App side, we can surface an error (or warning), at simulate time, to express that the protocol will still function properly, but may differ from your expectations based on the PD's rendering of the protocol.

Discussion/Future Work

At its smallest scope this python implementation of liquid tracking would just have to cover aspirate, dispense, blowout, and potentially droptip. This approach would be sufficient in many cases to supply the thermocycler with accurate volume information at runtime. Those cases that it wouldn't cover, (e.g. initial liquid volume in TC wells > 0), are common and not adequately addressed with the minimal solution.

Because of this, I propose that our first implementation aim for basic parity with Protocol Designer's underlying step-generation liquid tracking. This similarly only requires coverage of aspirate, dispense, blowout, and droptip. The main difference is that instead of just tracking gross volume in wells, we also introduce the concept of liquid identity. This is the mechanism that allows PD to track mixtures/locations of reagents across a protocol. This sort of information paired with liquid metadata would allow for a rich runtime experience that is much more adaptable and extendable for features that are down the road.

This approach gives us much more leverage for creating connective tissue between our two protocol creating api's. For example, using a PD-esque UI to populate your starting deck state, and exporting that to be referenced in a Python protocol. This would save us from strict reliance on the inevitable suite of utility functions required for an average Python user to populate their deck and its labware with reagents at the beginning of every protocol.

The opposite direction could also be fruitful by allowing the robot software to send live updates of the current liquid state at runtime to clients in a consistent format to the way it is described under the hood in PD.

┆Issue is synchronized with this Wrike Task by Unito

[sfoster1]

Option 2 does seem like the best one to me. In addition to your notes about the options, it doesn't seem impossible to me that the two liquid tracking components might at some point want to diverge - the goal of the python API in the long term is to enable arbitrary complex protocols with integration for external hardware and so on, and it might end up requiring different liquid handling. Keeping the two modules separate therefore means you don't have to have a bunch of switching logic in the shared module that would rapidly end up with option 2 anyway. And once these are two separate modules, we should always have consistency within a stack.

I agree with the rest of your points here, but I'd like to bring up something else: this could be a solid impetus to execute planned refactors around making the python protocol API and the json protocol executors true siblings. Specifically, factoring the control code and deck coordinate handling into an executor layer that is shared by the protocol API and json protocols, which doesn't have its own liquid tracking - in favor of the upstream interface providing it directly - and can be leaner and faster.

In the specific case of liquid tracking, we make the json protocol handlers and the python protocol API responsible for liquid tracking on their own. That lets the json protocol handlers just look at the data the protocol designer already calculated and avoid duplicating work. There's no extra information the runtime liquid level tracking can give because we don't actually have feedback.

[IanLondon]

Thanks for the thoughtful write-up!

Like Seth pointed out, we might want the implementations to be allowed to diverge. PAPI vs PD/JSON are different tools with different uses, and even if they go in the same direction they might develop distinctly differently, and at different rates.

IMO liquid tracking is a stylized cartoon which we have made to help users design and view their protocols. It's not a source of truth and doesn't need to be strictly consistent across our applications (or their versions). It's not suitable for execution b/c it's only accurate in a narrow lane (boundaries of which are far from being well-defined) and even within that happy path we don't empirically validate the correctness of liquid tracking. And that's OK, because accurately simulating reality isn't what our liquid tracking needs to do.

I like the idea that the JSON executor doesn't do liquid tracking and expects that liquid-related problems were dealt with responsibly in the app which created the JSON protocol. Similarly, a Python execution layer could just do dumbly run the commands it is given, and not track liquids itself. (Would be nice if PAPI was the same execution layer as JSON, like Seth said -- right now AFAIK there's no similar "executor" for Python protocols. Would that be the structure that Toma talked about way back: calling the Python methods would emit JSON commands instead of immediately interacting with the drivers?)

User stories

I might have missed some, but here's an attempt to distill some user stories out of the RFC text:

• As a PAPI user, I want to track multiple liquid types across a protocol, so that I have more information about what is happening in my protocol

• As a PAPI user, I want a (PD-esque?) UI to set up my starting deck state in my Python protocol, so that I can use liquid tracking information without having to define the liquid state in every single well in Python

• As a person running a Python protocol on my robot, I want to visualize the initial liquid state so I know where to put liquids on the deck

• As a person running a Python protocol on my robot, I want to see the liquid state update as the protocol progresses so I have a clearer understanding of where the protocol is at

[pantslakz]

Heyas - @howisthisnamenottakenyet and I were talking about the sample volume field for Thermocycler in PD - and debating if its something which populate automagically (given the "liquid tracking" capabilities of PD), or if user should manually input the number.

During that convo we landed on the approach to have the user input the number in manually for two reasons:

1. pending on the type of liquid and how the liquids end up in the well (single/multi aspirate), we can't be certain that the actual volume in the well will equal the calculated volume based on transfers
2. inputting volume field for a thermocycler is an existing standard behaviour and so we thought it wouldn't be asking the user too much to manually input for now.

So it's interesting to learn that its this same field which has sparked the discussion around volume tracking in API. It's also worth noting that HW is starting to do more testing around pipetting accuracy - however I don't believe they're considering different liquid classes at this time.

cc @mikec116

[umbhau]

Few things discussed offline with @b-cooper:

• Ensuring liquids, etc., can exist independently of the OT-2 and its timeline (e.g. if you wanted to track a specific instance of a liquid across various equipment)
• Thinking about how states could function as a 'contract' between multiple protocols (even JSON and Python protocols)
• Being able to reduce (at least one-way) both Python API and Protocol Designer protocols down to a consistent set of steps rendered on the same timeline

Overall, really like this RFC.

[theosanderson]

It may be useful to consider a user story featuring a user like my past self. I wanted to use a piece of labware (e.g. 50ml falcon) with wells so tall that the pipetting position would have to be different when it is full and empty, necessitating knowing how much liquid there was in it.

[yfukai]

Hello, as OT-2 users considering automated protocol generation, we are wondering if it is possible to simulate the liquid volume change via (JSON or Python) protocols, which would help the protocol validation and test-driven development a lot. Is there any progress related to this issue?
Thanks!

[b-cooper]

Hello, as OT-2 users considering automated protocol generation, we are wondering if it is possible to simulate the liquid volume change via (JSON or Python) protocols, which would help the protocol validation and test-driven development a lot. Is there any progress related to this issue?
Thanks!

@yfukai Yes, there is current ongoing work that aims to provide liquid volume state tracking across our software for a given protocol.

We are in the process of restructuring our protocol analysis/execution layer that runs on the robot (referred to as the Protocol Engine). The Protocol Engine work will pave the way for many features including the ability to track liquid volume information for a given pipetting command and expose new interfaces to this liquid tracking from the Python API.

As far as JSON protocols, our Protocol Designer currently has first-class support for tracking liquid volume change across the timeline of the protocol. When creating a protocol within the Protocol Designer GUI, you can establish the initial location of liquids and each successive command will effect that inspectable volume. If you'd like to adapt portions of that logic, they are contained within the step-generation (data-layer) and protocol-designer (view-layer). If one particular function would be useful to you in terms of JSON protocol validation (e.g. input JSON protocol, output liquid state of all labware on deck for the whole timeline of the protocol), we can potentially add it as a utility function to our step-generation project.

For Python protocols you will likely have to wait a couple of months for the current work on the Protocol Engine to be completed and released to take advantage of any new liquid-specific functionality.

If you can add any detail about your ideal protocol validation workflow for "simulating liquid volume change", it's helpful info for us to prioritize these sorts of low-level features.

[yfukai]

Hello @b-cooper, thank you for your kind and detailed reply! I genuinely appreciate it. The idea of restructuring the protocol execution sounds wonderful, and we are looking forward to any update.
Yes, we imagined something like the Protocol Designer, but one which can be executed from the command line or a (possibly Python or JavaScript) program and can read the JSON (and perhaps Python) protocol (as the JSON schema for Protocol Designer input seems yet to be documented).

input JSON protocol, output liquid state of all labware on deck for the whole timeline of the protocol
we can potentially add it as a utility function to our step-generation project

This update sounds quite great and will be helpful for validation of protocols auto-generated in scripts, etc. (and there seem to be similar needs in the Japanese research community, at least).

What we are trying to do is, for example, generating protocols dynamically from tables. I believe an emulator will be essential for test-driven development of those programs in many cases.

[yfukai]

Hello @b-cooper, would you mind letting us know the update on this? We're quite looking forward to implementing liquid volume tracking that we can programmatically access.

[yfukai]

Is this still actively considered?

[b-cooper]

Hi @yfukai, I'm sorry to have kept you hanging.

I'm happy to say that we've been hard at work extending the functionality of the software stack including key portions of infrastructure required to unlock programmatically accessible liquid volume tracking.

We are now kicking off development on the introduction of basic liquid setup specification in the Python Protocol API (Protocol Designer already has this) as well as within the Protocol Engine.

This feature addition will be released along with an addition to the Run Setup flow in the Opentrons App, that visualizes the liquid setup specified in the protocol.

Once this first feature is released, work will continue this year, with the aim to allow for more inspectability of expected liquid state across the entire timeline of a protocol run. We hope that this programmatic access to mid-protocol liquid-state will provide the tools necessary to allow for more powerful and flexible protocols in our ecosystem.

Thank you for your feedback and patient support, it has been very helpful in our journey to make the robot more and more flexible and easy-to-use.

[bndo]

Inadvertent closing! @b-cooper's update above is still valid.

[rick-osmo]

Hi, what is the current timeline for implementation of this feature?