Skip to content

Latest commit

 

History

History
665 lines (512 loc) · 16.2 KB

mep-0001.md

File metadata and controls

665 lines (512 loc) · 16.2 KB
MEP Title Discussion Implementation
1
Representation as a Python File
XXX

Representation of an App as a Python File

Abstract

This MEP proposes a representation of Marimo apps as structured Python files. The representation is legible and easily versionable through source control. It is designed to allow importing Marimo apps as Python modules, to access cells, names, or the DAG itself, while also functioning as an executable script. We strive for a minimal representation with these constraints in mind.

Motivation

Marimo apps are dataflow programs, implemented in Python and executed by the marimo Python library. It makes sense for these apps to be stored on disk as pure Python files; if an app were a pure Python file, then:

  1. marimo could load it using an import statement instead of parsing a new file format
  2. users wouldn't need to learn a new file format
  3. users could format these files however they like, and marimo would still be able to read them
  4. they could be executed as scripts using python
  5. they could be imported into other marimo apps or Python modules using the familiar import statement
  6. users could easily write or edit them using their text editor of choice

Criteria

We seek a representation that has the following properties, ordered from most important to least:

  1. git diff friendly
  2. easy for humans to read
    1. Pythonic
    2. cell ordering (in column) or arrangement (in grid)
    3. cell names (given by user, or automatically generated)
    4. cell refs and defs
    5. dataflow structure
  3. usable as a Python module
    1. for users to use ("remix") cells, names, or the entire graph in other Python programs
  4. executable as a Python script
  5. editable with a text editor

Example

Three cells, in a program called "numerics". Each cells can be understood as a map between references and definitions ("refs" and "defs").

cell refs defs 
import marimo as mo
import numpy as np
 {} {mo, np} 
def matmul(X, Y):
  return np.matmul(X, y)
 {np} {matmul} 
Z = matmul(np.random.randn(4, 4), np.random.randn(4, 4))
mo.md(f"You calculated {Z}")
 {matmul, mo, np} {Z}

Saved to numerics.py:

import marimo

__generated_with = "0.0.1"
app = marimo.App()


@app.cell
def __():
    import marimo as mo
    import numpy as np
    return mo, np


@app.cell
def __(np):
    def matmul(X, Y):
        return np.matmul(X, Y)
    return matmul,


@app.cell
def calculate(matmul, mo, np):
    from other_app import _data
    Z = matmul(np.random.randn(4, 4), _data())
    mo.md(f"You calculated {Z}")
    return Z,


if __name__ == '__main__':
    app.run()

Specification

A marimo file contains two names that will be used either internally by the marimo library, or by users:

  • app: The only public facing attribute. The app object will be extended with methods and attributes that provide access to the app's defs.
  • (optional) __generated_with: the version of marimo used to generate the app

Because the module is a pure Python file, it may be formatted in any way the user likes. Users should not rely on the ability to include arbitrary comments and logic to marimo, because these will be overwritten if/when using the frontend to save edits (on the other hand, they may safely rely on custom contents if they only use marimo run).

Organization

The module's code is organized in the following order:

  • Version. __generated_with, the version of marimo used to generate the module
  • App. A creation of an empty app.
  • Cells. Definitions of cells as functions, decorated with @app.cell.
  • Syntax errors. A list of cells with syntax errors, included only when at least one cell has a syntax error, and interleaved with cell definitions.
  • Main: An if __name__ == '__main__' guarded section for running the app.

Cells

Cells are included in the module as functions, decorated with app.cell. A cell function is a mapping from the cell's refs, which it takes as arguments, to its defs, which it returns.

Cells are defined in the order that they are arranged in the frontend.

The body of the cell (excluding the return statement) is dedented and pasted verbatim into the frontend, meaning that all formatting is preserved.

Users may choose to name their cell functions, either in the frontend or by directly editing the generated module. Cells which are left unnamed by the user are given the placeholder name _. Named cells can be imported by other scripts/libraries from the module top-level.

Cells have the following restrictions on their names:

  • No cell can be named app.
  • No cell can be named marimo.
  • No cell can be named a reserved Python keyword.
  • Names cannot begin with two underscores: these are reserved for unnamed cells and for future use by marimo's codegen and loader.

App

The app object has three members that must remain in future versions to ensure backward compatibility:

  • _functions, a list of the functions used to make the app, used by the library to instantiate the module.
  • _add_unparsable_cell, used in the module to register cells that have syntax errors.
  • run, which executes the apps and returns its outputs (a dict mapping cell name to output) and its defs (a dict mapping def name to value).

In the future the app object may be extended with additional public members, such as a functions object that provides access to function defs that were defined in refless cells (or at least cells whose only refs are imports), or a classes object providing access to class defs, and code transformation utilities (eg, to push refs such as imports down into a cell).

Handling Syntax Errors

A generated module must always be importable, even if one or more cells have syntax errors, so that the server can extract cell codes and load them into the editor.

Cells with syntax errors cannot be defined as functions (if they were, importing the module would raise a SyntaxError exception, making it impossible to load the app in the marimo frontend). Instead, they are included in the file via the function _add_unparsable_cell. This function takes the cell's code as a string, and inserts it into an internal list that combines valid cells and unparsable ones (so that the order of cells is preserved).

Example:

import marimo

__generated_with = "0.0.1"
app = marimo.App()


@app.cell
def _():
    import numpy as np
    return np,


app._add_unparsable_cell(
    """
    _ error
    """
)


@app.cell
def _():
    'all good'
    return


app._add_unparsable_cell(
    """
    _ another_error
    _ and \"\"\"another\"\"\"
    """,
)


if __name__ == '__main__':
    app.run()

Running as a Script

A marimo-generated module can be executed as a regular Python script, e.g.,

python numerics.py

This will execute app.run(), which will in turn execute the DAG. This may be useful if the dag has side-effects such as writing disk or printing to stdout.

(It could be useful if the generated HTML was printed to standard out, but it's not clear that the generated HTML is useful outside the marimo frontend.)

In the future we may lift top-level names ("refless defs", ie defs that do not depend on refs) and make them optional parameters of the main program. This would be helpful when stiching together marimo apps with pipelining tools (which we may ourselves build).

Namespacing

The names of cells are added to the cell's top-level namespace. This makes it possible for users to import a cell directly from a module (from numerics import calculate), at the cost of polluting the top-level namespace.

The specification takes care to avoid name clashes with cell names. The specification reserves the the following names for itself:

  • marimo
  • app
  • any name starting with two underscores (used for internal variables and unnamed cells)

Cells are forbidden from having the same name as a reserved name.

Because users can have arbitrary names, this means that should we wish to add public names in the future, these names will have to be nested under the app object.

Extensibility

The file must always export an app object that has the three attributes mentioned in the app section.

Additional functionality can be added by adding members to the App class, or by adding members to the module whose names start with __. No other global names may be added.

Backwards Compatibility

The file format is guaranteed to be backwards compatible: marimo will always be able to open modules generated by older versions of marimo, because all it needs are the cell functions (app._functions) and the cells with syntax errors (app._unparsable_cells).

Conversely, for as long as we only depend on these two names, marimo should be forward compatible with modules generated by newer versions of marimo (i.e., an old version of marimo should be able to load a module generated by a newer version).

The version of marimo used to generate the module is included in the file in case a backward incompatible change is made. Any version of marimo that introduces the backward incompatible change should be bundled with a loader that can read modules generated by older versions.

Evaluation

Evaluation against the criteria:

  1. git diff friendly

Let's consider different kinds of modifications made to an app, and the modifications that would be made to the generated module as a result.

modification to app modification to module clean diff?
modify an existing cell modification to a single function (body, signature, returns) Y
move a cell (up or down) function moved to a different location in the file Y
rename a cell a function's name is changed Y
introduce a syntax error function replaced with a call to `_add_unparsable_cell` Sort of
change marimo version version number in `__generated_with` changes Y

Verdict: A. Very clean diffs!

  1. easy for humans to read
    1. Pythonic

Yes, fairly Pythonic. Unnamed cells are a little cryptic, but it's easy to get used to them.

  2. cell ordering (in column) or arrangement (in grid)

Easy to read off, since cells are defined in presentation order. However, it is not obvious how to extend this format to accommodate grid layouts in a way that remains readable.

  3. cell names (given by user, or automatically generated)

Easy to read off, and to distinguish between named and unnamed cells.

  4. cell refs and defs

Easy to read off from signature and returns.

  5. dataflow structure

Not at all evident. Would need to rely on an external program to visualize the DAG, and even still it would be awkward to visualize DAGs with unnamed cells.

Verdict: B+. Very readable on all accounts except for dataflow structure, on which we totally fail.

  1. usable as a Python module
    1. for users to use ("remix") cells, names, or the entire graph in other Python programs

Verdict: B+. Cells are accessible at top-level, which is nice. Remixing APIs can be included in the future under the app object, which is okay.

  1. executable as a Python script

Verdict: A+. Can execute with python directly.

  1. editable with a text editor.

Verdict: A. The specification is simple, flexible, and totally free of magical tokens.

Final Verdict

criterion marimo's grade jupyter's grade streamlit's grade
clean diffs A F A+
readability B+ F A+
usable as a Python module B+ F C
executable as a Python script A F F
editable with a text editor A F- A+

Alternatives Considered

  1. Considered including cells in a function namespace, so that there would be no name conflicts. But this feels too complicated.
def _make_app():
    app = marimo._App()

    @app.cell
    def __():
        import marimo as mo
        import numpy as np
        return mo, np

    @app.cell
    def __(np):
        def matmul(X, Y):
            return np.matmul(X, Y)
        return matmul,

    @app.cell
    def calculate(matmul, mo, np):
        from other_app import _data
        Z = matmul(np.random.randn(4, 4), _data())
        mo.md(f"You calculated {Z}")
        return Z,

    return app
  1. No decorators, a function that returns cell in presentation order, and app made after. But too complicated, plus where to put unparsable cells?
"""a marimo app"""

import marimo


_marimo_version = 0.0.1


def _cells():
    def a():
        import marimo as mo
        import numpy as np
        return mo, np

    def b(np):
        def matmul(X, Y):
            return np.matmul(X, Y)
        return matmul,

    def c(matmul, mo, np):
        Z = matmul(np.random.randn(4, 4), np.random.randn(4, 4))
        mo.md(f"You calculated {Z}")
        return Z,

    return b, c, a


app = marimo._make_app([b, c, a])


if __name__ == '__main__':
    app.run()

  1. Cells defined in a topological ordering of the DAG. While this conveys the program order, it makes the file difficult for the author of the app to read, and can lead to messy diffs as small changes (such as renaming cells or swapping contents of cells, depending on the implementation) can lead to large and confusing diffs.

  2. Unnamed cells given a default name, such as __a, __b. This would make it easier to visualize the program as a DAG, since we would have names to refer to, but it would pollute the reader's symbol table.

  3. Explicitly defining the DAG, as sequence of function calls, under the __main__ section. This was supposed to help readability, since it documents the graph; however, it ended up hurting readability because DAGs can become very verbose (due to the need for namespacing) and obtuse (when users don't name their cells, which will be the default). Moreover it was misleading because it suggested that sibling cells ran in a deterministic order, which is false.

  4. A flat format in which the program was stored as a script, using comments to separate cells instead of encapsulating them in functions. This is a smaller representation than what we've proposed in this MEP, but it makes it unwieldy to use the script as a module. It would also require us to add magic comments or symbols to parse the code into cells, making the format brittle.

  5. Leave comments documenting which cells are involved in cycles, which names are multiply defined and by which cells, and which cells try to delete their refs. We can add these things later, without affecting compatibility, since they are just comments.

Implementation

https://github.com/marimo-team/prototype/tree/mep-0001-revision

Future Considerations

  • Invoking a marimo app as a script with optional parameters, one for each mo.Name object:
python app.py -n x 1.0 -n y "hello"

where the syntax is -n [name] [value]. This lets one override the default values of UI elements.

  • Designate a special member or string whose value becomes help documentation for the app (docstring, or for the script ...)
  • How can we extend this format to handle grid layouts? The cell decorator could be extended to optionally take coordinates/width/height of a cell, and cells could be defined in some flattened order (e.g, row-wise)
  • Communicating the DAG structure in a comment
  • A tool (or a marimo app) for visualizing the DAG
  • A tool that tells you if your app has any problems, like cyclic references or multiply defined cells ... (marimo check)