Introduction and README.rst

[clozinski]

I volunteered to work on this project.
I think that the best way to understand Python is to
understand how the Python interpreter is written. Starting with cPython is just
too difficult. Byterun is a great place to start. A great way to understand software is to document it.
The first thing I would like to change
in the README.rst is to link to the excellent introductory article on Byterun.
http://www.aosabook.org/en/500L/a-python-interpreter-written-in-python.html

Does anyone have any preferences on how I write documentation?
In the comments? Separate docs? Read the docs? Elsewhere?
Is there anyone who would review my pull requests?

Of course one does not really understand until one has engaged with the software:
"I hear and I forget
I see and I remember
I do and I understand."

Then the next thing to do would be to process the pull requests.
I see there are three pull requests. In due course I plan to review them, although it will take a while until I get to it. This is not my day job. First I have to read and understand the code.
Is there anything else which needs to be done to Byterun?
Thanks for all the hard work already done on Byterun.
Chris

[clozinski]

Documentation Plans.

I think that the article on Byterun
A Python Interpreter Written in Python does a great job introducing Byterun, but it is not really software documentation. And in particular it does not cover the data structures for blocks.

The software documentation needs to cover the data structures, and what needs to be true about them. For example frames have local can global variables, and there is a certain way that they are initialized. I know what local and globals are. I can see the code that initializes them, but I am not quite sure why it is done that way. That all needs to be explained.

"Show me your [code] and conceal your [data structures], and I shall continue to be mystified. Show me your [data structures], and I won't usually need your [code]; it'll be obvious.''

And while the Byterun article focuses on how Byte codes are interpreted, I would rather know how the details relate to the Python language itself.

And then there are gnarly bits of code like the implementation of closures which certainly need explanation.

def make_cell(value):
    # Thanks to Alex Gaynor for help with this bit of twistiness.
    # Construct an actual cell object by creating a closure right here,
    # and grabbing the cell object out of the function we create.
    fn = (lambda x: lambda: x)(value)
    if PY3:
        return fn.__closure__[0]
    else:
        return fn.func_closure[0]

So I think what makes sense for documentation is a separate article somewhere. My first instinct is to write it in Libre Office, but that is not so good for collaboration. And lots of this must be documented in the cPython world, so many links to relevant content would be great.

I find that I often approach open source projects by documenting them. Explaining helps one understand. And the senior people on the project are always quick to point out mistakes.

[llllllllll]

Thank you for deciding to add documentation! If you open pull requests with documentation I will review them. I have been meaning to get to the older prs but it keeps getting bumped from my queue, partly because I don't think the authors are still waiting on them. I think that sphinx would probably be the best choice for documentation for this project. It is the de facto standard for Python projects so most contributors would be able to read and modify the source. Sphinx also has a tool called "intersphinx" for linking documentation across projects. This will allow you to generate links to the official CPython docs where needed. readthedocs has tools for building and hosting sphinx projects so it would be easy to make these visible.

To answer your particular question about the make_cell function: cell is a object like any other. We would love to be able to call Cell(x) to get a cell wrapping x, but, Python does not expose this type to us. A first workaround to try is: type(f.__closure__[0])(x) to get the type of the cell in some function and then call the constructor for the type; however, the constructor is not exposed to python so this call will fail. What we need to do is make Python construct the cell we want for us and then extract it from the function. We do this by creating a new closure. Expanding the lambdas may help:

def outer(value):
    def inner():
        return value
    return inner

fn = outer(value)

this outer function creates a closure inner() which closes over the variable value and creates a cell for us. This looks like it is more complicated than it needs to be because make_cell itself can act as outer. It is 4:38 and I am not at a computer with an interpreter ready but I believe you could just replace the fn = ... line with:

def fn():
    value  # explicitly reference `value` to force it to be a free variable

and the rest of the code should work as expected (for the same reasons)

[clozinski]

Here is the branch with the updated README.rst file.
https://github.com/clozinski/byterun/tree/ImproveReadMe

I hope you like the text.

For some reason the hyperlinks are not working correctly. They just display the original text strings, and are not converted into tags.
They work fine on my machine in MacDown.
They work fine if I copy them to my other repositories.
Links from other repositories break when I copy them to the Byterun fork.

I have no idea why my markdonw links are not working. An extensive Google search did not reveal anything. Any suggestions?

I could just get rid of the text part, but better to do it correctly. One learns something in the process.

Chris

[clozinski]

Well the great news is that the introductory article by Alison about Byterun is in a book that is under the creative commons license. So I can extract the documentation parts and convert it to Sphinx. Then, if it is okay with Ned, what I will do is, is to change the license on this repository to say that the documentation is under a creative commons license.

So here is what the documentation should cover.
Byte Run Classes
• Frame
• Function
• Method
• Cell/Closure
• Generator
• Block?

Then after that, the Byte codes need documentation. Some are very simple, one liners. Others involve some software archeology to better understand. I am more interested in documenting what each byte code should be doing, rather than the specifics of how it is implemented. Maybe a bit of both. I will have to look to find the documentation of the cPython Bytecodes.

Maybe a good place to start is with the blocks class. There is no blocks class. Instead the information is spread out in at least two byte codes. One to push a block and one to pop it. Maybe additional byte codes refer to blocks. It would be very useful to me to bring that information all into one place. The documentation should list what is included in blocks and how they are used. What happens to a block when an exception is thrown.

As for the link problem, I will just remove the text and submit the pull request with raw links.

Chris

[llllllllll]

The reason those links are not rendering correctly is because the readme is restructured text instead of markdown. The syntax for links in ReST is:

`link text <link target>`_

I think that you shouldn't worry too much about the distinction between what is a "byterun class" vs a CPython concept. The impetus for byterun was to understand how the CPython interpreter worked by re implementing parts of it it. All of the abstractions used are emulating abstractions present in CPython. With that in mind, I think that you may want to either pull from or link to a lot of the standard CPython development documentation. For example, all of the instructions are documented in the dis docs. In my opinion, some of the descriptions are vague so we could extend the documentation where needed but it would be good to link to the official source. I have some comments about your readme proposal but it would be easiest to discuss them in a pull request. It's okay if you aren't done, we can at least start the dialog.

Re blocks: a block is the state required to ensure that exception handlers, finally blocks, and __exit__ is invoked. You are correct that the opcodes that refer to blocks are just the push/pop instructions.

[clozinski]

The impetus for byterun was to understand how the CPython interpreter worked by re implementing parts of it it. All of the abstractions used are emulating abstractions present in CPython. With that in mind, I think that you may want to either pull from or link to a lot of the standard CPython development documentation.

Thank you for that bit of guidance. It crystalizes a thought I had in the back of my head that my goal here is to get the abstract concepts correct. Sometimes the code does the right thing, the example of setting the globals and locals comes to mind, but it is not initially clear what the abstraction is, why that is the right thing to do. So in writing, I will try to get the abstractions correct. Once I manage to do that, my understanding of the VM will have improved.