Executable python files and --jobs command line argument

[obtitus]

• Made some of the python files executable
• Added --jobs -j to limit maximum number of parallel processes. I had some bad experience with a large project (my master thesis) where some parts required a huge amount of memory, after a clean; 8 pythontex processes would then grab 8 times this much memory. This option can also be useful when running on large clusters/servers, where you don't necessarily want to grab all 64 cores even though our document is large enough to do so.

Some options on the latex side should also be considered, a few cases:

• Mark the block as "do not run in parallel" (e.g. due to being memory intensive, or if utilizing some non-thread safe external library/application)
• Mark the block as "I will generate x parallel processes", then pythontex should handle this to avoid taxing the system.

[gpoore]

• Executable files: Good suggestion. I sometimes miss these things spending so much time in Windows.
• Maximum number of processes: I knew that issues were possible in principle, but hadn't considered your use case or the possibility of running on clusters. I think the name --jobs makes sense, and it's also what SCons uses for this sort of thing. Something like --maxjobs might be just a little clearer, but then the one-letter short form wouldn't be as obvious, so it's probably not a good alternative.

I will put both of these in v0.13 final.

In terms of the other suggestions, I agree with both. I started working on ways to allow per-session options last summer, and will see if I can get that operational by this coming summer. I want to allow rerun to be set on a per-session basis, so that a session can be disabled, etc. I will add these to the list of per-session options.

There are two ways per-session options could be managed. One would be to enter them after the session name, in any chunk of code belonging to a given session. Something like this:

\begin{pycode}[session_name, parallel=false]
...
\end{pycode}

Another option would be to have a command. Something like this:

\setpythontexsession{session_name}{parallel=false}

Any thoughts on the syntax for per-session options?

Also, there's a bit of a tangent discussion on the future of PythonTeX in #34 (comment), and I'd be interested in any thoughts you have on that.

[obtitus]

The naming --jobs is from make, I am guessing that is where SCons got it from

-j [N], --jobs[=N]          Allow N jobs at once; infinite jobs with no arg.

parallel https://www.gnu.org/software/parallel/man.html#options allows for "--max-procs N" and "-P N" in addition to "-j" and "--jobs", we could consider the same.

I think the second one is the most realistic, the feature would be so obscure that the extra typing would not be a problem. The problem with the first one is how to handle this:

\begin{pycode}[session_a, parallel=false]
...
\end{pycode}
...
\begin{pycode}[session_a, parallel=true] % should we throw an error?
...
\end{pycode}

If we where talking only python we might be able to handle the above case gracefully, but it would not be worth it.

When it comes to another markdown language, I don't really think the world needs another markdown language (are there not enough?). A hook into say sphinx would in my mind be more useful. I am thinking something like pngmath, but allowing arbitrary latex code (pycode included) to be brought in as an image or rendered in some other way. Note that matplotlib already have hooks for including generated plots and code (http://matplotlib.org/sampledoc/extensions.html). But feel free to go for it!

[gpoore]

I've run into a complication with the permissions. Permissions get changed any time I transfer files between Windows and Ubuntu. So trying to set permissions in the commits may not be practical.

As an alternative, how about having the installer chmod 755 all the executable scripts during installation (if the OS isn't Windows)?

[obtitus]

The installer already does chmod 755 as far as I can see. I changed the permissions so that I could easily run the scripts without installing, in addition, the installer script itself should be executable. But if this causes problems, then forget about it.

[gpoore]

The installer doesn't chmod the pythontex*.py and depythontex*.py, only the versions without numbers in the names, so I will fix that. If I figure out a way to change the permissions in a cross-platform manner, I'll let you know.

[obtitus]

Just a small detail:
In your commit, you have called multiprocessing.cpu_count() in the if verbose block without a try except block for the possible "NotImplementedError" exception, which is a potential bug for systems without multiprocessing support.

In regards to the poor support for classes in octave, do you really need a class? We could use a struct and some functions? Octave seems to be more lenient in allowing functions to be defined as part of a script instead of in a separate file.

[gpoore]

I'll fix the try/except before the final 0.13 release...thanks!

For Octave, I'm happy to stick with a struct and some functions. After looking at this, this, and similar sources, it looks like it isn't possible to do what I would want even with a class. Apparently, methods are called like normal functions, with the object as the first argument. It should be possible to create something just as good only using functions defined in the template, which would be a lot simpler in terms of the number of files involved. I have the dependencies and created files tracking mostly working using this approach.