import_scad('MCAD') fails with the last version of OpenSCAD and solidpython on march 2021

[fredericcassin]

The error occurs on every omport_scad attempt on any library folder I try.
The error with the wiki sample code trace the error:

Unable to import SCAD module: 12ptStar from shapes.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: bearing from hardware.scad, with error: invalid syntax (, line 3)
Unable to import SCAD module: nut from hardware.scad, with error: invalid syntax (, line 3)
Unable to import SCAD module: washer from hardware.scad, with error: invalid syntax (, line 3)
Unable to import SCAD module: rodnut from hardware.scad, with error: invalid syntax (, line 3)
Unable to import SCAD module: rodwasher from hardware.scad, with error: invalid syntax (, line 3)
Unable to import SCAD module: duplicate from multiply.scad, with error: duplicate argument 'no' in function definition (, line 3)
Unable to import SCAD module: 8bit_str from letter_necklace.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 8bit_char from bitmap.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 8bit_str from bitmap.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 8bit_polyfont from fonts.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 3dtri_sides2coord from 3d_triangle.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 3dtri_centerOfGravityCoord from 3d_triangle.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 3dtri_centerOfcircumcircle from 3d_triangle.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 3dtri_radiusOfcircumcircle from 3d_triangle.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 3dtri_radiusOfIn_circle from 3d_triangle.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 3dtri_centerOfIn_circle from 3d_triangle.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 3dtri_draw from 3d_triangle.scad, with error: invalid syntax (, line 2)
Unable to import SCAD module: 3dtri_rnd_draw from 3d_triangle.scad, with error: invalid syntax (, line 2)

Found OpenSCAD code that's not compatible with Python.
Imported OpenSCAD code using from
can be accessed with from_ in SolidPython

['2Dshapes', '3d_triangle', 'class', 'delattr', 'dict', 'dir', 'doc', 'eq', 'format', 'ge', 'getattribute', 'gt', 'hash', 'init', 'init_subclass', 'le', 'lt', 'ne', 'new', 'reduce', 'reduce_ex', 'repr', 'setattr', 'sizeof', 'str', 'subclasshook', 'bearing', 'bitmap', 'boxes', 'constants', 'curves', 'fonts', 'gears', 'gridbeam', 'hardware', 'involute_gears', 'layouts', 'lego_compatibility', 'libtriangles', 'materials', 'math', 'metric_fastners', 'motors', 'multiply', 'nuts_and_bolts', 'polyholes', 'regular_shapes', 'screw', 'servos', 'shapes', 'stepper', 'teardrop', 'transformations', 'triangles', 'trochoids', 'units', 'unregular_shapes', 'utilities']

[etjones]

Thanks for reporting this. I think this is all working as it's 'supposed' to; there is quite a lot of OpenSCAD code in MCAD that the current parser won't handle correctly, and the import is reporting on all of that.

That said, the correct thing to do would be for us to parse all OpenSCAD code correctly, rather than just complaining about stuff we didn't handle before. I'll take a pass at this and see if I can make our OpenSCAD parser more capable.

[etjones]

I just took a look at this. I'm intending to improve the parse so that we can safely import any valid OpenSCAD code. Until then, it's worth noting that MCAD contains hundreds of nested modules & functions so when you do, for instance:

from solid import import_scad

# This import will print a list of OpenSCAD functions/modules that weren't imported
MCAD = import_scad('MCAD')

# However, there's a LOT here that does get imported successfully  you can look through the MCAD code, 
# or look at the namespace to find, for example `MCAD.lego_compatibility.post()`
print(MCAD.__dict__.keys())

[jeff-dh]

I had the same issue and spent some time on it.

These are actually two separate issues:

1. OpenSCAD seems to allow identifiers that start with a digit (e.g. 12ptStar from MCAD/shapes.scad) and since SolidPython creates Python classes to "wrap" all functions and modules using the same identifiers -- as long as they are no python keyword -- the wrapper class is not valid python syntax and the import fails.
   I fixed this issue extending the _[un]subbed_keyword to "escape" identifiers starting with a digit in this branch.
   I think this should work except for the case that you want to import an OpenSCAD identifier starting with _[0-9] (same issue with python keywords, the OpenSCAD identifier from_ will not work with the _unsubbed_keyword function and I assume it will cause a OpenSCAD compile error/warning).

2. The parse_scad_callables function uses regular expressions that are not capable of correctly parsing OpenSCAD function and module declarations. This is for example the case for the nut module from MCAD/hardware.scad. It gets confused when there's a function call in the body of a function / module that's in the same line as the header.
   I wrote a python openscad parser using ply and integrated it into SolidPython in this branch. This seems to fix the issue.

Both branches seem to work (tested with MCAD, BOSL, agentscad, scad-utils) but are for now only tested in a very small development environment.

Feel free to use those branches, I might request a pull request after some more usage.

[jeff-dh]

Btw: the parse_scad_callables function seems to have another issue:

When trying to import the BOSL Library (The Belfry OpenScad Library https://github.com/revarbat/BOSL) SolidPython freezes.
I tracked it down into the parse_scad_callables function and to the BOSL/transforms.scad file. When I was printf-debugging it I couldn't get a handle on it (and decided to try out ply ;).

But this is an issue on master.

[etjones]

Oh man, this sounds like great work! I’m away from a computer for a couple days, but I’m really excited about having a valid parser as opposed to the kludgy regex hack that we use now. I’ll follow up on this when I’m back home.

…

On May 16, 2021, at 12:09 PM, jeff-dh ***@***.***> wrote:  Btw: the parse_scad_callables function has another issue: When trying to import the BOSL Library (The Belfry OpenScad Library https://github.com/revarbat/BOSL) SolidPython freezes. I tracked it down into the parse_scad_callables function and to the BOSL/transforms.scad file. When I was printf-debugging it I couldn't get a handle on it (and decided to try out ply ;). But this is an issue on master. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[jeff-dh]

There are (at least ;) two more things to mention about that branch:

1. I added the "correct" handling of optional arguments lately (e.g. non optional arguments are added as args and optional arguments as kwargs into the callables list). But I stumbled over commit 4a4872f... if that's the case (that all arguments are optional in OpenSCAD) it needs a little adjustment.

2. At the moment the $-symbol is simply ignored by the lexer (meaning it is treated similar to whitespace). I wonder if we need to parse it to handle $fn, $fa, $fs parameters correctly. I don't know how this is handled yet but saw something like a mapping(?) to "segments" in the code. This might need a little adjustment.
   At the moment this module:
   module blub(a, b, $fn) {};
   is treatet as:
   module blub(a, b, fn) {};

[etjones]

Hey @jeff-dh, would you be willing to open a pull request with the use_py_scadparser branch? Using a real parser is absolutely the right solution for this problem, and I'm pretty amped you did the heavy lifting there; I've taken some stabs at it in the past but I got out of my depth.

I took a quick look at your branch. What version of ply are you using? PyPi has version 3.11, but it looks like there's a newer 4.0 version that isn't accessible to dependency management yet (http://www.dabeaz.com/ply/, Current Status heading). I'm committed to keeping SolidPython completely installable with pip install solidpython, so if we can stick to the PyPi-available Ply v3.11, that's my strong preference.

In my copy of your branch, I was able to add Ply to SolidPython's dependencies with poetry add ply, but then I got a bunch of parse errors in SolidPython's test suite with $SOLIDPYTHON_DIR/solid/run_all_tests.sh. Have you managed to get the tests running on your install?

[jeff-dh]

just a quick reply for now: ply.__version__ -> 3.11

[jeff-dh]

In my copy of your branch, I was able to add Ply to SolidPython's dependencies with poetry add ply, but then I got a bunch of parse errors in SolidPython's test suite with $SOLIDPYTHON_DIR/solid/run_all_tests.sh. Have you managed to get the tests running on your install?

I never tried it ;) I'll have a look at it soon

[jeff-dh]

the parameter of the parse_scad_callables changed it used to be scad_contents and now is filename could that be the reason?

[etjones]

That’s exactly the reason! There was also a complaint about an unclosed file handle that could be fixed with a with path.open() as f: context handler.

[jeff-dh]

On Tue, 18 May 2021 19:01:34 -0700 Evan Jones ***@***.***> wrote: That’s exactly the reason! There was also a complaint about an unclosed file handle that could be fixed with a `with path.open() as f:` context handler.

If you don't want to wait for a patch: py_scadparser/scad_parser.py : 249 ;)