Minify projects

[GarboMuffin]

Scratch projects tend to be very large, we should try to address that with lossy compression that doesn't affect project behavior

• project.json: We should be able to very aggressively compress by pretending the editor doesn't exist
• *.png: we could throw the files into a lossless compressor eg. oxipng
• *.svg: we could throw the files into svgo with a few things turned off

[CST1229]

Something that might be worth adding is something like the project.json size reducer, especially compressing identifiers.

[GarboMuffin]

That's vaguely what I'm referring to for project.json. if we pretend that the editor doesn't exist and decide to effectively become a project obfuscator, we can do some much, much more aggressive things than that project

[apple502j]

Compressions that do not affect the editor (thus can be applied to all projects):

• ID shortening
• Trimming meta.agent and meta.vm (they are not required)
• Using literal true/false in block.mutation
• Using numbers in num_primitive and text_primitive if it allows
• Using default values

Compressions that affect the editor in some way (but acceptable in immutable projects):

• Removing comments
• Removing broadcasts and local variables so that the fallback code will create it in the runtime
• Removing invisible monitors
• Removing extensions. This is used in WWW but not in VM, loading extension relies on opcode.
• Removing block.x/block.y
• Removing shadow block
• Removing unused variables or lists
• Removing dead code/orphaned blocks

Compression that violates the Scratch 3.0 file format and VM implementation but is acceptable:

• Shortening keys
• Removing target.isStage; the first element of the list must be the stage so this is redundant
• Removing asset.md5ext which is just ${assetId}.${dataFormat}
• Using sequential ID instead of hash for filenames like Scratch 2.0
• Storing color_primitive as CSS color, decimal value, RGB or RRGGBB, whichever is shorter
• Flattening block.inputs
• Changing block.mutation, currently this is JSON representation of XMLDOM
• Using 1/0 instead of boolean
• Removing name and layerOrder from the stage

[Samq64]

Another easy way to add lossy compression is to add an option to initialize all variables to '' and lists to []. That would save a lot of space, but it has to be optional because a lot of projects rely on those values and don't regenerate them.

[GarboMuffin]

sb3 project.json compression has landed

• https://scratch.mit.edu/projects/10128407/: 4.8MB to 3.0MB
• https://scratch.mit.edu/projects/60917032/: 3.0MB to 1.8MB
• https://scratch.mit.edu/projects/413072152/: 3.1MB to 1.9MB (after gzip: 538KB to 261KB)

There's probably a couple hundred more KB that could be saved without breaking projects. Beyond that point, we would probably be better off making a binary file format instead of using JSON.

Note that many of these projects contain many large assets that make any project.json optimizations almost meaningless. Some of those will be investigated next.

[lisa-wolfgang]

Note that many of these projects contain many large assets that make any project.json optimizations almost meaningless.

Yes, especially audio files. Scratch projects can play .mp3 files as long as they have the same filenames as the .wav assets they were converted from, so automatically converting all audio to .mp3 at a constant (reduced) bitrate would be very beneficial. (I've already done this manually for a project and haven't had issues with editing the files either.)