Planned changes to Mindcode

[cardillan]

Preamble

This issue quite comprehensively describes planned upcoming changes to Mindcode. I'll be very grateful for any comments on them (both on the planned syntax and the mechanisms behind that). I don't have any experience in designing languages, and it is quite possible that a lot of things I envision could be designed better.

General changes to the existing syntax are described here: #148. This issue also contains list of features that were completed as of Release 2.4.0.

Change in default language target

Default language target will be bumped to ML8A - corresponding to Mindustry 8 Logic - as soon as Mindustry 8 is released.

Itinerary of planned changes

The following chapters contain individual planned changes. There isn't any fixed implementation schedule, but the imagined order of implementation is about this:

• Immediate
  • Rewriting ANLTR grammar, parser and compiler (a prerequisite for developing Mindcode further)
  • Removing deprecated functionalities
  • Promoting experimental features to standard status
  • Explicit variable declarations (optional under relaxed syntax)
  • Support for using external variables as out variables in for-each loops and output function parameters
• Near horizon
  • Processor-variables backed arrays, including list iteration/vararg passing
  • Code blocks
  • Include files/modules/namespaces
  • Refactoring of Mindustry Logic functions
• Medium horizon
  • Type system
  • Records/structures
  • Local compiler options
  • Short-circuit boolean evaluation
• Far horizon
  • fallthrough and yield in case expressions
  • Memory model
  • Enums
  • Function pointers/array pointers
  • Range checking operator in boolean expressions

Explicit variable declarations

See #150.

External variables

In addition to allocating a heap in a linked block or a variable, it will be possible to allocate it in a parameter variable as well.

param HEAP = cell1;
allocate heap in HEAP[32 ... 64];

In addition to the current way of allocating global variables on a heap, it will be possible to allocate global variables at specific index outside the heap and possibly in a different memory block. The index must be a compile time integer constant.

const count_index = 10;

allocate $state in HEAP[0];
allocate $count in cell2[count_index];

allocate $error in HEAP[32];                // Error: indexes 32 .. 63 are reseved for automatic heap.

It will also be possible to allocate an array of a fixed size, either on a heap, or in a different memory block. Arrays will always be indexed starting at 0. If the array's position in a memory block doesn't start at 0, the indexes will be adjusted as necessary when accessing the array.

allocate array $arr[10];                    // Allocates an array with 10 items on the heap
allocate array $external in cell1[16..32];  // Allocates an array in the cell1 at given indexes.
                                            // The array has indexes 0-15, which are mapped to 16-31. 

It will be allowed to allocate two external variables pointing to the same memory slot, at it is impossible to prevent this happening generally (memory blocks can be obtained through getlink and other means, and there's no general algorithm to determine whether two blocks obtained programmatically point to the same memory block or not). So, for example, the following will be legal:

allocate array $data in bank1[0 ... 256];
allocate $first in bank1[0];
allocate $last in bank1[255];
allocate array $overlapping in bank1[128 ... 512];

Support for dynamic indexes in external variables might be added one day. It shouldn't require changes to the syntax.

head_index = 0;
allocate $head in bank1[head_index];
allocate array $data in bank1[0 ... 512];

$head = 0;
head_index = (head_index + 1) % 512;

Processor-variables backed arrays

It will be possible to create a fixed-size array, where individual elements are kept in a processor variable. When accessing an element at a constant index, the variable will be accessed directly. When accessing an element using a dynamic index, the proper variable for the element will be read/written using a compiler-generated function.

Arrays will be global and static: the array cannot be assigned to a variable and cannot be passed as an argument to a function, except as a vararg argument (and maybe additional support to inline functions).

Example:

array units[10];            // 10 elements
array data[] = (1, 2, 3);   // 3 elements

Assignments between arrays, especially when they have the same size, might be supported.

Array iteration loops will be supported:

array units[10];

for unit in units
    ...
end;

Code blocks

It will be possible to declare a code block using the following syntax:

begin
    print("This is a code block");
    a = b * c;
end;

The main use case for this is to create a main code block, and to apply a compiler/optimizer option to the entire code block.

Include files/modules/namespaces

See #149.

Refactoring of Mindustry Logic functions

Implementing namespaces would allow better organization of Mindcode functions into related groups (e.g. separate namespace for world-processor functions).

Current system would be accessible in one namespace for backwards compatibility, new systems would come with their own namespaces.

Type system

Types will provide more information to the compiler/optimizer, type checking and enforcing will make programs safer.

Possible types:

• void: no value (A void function). Type of expressions/statements not having a value, e.g. while loop.
• var: corresponds exactly to the Mindustry Processor variable
• int: an integer (Mindustry Processor long)
• float: a float (Mindustry Processor double)
• string
• block: a block. Linked variable would be of type block.
• unit: a unit
• block_t: block type - @switch
• unit_t: unit type - @mono
• item: item type - @coal
• liquid: liquid type - @water
• property: sensable property - @totalItems
• Probably a few others I can't think of now

Records/structures

Record will be a user-defined type consisting of members. Each member of a record/structure will be stored in a processor variable.

Nested records will probably be supported.

Main benefits:

• Array of records: more efficient when accessing all members of the record at once
• Multivalued Function return values/function output parameters

Possibly a Vector record consisting of x, y members, with arithmetic operations defined on them

Local compiler options

It will be possible to apply different compiler/optimizer options to a block of code using the #set local directive:

#set local option = value, option = value, ...;

List of options that can be applied this way will be limited. An example is the goal option.

The #set local directive will be applied to the next expression, for example to a loop. Therefore

#set local goal = size;
for i in 0 ... 512
    bank1[i] = 0; 
end;

will prevent the loop from being unrolled, but will allow other loops in the program to be unrolled. Code blocks can be used to apply the directive to several expressions at once. Using #set local before function declaration will change the settings for the entire function.

Short-circuit boolean evaluation

Short-circuit boolean evaluation will be introduced into the language in some form. The current plan is to use and and or operators for short-circuiting, and && and || operators for full boolean evaluation. If you know you want to avoid short-circuiting in some expression, it is advisable to use the latter form.

fallthrough and yield in case expressions

The fallthrough keyword can be used in a when branch in case expression to transfer the control to the next branch:

case number
    when 1 then 
        print "one";
        fallthrough;
    when 2 then
        print "two";
end;

When number is 1, the code will print "onetwo". When number is 2, the output will be just "two". fallthrough can be used anywhere in the when branch, for example inside an if statement.

The yield keyword can be used in a when branch in case expression to terminate the evaluation of the case expression and provide the resulting value (the code will output "even"):

number = 4;
output = case number
    when 1 .. 10 then
        if number % 2 == 0 then
            yield "even";
        end;
        print("not even\n");
        "odd"; 
    else
        "too big";
end;
print(output);

The value after yield will be optional; when not given, the resulting value of the case expression will be null.

Memory model

Optimizations of external variables are planned for the future. These should prevent unnecessary reads and writes from/to memory blocks. To assist the compiler with these optimizations, it will be possible to assign a memory block or a variable one of three possible memory models:

• volatile: this means that the external memory may be modified by another processor, and therefore reads and writes from/to this memory must not be eliminated from the code.
• aliased: this means that the external memory will not be modified by another processor, but the memory block might be accessed using two different variables.
• restricted: this means the memory block will never be accessed using two different variables.

A memory model can be assigned to a memory block, or to a concrete slot inside a memory block. Model assigned to a slot has precedence over model assigned to the whole block. Conflicting assignment on a block or slot level cause compilation error. Declaring a variable as restricted when it can be inferred from a code it is actually aliased causes compilation error.

Example:

param MEMORY1 = bank1;
param MEMORY2 = bank2;

allocate $state in MEMORY1[0];

declare MEMORY1 restricted;
declare MEMORY2[0 ... 256] volatile;
declare MEMORY2[256 ... 512] aliased;

declare $state volatile;

allocate $state2 in MEMORY1[0];
declare $state2 restricted;             // Error: we know $state2 is an alias to $state

Enums

It will be possible to declare enums using the enum keyword:

enum enum_name is element1, element2, element3 end;

Mindcode will assign values to the enums as it sees fit. There are no guarantees on the numbers whatsoever, except preserving the declaration order. They could be instruction addresses inside a case expression, for example, if there's just one case expression.

Mindcode will provide functions to access enum properties (e.g. element.name, element.next, element.previous). They might be a bit costly to use.

Support for enums in list iteration Loops: for i in enum_name.

Function pointers

Function pointers would allow storing a function address in a variable and calling the function through that variable. The internal mechanism is briefly described in the road map.

Assigning a function address to a variable could happen in several ways:

def foo()
    print("foo");
end;

fptr1 = foo;                // Note: no parentheses

fptr2 = def bar() 
    print("bar");
end;

// Lambda syntax
fptr3 = () -> print("baz");
fptr4 = n -> print("n");    // No parentheses required for lambdas with single parameter 
fptr5 = (x, y) -> x + y;

// Code block needs to be used if the lambda is more than one expression
fptr6 = n -> begin
    sum = 0;
    while n > 0 do
        sum += n--;
    end;
    return n;               // Lambda is just another way of writing functions, return is allowed
end;

// Code block is not needed if the lambda is just one expression, even a complex one
fptr7 = n-> while n > 0
    println(n--);
end;

// Lambdas with output parameters
fptr8 = (x, out y) -> begin y = x * x; return y + x + 1; end;

// Calls via the function pointers
fptr3();
fptr4("Hello");
print(fptr5(5, 7));

// Will print numbers 10 to 1 in descending order and then null, as loops always returns null.  
print(fptr7(10));

// Mindcode lambdas will probably have read-only access to the outside variables:
status1 = () -> printf("Running: $count reactors active\n");
status2 = () -> printf("Shutdown: cryofluid level $level too low\n");

print_status(running ? status1 : status2);

def print_status(fptr_status)
    ...
    println("Power plant status:");
    printf("Energy consumption: $consumption\n");
    fptr_status();
    printf("Battery level: $battery\n");
end;

Range checking operator in boolean expressions

The case expression supports specifying ranges in when branches. Similarly, the in operator will allow testing ranges in boolean expressions:

if i in 5 .. 10 then
    println("In range");
else
    println("Outside range");
end;

print(i in 5 .. 10 ? "In range" : "Outside range");

[cardillan]

Two more brief ideas:

• Support for combining source files. Probably by including files at the source level (e.g. C-like #include). Implementing it with ANTLR won't be straightforward, unfortunately.
• Support for Ruby-like condition modifier, e.g print("debug") if debug;. Ifs with just one statement are hard to write otherwise. Or maybe we could abuse the ternary operator by making the else branch optional: debug ? print("debug"); I may like the latter a bit more.

[JeanJPNM]

I think that having an approach similar to python modules can be more advantageous, because #include usually relies on other compiler directives like #ifdef to not generate the same code twice, which can happen when there are cyclic dependencies

graph BT;
    A-->B;
    A-->C;
    B-->A;

Loading

or when two files share a dependency.

graph BT;
    A-->B;
    A-->C;
    B-->D;
    C-->D;

Loading

With modules, the reasoning is simpler because you can guarantee that each module is only included once,
with the order being the same as a depth first search on the module graph.

In practice, you can do the same with #include, but it brings the previously mentioned connotations from
its most famous implementation in C and C++. So people may expected it to behave in the same way because of its name.

[cardillan]

That's a good suggestion! I've already read about your plans to add modules to mlogjs some time ago and my solution is now posted at the bottom of this discussion. Your project has inspired me a lot during the development of Mindcode - thanks :)

[tom-leys]

Wow! You have a lot of ideas for improvement.

Should global variables be explicitly declared (e.g. declare STATUS global) instead of relying on the upper-case convention for global variable names?

I dislike the current system. I'd prefer that all variables declared outside a method (in the top level) are considered global.

There should be a main function (def main) (case insensitive) which is the program's entry point. Rather than putting all code at the top (global) level.

If you don't have main, then you might make every line of code at global scope automatially "in main"- this is a copy of what C# does recently.

So if you have a variable in main, it's local. If it is outside of main, it's global. Done.

The $ prefix of external variables is the same as the $ formatting character in the printf function. Should one of them be changed, and how?

Never used external variables (don't know what these are) but sounds like you want a declare statement for these (based on later ideas you mention)

I'd like to add a variant of printf which would output a newline character in addition to the normal output. What's a good name for such a function: printfln, printlnf or something else entirely?

Yeah, I think printlnf makes sense here. Or just make all print statements printf (support $). An alternative - Python lets you make any string formatted with f"blah blah" (note the f prefix) so you could go that way - i.e str = f"hello $var"; println(str) is valid.

I plan defining processor-variables backed arrays by just the array keyword. Should arrays be declared using the allocate or declare keywords instead?

Dislike the word allocate here because two CPUs might be sharing memory and it sounds like this CPU is authorative when it is not.

Declare makes sense. Though we might have a lot of declares... You'll need declare memory and other similar combos.

Mindcode currently has a list of variable names that may represent blocks linked to the processor, such as cell1 or switch5. This list needs to be kept in sync with list of existing blocks in Mindustry, and adding new types of blocks may require releasing a new version of Mindcode. Block types defined by mods are entirely unsupported. Would it be better to require explicitly declaring such variables (declare cell1 linked) and treat other variable names as normal?

I think it is good form to copy these linked components to named variables in the code anyway. You can achieve both with the right declare statements.

There is a gotcha here though.

My prefered code structure is

<CONSTS>

loop{
  <COPY LINKS>
  code
}

If you don't put the copy of linked components in the loop, those links will break if the relevant structures are built after the CPU starts looping (or if they are destroyed and replaced).

[tom-leys]

Enums

Might be useful to have a function say int() which outputs 0,1,2,3 for different enum labels. And one say str() which outputs the relevant tag as a string for debugging.

(Pointers to arrays might, perhaps, come one day).

You could achieve this (probably) with a "pointer" being a pair of - a link to the relevant array function (jump table) and the length of the array. From there you can dynamically compute the jumps required. Length can also be used for range checks.

Structs would indeed be nice to have.

[cardillan]

@tom-leys Thanks for your feedback! Having these conversations helps really a lot.

I think it is time I flesh out my objectives explicitly here:

• Using variables without declaration unless they're special in some way.
  • Global variables and external variables (i.e. variables in memory cells/banks) may be considered special.
• Using linked blocks without declaration, if possible.
• Supporting simple programs with no boilerplate at all.
  • By simple I mean a program which is just a sequence of expressions/statements. No functions, no external variables. Global variables wouldn't be needed in such a program.
• Not mangling variable names when compiling to mlog when possible.

There should be a main function (def main) (case insensitive) which is the program's entry point. Rather than putting all code at the top (global) level.

If you don't have main, then you might make every line of code at global scope automatially "in main"- this is a copy of what C# does recently.

This makes sense, and having a main function would automatically enforce a nice structure of declarations, but I don't want to have to support two different ways of doing this. Given the choice I'll opt for global scope main code. A small plus is that variable names in functions are mangled, while variable names outside functions are not, which is consistent and feels intuitive.

Code blocks will allow you to enclose the main code in begin ... end; incidentally this is how Borland Pascal did it back in the day (feels old :D). It won't add any functionality at all, it just might look nicer.

So if you have a variable in main, it's local. If it is outside of main, it's global. Done.

As there would be no code outside of main, global variables would have to be declared anyway - there would be no other way to put them there. So having to declare globals explicitly seems natural now.

Never used external variables (don't know what these are) but sounds like you want a declare statement for these (based on later ideas you mention)

External variables represent memory cell/bank slots. More about them is here: Variables and constants.

In Mindcode as designed by @francois, no variables were declared, even the external ones. So they needed the prefix to be identifiable. I still kinda like it - they're such a different beasts that having them stand out in the code seems useful to me.

I'm now tending towards declaring them explicitly, in which case the prefix can be dropped. Again, users can choose whatever convention they want for them.

An alternative - Python lets you make any string formatted with f"blah blah" (note the f prefix) so you could go that way - i.e str = f"hello $var"; println(str) is valid.

I originally wanted to do this, but as there isn't any string manipulation in mlog whatsoever, everything needs to be resolved at the compile time. I couldn't handle the f"..." strings being assigned to a variable or a function parameter, for example. There's an exception for inline functions, I think, where a string passed as an argument can be later passed to printf.

However, maybe I could keep just print and println, and if the first argument is a formattable string, it would trigger the formatting variant. I might like it more if the prefix of such string was $, as this would correspond with the formatting directive inside the string. So, for example println($"Position: $x, $y");. Passing the formattable strings to an inline function might be more problematic, though.

A global replace of printf(" with print($" would handle this transition. And I could still keep printf around and just make it deprecated.

Dislike the word allocate here because two CPUs might be sharing memory and it sounds like this CPU is authorative when it is not.

Declare makes sense. Though we might have a lot of declares... You'll need declare memory and other similar combos.

I'll keep allocate for allocating heap and stack in external memory. I'll use declare for everything else - globals, external variables, external/internal arrays. var would be shorter, but declare feels better, as not everything declared like this will be a variable (arrays are more like global static structures than variables).

I think it is good form to copy these linked components to named variables in the code anyway. You can achieve both with the right declare statements.

There's a particular problem, though: say I use wave1 as a variable. However, there's a block which can be linked to a processor under that name. When that happens, even by mistake, your wave1 variable becomes a linked block - and the program will probably break down. This is why I prefer to use an automatic mechanism for identifying linked blocks. I'll probably use another mechanism for adding a new word to the list of reserved words that may represent a linked block. It would be either declare balloon linked, or maybe #linked balloon, both implicitly making balloon1 a linked variable. I prefer the latter, as the command basically extends Mindcode's metadata, so it is more like a compiler directive.

I might allow to declare linked blocks explicitly, say declare switch1 linked, but it might be confusing, as it would do precisely nothing. Perhaps a strict syntax option could be added which would require all linked blocks to be explicitly declared, and that would prevent using variable names reserved for linked blocks, but undeclared, from being used an any way.

If you don't put the copy of linked components in the loop, those links will break if the relevant structures are built after the CPU starts looping (or if they are destroyed and replaced).

Yes, I've run into this as well. I don't think this can be resolved at the language level, but I'll add a chapter to tips and tricks.

I usually load blocks dynamically to solve both delayed building and dynamic block linking (where the name isn't fixed anyway) issues:

while MEMORY == null || DISPLAY == null
    print("Item/Liquid/Energy Level Display")
    n = @links
    while n > 0
        n = n - 1
        block = getlink(n)
        print("\nFound: ", block)
        case block.type
            when @large-logic-display       DISPLAY = block
            when @message                   MESSAGE = block
            when @switch                    SWITCH = block
            when @memory-bank, @memory-cell MEMORY = block
        end
    end
    
    printf("\nMessage: $MESSAGE\nSwitch: $SWITCH\nMemory: $MEMORY\nDisplay: $DISPLAY")
    if MEMORY == null || DISPLAY == null
        print("\nNo memory or display.")
    end
    printflush(MESSAGE)
end

[calestialgem]

If a main function feels like too much boilerplate, how about an entrypoint scope like in HLML. Putting globals and locals of main in the same scope feels like a step backwards.

// globals
entrypoint
  // locals
  // code
end

It is writing only 2 extra keywords to gain better structure.

[cardillan]

This could work - I'd only prefer the begin keyword, making it a code block:

declare x;   // global
begin
    declare y; // local (optional)
    // main code
end;

I strongly prefer to let users just type in a few statements into the web app and see them compiled. However, this might perhaps be helped a bit if clicking the "start with a new script" link would load something like this into the editor:

begin
    // Type your code here:
    <cursor here>
end;

It would be very important to produce a clear error message if the begin/end were missing. I'm not sure ANTLR will produce that on its own, maybe it will need some help.

Now, since today, we have remarks. And parameters would be declared in the global scope. So we'd have to allow specifying remarks in the global scope too:

allocate heap in cell1;             // doesn't produce any mlog code
remark("Configurable options:");    // generates the remark
param MIN = 10;                     // generates set instruction
param MAX = 100;                    // this too

declare x;                          // doesn't produce any mlog code

remark("Don't modify anything below this line.");
begin
    for i in MIN .. MAX
        print("Here is some actual code");
    end;
end;

It shouldn't be a problem, I guess.

Without the main function/code block, declaring global variables would have to be explicit (declare x global;). That's not very nice.

[cardillan]

If you don't put the copy of linked components in the loop, those links will break if the relevant structures are built after the CPU starts looping (or if they are destroyed and replaced).

There actually might be a way to help this at the language level. I currently envision that linked blocks might be declared in the language either as parameters (and therefore changeable in the mlog code), or as directly used linked blocks. The syntax might look like this:

param memory = cell1;
param display = display1;
linked switch1;

begin 
    ...
end;

So the compiler knows which external blocks are expected, and can automatically generate a loop that would wait for them to materialize, essentially something like this:

do
    memory = cell1;
    display = display1;
loop while memory == null or display == null or switch1 == null;

// main code block

I plan to suggest a feature to Anuken where blocks could have tags settable by a processor which would appear on mouseover. If this gets accepted, such code could include setting the processor's tag to something like "Waiting for cell1, display1 and switch1 to be connected."

[cardillan]

Mindcode modules: moved to #149.

[cardillan]

I'll create issues for individual topic where some concrete idea has already emerged. Relevant passages in the original post will be replaced with a reference to the specific issue.

[cardillan]

Updated the initial post to reflect current developments.

[cardillan]

I've updated the priorities a bit. I want to introduce processor-variables backed arrays as soon as possible, since I believe that currently it is a feature which will bring the greatest benefit.

While pondering the implementation I realized that arrays of records/structures would be very useful performance-wise. Storing structure members in several separate variables would mean that retrieving each member at given index would require separate function call, while accessing a structure in an array would require just one function call. There will still be difficulties and optimization challenges, but structures will have additional benefits, e.g. for function return values.