Open
Description
It used to be that we only pretokenised tokens, and now we tokenise Strings too. We could also pretokenise ints/floats in a similar way as parsing these has a nonzero overhead involving a multiplication by 10 for each digit.
Having a token for int8/int32 would probably be a good start. Tokenising floating point properly would require a double to be stored (so 8 bytes) or it'd affect the final value so likely would make the pretokenised code bigger. There's always an option to store floating point base 10 (maybe 24 bit mantissa + 8 bit exponent) which would accurately store the vast majority of floating point values used in code.