You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It used to be that we only pretokenised tokens, and now we tokenise Strings too. We could also pretokenise ints/floats in a similar way as parsing these has a nonzero overhead involving a multiplication by 10 for each digit.
Having a token for int8/int32 would probably be a good start. Tokenising floating point properly would require a double to be stored (so 8 bytes) or it'd affect the final value so likely would make the pretokenised code bigger. There's always an option to store floating point base 10 (maybe 24 bit mantissa + 8 bit exponent) which would accurately store the vast majority of floating point values used in code.
The text was updated successfully, but these errors were encountered:
It used to be that we only pretokenised tokens, and now we tokenise Strings too. We could also pretokenise ints/floats in a similar way as parsing these has a nonzero overhead involving a multiplication by 10 for each digit.
Having a token for int8/int32 would probably be a good start. Tokenising floating point properly would require a double to be stored (so 8 bytes) or it'd affect the final value so likely would make the pretokenised code bigger. There's always an option to store floating point base 10 (maybe 24 bit mantissa + 8 bit exponent) which would accurately store the vast majority of floating point values used in code.
The text was updated successfully, but these errors were encountered: