Adding new tokens

[SatyamSingh8306]

Summary

Adds support and verifying for various tokens in the lexer.py, including bitwise operators (BITWISE_AND, BITWISE_OR, BITWISE_XOR, BITWISE_NOT), assignment operators (ASSIGN_AND, ASSIGN_OR, ASSIGN_XOR, ASSIGN_MOD, ASSIGN_SHIFT_LEFT, ASSIGN_SHIFT_RIGHT), logical operators (LOGICAL_AND, LOGICAL_OR), and data types (DOUBLE, UNSIGNED_INT). These updates expand the lexers.py capabilities to identify and process these tokens accurately.

Related Issue

closes #61
fixes #61
resolves #61

Changes

• Added token patterns for DOUBLE and UNSIGNED_INT to the TOKENS dictionary.
• Ensured all bitwise and assignment operators are present and correctly recognized.
• Updated lexer functionality to handle these tokens during parsing.
• Verified existing tokens and resolved any inconsistencies.

Shader Sample (Optional)

N/A

Testing

• Added unit tests for each new token in the lexer.
• Manually tested by running the lexer on a set of code snippets containing the new tokens.
• Verified that existing functionality remains unaffected by the changes.

Checklist

• Tests cover new or changed code (or reason why not)
• Linked the issue with a closing keyword (if applicable)
• Only touched files related to the issue
• Everything builds and runs locally

[NripeshN]

Could you please add the tests for this too? You can add them here: tests/test_translator/test_lexer.py

[SatyamSingh8306]

Could you please add the tests for this too? You can add them here: tests/test_translator/test_lexer.py
ok

[github-actions]

This pull request has been marked as stale because it has been inactive for more than 14 days. Please update this pull request or it will be automatically closed in 7 days.