Where does tokenization and analysis occur?

[MatthewWilliamson4040]

I've been looking for where the magic happens. Tokenization and the analysis of tokens. In simpler terms, where's the code that determines whether the token in question is a comment, a variable, and likely falls through to just plain text?

I'm trying to locate this for Issue #1534 I've located most other bits of code I need to make it happen, but I am going to need to modify the behavior to teach the program how to recognize some new types of tokens.

I've been digging for a while and was hoping someone more familiar with the project could just point me right to it. I feel I've spent enough time digging to justify asking at this point. Thank you in advance.

[MatthewWilliamson4040]

Okay I partially found my own answer here. I did finally find under lib/custom/modes the files apache.js, elixir.js, and so on. I notice there are a lot of languages missing from this folder yet it still operates on them. What am I missing here?