support high level function call from parser to get a single token one by one

[asmwarrior]

I see this library have an interface preprocess, which read a file and write an output file of whole results.

For a high level parser, it would better to supply a function like "get token" or "peek token", so the parser can use it directly.

[john-blackburn]

I see what you mean and it should be possible to do it. I know that compilers don't actually call the preprocessor to create an output file then start lexing that output file. Rather they get the actual tokens from the preprocessor (with everything expanded) and use those tokens. This avoids having to lex twice but it means the preprocessor and compiler (or any other program that might need the post processed output) have to agree on a format for the tokens.

When it lexes a file, my preprocessor creates a list<list<Token>> with Token being a struct. Each list of Tokens is a line in the original file and then the list of list of Token is the entire file. It then goes through this data structure and expands each line. Each line (a list<Token>) is then written using writeLine which dumps to an output file or into a string. So in principle I could return the entire list of list of Tokens to the caller who can then interrogate and pull out Tokens as needed. He then needs to understand the Token struct and the constants in const.h. However, although a list<list<Token>> is available for a single file I'm note sure it is there for all files which have been (recursively) included. So some refactoring may be needed...

I will think about this some more. I agree it's very important if the preprocessor is to be useful in practice.

[asmwarrior]

Yes, if I would like to use your preprocessor, I need such interface. I can use it to get all the file dependencies. I can use it to feed the tokens one by one to some high level parsers.