tokenizer-next

iterator based tokenizer for writing parsers

npm i tokenizer-next

pnpm add tokenizer-next

yarn add tokenizer-next

API

# createTokenizer(regexps) – Create a {@link TokenizerFactory} for the given RegExps.

src/index.ts#L19

To capture, RegExps must use a named group.

const tokenize = createTokenizer(
  /(?<ident>[a-z]+)/, // named groups determine token `group`
  /(?<number>[0-9]+)/
)

# regexps – RegExps to match.

RegExp []

createTokenizer(regexps)  =>

TokenizerFactory

# Token – Token interface

src/match-to-token/dist/types/token.d.ts#L5

# constructor(value, group, source)

src/match-to-token/dist/types/token.d.ts#L23

# new Token()

Token

# value

string

# group

string

# source

RegExpMatchArrayLike

# group – The group it matched.

src/match-to-token/dist/types/token.d.ts#L10

string

# source – The input string.

src/match-to-token/dist/types/token.d.ts#L22

RegExpMatchArrayLike

# index
src/match-to-token/dist/types/token.d.ts#L18
# value
src/match-to-token/dist/types/token.d.ts#L14
# as(value, group)

src/match-to-token/dist/types/token.d.ts#L25

# value

string

# group

string

as(value, group)  =>

Token

# is(group, value)

src/match-to-token/dist/types/token.d.ts#L24

# group

string

# value

string

is(group, value)  =>

boolean

# create(value, group, source)

src/match-to-token/dist/types/token.d.ts#L6

# value

string

# group

string

# source

RegExpMatchArrayLike

create(value, group, source)  =>

Token

# RegExpMatchArrayLike

src/match-to-token/dist/types/index.d.ts#L2

# index

src/match-to-token/dist/types/index.d.ts#L3

number

# input

src/match-to-token/dist/types/index.d.ts#L4

string

# Token

src/match-to-token/dist/types/index.d.ts#L6

MatchToken & string

# TokenizerCallableIterable – Can be called to return next <a href="https://github.com/stagas/match-to-token#token">Token</a> or can be used as an Iterable on for-of and spread operations.

src/index.ts#L74

# ()

()  =>

Token

& Iterable<Token>

# TokenizerFactory

src/index.ts#L67

# (input) – Create a {@link TokenizerCallableIterable} for given input string.

// using next()
const next = tokenize('hello 123')
console.log(next()) // => {group: 'ident', value: 'hello', index: 0}
console.log(next()) // => {group: 'number', value: '123', index: 6}
console.log(next()) // => undefined

// using for of
for (const token of tokenize('hello 123')) {
  console.log(token)
  // => {group: 'ident', value: 'hello', index: 0}
  // => {group: 'number', value: '123', index: 6}
}

// using spread
const tokens = [...tokenize('hello 123')]
console.log(tokens)
// => [
//   {group: 'ident', value: 'hello', index: 0},
//   {group: 'number', value: '123', index: 6}
// ]

# input – The string to tokenize.

string

(input)  =>

TokenizerCallableIterable

# createTokenizer(regexps) – Create a {@link TokenizerFactory} for the given RegExps.

src/index.ts#L19

To capture, RegExps must use a named group.

const tokenize = createTokenizer(
  /(?<ident>[a-z]+)/, // named groups determine token `group`
  /(?<number>[0-9]+)/
)

# regexps – RegExps to match.

RegExp []

createTokenizer(regexps)  =>

TokenizerFactory

Credits

• match-to-token by stagas – transform a RegExp named group match to a more useful object

Contributing

Fork or edit and submit a PR.

All contributions are welcome!

License

MIT © 2022 stagas