Overview 
A regex that tokenizes JavaScript.
var jsTokens = require("js-tokens").default
var jsString = "var foo=opts.foo;\n..."
jsString.match(jsTokens)
// ["var", " ", "foo", "=", "opts", ".", "foo", ";", "\n", ...]
Installation
npm install js-tokens
import jsTokens from "js-tokens"
// or:
var jsTokens = require("js-tokens").default
Usage
jsTokens
A regex with the g flag that matches JavaScript tokens.
The regex always matches, even invalid JavaScript and the empty string.
The next match is always directly after the previous.
var token = matchToToken(match)
import {matchToToken} from "js-tokens"
// or:
var matchToToken = require("js-tokens").matchToToken
Takes a match returned by jsTokens.exec(string), and returns a {type: String, value: String} object. The following types are available:
- string
- comment
- regex
- number
- name
- punctuator
- whitespace
- invalid
Multi-line comments and strings also have a closed property indicating if the
token was closed or not (see below).
Comments and strings both come in several flavors. To distinguish them, check if
the token starts with //, /*, ', " or `.
Names are ECMAScript IdentifierNames, that is, including both identifiers and keywords. You may use is-keyword-js to tell them apart.
Whitespace includes both line terminators and other whitespace.
ECMAScript support
The intention is to always support the latest ECMAScript version whose feature set has been finalized.
If adding support for a newer version requires changes, a new version with a major verion bump will be released.
Currently, ECMAScript 2018 is supported.
Invalid code handling
Unterminated strings are still matched as strings. JavaScript strings cannot contain (unescaped) newlines, so unterminated strings simply end at the end of the line. Unterminated template strings can contain unescaped newlines, though, so they go on to the end of input.
Unterminated multi-line comments are also still matched as comments. They simply go on to the end of the input.
Unterminated regex literals are likely matched as division and whatever is inside the regex.
Invalid ASCII characters have their own capturing group.
Invalid non-ASCII characters are treated as names, to simplify the matching of names (except unicode spaces which are treated as whitespace). Note: See also the ES2018 section.
Regex literals may contain invalid regex syntax. They are still matched as regex literals. They may also contain repeated regex flags, to keep the regex simple.
Strings may contain invalid escape sequences.
Limitations
Tokenizing JavaScript using regexes—in fact, one single regex—won’t be perfect. But that’s not the point either.
You may compare jsTokens with esprima by using esprima-compare.js.
See npm run esprima-compare!
Template string interpolation
Template strings are matched as single tokens, from the starting ` to the
ending `, including interpolations (whose tokens are not matched
individually).
Matching template string interpolations requires recursive balancing of { and
}—something that JavaScript regexes cannot do. Only one level of nesting is
supported.
Division and regex literals collision
Consider this example:
var g = 9.82
var number = bar / 2/g
var regex = / 2/g
A human can easily understand that in the number line we’re dealing with
division, and in the regex line we’re dealing with a regex literal. How come?
Because humans can look at the whole code to put the / characters in context.
A JavaScript regex cannot. It only sees forwards. (Well, ES2018 regexes can also
look backwards. See the ES2018 section).
When the jsTokens regex scans throught the above, it will see the following
at the end of both the number and regex rows:
/ 2/g
It is then impossible to know if that is a regex literal, or part of an expression dealing with division.
Here is a similar case:
foo /= 2/g
foo(/= 2/g)
The first line divides the foo variable with 2/g. The second line calls the
foo function with the regex literal /= 2/g. Again, since jsTokens only
sees forwards, it cannot tell the two cases apart.
There are some cases where we can tell division and regex literals apart, though.
First off, we have the simple cases where there’s only one slash in the line:
var foo = 2/g
foo /= 2
Regex literals cannot contain newlines, so the above cases are correctly identified as division. Things are only problematic when there are more than one non-comment slash in a single line.
Secondly, not every character is a valid regex flag.
var number = bar / 2/e
The above example is also correctly identifie