Skip to content

Use parser to parse tsconfig json instead of using Json.parse #12336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 34 commits into from
Jun 15, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
cca98c3
Parse json using our own parser
sheetalkamat Nov 11, 2016
b3f816b
Reorganize functions so commandline parser can include parser.
sheetalkamat Nov 14, 2016
0b3074f
Keep the original api and add new api that handles the JsonNode
sheetalkamat Nov 16, 2016
25be912
Merge branch 'master' into ownJsonParsing
sheetalkamat Nov 23, 2016
005123f
Handle typingOptions to typeAcquisition rename when converting parsed…
sheetalkamat Nov 23, 2016
6419de9
Merge branch 'master' into ownJsonParsing
sheetalkamat Nov 23, 2016
0dd944a
Clean up api so that parsing returns JsonSourceFile
sheetalkamat Nov 24, 2016
c7744a8
Merge branch 'master' into ownJsonParsing
sheetalkamat Nov 30, 2016
f83e754
Merge branch 'master' into ownJsonParsing
sheetalkamat Dec 15, 2016
236e15a
Merge branch 'master' into ownJsonParsing
sheetalkamat Dec 21, 2016
0bbbc51
Merge branch 'master' into ownJsonParsing
sheetalkamat Jan 17, 2017
d3ce95f
Merge branch 'master' into ownJsonParsing
sheetalkamat Jan 23, 2017
911511e
Report option diagnostic in tsconfig.json if possible
sheetalkamat Nov 29, 2016
98bd310
Update the protocol to return file name in compiler diagnostics and p…
sheetalkamat Nov 29, 2016
cb1b164
Encorporated PR feedback
sheetalkamat Feb 8, 2017
6f568b3
Merge branch 'master' into ownJsonParsing
sheetalkamat Apr 17, 2017
d7e9609
Merge branch 'master' into ownJsonParsing
sheetalkamat May 11, 2017
b60de87
Merge branch 'master' into ownJsonParsing
sheetalkamat May 11, 2017
c97b389
Simplifying the json conversion notifier
sheetalkamat May 11, 2017
ec2f967
Merge branch 'master' into ownJsonParsing
sheetalkamat May 15, 2017
ea60e99
Get configFiles as part of file names
sheetalkamat May 15, 2017
7cf93f9
Report config file errors as part of syntactic diagnostic on the file
sheetalkamat May 23, 2017
20570b0
Merge pull request #15867 from Microsoft/configFileInListOfProjectFiles
sheetalkamat May 23, 2017
f1ea38d
Merge branch 'master' into ownJsonParsing
sheetalkamat May 24, 2017
5501892
Fix the build break of typings installer
sheetalkamat May 24, 2017
16fd947
Reorganized code to keep createBundle and updateBundle in factory.ts
sheetalkamat May 26, 2017
7bd9e09
Make configFile on compiler options as non enumerable
sheetalkamat May 27, 2017
7db76ed
Always return empty object when converting the json sourcefile into a…
sheetalkamat May 27, 2017
8d771ca
Merge branch 'master' into ownJsonParsing
sheetalkamat May 30, 2017
8f4f6c5
Merge branch 'master' into ownJsonParsing
sheetalkamat Jun 1, 2017
d680720
Merge branch 'master' into ownJsonParsing
sheetalkamat Jun 5, 2017
c4ad151
Merge branch 'master' into ownJsonParsing
sheetalkamat Jun 6, 2017
48189c8
Merge branch 'master' into ownJsonParsing
sheetalkamat Jun 6, 2017
09f0b34
Merge branch 'master' into ownJsonParsing
sheetalkamat Jun 15, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions src/compiler/checker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@ namespace ts {
return symbol.id;
}

export function isInstantiatedModule(node: ModuleDeclaration, preserveConstEnums: boolean) {
const moduleState = getModuleInstanceState(node);
return moduleState === ModuleInstanceState.Instantiated ||
(preserveConstEnums && moduleState === ModuleInstanceState.ConstEnumOnly);
}

export function createTypeChecker(host: TypeCheckerHost, produceDiagnostics: boolean): TypeChecker {
// Cancellation that controls whether or not we can cancel in the middle of type checking.
// In general cancelling is *not* safe for the type checker. We might be in the middle of
Expand Down
780 changes: 624 additions & 156 deletions src/compiler/commandLineParser.ts

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions src/compiler/core.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2139,6 +2139,8 @@ namespace ts {
return ScriptKind.TS;
case Extension.Tsx:
return ScriptKind.TSX;
case ".json":
return ScriptKind.JSON;
default:
return ScriptKind.Unknown;
}
Expand Down
10 changes: 9 additions & 1 deletion src/compiler/diagnosticMessages.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
{
"Unterminated string literal.": {
"category": "Error",
"code": 1002
Expand Down Expand Up @@ -899,6 +899,14 @@
"category": "Error",
"code": 1326
},
"String literal with double quotes expected.": {
"category": "Error",
"code": 1327
},
"Property value can only be string literal, numeric literal, 'true', 'false', 'null', object literal or array literal.": {
"category": "Error",
"code": 1328
},

"Duplicate identifier '{0}'.": {
"category": "Error",
Expand Down
64 changes: 64 additions & 0 deletions src/compiler/emitter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,70 @@ namespace ts {
const delimiters = createDelimiterMap();
const brackets = createBracketsMap();

/*@internal*/
/**
* Iterates over the source files that are expected to have an emit output.
*
* @param host An EmitHost.
* @param action The action to execute.
* @param sourceFilesOrTargetSourceFile
* If an array, the full list of source files to emit.
* Else, calls `getSourceFilesToEmit` with the (optional) target source file to determine the list of source files to emit.
*/
export function forEachEmittedFile(
host: EmitHost, action: (emitFileNames: EmitFileNames, sourceFileOrBundle: SourceFile | Bundle, emitOnlyDtsFiles: boolean) => void,
sourceFilesOrTargetSourceFile?: SourceFile[] | SourceFile,
emitOnlyDtsFiles?: boolean) {

const sourceFiles = isArray(sourceFilesOrTargetSourceFile) ? sourceFilesOrTargetSourceFile : getSourceFilesToEmit(host, sourceFilesOrTargetSourceFile);
const options = host.getCompilerOptions();
if (options.outFile || options.out) {
if (sourceFiles.length) {
const jsFilePath = options.outFile || options.out;
const sourceMapFilePath = getSourceMapFilePath(jsFilePath, options);
const declarationFilePath = options.declaration ? removeFileExtension(jsFilePath) + Extension.Dts : "";
action({ jsFilePath, sourceMapFilePath, declarationFilePath }, createBundle(sourceFiles), emitOnlyDtsFiles);
}
}
else {
for (const sourceFile of sourceFiles) {
const jsFilePath = getOwnEmitOutputFilePath(sourceFile, host, getOutputExtension(sourceFile, options));
const sourceMapFilePath = getSourceMapFilePath(jsFilePath, options);
const declarationFilePath = !isSourceFileJavaScript(sourceFile) && (emitOnlyDtsFiles || options.declaration) ? getDeclarationEmitOutputFilePath(sourceFile, host) : undefined;
action({ jsFilePath, sourceMapFilePath, declarationFilePath }, sourceFile, emitOnlyDtsFiles);
}
}
}

function getSourceMapFilePath(jsFilePath: string, options: CompilerOptions) {
return options.sourceMap ? jsFilePath + ".map" : undefined;
}

// JavaScript files are always LanguageVariant.JSX, as JSX syntax is allowed in .js files also.
// So for JavaScript files, '.jsx' is only emitted if the input was '.jsx', and JsxEmit.Preserve.
// For TypeScript, the only time to emit with a '.jsx' extension, is on JSX input, and JsxEmit.Preserve
function getOutputExtension(sourceFile: SourceFile, options: CompilerOptions): Extension {
if (options.jsx === JsxEmit.Preserve) {
if (isSourceFileJavaScript(sourceFile)) {
if (fileExtensionIs(sourceFile.fileName, Extension.Jsx)) {
return Extension.Jsx;
}
}
else if (sourceFile.languageVariant === LanguageVariant.JSX) {
// TypeScript source file preserving JSX syntax
return Extension.Jsx;
}
}
return Extension.Js;
}

function getOriginalSourceFileOrBundle(sourceFileOrBundle: SourceFile | Bundle) {
if (sourceFileOrBundle.kind === SyntaxKind.Bundle) {
return updateBundle(sourceFileOrBundle, sameMap(sourceFileOrBundle.sourceFiles, getOriginalSourceFile));
}
return getOriginalSourceFile(sourceFileOrBundle);
}

/*@internal*/
// targetSourceFile is when users only want one file in entire project to be emitted. This is used in compileOnSave feature
export function emitFiles(resolver: EmitResolver, host: EmitHost, targetSourceFile: SourceFile, emitOnlyDtsFiles?: boolean, transformers?: TransformerFactory<SourceFile>[]): EmitResult {
Expand Down
191 changes: 0 additions & 191 deletions src/compiler/factory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2287,14 +2287,6 @@ namespace ts {
return range;
}

/**
* Gets flags that control emit behavior of a node.
*/
export function getEmitFlags(node: Node): EmitFlags | undefined {
const emitNode = node.emitNode;
return emitNode && emitNode.flags;
}

/**
* Sets flags that control emit behavior of a node.
*/
Expand Down Expand Up @@ -3809,16 +3801,6 @@ namespace ts {
return node;
}

export function skipPartiallyEmittedExpressions(node: Expression): Expression;
export function skipPartiallyEmittedExpressions(node: Node): Node;
export function skipPartiallyEmittedExpressions(node: Node) {
while (node.kind === SyntaxKind.PartiallyEmittedExpression) {
node = (<PartiallyEmittedExpression>node).expression;
}

return node;
}

function updateOuterExpression(outerExpression: OuterExpression, expression: Expression) {
switch (outerExpression.kind) {
case SyntaxKind.ParenthesizedExpression: return updateParen(outerExpression, expression);
Expand Down Expand Up @@ -4239,177 +4221,4 @@ namespace ts {
Debug.assertNode(node, isExpression);
return <Expression>node;
}

export interface ExternalModuleInfo {
externalImports: (ImportDeclaration | ImportEqualsDeclaration | ExportDeclaration)[]; // imports of other external modules
externalHelpersImportDeclaration: ImportDeclaration | undefined; // import of external helpers
exportSpecifiers: Map<ExportSpecifier[]>; // export specifiers by name
exportedBindings: Identifier[][]; // exported names of local declarations
exportedNames: Identifier[]; // all exported names local to module
exportEquals: ExportAssignment | undefined; // an export= declaration if one was present
hasExportStarsToExportValues: boolean; // whether this module contains export*
}

export function collectExternalModuleInfo(sourceFile: SourceFile, resolver: EmitResolver, compilerOptions: CompilerOptions): ExternalModuleInfo {
const externalImports: (ImportDeclaration | ImportEqualsDeclaration | ExportDeclaration)[] = [];
const exportSpecifiers = createMultiMap<ExportSpecifier>();
const exportedBindings: Identifier[][] = [];
const uniqueExports = createMap<boolean>();
let exportedNames: Identifier[];
let hasExportDefault = false;
let exportEquals: ExportAssignment = undefined;
let hasExportStarsToExportValues = false;

for (const node of sourceFile.statements) {
switch (node.kind) {
case SyntaxKind.ImportDeclaration:
// import "mod"
// import x from "mod"
// import * as x from "mod"
// import { x, y } from "mod"
externalImports.push(<ImportDeclaration>node);
break;

case SyntaxKind.ImportEqualsDeclaration:
if ((<ImportEqualsDeclaration>node).moduleReference.kind === SyntaxKind.ExternalModuleReference) {
// import x = require("mod")
externalImports.push(<ImportEqualsDeclaration>node);
}

break;

case SyntaxKind.ExportDeclaration:
if ((<ExportDeclaration>node).moduleSpecifier) {
if (!(<ExportDeclaration>node).exportClause) {
// export * from "mod"
externalImports.push(<ExportDeclaration>node);
hasExportStarsToExportValues = true;
}
else {
// export { x, y } from "mod"
externalImports.push(<ExportDeclaration>node);
}
}
else {
// export { x, y }
for (const specifier of (<ExportDeclaration>node).exportClause.elements) {
if (!uniqueExports.get(specifier.name.text)) {
const name = specifier.propertyName || specifier.name;
exportSpecifiers.add(name.text, specifier);

const decl = resolver.getReferencedImportDeclaration(name)
|| resolver.getReferencedValueDeclaration(name);

if (decl) {
multiMapSparseArrayAdd(exportedBindings, getOriginalNodeId(decl), specifier.name);
}

uniqueExports.set(specifier.name.text, true);
exportedNames = append(exportedNames, specifier.name);
}
}
}
break;

case SyntaxKind.ExportAssignment:
if ((<ExportAssignment>node).isExportEquals && !exportEquals) {
// export = x
exportEquals = <ExportAssignment>node;
}
break;

case SyntaxKind.VariableStatement:
if (hasModifier(node, ModifierFlags.Export)) {
for (const decl of (<VariableStatement>node).declarationList.declarations) {
exportedNames = collectExportedVariableInfo(decl, uniqueExports, exportedNames);
}
}
break;

case SyntaxKind.FunctionDeclaration:
if (hasModifier(node, ModifierFlags.Export)) {
if (hasModifier(node, ModifierFlags.Default)) {
// export default function() { }
if (!hasExportDefault) {
multiMapSparseArrayAdd(exportedBindings, getOriginalNodeId(node), getDeclarationName(<FunctionDeclaration>node));
hasExportDefault = true;
}
}
else {
// export function x() { }
const name = (<FunctionDeclaration>node).name;
if (!uniqueExports.get(name.text)) {
multiMapSparseArrayAdd(exportedBindings, getOriginalNodeId(node), name);
uniqueExports.set(name.text, true);
exportedNames = append(exportedNames, name);
}
}
}
break;

case SyntaxKind.ClassDeclaration:
if (hasModifier(node, ModifierFlags.Export)) {
if (hasModifier(node, ModifierFlags.Default)) {
// export default class { }
if (!hasExportDefault) {
multiMapSparseArrayAdd(exportedBindings, getOriginalNodeId(node), getDeclarationName(<ClassDeclaration>node));
hasExportDefault = true;
}
}
else {
// export class x { }
const name = (<ClassDeclaration>node).name;
if (!uniqueExports.get(name.text)) {
multiMapSparseArrayAdd(exportedBindings, getOriginalNodeId(node), name);
uniqueExports.set(name.text, true);
exportedNames = append(exportedNames, name);
}
}
}
break;
}
}

const externalHelpersModuleName = getOrCreateExternalHelpersModuleNameIfNeeded(sourceFile, compilerOptions, hasExportStarsToExportValues);
const externalHelpersImportDeclaration = externalHelpersModuleName && createImportDeclaration(
/*decorators*/ undefined,
/*modifiers*/ undefined,
createImportClause(/*name*/ undefined, createNamespaceImport(externalHelpersModuleName)),
createLiteral(externalHelpersModuleNameText));

if (externalHelpersImportDeclaration) {
externalImports.unshift(externalHelpersImportDeclaration);
}

return { externalImports, exportSpecifiers, exportEquals, hasExportStarsToExportValues, exportedBindings, exportedNames, externalHelpersImportDeclaration };
}

function collectExportedVariableInfo(decl: VariableDeclaration | BindingElement, uniqueExports: Map<boolean>, exportedNames: Identifier[]) {
if (isBindingPattern(decl.name)) {
for (const element of decl.name.elements) {
if (!isOmittedExpression(element)) {
exportedNames = collectExportedVariableInfo(element, uniqueExports, exportedNames);
}
}
}
else if (!isGeneratedIdentifier(decl.name)) {
if (!uniqueExports.get(decl.name.text)) {
uniqueExports.set(decl.name.text, true);
exportedNames = append(exportedNames, decl.name);
}
}
return exportedNames;
}

/** Use a sparse array as a multi-map. */
function multiMapSparseArrayAdd<V>(map: V[][], key: number, value: V): V[] {
let values = map[key];
if (values) {
values.push(value);
}
else {
map[key] = values = [value];
}
return values;
}
}
39 changes: 36 additions & 3 deletions src/compiler/parser.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
/// <reference path="utilities.ts"/>
/// <reference path="scanner.ts"/>
/// <reference path="factory.ts"/>

namespace ts {
let NodeConstructor: new (kind: SyntaxKind, pos: number, end: number) => Node;
Expand Down Expand Up @@ -466,6 +465,15 @@ namespace ts {
return Parser.parseIsolatedEntityName(text, languageVersion);
}

/**
* Parse json text into SyntaxTree and return node and parse errors if any
* @param fileName
* @param sourceText
*/
export function parseJsonText(fileName: string, sourceText: string): JsonSourceFile {
return Parser.parseJsonText(fileName, sourceText);
}

// See also `isExternalOrCommonJsModule` in utilities.ts
export function isExternalModule(file: SourceFile): boolean {
return file.externalModuleIndicator !== undefined;
Expand Down Expand Up @@ -632,9 +640,34 @@ namespace ts {
return isInvalid ? entityName : undefined;
}

export function parseJsonText(fileName: string, sourceText: string): JsonSourceFile {
initializeState(sourceText, ScriptTarget.ES2015, /*syntaxCursor*/ undefined, ScriptKind.JSON);
// Set source file so that errors will be reported with this file name
sourceFile = createSourceFile(fileName, ScriptTarget.ES2015, ScriptKind.JSON);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably shouldn't use createSourceFile, since that initializes a lot of variables these will never use.

const result = <JsonSourceFile>sourceFile;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Be sure to use result instead of sourceFile in property assignments as it is correctly typed.


// Prime the scanner.
nextToken();
if (token() === SyntaxKind.EndOfFileToken) {
sourceFile.endOfFileToken = <EndOfFileToken>parseTokenNode();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need an EndOfFileToken?

}
else if (token() === SyntaxKind.OpenBraceToken ||
lookAhead(() => token() === SyntaxKind.StringLiteral)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this happen often? i am assuming this is to handle [ "compilerOptions" : ..., but do we need to ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to handle one of the failing test case which was used to sanitize the tsconfig.json

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you give a comment why we need a look ahead to be string literal? also would the the property is allow to be identifier since we use parseObjectLiteralExpression to parse the object?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to ECMA-404, the key for the property of a JSON object must be a string literal.

result.jsonObject = parseObjectLiteralExpression();
sourceFile.endOfFileToken = parseExpectedToken(SyntaxKind.EndOfFileToken, /*reportAtCurrentPosition*/ false, Diagnostics.Unexpected_token);
}
else {
parseExpected(SyntaxKind.OpenBraceToken);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are only planning on using this to parse tsconfig.json, then this is fine. If we are planning on using this to parse other JSON files, there are other legal root objects for a JSON file (e.g. Array Literals, string literals, numeric literals, true, false, and null).

}

sourceFile.parseDiagnostics = parseDiagnostics;
clearState();
return result;
}

function getLanguageVariant(scriptKind: ScriptKind) {
// .tsx and .jsx files are treated as jsx language variant.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment probably should be removed

return scriptKind === ScriptKind.TSX || scriptKind === ScriptKind.JSX || scriptKind === ScriptKind.JS ? LanguageVariant.JSX : LanguageVariant.Standard;
return scriptKind === ScriptKind.TSX || scriptKind === ScriptKind.JSX || scriptKind === ScriptKind.JS || scriptKind === ScriptKind.JSON ? LanguageVariant.JSX : LanguageVariant.Standard;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JSON files can't contain JSX anyway.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we LanguageVariant.JSX be renamed ? since it is more than just TSX and JSX

}

function initializeState(_sourceText: string, languageVersion: ScriptTarget, _syntaxCursor: IncrementalParser.SyntaxCursor, scriptKind: ScriptKind) {
Expand All @@ -652,7 +685,7 @@ namespace ts {
identifierCount = 0;
nodeCount = 0;

contextFlags = scriptKind === ScriptKind.JS || scriptKind === ScriptKind.JSX ? NodeFlags.JavaScriptFile : NodeFlags.None;
contextFlags = scriptKind === ScriptKind.JS || scriptKind === ScriptKind.JSX || scriptKind === ScriptKind.JSON ? NodeFlags.JavaScriptFile : NodeFlags.None;
parseErrorBeforeNextFinishedNode = false;

// Initialize and prime the scanner before parsing the source elements.
Expand Down
Loading