Skip to content

Reuse program structure #3616

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
Closed

Reuse program structure #3616

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
226deec
reuse structure of the program if changes in files don't affect impor…
vladima Jun 23, 2015
39e832d
use existing information about module resolutions
vladima Jun 23, 2015
7a7d775
Merge remote-tracking branch 'origin/master' into reuseProgramStructure
vladima Jun 23, 2015
ba3eb0d
added Program.structureIsReused property, disallow reuse if target mo…
vladima Jun 24, 2015
16deccd
revert unintentional change
vladima Jun 24, 2015
9e81ac9
merge with master
vladima Jun 24, 2015
df508de
fix formatting in parser
vladima Jun 24, 2015
c968b36
addressed PR feedback
vladima Jun 25, 2015
66f6736
addressed PR feedback
vladima Jun 25, 2015
2685d40
addressed PR feedback
vladima Jul 9, 2015
6a502cd
Merge remote-tracking branch 'origin/master' into reuseProgramStructure
vladima Jul 9, 2015
e15c700
clean old program to prevent it from being captured into the closure
vladima Jul 9, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Jakefile.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,8 @@ var harnessSources = [
"services/patternMatcher.ts",
"versionCache.ts",
"convertToBase64.ts",
"transpile.ts"
"transpile.ts",
"reuseProgramStructure.ts"
].map(function (f) {
return path.join(unittestsDirectory, f);
})).concat([
Expand Down
21 changes: 6 additions & 15 deletions src/compiler/checker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -929,28 +929,19 @@ namespace ts {
// Escape the name in the "require(...)" clause to ensure we find the right symbol.
let moduleName = escapeIdentifier(moduleReferenceLiteral.text);

if (!moduleName) return;
if (!moduleName) {
return;
}
let isRelative = isExternalModuleNameRelative(moduleName);
if (!isRelative) {
let symbol = getSymbol(globals, '"' + moduleName + '"', SymbolFlags.ValueModule);
if (symbol) {
return symbol;
}
}
let fileName: string;
let sourceFile: SourceFile;
while (true) {
fileName = normalizePath(combinePaths(searchPath, moduleName));
sourceFile = forEach(supportedExtensions, extension => host.getSourceFile(fileName + extension));
if (sourceFile || isRelative) {
break;
}
let parentPath = getDirectoryPath(searchPath);
if (parentPath === searchPath) {
break;
}
searchPath = parentPath;
}

let fileName = getResolvedModuleFileName(getSourceFile(location), moduleReferenceLiteral.text);
let sourceFile = fileName && host.getSourceFile(fileName);
if (sourceFile) {
if (sourceFile.symbol) {
return sourceFile.symbol;
Expand Down
254 changes: 197 additions & 57 deletions src/compiler/program.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ namespace ts {

/** The version of the TypeScript compiler release */
export const version = "1.5.3";


var emptyArray: any[] = [];

export function findConfigFile(searchPath: string): string {
var fileName = "tsconfig.json";
while (true) {
Expand Down Expand Up @@ -143,7 +145,7 @@ namespace ts {
}
}

export function createProgram(rootNames: string[], options: CompilerOptions, host?: CompilerHost): Program {
export function createProgram(rootNames: string[], options: CompilerOptions, host?: CompilerHost, oldProgram?: Program): Program {
let program: Program;
let files: SourceFile[] = [];
let diagnostics = createDiagnosticCollection();
Expand All @@ -160,22 +162,39 @@ namespace ts {
host = host || createCompilerHost(options);

let filesByName = createFileMap<SourceFile>(fileName => host.getCanonicalFileName(fileName));

forEach(rootNames, name => processRootFile(name, /*isDefaultLib:*/ false));

// Do not process the default library if:
// - The '--noLib' flag is used.
// - A 'no-default-lib' reference comment is encountered in
// processing the root files.
if (!skipDefaultLib) {
processRootFile(host.getDefaultLibFileName(options), /*isDefaultLib:*/ true);

if (oldProgram) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just add an explanatory comment. Something to give hte intuition for why you are checking these specific options, and the types of options you'd put here in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

// check properties that can affect structure of the program or module resolution strategy
// if any of these properties has changed - structure cannot be reused
let oldOptions = oldProgram.getCompilerOptions();
if ((oldOptions.module !== options.module) ||
(oldOptions.noResolve !== options.noResolve) ||
(oldOptions.target !== options.target) ||
(oldOptions.noLib !== options.noLib)) {
oldProgram = undefined;
}
}

if (!tryReuseStructureFromOldProgram()) {
forEach(rootNames, name => processRootFile(name, false));
// Do not process the default library if:
// - The '--noLib' flag is used.
// - A 'no-default-lib' reference comment is encountered in
// processing the root files.
if (!skipDefaultLib) {
processRootFile(host.getDefaultLibFileName(options), true);
}
}

verifyCompilerOptions();

// unconditionally set oldProgram to undefined to prevent it from being captured in closure
oldProgram = undefined;

programTime += new Date().getTime() - start;

program = {
getRootFileNames: () => rootNames,
getSourceFile: getSourceFile,
getSourceFiles: () => files,
getCompilerOptions: () => options,
Expand Down Expand Up @@ -211,6 +230,70 @@ namespace ts {
return classifiableNames;
}

function tryReuseStructureFromOldProgram(): boolean {
if (!oldProgram) {
return false;
}

Debug.assert(!oldProgram.structureIsReused);

// there is an old program, check if we can reuse its structure
let oldRootNames = oldProgram.getRootFileNames();
if (!arrayIsEqualTo(oldRootNames, rootNames)) {
return false;
}

// check if program source files has changed in the way that can affect structure of the program
let newSourceFiles: SourceFile[] = [];
for (let oldSourceFile of oldProgram.getSourceFiles()) {
let newSourceFile = host.getSourceFile(oldSourceFile.fileName, options.target);
if (!newSourceFile) {
return false;
}

if (oldSourceFile !== newSourceFile) {
if (oldSourceFile.hasNoDefaultLib !== newSourceFile.hasNoDefaultLib) {
// value of no-default-lib has changed
// this will affect if default library is injected into the list of files
return false;
}

// check tripleslash references
if (!arrayIsEqualTo(oldSourceFile.referencedFiles, newSourceFile.referencedFiles, fileReferenceIsEqualTo)) {
// tripleslash references has changed
return false;
}

// check imports
collectExternalModuleReferences(newSourceFile);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this step seems odd. Are the imports not automatically collected?

if (!arrayIsEqualTo(oldSourceFile.imports, newSourceFile.imports, moduleNameIsEqualTo)) {
// imports has changed
return false;
}
// pass the cache of module resolutions from the old source file
newSourceFile.resolvedModules = oldSourceFile.resolvedModules;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you may have to check for no-default-lib.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

}
else {
// file has no changes - use it as is
newSourceFile = oldSourceFile;
}

// if file has passed all checks it should be safe to reuse it
newSourceFiles.push(newSourceFile);
}

// update fileName -> file mapping
for (let file of newSourceFiles) {
filesByName.set(file.fileName, file);
}

files = newSourceFiles;

oldProgram.structureIsReused = true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this give us at this point? we have copied the data, there should not be any dependency, correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this flag is currently used only in tests


return true;
}

function getEmitHost(writeFileCallback?: WriteFileCallback): EmitHost {
return {
getCanonicalFileName: fileName => host.getCanonicalFileName(fileName),
Expand Down Expand Up @@ -370,6 +453,62 @@ namespace ts {

function processRootFile(fileName: string, isDefaultLib: boolean) {
processSourceFile(normalizePath(fileName), isDefaultLib);
}

function fileReferenceIsEqualTo(a: FileReference, b: FileReference): boolean {
return a.fileName === b.fileName;
}

function moduleNameIsEqualTo(a: LiteralExpression, b: LiteralExpression): boolean {
return a.text === b.text;
}

function collectExternalModuleReferences(file: SourceFile): void {
if (file.imports) {
return;
}

let imports: LiteralExpression[];
for (let node of file.statements) {
switch (node.kind) {
case SyntaxKind.ImportDeclaration:
case SyntaxKind.ImportEqualsDeclaration:
case SyntaxKind.ExportDeclaration:
let moduleNameExpr = getExternalModuleName(node);
if (!moduleNameExpr || moduleNameExpr.kind !== SyntaxKind.StringLiteral) {
break;
}
if (!(<LiteralExpression>moduleNameExpr).text) {
break;
}

(imports || (imports = [])).push(<LiteralExpression>moduleNameExpr);
break;
case SyntaxKind.ModuleDeclaration:
if ((<ModuleDeclaration>node).name.kind === SyntaxKind.StringLiteral && (node.flags & NodeFlags.Ambient || isDeclarationFile(file))) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that this part has been around already before your change, but i think we do not need the fork. we just need a helper, given a node find all the external module references inside, if we find an external module declaration recursively call the same routine on the module body.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer to make it a separte PR

// TypeScript 1.0 spec (April 2014): 12.1.6
// An AmbientExternalModuleDeclaration declares an external module.
// This type of declaration is permitted only in the global module.
// The StringLiteral must specify a top - level external module name.
// Relative external module names are not permitted
forEachChild((<ModuleDeclaration>node).body, node => {
if (isExternalModuleImportEqualsDeclaration(node) &&
getExternalModuleImportEqualsDeclarationExpression(node).kind === SyntaxKind.StringLiteral) {
let moduleName = <LiteralExpression>getExternalModuleImportEqualsDeclarationExpression(node);
// TypeScript 1.0 spec (April 2014): 12.1.6
// An ExternalImportDeclaration in anAmbientExternalModuleDeclaration may reference other external modules
// only through top - level external module names. Relative external module names are not permitted.
if (moduleName) {
(imports || (imports = [])).push(moduleName);
}
}
});
}
break;
}
}

file.imports = imports || emptyArray;
}

function processSourceFile(fileName: string, isDefaultLib: boolean, refFile?: SourceFile, refPos?: number, refEnd?: number) {
Expand Down Expand Up @@ -487,57 +626,58 @@ namespace ts {
processSourceFile(normalizePath(referencedFileName), /* isDefaultLib */ false, file, ref.pos, ref.end);
});
}

function processImportedModules(file: SourceFile, basePath: string) {
forEach(file.statements, node => {
if (node.kind === SyntaxKind.ImportDeclaration || node.kind === SyntaxKind.ImportEqualsDeclaration || node.kind === SyntaxKind.ExportDeclaration) {
let moduleNameExpr = getExternalModuleName(node);
if (moduleNameExpr && moduleNameExpr.kind === SyntaxKind.StringLiteral) {
let moduleNameText = (<LiteralExpression>moduleNameExpr).text;
if (moduleNameText) {
let searchPath = basePath;
let searchName: string;
while (true) {
searchName = normalizePath(combinePaths(searchPath, moduleNameText));
if (forEach(supportedExtensions, extension => findModuleSourceFile(searchName + extension, moduleNameExpr))) {
break;
}
let parentPath = getDirectoryPath(searchPath);
if (parentPath === searchPath) {
break;
}
searchPath = parentPath;
}
}
}
}
else if (node.kind === SyntaxKind.ModuleDeclaration && (<ModuleDeclaration>node).name.kind === SyntaxKind.StringLiteral && (node.flags & NodeFlags.Ambient || isDeclarationFile(file))) {
// TypeScript 1.0 spec (April 2014): 12.1.6
// An AmbientExternalModuleDeclaration declares an external module.
// This type of declaration is permitted only in the global module.
// The StringLiteral must specify a top - level external module name.
// Relative external module names are not permitted
forEachChild((<ModuleDeclaration>node).body, node => {
if (isExternalModuleImportEqualsDeclaration(node) &&
getExternalModuleImportEqualsDeclarationExpression(node).kind === SyntaxKind.StringLiteral) {

let nameLiteral = <LiteralExpression>getExternalModuleImportEqualsDeclarationExpression(node);
let moduleName = nameLiteral.text;
if (moduleName) {
// TypeScript 1.0 spec (April 2014): 12.1.6
// An ExternalImportDeclaration in anAmbientExternalModuleDeclaration may reference other external modules
// only through top - level external module names. Relative external module names are not permitted.
let searchName = normalizePath(combinePaths(basePath, moduleName));
forEach(supportedExtensions, extension => findModuleSourceFile(searchName + extension, nameLiteral));
}
}
});

function processImportedModules(file: SourceFile, basePath: string) {
collectExternalModuleReferences(file);
if (file.imports.length) {
file.resolvedModules = {};
let oldSourceFile = oldProgram && oldProgram.getSourceFile(file.fileName);
for (let moduleName of file.imports) {
resolveModule(moduleName, oldSourceFile && oldSourceFile.resolvedModules);
}
});

}
else {
// no imports - drop cached module resolutions
file.resolvedModules = undefined;
}
return;

function findModuleSourceFile(fileName: string, nameLiteral: Expression) {
return findSourceFile(fileName, /* isDefaultLib */ false, file, nameLiteral.pos, nameLiteral.end - nameLiteral.pos);
}

function resolveModule(moduleNameExpr: LiteralExpression, existingResolutions: Map<string>): void {
let searchPath = basePath;
let searchName: string;

if (existingResolutions && hasProperty(existingResolutions, moduleNameExpr.text)) {
let fileName = existingResolutions[moduleNameExpr.text];
// use existing resolution
setResolvedModuleName(file, moduleNameExpr.text, fileName);
if (fileName) {
findModuleSourceFile(fileName, moduleNameExpr);
}
return;
}

while (true) {
searchName = normalizePath(combinePaths(searchPath, moduleNameExpr.text));
let referencedSourceFile = forEach(supportedExtensions, extension => findModuleSourceFile(searchName + extension, moduleNameExpr));
if (referencedSourceFile) {
setResolvedModuleName(file, moduleNameExpr.text, referencedSourceFile.fileName);
return;
}

let parentPath = getDirectoryPath(searchPath);
if (parentPath === searchPath) {
break;
}
searchPath = parentPath;
}
// mark reference as non-resolved
setResolvedModuleName(file, moduleNameExpr.text, undefined);
}
}

function computeCommonSourceDirectory(sourceFiles: SourceFile[]): string {
Expand Down
14 changes: 13 additions & 1 deletion src/compiler/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1272,8 +1272,11 @@ namespace ts {
// Stores a line map for the file.
// This field should never be used directly to obtain line map, use getLineMap function instead.
/* @internal */ lineMap: number[];

/* @internal */ classifiableNames?: Map<string>;
// Stores a mapping 'external module reference text' -> 'resolved file name' | undefined
// Content of this fiels should never be used directly - use getResolvedModuleFileName/setResolvedModuleFileName functions instead
/* @internal */ resolvedModules: Map<string>;
/* @internal */ imports: LiteralExpression[];
}

export interface ScriptReferenceHost {
Expand All @@ -1300,6 +1303,12 @@ namespace ts {
}

export interface Program extends ScriptReferenceHost {

/**
* Get a list of root file names that were passed to a 'createProgram'
*/
getRootFileNames(): string[]

/**
* Get a list of files in the program
*/
Expand Down Expand Up @@ -1340,6 +1349,9 @@ namespace ts {
/* @internal */ getIdentifierCount(): number;
/* @internal */ getSymbolCount(): number;
/* @internal */ getTypeCount(): number;

// For testing purposes only.
/* @internal */ structureIsReused?: boolean;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment that this is for tests ony.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

}

export interface SourceMapSpan {
Expand Down
Loading