Supports incremental build in tsc --b --w mode by sheetalkamat · Pull Request #29161 · microsoft/TypeScript

sheetalkamat · 2018-12-26T20:33:56Z

Some notes:

Since we use builder, the first time program is created, its going to emit all the files and not just changed - just like it does in tsc --watch mode
We do not cache source files yet and that's something to be looked at later.
Two things can be done:
- To keep source files for program thats being updated in last few cycles
- To release the source files if size exceeds or program is not used for few build cycles
- Adding this to Create tsbuild public API and use incremental build in tsbuild --watch mode #28640
We use hash from source file's text as the version.
Because in tsbuild we get semantic diagnostics before emit, the EmitAndSemanticDiagnosticsBuilderProgram's existing assumption was not satisfied (that is changed files are already emitted before getting semantic diagnostics). So in such case, we store list of files to emit later so as to help with that case.
In this we also have a new Builder method to release the program and its source files to ensure that we aren't holding on program after emit and/or semantic diagnostics are complete. That way the builder state can be in memory for comparison and finding out what are new changes that need emitting while creating new program but we aren't holding onto these costly (memory wise) resources. (all the program methods are thus available only till program is released.

Here are the perf numbers:
Local change::

File change detected. Starting incremental compilation...
Project 'SubProject 4' is out of date because oldest output 'someOutputFile.js' is older than newest input 'SubProject 2'
Building project 'SubProject 4'...
Updating unchanged output timestamps of project 'SubProject 4'...
Before After
Files: 8305 Files: 8305
Lines: 509633 Lines: 509633
Nodes: 1741390 Nodes: 1741390
Identifiers: 563590 Identifiers: 563590
Symbols: 597260 Symbols: 316035
Types: 175190 Types: 232
Memory used: 1418992K Memory used: 1859282K
I/O Read time: 2.83s I/O Read time: 3.85s
Parse time: 2.43s Parse time: 3.42s
Program time: 23.45s Program time: 33.07s
Bind time: 1.56s Bind time: 1.71s
Check time: 11.86s Check time: 0.04s
transformTime time: 13.72s transformTime time: 0.25s
Source Map time: 2.58s Source Map time: 0.01s
commentTime time: 1.46s commentTime time: 0.00s
printTime time: 27.08s printTime time: 0.28s
Emit time: 27.08s Emit time: 0.28s
I/O Write time: 16.16s I/O Write time: 0.04s
Total time: 63.96s Total time: 35.09s
Building project 'SubProject 5'...
Updating unchanged output timestamps of project 'SubProject 5'...
Before After
Files: 1875 Files: 1875
Lines: 207731 Lines: 207731
Nodes: 632955 Nodes: 632955
Identifiers: 212573 Identifiers: 212573
Symbols: 162334 Symbols: 125245
Types: 22975 Types: 71
Memory used: 1095817K Memory used: 311574K
I/O Read time: 1.13s I/O Read time: 0.74s
Parse time: 2.03s Parse time: 0.87s
Program time: 9.64s Program time: 5.97s
Bind time: 0.67s Bind time: 0.48s
Check time: 2.17s
transformTime time: 1.22s
Source Map time: 0.77s
commentTime time: 0.34s
printTime time: 4.16s
Emit time: 4.16s
I/O Write time: 1.92s
Total time: 16.64s Total time: 6.45s
Building project 'SubProject 6'...
Updating unchanged output timestamps of project 'SubProject 6'...
Before After
Files: 2396 Files: 2396
Lines: 221677 Lines: 221677
Nodes: 669898 Nodes: 669898
Identifiers: 221017 Identifiers: 221017
Symbols: 189874 Symbols: 143939
Types: 30412 Types: 71
Memory used: 1442324K Memory used: 333909K
I/O Read time: 0.92s I/O Read time: 0.91s
Parse time: 1.40s Parse time: 1.17s
Program time: 8.91s Program time: 7.73s
Bind time: 0.43s Bind time: 0.39s
Check time: 1.74s
transformTime time: 0.67s
Source Map time: 0.18s
commentTime time: 0.10s
printTime time: 1.67s
Emit time: 1.67s
I/O Write time: 1.40s
Total time: 12.74s Total time: 8.12s
Building project 'SubProject 7'...
Updating unchanged output timestamps of project 'SubProject 7'...
Before After
Files: 9326 Files: 9326
Lines: 552828 Lines: 552830
Nodes: 1896877 Nodes: 1896879
Identifiers: 605603 Identifiers: 605605
Symbols: 608250 Symbols: 330625
Types: 173986 Types: 71
Memory used: 2367385K Memory used: 953536K
I/O Read time: 4.22s I/O Read time: 3.84s
Parse time: 3.18s Parse time: 2.02s
Program time: 36.41s Program time: 27.79s
Bind time: 2.73s Bind time: 1.19s
Check time: 13.15s
transformTime time: 15.36s
Source Map time: 2.35s
commentTime time: 1.52s
printTime time: 28.95s
Emit time: 28.95s
I/O Write time: 13.41s
Total time: 81.23s Total time: 28.98s
Building project 'SubProject 8'...
Updating unchanged output timestamps of project 'SubProject 8'...
Before After
Files: 6146 Files: 6146
Lines: 367180 Lines: 367182
Nodes: 1201703 Nodes: 1201705
Identifiers: 386452 Identifiers: 386454
Symbols: 406182 Symbols: 232990
Types: 94077 Types: 71
Memory used: 1704251K Memory used: 1333457K
I/O Read time: 3.06s I/O Read time: 2.50s
Parse time: 3.61s Parse time: 2.06s
Program time: 31.86s Program time: 19.05s
Bind time: 1.76s Bind time: 0.77s
Check time: 10.90s
transformTime time: 7.64s
Source Map time: 0.96s
commentTime time: 0.91s
printTime time: 13.81s
Emit time: 13.81s
I/O Write time: 6.67s
Total time: 58.33s Total time: 19.82s
Building project 'SubProject 9'...
Updating unchanged output timestamps of project 'SubProject 9'...
Before After
Files: 3781 Files: 3781
Lines: 264899 Lines: 264899
Nodes: 834360 Nodes: 834360
Identifiers: 278052 Identifiers: 278052
Symbols: 261052 Symbols: 178924
Types: 52797 Types: 71
Memory used: 2175398K Memory used: 1620896K
I/O Read time: 1.67s I/O Read time: 1.51s
Parse time: 1.98s Parse time: 1.19s
Program time: 16.03s Program time: 11.75s
Bind time: 0.77s Bind time: 0.49s
Check time: 4.86s
transformTime time: 4.24s
Source Map time: 0.40s
commentTime time: 0.21s
printTime time: 6.63s
Emit time: 6.63s
I/O Write time: 2.03s
Total time: 28.29s Total time: 12.24s
Solution Total Time:
Before: 261.19, After: 110.7

Local change:

File change detected. Starting incremental compilation...
Project 'SubProject 4' is out of date because oldest output 'someOutputFile.js' is older than newest input 'SubProject 2'
Building project 'SubProject 4'...
Updating unchanged output timestamps of project 'SubProject 4'...
Before After
Files: 8305 Files: 8305
Lines: 509635 Lines: 509635
Nodes: 1741406 Nodes: 1741406
Identifiers: 563597 Identifiers: 563597
Symbols: 597262 Symbols: 315957
Types: 175191 Types: 114
Memory used: 3243603K Memory used: 690192K
I/O Read time: 3.23s I/O Read time: 3.28s
Parse time: 4.00s Parse time: 2.31s
Program time: 35.91s Program time: 24.59s
Bind time: 1.75s Bind time: 1.27s
Check time: 15.76s Check time: 0.01s
transformTime time: 15.10s transformTime time: 0.00s
Source Map time: 2.93s Source Map time: 0.00s
commentTime time: 1.85s commentTime time: 0.00s
printTime time: 30.68s printTime time: 0.00s
Emit time: 30.68s Emit time: 0.00s
I/O Write time: 14.47s I/O Write time: 0.00s
Total time: 84.10s Total time: 25.87s
Project 'SubProject 5' is up to date with .d.ts files from its dependencies
Updating output timestamps of project 'SubProject 5'...
Project 'SubProject 6' is up to date with .d.ts files from its dependencies
Updating output timestamps of project 'SubProject 6'...
Project 'SubProject 7' is up to date with .d.ts files from its dependencies
Updating output timestamps of project 'SubProject 7'...
Project 'SubProject 8' is up to date with .d.ts files from its dependencies
Updating output timestamps of project 'SubProject 8'...
Project 'SubProject 9' is up to date with .d.ts files from its dependencies
Updating output timestamps of project 'SubProject 9'...
Solution Total Time:
Before: 84.1, After: 25.87
Found 0 errors. Watching for file changes.

…ide detailed information on watch invokations

…ncremental builds

…on to set Program in state to undefined for storing.

This helps in storing the semantic diagnostics

… of incremental build This ensures that after `tsbuild` after incremental build of `tsbuild -w` doesnt result in unnecessary rebuilds

…oject reference with prepend since it cant tell changes in js/map files

…dling These will fail since its still TODO

…state This helps us revert to state where we pretend as if emit is not done (since we do not do emit if there are errors)

sheetalkamat · 2019-01-10T17:30:22Z

Ping @sandersn @rbuckton @weswigham @DanielRosenwasser Would like to get this in soon.

sandersn

I looked over everything but I need to learn more about watch mode + tsbuild in person before signing off.

sheetalkamat · 2019-01-17T20:52:50Z

@sandersn comments?

xaviergonz · 2019-01-17T23:06:03Z

Just wondering, wouldn't it be even faster if the cache was also based on file size as first check before doing the actual hash?

sheetalkamat added 11 commits December 21, 2018 12:53

Use watch factory instead of direct host functions in tsbuild to prov…
0cabb00
…ide detailed information on watch invokations

Make BuilderProgram as Program
9e05abc

Make SolutionBuilder handle BuilderProgram in preparation to handle i…
48baa42
…ncremental builds

Revert BuilderProgram to be redirected object to Program in preparati…
56a76d8
…on to set Program in state to undefined for storing.

Add method to release held Program in BuilderProgram
69193d9

Use oldProgram to create the new Program.
47f5106
This helps in storing the semantic diagnostics

Use emit builder to emit only changed files.
f1949bb

Update the timestamps of outputs that dont need to be written because…
7b290fd
… of incremental build This ensures that after `tsbuild` after incremental build of `tsbuild -w` doesnt result in unnecessary rebuilds

Handle prepend in incremental build. Always emit when program uses pr…
0d9038c
…oject reference with prepend since it cant tell changes in js/map files

Write the tests for incremental build and declaration emit errors han…
b360ff7
…dling These will fail since its still TODO

Handle declaration emit errors in tsbuild mode by backing up builder …
69abc12
…state This helps us revert to state where we pretend as if emit is not done (since we do not do emit if there are errors)

sheetalkamat mentioned this pull request Dec 26, 2018
Create tsbuild public API and use incremental build in tsbuild --watch mode #28640
Closed
9 tasks

sheetalkamat requested review from DanielRosenwasser, rbuckton, sandersn and weswigham December 26, 2018 20:55

ajafff reviewed Dec 26, 2018
View reviewed changes

Comment threadsrc/compiler/program.ts

sheetalkamat added 2 commits December 27, 2018 10:36

Use DirectoryStructureHost for fileExists and readFile
42484b5

Merge branch 'master' into incrementalBuild
1b37830

sandersn reviewed Jan 10, 2019
View reviewed changes

Comment threadsrc/compiler/core.ts Outdated

Fix typo
abc8618

sandersn reviewed Jan 10, 2019
View reviewed changes

Comment threadsrc/compiler/sys.ts

Merge branch 'master' into incrementalBuild
c1edbb8

sandersn self-assigned this Jan 14, 2019