Add StringMap, NumberMap, and StringSet data structures - #10230

Closed
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map
Closed

Add StringMap, NumberMap, and StringSet data structures#10230
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map

Conversation

@ghost

@ghostDeleted user (ghost) commented Aug 9, 2016

Copy link
Copy Markdown

This switches us to using native Maps and Sets where possible.
Since we can't guarantee that the native Map will exist, we instead have StringMap and NumberMap for string-keyed and number-keyed maps, with shims using objects and arrays.
Many remaining uses of the old Map may be better off as StringMap, but I've left alone anything that looks public-facing.
I've also mostly left declarationEmitter and emitter alone. When we merge the transforms branch in we should use the new maps there.

I'd like to assign Raghav Katyal (@rakatyal) to this but it looks like he's not yet on the available assignees list.

…e available, to replace the old Map, which is now ObjMap.
Comment threadsrc/compiler/dataStructures.ts Outdated
}

/** String-keyed Map. */
export interface SMap<V> extends MapCommon<string, V> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it will be easier to read if you just use StringMap instead of SMap and same as NumberMap for NMap

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On that topic, what should we name ObjMap? (The keys aren't objects. The map itself is an object, which is used like a StringMap.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumberKeyedMap, StringKeyedMap, ObjectBackedMap. With good autcomplete, there's no real reason for confusing or ambiguous hungarianesque notation.

Comment threadsrc/compiler/binder.ts Outdated
const typeLiteralSymbol = createSymbol(SymbolFlags.TypeLiteral, "__type");
addDeclarationToSymbol(typeLiteralSymbol, node, SymbolFlags.TypeLiteral);
typeLiteralSymbol.members = { [symbol.name]: symbol };
typeLiteralSymbol.members = singletonMap(symbol.name, symbol);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of singletonMap?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought it was neater than new SMap([[symbol.name, symbol]]).

@weswighamWesley Wigham (weswigham)Aug 9, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing about this is like a singleton in the design pattern sense (it's a loaded term), so the name is a bit confusing. createStringKeyedMap would be more straightforward, if you must wrap the constructor.

@weswigham

Copy link
Copy Markdown
Member

This appears to sort object type keys (which is causing what should be a transparent change to manifest in our baselines). With this change sort order is now insertion order (which arguably is nice), however this is likely a problem because the polyfills right now don't do the same sorting. (This may surface as node 0.10 failing CI with mismatching baselines once the other bugs are fixed)

Comment threadsrc/compiler/dataStructures.ts Outdated
export const SSet: SSetConstructor = Set ? Set : ShimSSet;

/** False iff there are any values in the set. */
export function setIsEmpty(set: SSet): boolean {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be isSetEmpty?

@rakatyal

Copy link
Copy Markdown

Could you also please share some numbers on how this effect both memory and performance in the typescript repo and other use cases?

@ghost

Copy link
Copy Markdown
Author

Overall, we get about a 10% reduction of both memory and total time. (We could get even more by changing the emitter to use StringMaps.)

(Note: since file system accesses are cached, I throw away the first run, which has a large I/O read time.)

VSCode

Testing on the latest checkout of Microsoft/vscode:

Constant:

Files: 1120
Lines: 312630
Nodes: 1591307
Identifiers: 496012
Symbols: 417268
Types: 114545
I/O write: 0.00s (--noEmit)
Emit time: 0.00s (--noEmit)

tsc -p src --noEmit --diagnostics:

Memory used: 704300K 679391K 679408K
I/O read: 0.18s 0.22s 0.19s
Parse time: 5.36s 5.58s 5.60s
Bind time: 1.51s 1.70s 1.64s
Check time: 10.44s 10.24s 9.79s
Total time: 17.30s 17.52s 17.02s

node ../TypeScript/built/local/tsc --noEmit --diagnostics -p src:

Memory used: 593708K 589821K 595477K
I/O read: 0.24s 0.19s 0.17s
Parse time: 5.43s 5.21s 5.01s
Bind time: 1.16s 1.25s 1.43s
Check time: 8.46s 8.97s 9.13s
Total time: 15.05s 15.43s 15.57s

Typescript services

Testing on the src/services directory (in this branch):

Files: 43
Lines: 91284
Nodes: 456582
Identifiers: 168783
Symbols: 156627
Types: 31431
I/O read: 0.04s 0.02s 0.03s 0.02s 0.01s 0.02s

tsc -p src/services --noEmit --diagnostics:

Memory used: 243354K 243332K 235689K
Parse time: 0.75s 0.78s 0.77s
Bind time: 0.46s 0.47s 0.46s
Check time: 3.28s 3.19s 3.24s
Total time: 4.49s 4.43s 4.47s

node ./built/local/tsc.js -p src/services --noEmit --diagnostics

Memory used: 216397K 216644K 217180K
Parse time: 0.79s 0.77s 0.84s
Bind time: 0.39s 0.36s 0.39s
Check time: 2.77s 2.73s 2.84s
Total time: 3.95s 3.87s 4.08s

Single file

Just a file with console.log("Hello world!")

Files: 2
Lines: 19010
Nodes: 98062
Identifiers: 35497
Symbols: 94435
Types: 11926

tsc --diagnostics --noEmit a.ts:

Memory used: 89009K 89023K 89243K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.25s 0.21s 0.21s
Bind time: 0.10s 0.10s 0.08s
Check time: 0.91s 0.99s 0.91s
Total time: 1.26s 1.30s 1.20s

node ../../../TypeScript/built/local/tsc.js --diagnostics --noEmit a.ts:

Memory used: 82048K 81071K 81771K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.21s 0.20s 0.20s
Bind time: 0.12s 0.09s 0.09s
Check time: 0.76s 0.64s 0.74s
Total time: 1.09s 0.93s 1.02s

Interactive test

What I did:

Go to vscode repo
`code src`
Use file navigator to go to vs/base/browser/ui/actionbar/actionbar.ts
Then type ctrl-t, "getZoomLevel"
View memory in task manager
Then go to settings and change "typescript.tsdk" to point to my local version, close all tabs, and start over.

"C:\\Program Files\\nodejs\\node_modules\\typescript\\lib": 272.2MB

"C:\\Users\\anhans\\work\\TypeScript\\built\\local": 265.7MB

@ghostghost changed the title Add SMap and NMap data structuresAdd StringMap, NumberMap, and StringSet data structuresAug 10, 2016
@mhegazy

Copy link
Copy Markdown
Contributor

Andy (Andrewkraft) (@Andy-MS) what were the numbers if we are using the shims and not the native implementation?

@ghost

Deleted user (ghost) commented Sep 28, 2016

Copy link
Copy Markdown
Author

The shims caused an increase in memory due to indirection: the shim was an object pointing to another object in dictionary mode.

With native maps:

Memory used: 611586K
Parse time: 4.88s
Bind time: 1.26s
Check time: 8.22s
Total time: 14.36s
Memory used: 613703K
Parse time: 5.01s
Bind time: 1.26s
Check time: 8.45s
Total time: 14.72s
Memory used: 613734K
Parse time: 4.99s
Bind time: 1.27s
Check time: 8.17s
Total time: 14.43s

With the shims always used:

Memory used: 835746K
Parse time: 5.34s
Bind time: 1.63s
Check time: 11.22s
Total time: 18.19s
Memory used: 820061K
Parse time: 5.40s
Bind time: 1.57s
Check time: 11.30s
Total time: 18.28s
Memory used: 830217K
Parse time: 5.41s
Bind time: 1.58s
Check time: 11.22s
Total time: 18.20s

I have created a new PR #11210 which fixes this issue.

@ghostghost closed this Sep 28, 2016
@microsoftMicrosoft (microsoft) locked and limited conversation to collaborators Jun 19, 2018
This pull request was closed.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@weswigham@rakatyal@mhegazy@yuit@9990859714@msftclas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Add StringMap, NumberMap, and StringSet data structures - #10230

Closed
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map
Closed

Add StringMap, NumberMap, and StringSet data structures#10230
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map

Conversation

@ghost

@ghostDeleted user (ghost) commented Aug 9, 2016

Copy link
Copy Markdown

This switches us to using native Maps and Sets where possible.
Since we can't guarantee that the native Map will exist, we instead have StringMap and NumberMap for string-keyed and number-keyed maps, with shims using objects and arrays.
Many remaining uses of the old Map may be better off as StringMap, but I've left alone anything that looks public-facing.
I've also mostly left declarationEmitter and emitter alone. When we merge the transforms branch in we should use the new maps there.

I'd like to assign Raghav Katyal (@rakatyal) to this but it looks like he's not yet on the available assignees list.

…e available, to replace the old Map, which is now ObjMap.
Comment threadsrc/compiler/dataStructures.ts Outdated
}

/** String-keyed Map. */
export interface SMap<V> extends MapCommon<string, V> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it will be easier to read if you just use StringMap instead of SMap and same as NumberMap for NMap

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On that topic, what should we name ObjMap? (The keys aren't objects. The map itself is an object, which is used like a StringMap.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumberKeyedMap, StringKeyedMap, ObjectBackedMap. With good autcomplete, there's no real reason for confusing or ambiguous hungarianesque notation.

Comment threadsrc/compiler/binder.ts Outdated
const typeLiteralSymbol = createSymbol(SymbolFlags.TypeLiteral, "__type");
addDeclarationToSymbol(typeLiteralSymbol, node, SymbolFlags.TypeLiteral);
typeLiteralSymbol.members = { [symbol.name]: symbol };
typeLiteralSymbol.members = singletonMap(symbol.name, symbol);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of singletonMap?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought it was neater than new SMap([[symbol.name, symbol]]).

@weswighamWesley Wigham (weswigham)Aug 9, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing about this is like a singleton in the design pattern sense (it's a loaded term), so the name is a bit confusing. createStringKeyedMap would be more straightforward, if you must wrap the constructor.

@weswigham

Copy link
Copy Markdown
Member

This appears to sort object type keys (which is causing what should be a transparent change to manifest in our baselines). With this change sort order is now insertion order (which arguably is nice), however this is likely a problem because the polyfills right now don't do the same sorting. (This may surface as node 0.10 failing CI with mismatching baselines once the other bugs are fixed)

Comment threadsrc/compiler/dataStructures.ts Outdated
export const SSet: SSetConstructor = Set ? Set : ShimSSet;

/** False iff there are any values in the set. */
export function setIsEmpty(set: SSet): boolean {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be isSetEmpty?

@rakatyal

Copy link
Copy Markdown

Could you also please share some numbers on how this effect both memory and performance in the typescript repo and other use cases?

@ghost

Copy link
Copy Markdown
Author

Overall, we get about a 10% reduction of both memory and total time. (We could get even more by changing the emitter to use StringMaps.)

(Note: since file system accesses are cached, I throw away the first run, which has a large I/O read time.)

VSCode

Testing on the latest checkout of Microsoft/vscode:

Constant:

Files: 1120
Lines: 312630
Nodes: 1591307
Identifiers: 496012
Symbols: 417268
Types: 114545
I/O write: 0.00s (--noEmit)
Emit time: 0.00s (--noEmit)

tsc -p src --noEmit --diagnostics:

Memory used: 704300K 679391K 679408K
I/O read: 0.18s 0.22s 0.19s
Parse time: 5.36s 5.58s 5.60s
Bind time: 1.51s 1.70s 1.64s
Check time: 10.44s 10.24s 9.79s
Total time: 17.30s 17.52s 17.02s

node ../TypeScript/built/local/tsc --noEmit --diagnostics -p src:

Memory used: 593708K 589821K 595477K
I/O read: 0.24s 0.19s 0.17s
Parse time: 5.43s 5.21s 5.01s
Bind time: 1.16s 1.25s 1.43s
Check time: 8.46s 8.97s 9.13s
Total time: 15.05s 15.43s 15.57s

Typescript services

Testing on the src/services directory (in this branch):

Files: 43
Lines: 91284
Nodes: 456582
Identifiers: 168783
Symbols: 156627
Types: 31431
I/O read: 0.04s 0.02s 0.03s 0.02s 0.01s 0.02s

tsc -p src/services --noEmit --diagnostics:

Memory used: 243354K 243332K 235689K
Parse time: 0.75s 0.78s 0.77s
Bind time: 0.46s 0.47s 0.46s
Check time: 3.28s 3.19s 3.24s
Total time: 4.49s 4.43s 4.47s

node ./built/local/tsc.js -p src/services --noEmit --diagnostics

Memory used: 216397K 216644K 217180K
Parse time: 0.79s 0.77s 0.84s
Bind time: 0.39s 0.36s 0.39s
Check time: 2.77s 2.73s 2.84s
Total time: 3.95s 3.87s 4.08s

Single file

Just a file with console.log("Hello world!")

Files: 2
Lines: 19010
Nodes: 98062
Identifiers: 35497
Symbols: 94435
Types: 11926

tsc --diagnostics --noEmit a.ts:

Memory used: 89009K 89023K 89243K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.25s 0.21s 0.21s
Bind time: 0.10s 0.10s 0.08s
Check time: 0.91s 0.99s 0.91s
Total time: 1.26s 1.30s 1.20s

node ../../../TypeScript/built/local/tsc.js --diagnostics --noEmit a.ts:

Memory used: 82048K 81071K 81771K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.21s 0.20s 0.20s
Bind time: 0.12s 0.09s 0.09s
Check time: 0.76s 0.64s 0.74s
Total time: 1.09s 0.93s 1.02s

Interactive test

What I did:

Go to vscode repo
`code src`
Use file navigator to go to vs/base/browser/ui/actionbar/actionbar.ts
Then type ctrl-t, "getZoomLevel"
View memory in task manager
Then go to settings and change "typescript.tsdk" to point to my local version, close all tabs, and start over.

"C:\\Program Files\\nodejs\\node_modules\\typescript\\lib": 272.2MB

"C:\\Users\\anhans\\work\\TypeScript\\built\\local": 265.7MB

@ghostghost changed the title Add SMap and NMap data structuresAdd StringMap, NumberMap, and StringSet data structuresAug 10, 2016
@mhegazy

Copy link
Copy Markdown
Contributor

Andy (Andrewkraft) (@Andy-MS) what were the numbers if we are using the shims and not the native implementation?

@ghost

Deleted user (ghost) commented Sep 28, 2016

Copy link
Copy Markdown
Author

The shims caused an increase in memory due to indirection: the shim was an object pointing to another object in dictionary mode.

With native maps:

Memory used: 611586K
Parse time: 4.88s
Bind time: 1.26s
Check time: 8.22s
Total time: 14.36s
Memory used: 613703K
Parse time: 5.01s
Bind time: 1.26s
Check time: 8.45s
Total time: 14.72s
Memory used: 613734K
Parse time: 4.99s
Bind time: 1.27s
Check time: 8.17s
Total time: 14.43s

With the shims always used:

Memory used: 835746K
Parse time: 5.34s
Bind time: 1.63s
Check time: 11.22s
Total time: 18.19s
Memory used: 820061K
Parse time: 5.40s
Bind time: 1.57s
Check time: 11.30s
Total time: 18.28s
Memory used: 830217K
Parse time: 5.41s
Bind time: 1.58s
Check time: 11.22s
Total time: 18.20s

I have created a new PR #11210 which fixes this issue.

@ghostghost closed this Sep 28, 2016
@microsoftMicrosoft (microsoft) locked and limited conversation to collaborators Jun 19, 2018
This pull request was closed.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@weswigham@rakatyal@mhegazy@yuit@9990859714@msftclas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add StringMap, NumberMap, and StringSet data structures - #10230

Closed
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map
Closed

Add StringMap, NumberMap, and StringSet data structures#10230
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map

Conversation

@ghost

@ghostDeleted user (ghost) commented Aug 9, 2016

Copy link
Copy Markdown

This switches us to using native Maps and Sets where possible.
Since we can't guarantee that the native Map will exist, we instead have StringMap and NumberMap for string-keyed and number-keyed maps, with shims using objects and arrays.
Many remaining uses of the old Map may be better off as StringMap, but I've left alone anything that looks public-facing.
I've also mostly left declarationEmitter and emitter alone. When we merge the transforms branch in we should use the new maps there.

I'd like to assign Raghav Katyal (@rakatyal) to this but it looks like he's not yet on the available assignees list.

…e available, to replace the old Map, which is now ObjMap.
Comment threadsrc/compiler/dataStructures.ts Outdated
}

/** String-keyed Map. */
export interface SMap<V> extends MapCommon<string, V> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it will be easier to read if you just use StringMap instead of SMap and same as NumberMap for NMap

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On that topic, what should we name ObjMap? (The keys aren't objects. The map itself is an object, which is used like a StringMap.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumberKeyedMap, StringKeyedMap, ObjectBackedMap. With good autcomplete, there's no real reason for confusing or ambiguous hungarianesque notation.

Comment threadsrc/compiler/binder.ts Outdated
const typeLiteralSymbol = createSymbol(SymbolFlags.TypeLiteral, "__type");
addDeclarationToSymbol(typeLiteralSymbol, node, SymbolFlags.TypeLiteral);
typeLiteralSymbol.members = { [symbol.name]: symbol };
typeLiteralSymbol.members = singletonMap(symbol.name, symbol);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of singletonMap?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought it was neater than new SMap([[symbol.name, symbol]]).

@weswighamWesley Wigham (weswigham)Aug 9, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing about this is like a singleton in the design pattern sense (it's a loaded term), so the name is a bit confusing. createStringKeyedMap would be more straightforward, if you must wrap the constructor.

@weswigham

Copy link
Copy Markdown
Member

This appears to sort object type keys (which is causing what should be a transparent change to manifest in our baselines). With this change sort order is now insertion order (which arguably is nice), however this is likely a problem because the polyfills right now don't do the same sorting. (This may surface as node 0.10 failing CI with mismatching baselines once the other bugs are fixed)

Comment threadsrc/compiler/dataStructures.ts Outdated
export const SSet: SSetConstructor = Set ? Set : ShimSSet;

/** False iff there are any values in the set. */
export function setIsEmpty(set: SSet): boolean {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be isSetEmpty?

@rakatyal

Copy link
Copy Markdown

Could you also please share some numbers on how this effect both memory and performance in the typescript repo and other use cases?

@ghost

Copy link
Copy Markdown
Author

Overall, we get about a 10% reduction of both memory and total time. (We could get even more by changing the emitter to use StringMaps.)

(Note: since file system accesses are cached, I throw away the first run, which has a large I/O read time.)

VSCode

Testing on the latest checkout of Microsoft/vscode:

Constant:

Files: 1120
Lines: 312630
Nodes: 1591307
Identifiers: 496012
Symbols: 417268
Types: 114545
I/O write: 0.00s (--noEmit)
Emit time: 0.00s (--noEmit)

tsc -p src --noEmit --diagnostics:

Memory used: 704300K 679391K 679408K
I/O read: 0.18s 0.22s 0.19s
Parse time: 5.36s 5.58s 5.60s
Bind time: 1.51s 1.70s 1.64s
Check time: 10.44s 10.24s 9.79s
Total time: 17.30s 17.52s 17.02s

node ../TypeScript/built/local/tsc --noEmit --diagnostics -p src:

Memory used: 593708K 589821K 595477K
I/O read: 0.24s 0.19s 0.17s
Parse time: 5.43s 5.21s 5.01s
Bind time: 1.16s 1.25s 1.43s
Check time: 8.46s 8.97s 9.13s
Total time: 15.05s 15.43s 15.57s

Typescript services

Testing on the src/services directory (in this branch):

Files: 43
Lines: 91284
Nodes: 456582
Identifiers: 168783
Symbols: 156627
Types: 31431
I/O read: 0.04s 0.02s 0.03s 0.02s 0.01s 0.02s

tsc -p src/services --noEmit --diagnostics:

Memory used: 243354K 243332K 235689K
Parse time: 0.75s 0.78s 0.77s
Bind time: 0.46s 0.47s 0.46s
Check time: 3.28s 3.19s 3.24s
Total time: 4.49s 4.43s 4.47s

node ./built/local/tsc.js -p src/services --noEmit --diagnostics

Memory used: 216397K 216644K 217180K
Parse time: 0.79s 0.77s 0.84s
Bind time: 0.39s 0.36s 0.39s
Check time: 2.77s 2.73s 2.84s
Total time: 3.95s 3.87s 4.08s

Single file

Just a file with console.log("Hello world!")

Files: 2
Lines: 19010
Nodes: 98062
Identifiers: 35497
Symbols: 94435
Types: 11926

tsc --diagnostics --noEmit a.ts:

Memory used: 89009K 89023K 89243K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.25s 0.21s 0.21s
Bind time: 0.10s 0.10s 0.08s
Check time: 0.91s 0.99s 0.91s
Total time: 1.26s 1.30s 1.20s

node ../../../TypeScript/built/local/tsc.js --diagnostics --noEmit a.ts:

Memory used: 82048K 81071K 81771K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.21s 0.20s 0.20s
Bind time: 0.12s 0.09s 0.09s
Check time: 0.76s 0.64s 0.74s
Total time: 1.09s 0.93s 1.02s

Interactive test

What I did:

Go to vscode repo
`code src`
Use file navigator to go to vs/base/browser/ui/actionbar/actionbar.ts
Then type ctrl-t, "getZoomLevel"
View memory in task manager
Then go to settings and change "typescript.tsdk" to point to my local version, close all tabs, and start over.

"C:\\Program Files\\nodejs\\node_modules\\typescript\\lib": 272.2MB

"C:\\Users\\anhans\\work\\TypeScript\\built\\local": 265.7MB

@ghostghost changed the title Add SMap and NMap data structuresAdd StringMap, NumberMap, and StringSet data structuresAug 10, 2016
@mhegazy

Copy link
Copy Markdown
Contributor

Andy (Andrewkraft) (@Andy-MS) what were the numbers if we are using the shims and not the native implementation?

@ghost

Deleted user (ghost) commented Sep 28, 2016

Copy link
Copy Markdown
Author

The shims caused an increase in memory due to indirection: the shim was an object pointing to another object in dictionary mode.

With native maps:

Memory used: 611586K
Parse time: 4.88s
Bind time: 1.26s
Check time: 8.22s
Total time: 14.36s
Memory used: 613703K
Parse time: 5.01s
Bind time: 1.26s
Check time: 8.45s
Total time: 14.72s
Memory used: 613734K
Parse time: 4.99s
Bind time: 1.27s
Check time: 8.17s
Total time: 14.43s

With the shims always used:

Memory used: 835746K
Parse time: 5.34s
Bind time: 1.63s
Check time: 11.22s
Total time: 18.19s
Memory used: 820061K
Parse time: 5.40s
Bind time: 1.57s
Check time: 11.30s
Total time: 18.28s
Memory used: 830217K
Parse time: 5.41s
Bind time: 1.58s
Check time: 11.22s
Total time: 18.20s

I have created a new PR #11210 which fixes this issue.

@ghostghost closed this Sep 28, 2016
@microsoftMicrosoft (microsoft) locked and limited conversation to collaborators Jun 19, 2018
This pull request was closed.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@weswigham@rakatyal@mhegazy@yuit@9990859714@msftclas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add StringMap, NumberMap, and StringSet data structures - #10230

Closed
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map
Closed

Add StringMap, NumberMap, and StringSet data structures#10230
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map

Conversation

@ghost

@ghostDeleted user (ghost) commented Aug 9, 2016

Copy link
Copy Markdown

This switches us to using native Maps and Sets where possible.
Since we can't guarantee that the native Map will exist, we instead have StringMap and NumberMap for string-keyed and number-keyed maps, with shims using objects and arrays.
Many remaining uses of the old Map may be better off as StringMap, but I've left alone anything that looks public-facing.
I've also mostly left declarationEmitter and emitter alone. When we merge the transforms branch in we should use the new maps there.

I'd like to assign Raghav Katyal (@rakatyal) to this but it looks like he's not yet on the available assignees list.

…e available, to replace the old Map, which is now ObjMap.
Comment threadsrc/compiler/dataStructures.ts Outdated
}

/** String-keyed Map. */
export interface SMap<V> extends MapCommon<string, V> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it will be easier to read if you just use StringMap instead of SMap and same as NumberMap for NMap

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On that topic, what should we name ObjMap? (The keys aren't objects. The map itself is an object, which is used like a StringMap.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumberKeyedMap, StringKeyedMap, ObjectBackedMap. With good autcomplete, there's no real reason for confusing or ambiguous hungarianesque notation.

Comment threadsrc/compiler/binder.ts Outdated
const typeLiteralSymbol = createSymbol(SymbolFlags.TypeLiteral, "__type");
addDeclarationToSymbol(typeLiteralSymbol, node, SymbolFlags.TypeLiteral);
typeLiteralSymbol.members = { [symbol.name]: symbol };
typeLiteralSymbol.members = singletonMap(symbol.name, symbol);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of singletonMap?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought it was neater than new SMap([[symbol.name, symbol]]).

@weswighamWesley Wigham (weswigham)Aug 9, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing about this is like a singleton in the design pattern sense (it's a loaded term), so the name is a bit confusing. createStringKeyedMap would be more straightforward, if you must wrap the constructor.

@weswigham

Copy link
Copy Markdown
Member

This appears to sort object type keys (which is causing what should be a transparent change to manifest in our baselines). With this change sort order is now insertion order (which arguably is nice), however this is likely a problem because the polyfills right now don't do the same sorting. (This may surface as node 0.10 failing CI with mismatching baselines once the other bugs are fixed)

Comment threadsrc/compiler/dataStructures.ts Outdated
export const SSet: SSetConstructor = Set ? Set : ShimSSet;

/** False iff there are any values in the set. */
export function setIsEmpty(set: SSet): boolean {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be isSetEmpty?

@rakatyal

Copy link
Copy Markdown

Could you also please share some numbers on how this effect both memory and performance in the typescript repo and other use cases?

@ghost

Copy link
Copy Markdown
Author

Overall, we get about a 10% reduction of both memory and total time. (We could get even more by changing the emitter to use StringMaps.)

(Note: since file system accesses are cached, I throw away the first run, which has a large I/O read time.)

VSCode

Testing on the latest checkout of Microsoft/vscode:

Constant:

Files: 1120
Lines: 312630
Nodes: 1591307
Identifiers: 496012
Symbols: 417268
Types: 114545
I/O write: 0.00s (--noEmit)
Emit time: 0.00s (--noEmit)

tsc -p src --noEmit --diagnostics:

Memory used: 704300K 679391K 679408K
I/O read: 0.18s 0.22s 0.19s
Parse time: 5.36s 5.58s 5.60s
Bind time: 1.51s 1.70s 1.64s
Check time: 10.44s 10.24s 9.79s
Total time: 17.30s 17.52s 17.02s

node ../TypeScript/built/local/tsc --noEmit --diagnostics -p src:

Memory used: 593708K 589821K 595477K
I/O read: 0.24s 0.19s 0.17s
Parse time: 5.43s 5.21s 5.01s
Bind time: 1.16s 1.25s 1.43s
Check time: 8.46s 8.97s 9.13s
Total time: 15.05s 15.43s 15.57s

Typescript services

Testing on the src/services directory (in this branch):

Files: 43
Lines: 91284
Nodes: 456582
Identifiers: 168783
Symbols: 156627
Types: 31431
I/O read: 0.04s 0.02s 0.03s 0.02s 0.01s 0.02s

tsc -p src/services --noEmit --diagnostics:

Memory used: 243354K 243332K 235689K
Parse time: 0.75s 0.78s 0.77s
Bind time: 0.46s 0.47s 0.46s
Check time: 3.28s 3.19s 3.24s
Total time: 4.49s 4.43s 4.47s

node ./built/local/tsc.js -p src/services --noEmit --diagnostics

Memory used: 216397K 216644K 217180K
Parse time: 0.79s 0.77s 0.84s
Bind time: 0.39s 0.36s 0.39s
Check time: 2.77s 2.73s 2.84s
Total time: 3.95s 3.87s 4.08s

Single file

Just a file with console.log("Hello world!")

Files: 2
Lines: 19010
Nodes: 98062
Identifiers: 35497
Symbols: 94435
Types: 11926

tsc --diagnostics --noEmit a.ts:

Memory used: 89009K 89023K 89243K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.25s 0.21s 0.21s
Bind time: 0.10s 0.10s 0.08s
Check time: 0.91s 0.99s 0.91s
Total time: 1.26s 1.30s 1.20s

node ../../../TypeScript/built/local/tsc.js --diagnostics --noEmit a.ts:

Memory used: 82048K 81071K 81771K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.21s 0.20s 0.20s
Bind time: 0.12s 0.09s 0.09s
Check time: 0.76s 0.64s 0.74s
Total time: 1.09s 0.93s 1.02s

Interactive test

What I did:

Go to vscode repo
`code src`
Use file navigator to go to vs/base/browser/ui/actionbar/actionbar.ts
Then type ctrl-t, "getZoomLevel"
View memory in task manager
Then go to settings and change "typescript.tsdk" to point to my local version, close all tabs, and start over.

"C:\\Program Files\\nodejs\\node_modules\\typescript\\lib": 272.2MB

"C:\\Users\\anhans\\work\\TypeScript\\built\\local": 265.7MB

@ghostghost changed the title Add SMap and NMap data structuresAdd StringMap, NumberMap, and StringSet data structuresAug 10, 2016
@mhegazy

Copy link
Copy Markdown
Contributor

Andy (Andrewkraft) (@Andy-MS) what were the numbers if we are using the shims and not the native implementation?

@ghost

Deleted user (ghost) commented Sep 28, 2016

Copy link
Copy Markdown
Author

The shims caused an increase in memory due to indirection: the shim was an object pointing to another object in dictionary mode.

With native maps:

Memory used: 611586K
Parse time: 4.88s
Bind time: 1.26s
Check time: 8.22s
Total time: 14.36s
Memory used: 613703K
Parse time: 5.01s
Bind time: 1.26s
Check time: 8.45s
Total time: 14.72s
Memory used: 613734K
Parse time: 4.99s
Bind time: 1.27s
Check time: 8.17s
Total time: 14.43s

With the shims always used:

Memory used: 835746K
Parse time: 5.34s
Bind time: 1.63s
Check time: 11.22s
Total time: 18.19s
Memory used: 820061K
Parse time: 5.40s
Bind time: 1.57s
Check time: 11.30s
Total time: 18.28s
Memory used: 830217K
Parse time: 5.41s
Bind time: 1.58s
Check time: 11.22s
Total time: 18.20s

I have created a new PR #11210 which fixes this issue.

@ghostghost closed this Sep 28, 2016
@microsoftMicrosoft (microsoft) locked and limited conversation to collaborators Jun 19, 2018
This pull request was closed.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@weswigham@rakatyal@mhegazy@yuit@9990859714@msftclas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Add StringMap, NumberMap, and StringSet data structures - #10230

Closed
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map
Closed

Add StringMap, NumberMap, and StringSet data structures#10230
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map

Conversation

@ghost

@ghostDeleted user (ghost) commented Aug 9, 2016

Copy link
Copy Markdown

This switches us to using native Maps and Sets where possible.
Since we can't guarantee that the native Map will exist, we instead have StringMap and NumberMap for string-keyed and number-keyed maps, with shims using objects and arrays.
Many remaining uses of the old Map may be better off as StringMap, but I've left alone anything that looks public-facing.
I've also mostly left declarationEmitter and emitter alone. When we merge the transforms branch in we should use the new maps there.

I'd like to assign Raghav Katyal (@rakatyal) to this but it looks like he's not yet on the available assignees list.

…e available, to replace the old Map, which is now ObjMap.
Comment threadsrc/compiler/dataStructures.ts Outdated
}

/** String-keyed Map. */
export interface SMap<V> extends MapCommon<string, V> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it will be easier to read if you just use StringMap instead of SMap and same as NumberMap for NMap

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On that topic, what should we name ObjMap? (The keys aren't objects. The map itself is an object, which is used like a StringMap.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumberKeyedMap, StringKeyedMap, ObjectBackedMap. With good autcomplete, there's no real reason for confusing or ambiguous hungarianesque notation.

Comment threadsrc/compiler/binder.ts Outdated
const typeLiteralSymbol = createSymbol(SymbolFlags.TypeLiteral, "__type");
addDeclarationToSymbol(typeLiteralSymbol, node, SymbolFlags.TypeLiteral);
typeLiteralSymbol.members = { [symbol.name]: symbol };
typeLiteralSymbol.members = singletonMap(symbol.name, symbol);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of singletonMap?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought it was neater than new SMap([[symbol.name, symbol]]).

@weswighamWesley Wigham (weswigham)Aug 9, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing about this is like a singleton in the design pattern sense (it's a loaded term), so the name is a bit confusing. createStringKeyedMap would be more straightforward, if you must wrap the constructor.

@weswigham

Copy link
Copy Markdown
Member

This appears to sort object type keys (which is causing what should be a transparent change to manifest in our baselines). With this change sort order is now insertion order (which arguably is nice), however this is likely a problem because the polyfills right now don't do the same sorting. (This may surface as node 0.10 failing CI with mismatching baselines once the other bugs are fixed)

Comment threadsrc/compiler/dataStructures.ts Outdated
export const SSet: SSetConstructor = Set ? Set : ShimSSet;

/** False iff there are any values in the set. */
export function setIsEmpty(set: SSet): boolean {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be isSetEmpty?

@rakatyal

Copy link
Copy Markdown

Could you also please share some numbers on how this effect both memory and performance in the typescript repo and other use cases?

@ghost

Copy link
Copy Markdown
Author

Overall, we get about a 10% reduction of both memory and total time. (We could get even more by changing the emitter to use StringMaps.)

(Note: since file system accesses are cached, I throw away the first run, which has a large I/O read time.)

VSCode

Testing on the latest checkout of Microsoft/vscode:

Constant:

Files: 1120
Lines: 312630
Nodes: 1591307
Identifiers: 496012
Symbols: 417268
Types: 114545
I/O write: 0.00s (--noEmit)
Emit time: 0.00s (--noEmit)

tsc -p src --noEmit --diagnostics:

Memory used: 704300K 679391K 679408K
I/O read: 0.18s 0.22s 0.19s
Parse time: 5.36s 5.58s 5.60s
Bind time: 1.51s 1.70s 1.64s
Check time: 10.44s 10.24s 9.79s
Total time: 17.30s 17.52s 17.02s

node ../TypeScript/built/local/tsc --noEmit --diagnostics -p src:

Memory used: 593708K 589821K 595477K
I/O read: 0.24s 0.19s 0.17s
Parse time: 5.43s 5.21s 5.01s
Bind time: 1.16s 1.25s 1.43s
Check time: 8.46s 8.97s 9.13s
Total time: 15.05s 15.43s 15.57s

Typescript services

Testing on the src/services directory (in this branch):

Files: 43
Lines: 91284
Nodes: 456582
Identifiers: 168783
Symbols: 156627
Types: 31431
I/O read: 0.04s 0.02s 0.03s 0.02s 0.01s 0.02s

tsc -p src/services --noEmit --diagnostics:

Memory used: 243354K 243332K 235689K
Parse time: 0.75s 0.78s 0.77s
Bind time: 0.46s 0.47s 0.46s
Check time: 3.28s 3.19s 3.24s
Total time: 4.49s 4.43s 4.47s

node ./built/local/tsc.js -p src/services --noEmit --diagnostics

Memory used: 216397K 216644K 217180K
Parse time: 0.79s 0.77s 0.84s
Bind time: 0.39s 0.36s 0.39s
Check time: 2.77s 2.73s 2.84s
Total time: 3.95s 3.87s 4.08s

Single file

Just a file with console.log("Hello world!")

Files: 2
Lines: 19010
Nodes: 98062
Identifiers: 35497
Symbols: 94435
Types: 11926

tsc --diagnostics --noEmit a.ts:

Memory used: 89009K 89023K 89243K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.25s 0.21s 0.21s
Bind time: 0.10s 0.10s 0.08s
Check time: 0.91s 0.99s 0.91s
Total time: 1.26s 1.30s 1.20s

node ../../../TypeScript/built/local/tsc.js --diagnostics --noEmit a.ts:

Memory used: 82048K 81071K 81771K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.21s 0.20s 0.20s
Bind time: 0.12s 0.09s 0.09s
Check time: 0.76s 0.64s 0.74s
Total time: 1.09s 0.93s 1.02s

Interactive test

What I did:

Go to vscode repo
`code src`
Use file navigator to go to vs/base/browser/ui/actionbar/actionbar.ts
Then type ctrl-t, "getZoomLevel"
View memory in task manager
Then go to settings and change "typescript.tsdk" to point to my local version, close all tabs, and start over.

"C:\\Program Files\\nodejs\\node_modules\\typescript\\lib": 272.2MB

"C:\\Users\\anhans\\work\\TypeScript\\built\\local": 265.7MB

@ghostghost changed the title Add SMap and NMap data structuresAdd StringMap, NumberMap, and StringSet data structuresAug 10, 2016
@mhegazy

Copy link
Copy Markdown
Contributor

Andy (Andrewkraft) (@Andy-MS) what were the numbers if we are using the shims and not the native implementation?

@ghost

Deleted user (ghost) commented Sep 28, 2016

Copy link
Copy Markdown
Author

The shims caused an increase in memory due to indirection: the shim was an object pointing to another object in dictionary mode.

With native maps:

Memory used: 611586K
Parse time: 4.88s
Bind time: 1.26s
Check time: 8.22s
Total time: 14.36s
Memory used: 613703K
Parse time: 5.01s
Bind time: 1.26s
Check time: 8.45s
Total time: 14.72s
Memory used: 613734K
Parse time: 4.99s
Bind time: 1.27s
Check time: 8.17s
Total time: 14.43s

With the shims always used:

Memory used: 835746K
Parse time: 5.34s
Bind time: 1.63s
Check time: 11.22s
Total time: 18.19s
Memory used: 820061K
Parse time: 5.40s
Bind time: 1.57s
Check time: 11.30s
Total time: 18.28s
Memory used: 830217K
Parse time: 5.41s
Bind time: 1.58s
Check time: 11.22s
Total time: 18.20s

I have created a new PR #11210 which fixes this issue.

@ghostghost closed this Sep 28, 2016
@microsoftMicrosoft (microsoft) locked and limited conversation to collaborators Jun 19, 2018
This pull request was closed.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@weswigham@rakatyal@mhegazy@yuit@9990859714@msftclas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add StringMap, NumberMap, and StringSet data structures - #10230

Closed
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map
Closed

Add StringMap, NumberMap, and StringSet data structures#10230
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map

Conversation

@ghost

@ghostDeleted user (ghost) commented Aug 9, 2016

Copy link
Copy Markdown

This switches us to using native Maps and Sets where possible.
Since we can't guarantee that the native Map will exist, we instead have StringMap and NumberMap for string-keyed and number-keyed maps, with shims using objects and arrays.
Many remaining uses of the old Map may be better off as StringMap, but I've left alone anything that looks public-facing.
I've also mostly left declarationEmitter and emitter alone. When we merge the transforms branch in we should use the new maps there.

I'd like to assign Raghav Katyal (@rakatyal) to this but it looks like he's not yet on the available assignees list.

…e available, to replace the old Map, which is now ObjMap.
Comment threadsrc/compiler/dataStructures.ts Outdated
}

/** String-keyed Map. */
export interface SMap<V> extends MapCommon<string, V> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it will be easier to read if you just use StringMap instead of SMap and same as NumberMap for NMap

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On that topic, what should we name ObjMap? (The keys aren't objects. The map itself is an object, which is used like a StringMap.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumberKeyedMap, StringKeyedMap, ObjectBackedMap. With good autcomplete, there's no real reason for confusing or ambiguous hungarianesque notation.

Comment threadsrc/compiler/binder.ts Outdated
const typeLiteralSymbol = createSymbol(SymbolFlags.TypeLiteral, "__type");
addDeclarationToSymbol(typeLiteralSymbol, node, SymbolFlags.TypeLiteral);
typeLiteralSymbol.members = { [symbol.name]: symbol };
typeLiteralSymbol.members = singletonMap(symbol.name, symbol);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of singletonMap?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought it was neater than new SMap([[symbol.name, symbol]]).

@weswighamWesley Wigham (weswigham)Aug 9, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing about this is like a singleton in the design pattern sense (it's a loaded term), so the name is a bit confusing. createStringKeyedMap would be more straightforward, if you must wrap the constructor.

@weswigham

Copy link
Copy Markdown
Member

This appears to sort object type keys (which is causing what should be a transparent change to manifest in our baselines). With this change sort order is now insertion order (which arguably is nice), however this is likely a problem because the polyfills right now don't do the same sorting. (This may surface as node 0.10 failing CI with mismatching baselines once the other bugs are fixed)

Comment threadsrc/compiler/dataStructures.ts Outdated
export const SSet: SSetConstructor = Set ? Set : ShimSSet;

/** False iff there are any values in the set. */
export function setIsEmpty(set: SSet): boolean {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be isSetEmpty?

@rakatyal

Copy link
Copy Markdown

Could you also please share some numbers on how this effect both memory and performance in the typescript repo and other use cases?

@ghost

Copy link
Copy Markdown
Author

Overall, we get about a 10% reduction of both memory and total time. (We could get even more by changing the emitter to use StringMaps.)

(Note: since file system accesses are cached, I throw away the first run, which has a large I/O read time.)

VSCode

Testing on the latest checkout of Microsoft/vscode:

Constant:

Files: 1120
Lines: 312630
Nodes: 1591307
Identifiers: 496012
Symbols: 417268
Types: 114545
I/O write: 0.00s (--noEmit)
Emit time: 0.00s (--noEmit)

tsc -p src --noEmit --diagnostics:

Memory used: 704300K 679391K 679408K
I/O read: 0.18s 0.22s 0.19s
Parse time: 5.36s 5.58s 5.60s
Bind time: 1.51s 1.70s 1.64s
Check time: 10.44s 10.24s 9.79s
Total time: 17.30s 17.52s 17.02s

node ../TypeScript/built/local/tsc --noEmit --diagnostics -p src:

Memory used: 593708K 589821K 595477K
I/O read: 0.24s 0.19s 0.17s
Parse time: 5.43s 5.21s 5.01s
Bind time: 1.16s 1.25s 1.43s
Check time: 8.46s 8.97s 9.13s
Total time: 15.05s 15.43s 15.57s

Typescript services

Testing on the src/services directory (in this branch):

Files: 43
Lines: 91284
Nodes: 456582
Identifiers: 168783
Symbols: 156627
Types: 31431
I/O read: 0.04s 0.02s 0.03s 0.02s 0.01s 0.02s

tsc -p src/services --noEmit --diagnostics:

Memory used: 243354K 243332K 235689K
Parse time: 0.75s 0.78s 0.77s
Bind time: 0.46s 0.47s 0.46s
Check time: 3.28s 3.19s 3.24s
Total time: 4.49s 4.43s 4.47s

node ./built/local/tsc.js -p src/services --noEmit --diagnostics

Memory used: 216397K 216644K 217180K
Parse time: 0.79s 0.77s 0.84s
Bind time: 0.39s 0.36s 0.39s
Check time: 2.77s 2.73s 2.84s
Total time: 3.95s 3.87s 4.08s

Single file

Just a file with console.log("Hello world!")

Files: 2
Lines: 19010
Nodes: 98062
Identifiers: 35497
Symbols: 94435
Types: 11926

tsc --diagnostics --noEmit a.ts:

Memory used: 89009K 89023K 89243K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.25s 0.21s 0.21s
Bind time: 0.10s 0.10s 0.08s
Check time: 0.91s 0.99s 0.91s
Total time: 1.26s 1.30s 1.20s

node ../../../TypeScript/built/local/tsc.js --diagnostics --noEmit a.ts:

Memory used: 82048K 81071K 81771K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.21s 0.20s 0.20s
Bind time: 0.12s 0.09s 0.09s
Check time: 0.76s 0.64s 0.74s
Total time: 1.09s 0.93s 1.02s

Interactive test

What I did:

Go to vscode repo
`code src`
Use file navigator to go to vs/base/browser/ui/actionbar/actionbar.ts
Then type ctrl-t, "getZoomLevel"
View memory in task manager
Then go to settings and change "typescript.tsdk" to point to my local version, close all tabs, and start over.

"C:\\Program Files\\nodejs\\node_modules\\typescript\\lib": 272.2MB

"C:\\Users\\anhans\\work\\TypeScript\\built\\local": 265.7MB

@ghostghost changed the title Add SMap and NMap data structuresAdd StringMap, NumberMap, and StringSet data structuresAug 10, 2016
@mhegazy

Copy link
Copy Markdown
Contributor

Andy (Andrewkraft) (@Andy-MS) what were the numbers if we are using the shims and not the native implementation?

@ghost

Deleted user (ghost) commented Sep 28, 2016

Copy link
Copy Markdown
Author

The shims caused an increase in memory due to indirection: the shim was an object pointing to another object in dictionary mode.

With native maps:

Memory used: 611586K
Parse time: 4.88s
Bind time: 1.26s
Check time: 8.22s
Total time: 14.36s
Memory used: 613703K
Parse time: 5.01s
Bind time: 1.26s
Check time: 8.45s
Total time: 14.72s
Memory used: 613734K
Parse time: 4.99s
Bind time: 1.27s
Check time: 8.17s
Total time: 14.43s

With the shims always used:

Memory used: 835746K
Parse time: 5.34s
Bind time: 1.63s
Check time: 11.22s
Total time: 18.19s
Memory used: 820061K
Parse time: 5.40s
Bind time: 1.57s
Check time: 11.30s
Total time: 18.28s
Memory used: 830217K
Parse time: 5.41s
Bind time: 1.58s
Check time: 11.22s
Total time: 18.20s

I have created a new PR #11210 which fixes this issue.

@ghostghost closed this Sep 28, 2016
@microsoftMicrosoft (microsoft) locked and limited conversation to collaborators Jun 19, 2018
This pull request was closed.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@weswigham@rakatyal@mhegazy@yuit@9990859714@msftclas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add StringMap, NumberMap, and StringSet data structures - #10230

Closed
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map
Closed

Add StringMap, NumberMap, and StringSet data structures#10230
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map

Conversation

@ghost

@ghostDeleted user (ghost) commented Aug 9, 2016

Copy link
Copy Markdown

This switches us to using native Maps and Sets where possible.
Since we can't guarantee that the native Map will exist, we instead have StringMap and NumberMap for string-keyed and number-keyed maps, with shims using objects and arrays.
Many remaining uses of the old Map may be better off as StringMap, but I've left alone anything that looks public-facing.
I've also mostly left declarationEmitter and emitter alone. When we merge the transforms branch in we should use the new maps there.

I'd like to assign Raghav Katyal (@rakatyal) to this but it looks like he's not yet on the available assignees list.

…e available, to replace the old Map, which is now ObjMap.
Comment threadsrc/compiler/dataStructures.ts Outdated
}

/** String-keyed Map. */
export interface SMap<V> extends MapCommon<string, V> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it will be easier to read if you just use StringMap instead of SMap and same as NumberMap for NMap

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On that topic, what should we name ObjMap? (The keys aren't objects. The map itself is an object, which is used like a StringMap.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumberKeyedMap, StringKeyedMap, ObjectBackedMap. With good autcomplete, there's no real reason for confusing or ambiguous hungarianesque notation.

Comment threadsrc/compiler/binder.ts Outdated
const typeLiteralSymbol = createSymbol(SymbolFlags.TypeLiteral, "__type");
addDeclarationToSymbol(typeLiteralSymbol, node, SymbolFlags.TypeLiteral);
typeLiteralSymbol.members = { [symbol.name]: symbol };
typeLiteralSymbol.members = singletonMap(symbol.name, symbol);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of singletonMap?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought it was neater than new SMap([[symbol.name, symbol]]).

@weswighamWesley Wigham (weswigham)Aug 9, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing about this is like a singleton in the design pattern sense (it's a loaded term), so the name is a bit confusing. createStringKeyedMap would be more straightforward, if you must wrap the constructor.

@weswigham

Copy link
Copy Markdown
Member

This appears to sort object type keys (which is causing what should be a transparent change to manifest in our baselines). With this change sort order is now insertion order (which arguably is nice), however this is likely a problem because the polyfills right now don't do the same sorting. (This may surface as node 0.10 failing CI with mismatching baselines once the other bugs are fixed)

Comment threadsrc/compiler/dataStructures.ts Outdated
export const SSet: SSetConstructor = Set ? Set : ShimSSet;

/** False iff there are any values in the set. */
export function setIsEmpty(set: SSet): boolean {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be isSetEmpty?

@rakatyal

Copy link
Copy Markdown

Could you also please share some numbers on how this effect both memory and performance in the typescript repo and other use cases?

@ghost

Copy link
Copy Markdown
Author

Overall, we get about a 10% reduction of both memory and total time. (We could get even more by changing the emitter to use StringMaps.)

(Note: since file system accesses are cached, I throw away the first run, which has a large I/O read time.)

VSCode

Testing on the latest checkout of Microsoft/vscode:

Constant:

Files: 1120
Lines: 312630
Nodes: 1591307
Identifiers: 496012
Symbols: 417268
Types: 114545
I/O write: 0.00s (--noEmit)
Emit time: 0.00s (--noEmit)

tsc -p src --noEmit --diagnostics:

Memory used: 704300K 679391K 679408K
I/O read: 0.18s 0.22s 0.19s
Parse time: 5.36s 5.58s 5.60s
Bind time: 1.51s 1.70s 1.64s
Check time: 10.44s 10.24s 9.79s
Total time: 17.30s 17.52s 17.02s

node ../TypeScript/built/local/tsc --noEmit --diagnostics -p src:

Memory used: 593708K 589821K 595477K
I/O read: 0.24s 0.19s 0.17s
Parse time: 5.43s 5.21s 5.01s
Bind time: 1.16s 1.25s 1.43s
Check time: 8.46s 8.97s 9.13s
Total time: 15.05s 15.43s 15.57s

Typescript services

Testing on the src/services directory (in this branch):

Files: 43
Lines: 91284
Nodes: 456582
Identifiers: 168783
Symbols: 156627
Types: 31431
I/O read: 0.04s 0.02s 0.03s 0.02s 0.01s 0.02s

tsc -p src/services --noEmit --diagnostics:

Memory used: 243354K 243332K 235689K
Parse time: 0.75s 0.78s 0.77s
Bind time: 0.46s 0.47s 0.46s
Check time: 3.28s 3.19s 3.24s
Total time: 4.49s 4.43s 4.47s

node ./built/local/tsc.js -p src/services --noEmit --diagnostics

Memory used: 216397K 216644K 217180K
Parse time: 0.79s 0.77s 0.84s
Bind time: 0.39s 0.36s 0.39s
Check time: 2.77s 2.73s 2.84s
Total time: 3.95s 3.87s 4.08s

Single file

Just a file with console.log("Hello world!")

Files: 2
Lines: 19010
Nodes: 98062
Identifiers: 35497
Symbols: 94435
Types: 11926

tsc --diagnostics --noEmit a.ts:

Memory used: 89009K 89023K 89243K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.25s 0.21s 0.21s
Bind time: 0.10s 0.10s 0.08s
Check time: 0.91s 0.99s 0.91s
Total time: 1.26s 1.30s 1.20s

node ../../../TypeScript/built/local/tsc.js --diagnostics --noEmit a.ts:

Memory used: 82048K 81071K 81771K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.21s 0.20s 0.20s
Bind time: 0.12s 0.09s 0.09s
Check time: 0.76s 0.64s 0.74s
Total time: 1.09s 0.93s 1.02s

Interactive test

What I did:

Go to vscode repo
`code src`
Use file navigator to go to vs/base/browser/ui/actionbar/actionbar.ts
Then type ctrl-t, "getZoomLevel"
View memory in task manager
Then go to settings and change "typescript.tsdk" to point to my local version, close all tabs, and start over.

"C:\\Program Files\\nodejs\\node_modules\\typescript\\lib": 272.2MB

"C:\\Users\\anhans\\work\\TypeScript\\built\\local": 265.7MB

@ghostghost changed the title Add SMap and NMap data structuresAdd StringMap, NumberMap, and StringSet data structuresAug 10, 2016
@mhegazy

Copy link
Copy Markdown
Contributor

Andy (Andrewkraft) (@Andy-MS) what were the numbers if we are using the shims and not the native implementation?

@ghost

Deleted user (ghost) commented Sep 28, 2016

Copy link
Copy Markdown
Author

The shims caused an increase in memory due to indirection: the shim was an object pointing to another object in dictionary mode.

With native maps:

Memory used: 611586K
Parse time: 4.88s
Bind time: 1.26s
Check time: 8.22s
Total time: 14.36s
Memory used: 613703K
Parse time: 5.01s
Bind time: 1.26s
Check time: 8.45s
Total time: 14.72s
Memory used: 613734K
Parse time: 4.99s
Bind time: 1.27s
Check time: 8.17s
Total time: 14.43s

With the shims always used:

Memory used: 835746K
Parse time: 5.34s
Bind time: 1.63s
Check time: 11.22s
Total time: 18.19s
Memory used: 820061K
Parse time: 5.40s
Bind time: 1.57s
Check time: 11.30s
Total time: 18.28s
Memory used: 830217K
Parse time: 5.41s
Bind time: 1.58s
Check time: 11.22s
Total time: 18.20s

I have created a new PR #11210 which fixes this issue.

@ghostghost closed this Sep 28, 2016
@microsoftMicrosoft (microsoft) locked and limited conversation to collaborators Jun 19, 2018
This pull request was closed.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@weswigham@rakatyal@mhegazy@yuit@9990859714@msftclas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Add StringMap, NumberMap, and StringSet data structures - #10230

Closed
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map
Closed

Add StringMap, NumberMap, and StringSet data structures#10230
Deleted user (ghost) wants to merge 6 commits into
masterfrom
map

Conversation

@ghost

@ghostDeleted user (ghost) commented Aug 9, 2016

Copy link
Copy Markdown

This switches us to using native Maps and Sets where possible.
Since we can't guarantee that the native Map will exist, we instead have StringMap and NumberMap for string-keyed and number-keyed maps, with shims using objects and arrays.
Many remaining uses of the old Map may be better off as StringMap, but I've left alone anything that looks public-facing.
I've also mostly left declarationEmitter and emitter alone. When we merge the transforms branch in we should use the new maps there.

I'd like to assign Raghav Katyal (@rakatyal) to this but it looks like he's not yet on the available assignees list.

…e available, to replace the old Map, which is now ObjMap.
Comment threadsrc/compiler/dataStructures.ts Outdated
}

/** String-keyed Map. */
export interface SMap<V> extends MapCommon<string, V> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think it will be easier to read if you just use StringMap instead of SMap and same as NumberMap for NMap

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On that topic, what should we name ObjMap? (The keys aren't objects. The map itself is an object, which is used like a StringMap.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumberKeyedMap, StringKeyedMap, ObjectBackedMap. With good autcomplete, there's no real reason for confusing or ambiguous hungarianesque notation.

Comment threadsrc/compiler/binder.ts Outdated
const typeLiteralSymbol = createSymbol(SymbolFlags.TypeLiteral, "__type");
addDeclarationToSymbol(typeLiteralSymbol, node, SymbolFlags.TypeLiteral);
typeLiteralSymbol.members = { [symbol.name]: symbol };
typeLiteralSymbol.members = singletonMap(symbol.name, symbol);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the purpose of singletonMap?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought it was neater than new SMap([[symbol.name, symbol]]).

@weswighamWesley Wigham (weswigham)Aug 9, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing about this is like a singleton in the design pattern sense (it's a loaded term), so the name is a bit confusing. createStringKeyedMap would be more straightforward, if you must wrap the constructor.

@weswigham

Copy link
Copy Markdown
Member

This appears to sort object type keys (which is causing what should be a transparent change to manifest in our baselines). With this change sort order is now insertion order (which arguably is nice), however this is likely a problem because the polyfills right now don't do the same sorting. (This may surface as node 0.10 failing CI with mismatching baselines once the other bugs are fixed)

Comment threadsrc/compiler/dataStructures.ts Outdated
export const SSet: SSetConstructor = Set ? Set : ShimSSet;

/** False iff there are any values in the set. */
export function setIsEmpty(set: SSet): boolean {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be isSetEmpty?

@rakatyal

Copy link
Copy Markdown

Could you also please share some numbers on how this effect both memory and performance in the typescript repo and other use cases?

@ghost

Copy link
Copy Markdown
Author

Overall, we get about a 10% reduction of both memory and total time. (We could get even more by changing the emitter to use StringMaps.)

(Note: since file system accesses are cached, I throw away the first run, which has a large I/O read time.)

VSCode

Testing on the latest checkout of Microsoft/vscode:

Constant:

Files: 1120
Lines: 312630
Nodes: 1591307
Identifiers: 496012
Symbols: 417268
Types: 114545
I/O write: 0.00s (--noEmit)
Emit time: 0.00s (--noEmit)

tsc -p src --noEmit --diagnostics:

Memory used: 704300K 679391K 679408K
I/O read: 0.18s 0.22s 0.19s
Parse time: 5.36s 5.58s 5.60s
Bind time: 1.51s 1.70s 1.64s
Check time: 10.44s 10.24s 9.79s
Total time: 17.30s 17.52s 17.02s

node ../TypeScript/built/local/tsc --noEmit --diagnostics -p src:

Memory used: 593708K 589821K 595477K
I/O read: 0.24s 0.19s 0.17s
Parse time: 5.43s 5.21s 5.01s
Bind time: 1.16s 1.25s 1.43s
Check time: 8.46s 8.97s 9.13s
Total time: 15.05s 15.43s 15.57s

Typescript services

Testing on the src/services directory (in this branch):

Files: 43
Lines: 91284
Nodes: 456582
Identifiers: 168783
Symbols: 156627
Types: 31431
I/O read: 0.04s 0.02s 0.03s 0.02s 0.01s 0.02s

tsc -p src/services --noEmit --diagnostics:

Memory used: 243354K 243332K 235689K
Parse time: 0.75s 0.78s 0.77s
Bind time: 0.46s 0.47s 0.46s
Check time: 3.28s 3.19s 3.24s
Total time: 4.49s 4.43s 4.47s

node ./built/local/tsc.js -p src/services --noEmit --diagnostics

Memory used: 216397K 216644K 217180K
Parse time: 0.79s 0.77s 0.84s
Bind time: 0.39s 0.36s 0.39s
Check time: 2.77s 2.73s 2.84s
Total time: 3.95s 3.87s 4.08s

Single file

Just a file with console.log("Hello world!")

Files: 2
Lines: 19010
Nodes: 98062
Identifiers: 35497
Symbols: 94435
Types: 11926

tsc --diagnostics --noEmit a.ts:

Memory used: 89009K 89023K 89243K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.25s 0.21s 0.21s
Bind time: 0.10s 0.10s 0.08s
Check time: 0.91s 0.99s 0.91s
Total time: 1.26s 1.30s 1.20s

node ../../../TypeScript/built/local/tsc.js --diagnostics --noEmit a.ts:

Memory used: 82048K 81071K 81771K
I/O read: 0.00s 0.00s 0.00s
Parse time: 0.21s 0.20s 0.20s
Bind time: 0.12s 0.09s 0.09s
Check time: 0.76s 0.64s 0.74s
Total time: 1.09s 0.93s 1.02s

Interactive test

What I did:

Go to vscode repo
`code src`
Use file navigator to go to vs/base/browser/ui/actionbar/actionbar.ts
Then type ctrl-t, "getZoomLevel"
View memory in task manager
Then go to settings and change "typescript.tsdk" to point to my local version, close all tabs, and start over.

"C:\\Program Files\\nodejs\\node_modules\\typescript\\lib": 272.2MB

"C:\\Users\\anhans\\work\\TypeScript\\built\\local": 265.7MB

@ghostghost changed the title Add SMap and NMap data structuresAdd StringMap, NumberMap, and StringSet data structuresAug 10, 2016
@mhegazy

Copy link
Copy Markdown
Contributor

Andy (Andrewkraft) (@Andy-MS) what were the numbers if we are using the shims and not the native implementation?

@ghost

Deleted user (ghost) commented Sep 28, 2016

Copy link
Copy Markdown
Author

The shims caused an increase in memory due to indirection: the shim was an object pointing to another object in dictionary mode.

With native maps:

Memory used: 611586K
Parse time: 4.88s
Bind time: 1.26s
Check time: 8.22s
Total time: 14.36s
Memory used: 613703K
Parse time: 5.01s
Bind time: 1.26s
Check time: 8.45s
Total time: 14.72s
Memory used: 613734K
Parse time: 4.99s
Bind time: 1.27s
Check time: 8.17s
Total time: 14.43s

With the shims always used:

Memory used: 835746K
Parse time: 5.34s
Bind time: 1.63s
Check time: 11.22s
Total time: 18.19s
Memory used: 820061K
Parse time: 5.40s
Bind time: 1.57s
Check time: 11.30s
Total time: 18.28s
Memory used: 830217K
Parse time: 5.41s
Bind time: 1.58s
Check time: 11.22s
Total time: 18.20s

I have created a new PR #11210 which fixes this issue.

@ghostghost closed this Sep 28, 2016
@microsoftMicrosoft (microsoft) locked and limited conversation to collaborators Jun 19, 2018
This pull request was closed.
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@weswigham@rakatyal@mhegazy@yuit@9990859714@msftclas