Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions text/0085-source-hash-sourcemap-keying.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
- Start Date: 2023-04-06
- RFC Type: feature
- RFC PR: <link>
- RFC Status: draft

# Summary

This specification proposes the keying / discovering of source maps via
the source hashes of the generated JavaScript files. This is an alternative
lookup method to the proposed debug ID lookup. It's there to support the
`CallSite.getScriptHash` browser API that Chrome and Edge already ship.

# Motivation

We want to make processing / SourceMap-ing of JavaScript stack traces more
reliable. Our preferred solution for this involes unique Debug IDs but
supporting this requires bundler support as well. On the other hand source
hashes for minified files are now possible to support in Chrome and Edge
without toolchain changes.

# Background

Looking up source maps and minified files by source hash is possible because Chrome and Edge support a [callsite script hash lookup](https://github.com/MicrosoftEdge/DevTools/blob/main/explainers/SourceHash/explainer.md).
For at least those browsers it would be possible to associate source maps
purely by that content hash in recent versions.

The hashes are available via the `Error.prepareStackTrace` API by invoking
the `getScriptHash` method on a call site object. There is no fallback
possible for browsers not supporting script hashes.

The script hashes are documented to be SHA-256.

# Implementation

To support this the changes in our pipeline are limited.

## Protocol Changes

The `sourcemap` image type gains an additional key called `hashes` which
holds an object with hashes where each key corresponds to the name of a
hash algorithm and the value to the checksum in a format appropriate for
that hash. Today only `sha256` is specified where the value is a SHA-256
hash in hexadecimal format:

```json
{
"images": [
{
"type": "sourcemap",
"code_file": "http://example.com/file.min.js",
"hashes": {
"sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
]
}
```

## SDK Changes

On the SDK side we need to capture the content hashes of the referenced
frames and build up a map over time. The easiest solution for this is to
maintain a growing list of hashes and to rely on `Error.prepareStackTrace`
to capture this:

```javascript
const scriptHashes = {};
const originalPrepareStackTrace = Error.prepareStackTrace;

// we use `Error.prepareStackTrace` exclusively for capturing the script hashes
// and we rely on `err.stack` parsing for the actual stack trace. As documented
// getScriptHash() always returns a SHA-256 hash.
Error.prepareStackTrace = (err, callsites) => {
for (var callsite of callsites) {
if (!callsite.getScriptHash) {
const filename = callsite.getFileName();
if (!scriptHashes[filename]) {
scriptHashes[filename] = callsite.getScriptHash();
}
}
}
return originalPrepareStackTrace ? originalPrepareStackTrace(err, callsites) : err.stack;
};
```

The SDK can then feed the script hashes into the sent image list as specified
above.

## Toolchain / Source Map Changes

For establishing the mapping from minified JavaScript to the source map, the
source map needs to gain a content hash entry. As there is currently no
specification for how that mapping is supposed to be established, we would
reserve `x-sentry-file-hash-sha256` (header) and `x_sentry_fileHashSha256` (key):

```json
{
"file": "foo.min.js",
"x_sentry_fileHashSha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
```

Note that they use of the embedded key for now is likely discouraged for the
made up header as we do not plan on standardizing the file hashes.

To avoid re-calculating the SHA 256 hashes of the minified files constantly we
also want to add them to the manifest. For this we could leverage the digest
header, with the small suboptimal situation that the HTTP digest spec uses
a base64 hash:
Comment on lines +107 to +109

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just embrace the reality that the headers have nothing to do with HTTP anymore (even though they originally were inspired by that).

This is all a bit bikesheddy, but we should strive for consistency here between the protocol, the sourcemap JSON, and these headers. We use a mix of different names and string/objects all over the place.
I believe we are currently limited by headers being a String->String mapping. But we could relax that as well if needed.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would honestly just move all this stuff out of headers and keep toplevel, but as it stands these are actually still http headers in the current implementation. It really just overlays the HTTP fetch.


```json
{
"files/_/_/foo.min.js": {
"url": "~/foo.min.js",
"type": "minified_source",
"headers": {
"digest": "sha-256=LXEWQrcmsEQBYnyp+6wy9chTD7GQPMTbAiWHF5IaSIE=",
"sourcemap": "foo.min.map"
}
},
"files/_/_/foo.min.map": {
"url": "~/foo.min.map",
"type": "sourcemap",
"headers": {
"x-sentry-file-hash-sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
}
```

Upon uploading `sentry-cli` would establish the associations as today, but in addition
it would hash the file contents and add the necessary cross references into the
manifest. In this case we can get away without any file rewriting and just associate
the hashes with the manifest instead. Like with debug IDs we would require embedding
of sources into the source map.

## Sentry Mapping

On the Sentry side in addition to indexing by `debug_id` we add an additional database
index to record files indexed by it's SHA-256 content hash. For potential future proofing
this table should allow definiting a different hash than SHA-256 in the future.

# Drawbacks

A drawback of this solution is that non-code changes in source files
will generate files with identical hashes. Non-code changes for instance
can be whitespace or, more problematic, comments. The risk of this happening
is assumed to be pretty low by the original spec authors.

This also relies on `Error.prepareStackTrace` being under Sentry's control
to capture the script hashes which might not always be possible.

# Future Direction

This proposal leverages existing developments in the source map ecosystem and some potential
improvements might be added at one point.

## Source File Hashes

There is [a pending proposal](https://github.com/source-map/source-map-rfc/issues/21) to add a
`sourcesHash` array to the source map to add SHA-256 hashes of source files to the source map
which could then be used to access sources.
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions text/0085-source-hash-sourcemap-keying.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
- Start Date: 2023-04-06
- RFC Type: feature
- RFC PR: <link>
- RFC Status: draft

# Summary

This specification proposes the keying / discovering of source maps via
the source hashes of the generated JavaScript files. This is an alternative
lookup method to the proposed debug ID lookup. It's there to support the
`CallSite.getScriptHash` browser API that Chrome and Edge already ship.

# Motivation

We want to make processing / SourceMap-ing of JavaScript stack traces more
reliable. Our preferred solution for this involes unique Debug IDs but
supporting this requires bundler support as well. On the other hand source
hashes for minified files are now possible to support in Chrome and Edge
without toolchain changes.

# Background

Looking up source maps and minified files by source hash is possible because Chrome and Edge support a [callsite script hash lookup](https://github.com/MicrosoftEdge/DevTools/blob/main/explainers/SourceHash/explainer.md).
For at least those browsers it would be possible to associate source maps
purely by that content hash in recent versions.

The hashes are available via the `Error.prepareStackTrace` API by invoking
the `getScriptHash` method on a call site object. There is no fallback
possible for browsers not supporting script hashes.

The script hashes are documented to be SHA-256.

# Implementation

To support this the changes in our pipeline are limited.

## Protocol Changes

The `sourcemap` image type gains an additional key called `hashes` which
holds an object with hashes where each key corresponds to the name of a
hash algorithm and the value to the checksum in a format appropriate for
that hash. Today only `sha256` is specified where the value is a SHA-256
hash in hexadecimal format:

```json
{
"images": [
{
"type": "sourcemap",
"code_file": "http://example.com/file.min.js",
"hashes": {
"sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
]
}
```

## SDK Changes

On the SDK side we need to capture the content hashes of the referenced
frames and build up a map over time. The easiest solution for this is to
maintain a growing list of hashes and to rely on `Error.prepareStackTrace`
to capture this:

```javascript
const scriptHashes = {};
const originalPrepareStackTrace = Error.prepareStackTrace;

// we use `Error.prepareStackTrace` exclusively for capturing the script hashes
// and we rely on `err.stack` parsing for the actual stack trace. As documented
// getScriptHash() always returns a SHA-256 hash.
Error.prepareStackTrace = (err, callsites) => {
for (var callsite of callsites) {
if (!callsite.getScriptHash) {
const filename = callsite.getFileName();
if (!scriptHashes[filename]) {
scriptHashes[filename] = callsite.getScriptHash();
}
}
}
return originalPrepareStackTrace ? originalPrepareStackTrace(err, callsites) : err.stack;
};
```

The SDK can then feed the script hashes into the sent image list as specified
above.

## Toolchain / Source Map Changes

For establishing the mapping from minified JavaScript to the source map, the
source map needs to gain a content hash entry. As there is currently no
specification for how that mapping is supposed to be established, we would
reserve `x-sentry-file-hash-sha256` (header) and `x_sentry_fileHashSha256` (key):

```json
{
"file": "foo.min.js",
"x_sentry_fileHashSha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
```

Note that they use of the embedded key for now is likely discouraged for the
made up header as we do not plan on standardizing the file hashes.

To avoid re-calculating the SHA 256 hashes of the minified files constantly we
also want to add them to the manifest. For this we could leverage the digest
header, with the small suboptimal situation that the HTTP digest spec uses
a base64 hash:
Comment on lines +107 to +109

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just embrace the reality that the headers have nothing to do with HTTP anymore (even though they originally were inspired by that).

This is all a bit bikesheddy, but we should strive for consistency here between the protocol, the sourcemap JSON, and these headers. We use a mix of different names and string/objects all over the place.
I believe we are currently limited by headers being a String->String mapping. But we could relax that as well if needed.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would honestly just move all this stuff out of headers and keep toplevel, but as it stands these are actually still http headers in the current implementation. It really just overlays the HTTP fetch.


```json
{
"files/_/_/foo.min.js": {
"url": "~/foo.min.js",
"type": "minified_source",
"headers": {
"digest": "sha-256=LXEWQrcmsEQBYnyp+6wy9chTD7GQPMTbAiWHF5IaSIE=",
"sourcemap": "foo.min.map"
}
},
"files/_/_/foo.min.map": {
"url": "~/foo.min.map",
"type": "sourcemap",
"headers": {
"x-sentry-file-hash-sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
}
```

Upon uploading `sentry-cli` would establish the associations as today, but in addition
it would hash the file contents and add the necessary cross references into the
manifest. In this case we can get away without any file rewriting and just associate
the hashes with the manifest instead. Like with debug IDs we would require embedding
of sources into the source map.

## Sentry Mapping

On the Sentry side in addition to indexing by `debug_id` we add an additional database
index to record files indexed by it's SHA-256 content hash. For potential future proofing
this table should allow definiting a different hash than SHA-256 in the future.

# Drawbacks

A drawback of this solution is that non-code changes in source files
will generate files with identical hashes. Non-code changes for instance
can be whitespace or, more problematic, comments. The risk of this happening
is assumed to be pretty low by the original spec authors.

This also relies on `Error.prepareStackTrace` being under Sentry's control
to capture the script hashes which might not always be possible.

# Future Direction

This proposal leverages existing developments in the source map ecosystem and some potential
improvements might be added at one point.

## Source File Hashes

There is [a pending proposal](https://github.com/source-map/source-map-rfc/issues/21) to add a
`sourcesHash` array to the source map to add SHA-256 hashes of source files to the source map
which could then be used to access sources.
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions text/0085-source-hash-sourcemap-keying.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
- Start Date: 2023-04-06
- RFC Type: feature
- RFC PR: <link>
- RFC Status: draft

# Summary

This specification proposes the keying / discovering of source maps via
the source hashes of the generated JavaScript files. This is an alternative
lookup method to the proposed debug ID lookup. It's there to support the
`CallSite.getScriptHash` browser API that Chrome and Edge already ship.

# Motivation

We want to make processing / SourceMap-ing of JavaScript stack traces more
reliable. Our preferred solution for this involes unique Debug IDs but
supporting this requires bundler support as well. On the other hand source
hashes for minified files are now possible to support in Chrome and Edge
without toolchain changes.

# Background

Looking up source maps and minified files by source hash is possible because Chrome and Edge support a [callsite script hash lookup](https://github.com/MicrosoftEdge/DevTools/blob/main/explainers/SourceHash/explainer.md).
For at least those browsers it would be possible to associate source maps
purely by that content hash in recent versions.

The hashes are available via the `Error.prepareStackTrace` API by invoking
the `getScriptHash` method on a call site object. There is no fallback
possible for browsers not supporting script hashes.

The script hashes are documented to be SHA-256.

# Implementation

To support this the changes in our pipeline are limited.

## Protocol Changes

The `sourcemap` image type gains an additional key called `hashes` which
holds an object with hashes where each key corresponds to the name of a
hash algorithm and the value to the checksum in a format appropriate for
that hash. Today only `sha256` is specified where the value is a SHA-256
hash in hexadecimal format:

```json
{
"images": [
{
"type": "sourcemap",
"code_file": "http://example.com/file.min.js",
"hashes": {
"sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
]
}
```

## SDK Changes

On the SDK side we need to capture the content hashes of the referenced
frames and build up a map over time. The easiest solution for this is to
maintain a growing list of hashes and to rely on `Error.prepareStackTrace`
to capture this:

```javascript
const scriptHashes = {};
const originalPrepareStackTrace = Error.prepareStackTrace;

// we use `Error.prepareStackTrace` exclusively for capturing the script hashes
// and we rely on `err.stack` parsing for the actual stack trace. As documented
// getScriptHash() always returns a SHA-256 hash.
Error.prepareStackTrace = (err, callsites) => {
for (var callsite of callsites) {
if (!callsite.getScriptHash) {
const filename = callsite.getFileName();
if (!scriptHashes[filename]) {
scriptHashes[filename] = callsite.getScriptHash();
}
}
}
return originalPrepareStackTrace ? originalPrepareStackTrace(err, callsites) : err.stack;
};
```

The SDK can then feed the script hashes into the sent image list as specified
above.

## Toolchain / Source Map Changes

For establishing the mapping from minified JavaScript to the source map, the
source map needs to gain a content hash entry. As there is currently no
specification for how that mapping is supposed to be established, we would
reserve `x-sentry-file-hash-sha256` (header) and `x_sentry_fileHashSha256` (key):

```json
{
"file": "foo.min.js",
"x_sentry_fileHashSha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
```

Note that they use of the embedded key for now is likely discouraged for the
made up header as we do not plan on standardizing the file hashes.

To avoid re-calculating the SHA 256 hashes of the minified files constantly we
also want to add them to the manifest. For this we could leverage the digest
header, with the small suboptimal situation that the HTTP digest spec uses
a base64 hash:
Comment on lines +107 to +109

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just embrace the reality that the headers have nothing to do with HTTP anymore (even though they originally were inspired by that).

This is all a bit bikesheddy, but we should strive for consistency here between the protocol, the sourcemap JSON, and these headers. We use a mix of different names and string/objects all over the place.
I believe we are currently limited by headers being a String->String mapping. But we could relax that as well if needed.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would honestly just move all this stuff out of headers and keep toplevel, but as it stands these are actually still http headers in the current implementation. It really just overlays the HTTP fetch.


```json
{
"files/_/_/foo.min.js": {
"url": "~/foo.min.js",
"type": "minified_source",
"headers": {
"digest": "sha-256=LXEWQrcmsEQBYnyp+6wy9chTD7GQPMTbAiWHF5IaSIE=",
"sourcemap": "foo.min.map"
}
},
"files/_/_/foo.min.map": {
"url": "~/foo.min.map",
"type": "sourcemap",
"headers": {
"x-sentry-file-hash-sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
}
```

Upon uploading `sentry-cli` would establish the associations as today, but in addition
it would hash the file contents and add the necessary cross references into the
manifest. In this case we can get away without any file rewriting and just associate
the hashes with the manifest instead. Like with debug IDs we would require embedding
of sources into the source map.

## Sentry Mapping

On the Sentry side in addition to indexing by `debug_id` we add an additional database
index to record files indexed by it's SHA-256 content hash. For potential future proofing
this table should allow definiting a different hash than SHA-256 in the future.

# Drawbacks

A drawback of this solution is that non-code changes in source files
will generate files with identical hashes. Non-code changes for instance
can be whitespace or, more problematic, comments. The risk of this happening
is assumed to be pretty low by the original spec authors.

This also relies on `Error.prepareStackTrace` being under Sentry's control
to capture the script hashes which might not always be possible.

# Future Direction

This proposal leverages existing developments in the source map ecosystem and some potential
improvements might be added at one point.

## Source File Hashes

There is [a pending proposal](https://github.com/source-map/source-map-rfc/issues/21) to add a
`sourcesHash` array to the source map to add SHA-256 hashes of source files to the source map
which could then be used to access sources.
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions text/0085-source-hash-sourcemap-keying.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
- Start Date: 2023-04-06
- RFC Type: feature
- RFC PR: <link>
- RFC Status: draft

# Summary

This specification proposes the keying / discovering of source maps via
the source hashes of the generated JavaScript files. This is an alternative
lookup method to the proposed debug ID lookup. It's there to support the
`CallSite.getScriptHash` browser API that Chrome and Edge already ship.

# Motivation

We want to make processing / SourceMap-ing of JavaScript stack traces more
reliable. Our preferred solution for this involes unique Debug IDs but
supporting this requires bundler support as well. On the other hand source
hashes for minified files are now possible to support in Chrome and Edge
without toolchain changes.

# Background

Looking up source maps and minified files by source hash is possible because Chrome and Edge support a [callsite script hash lookup](https://github.com/MicrosoftEdge/DevTools/blob/main/explainers/SourceHash/explainer.md).
For at least those browsers it would be possible to associate source maps
purely by that content hash in recent versions.

The hashes are available via the `Error.prepareStackTrace` API by invoking
the `getScriptHash` method on a call site object. There is no fallback
possible for browsers not supporting script hashes.

The script hashes are documented to be SHA-256.

# Implementation

To support this the changes in our pipeline are limited.

## Protocol Changes

The `sourcemap` image type gains an additional key called `hashes` which
holds an object with hashes where each key corresponds to the name of a
hash algorithm and the value to the checksum in a format appropriate for
that hash. Today only `sha256` is specified where the value is a SHA-256
hash in hexadecimal format:

```json
{
"images": [
{
"type": "sourcemap",
"code_file": "http://example.com/file.min.js",
"hashes": {
"sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
]
}
```

## SDK Changes

On the SDK side we need to capture the content hashes of the referenced
frames and build up a map over time. The easiest solution for this is to
maintain a growing list of hashes and to rely on `Error.prepareStackTrace`
to capture this:

```javascript
const scriptHashes = {};
const originalPrepareStackTrace = Error.prepareStackTrace;

// we use `Error.prepareStackTrace` exclusively for capturing the script hashes
// and we rely on `err.stack` parsing for the actual stack trace. As documented
// getScriptHash() always returns a SHA-256 hash.
Error.prepareStackTrace = (err, callsites) => {
for (var callsite of callsites) {
if (!callsite.getScriptHash) {
const filename = callsite.getFileName();
if (!scriptHashes[filename]) {
scriptHashes[filename] = callsite.getScriptHash();
}
}
}
return originalPrepareStackTrace ? originalPrepareStackTrace(err, callsites) : err.stack;
};
```

The SDK can then feed the script hashes into the sent image list as specified
above.

## Toolchain / Source Map Changes

For establishing the mapping from minified JavaScript to the source map, the
source map needs to gain a content hash entry. As there is currently no
specification for how that mapping is supposed to be established, we would
reserve `x-sentry-file-hash-sha256` (header) and `x_sentry_fileHashSha256` (key):

```json
{
"file": "foo.min.js",
"x_sentry_fileHashSha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
```

Note that they use of the embedded key for now is likely discouraged for the
made up header as we do not plan on standardizing the file hashes.

To avoid re-calculating the SHA 256 hashes of the minified files constantly we
also want to add them to the manifest. For this we could leverage the digest
header, with the small suboptimal situation that the HTTP digest spec uses
a base64 hash:
Comment on lines +107 to +109

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just embrace the reality that the headers have nothing to do with HTTP anymore (even though they originally were inspired by that).

This is all a bit bikesheddy, but we should strive for consistency here between the protocol, the sourcemap JSON, and these headers. We use a mix of different names and string/objects all over the place.
I believe we are currently limited by headers being a String->String mapping. But we could relax that as well if needed.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would honestly just move all this stuff out of headers and keep toplevel, but as it stands these are actually still http headers in the current implementation. It really just overlays the HTTP fetch.


```json
{
"files/_/_/foo.min.js": {
"url": "~/foo.min.js",
"type": "minified_source",
"headers": {
"digest": "sha-256=LXEWQrcmsEQBYnyp+6wy9chTD7GQPMTbAiWHF5IaSIE=",
"sourcemap": "foo.min.map"
}
},
"files/_/_/foo.min.map": {
"url": "~/foo.min.map",
"type": "sourcemap",
"headers": {
"x-sentry-file-hash-sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
}
```

Upon uploading `sentry-cli` would establish the associations as today, but in addition
it would hash the file contents and add the necessary cross references into the
manifest. In this case we can get away without any file rewriting and just associate
the hashes with the manifest instead. Like with debug IDs we would require embedding
of sources into the source map.

## Sentry Mapping

On the Sentry side in addition to indexing by `debug_id` we add an additional database
index to record files indexed by it's SHA-256 content hash. For potential future proofing
this table should allow definiting a different hash than SHA-256 in the future.

# Drawbacks

A drawback of this solution is that non-code changes in source files
will generate files with identical hashes. Non-code changes for instance
can be whitespace or, more problematic, comments. The risk of this happening
is assumed to be pretty low by the original spec authors.

This also relies on `Error.prepareStackTrace` being under Sentry's control
to capture the script hashes which might not always be possible.

# Future Direction

This proposal leverages existing developments in the source map ecosystem and some potential
improvements might be added at one point.

## Source File Hashes

There is [a pending proposal](https://github.com/source-map/source-map-rfc/issues/21) to add a
`sourcesHash` array to the source map to add SHA-256 hashes of source files to the source map
which could then be used to access sources.
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions text/0085-source-hash-sourcemap-keying.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
- Start Date: 2023-04-06
- RFC Type: feature
- RFC PR: <link>
- RFC Status: draft

# Summary

This specification proposes the keying / discovering of source maps via
the source hashes of the generated JavaScript files. This is an alternative
lookup method to the proposed debug ID lookup. It's there to support the
`CallSite.getScriptHash` browser API that Chrome and Edge already ship.

# Motivation

We want to make processing / SourceMap-ing of JavaScript stack traces more
reliable. Our preferred solution for this involes unique Debug IDs but
supporting this requires bundler support as well. On the other hand source
hashes for minified files are now possible to support in Chrome and Edge
without toolchain changes.

# Background

Looking up source maps and minified files by source hash is possible because Chrome and Edge support a [callsite script hash lookup](https://github.com/MicrosoftEdge/DevTools/blob/main/explainers/SourceHash/explainer.md).
For at least those browsers it would be possible to associate source maps
purely by that content hash in recent versions.

The hashes are available via the `Error.prepareStackTrace` API by invoking
the `getScriptHash` method on a call site object. There is no fallback
possible for browsers not supporting script hashes.

The script hashes are documented to be SHA-256.

# Implementation

To support this the changes in our pipeline are limited.

## Protocol Changes

The `sourcemap` image type gains an additional key called `hashes` which
holds an object with hashes where each key corresponds to the name of a
hash algorithm and the value to the checksum in a format appropriate for
that hash. Today only `sha256` is specified where the value is a SHA-256
hash in hexadecimal format:

```json
{
"images": [
{
"type": "sourcemap",
"code_file": "http://example.com/file.min.js",
"hashes": {
"sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
]
}
```

## SDK Changes

On the SDK side we need to capture the content hashes of the referenced
frames and build up a map over time. The easiest solution for this is to
maintain a growing list of hashes and to rely on `Error.prepareStackTrace`
to capture this:

```javascript
const scriptHashes = {};
const originalPrepareStackTrace = Error.prepareStackTrace;

// we use `Error.prepareStackTrace` exclusively for capturing the script hashes
// and we rely on `err.stack` parsing for the actual stack trace. As documented
// getScriptHash() always returns a SHA-256 hash.
Error.prepareStackTrace = (err, callsites) => {
for (var callsite of callsites) {
if (!callsite.getScriptHash) {
const filename = callsite.getFileName();
if (!scriptHashes[filename]) {
scriptHashes[filename] = callsite.getScriptHash();
}
}
}
return originalPrepareStackTrace ? originalPrepareStackTrace(err, callsites) : err.stack;
};
```

The SDK can then feed the script hashes into the sent image list as specified
above.

## Toolchain / Source Map Changes

For establishing the mapping from minified JavaScript to the source map, the
source map needs to gain a content hash entry. As there is currently no
specification for how that mapping is supposed to be established, we would
reserve `x-sentry-file-hash-sha256` (header) and `x_sentry_fileHashSha256` (key):

```json
{
"file": "foo.min.js",
"x_sentry_fileHashSha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
```

Note that they use of the embedded key for now is likely discouraged for the
made up header as we do not plan on standardizing the file hashes.

To avoid re-calculating the SHA 256 hashes of the minified files constantly we
also want to add them to the manifest. For this we could leverage the digest
header, with the small suboptimal situation that the HTTP digest spec uses
a base64 hash:
Comment on lines +107 to +109

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just embrace the reality that the headers have nothing to do with HTTP anymore (even though they originally were inspired by that).

This is all a bit bikesheddy, but we should strive for consistency here between the protocol, the sourcemap JSON, and these headers. We use a mix of different names and string/objects all over the place.
I believe we are currently limited by headers being a String->String mapping. But we could relax that as well if needed.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would honestly just move all this stuff out of headers and keep toplevel, but as it stands these are actually still http headers in the current implementation. It really just overlays the HTTP fetch.


```json
{
"files/_/_/foo.min.js": {
"url": "~/foo.min.js",
"type": "minified_source",
"headers": {
"digest": "sha-256=LXEWQrcmsEQBYnyp+6wy9chTD7GQPMTbAiWHF5IaSIE=",
"sourcemap": "foo.min.map"
}
},
"files/_/_/foo.min.map": {
"url": "~/foo.min.map",
"type": "sourcemap",
"headers": {
"x-sentry-file-hash-sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
}
```

Upon uploading `sentry-cli` would establish the associations as today, but in addition
it would hash the file contents and add the necessary cross references into the
manifest. In this case we can get away without any file rewriting and just associate
the hashes with the manifest instead. Like with debug IDs we would require embedding
of sources into the source map.

## Sentry Mapping

On the Sentry side in addition to indexing by `debug_id` we add an additional database
index to record files indexed by it's SHA-256 content hash. For potential future proofing
this table should allow definiting a different hash than SHA-256 in the future.

# Drawbacks

A drawback of this solution is that non-code changes in source files
will generate files with identical hashes. Non-code changes for instance
can be whitespace or, more problematic, comments. The risk of this happening
is assumed to be pretty low by the original spec authors.

This also relies on `Error.prepareStackTrace` being under Sentry's control
to capture the script hashes which might not always be possible.

# Future Direction

This proposal leverages existing developments in the source map ecosystem and some potential
improvements might be added at one point.

## Source File Hashes

There is [a pending proposal](https://github.com/source-map/source-map-rfc/issues/21) to add a
`sourcesHash` array to the source map to add SHA-256 hashes of source files to the source map
which could then be used to access sources.
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions text/0085-source-hash-sourcemap-keying.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
- Start Date: 2023-04-06
- RFC Type: feature
- RFC PR: <link>
- RFC Status: draft

# Summary

This specification proposes the keying / discovering of source maps via
the source hashes of the generated JavaScript files. This is an alternative
lookup method to the proposed debug ID lookup. It's there to support the
`CallSite.getScriptHash` browser API that Chrome and Edge already ship.

# Motivation

We want to make processing / SourceMap-ing of JavaScript stack traces more
reliable. Our preferred solution for this involes unique Debug IDs but
supporting this requires bundler support as well. On the other hand source
hashes for minified files are now possible to support in Chrome and Edge
without toolchain changes.

# Background

Looking up source maps and minified files by source hash is possible because Chrome and Edge support a [callsite script hash lookup](https://github.com/MicrosoftEdge/DevTools/blob/main/explainers/SourceHash/explainer.md).
For at least those browsers it would be possible to associate source maps
purely by that content hash in recent versions.

The hashes are available via the `Error.prepareStackTrace` API by invoking
the `getScriptHash` method on a call site object. There is no fallback
possible for browsers not supporting script hashes.

The script hashes are documented to be SHA-256.

# Implementation

To support this the changes in our pipeline are limited.

## Protocol Changes

The `sourcemap` image type gains an additional key called `hashes` which
holds an object with hashes where each key corresponds to the name of a
hash algorithm and the value to the checksum in a format appropriate for
that hash. Today only `sha256` is specified where the value is a SHA-256
hash in hexadecimal format:

```json
{
"images": [
{
"type": "sourcemap",
"code_file": "http://example.com/file.min.js",
"hashes": {
"sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
]
}
```

## SDK Changes

On the SDK side we need to capture the content hashes of the referenced
frames and build up a map over time. The easiest solution for this is to
maintain a growing list of hashes and to rely on `Error.prepareStackTrace`
to capture this:

```javascript
const scriptHashes = {};
const originalPrepareStackTrace = Error.prepareStackTrace;

// we use `Error.prepareStackTrace` exclusively for capturing the script hashes
// and we rely on `err.stack` parsing for the actual stack trace. As documented
// getScriptHash() always returns a SHA-256 hash.
Error.prepareStackTrace = (err, callsites) => {
for (var callsite of callsites) {
if (!callsite.getScriptHash) {
const filename = callsite.getFileName();
if (!scriptHashes[filename]) {
scriptHashes[filename] = callsite.getScriptHash();
}
}
}
return originalPrepareStackTrace ? originalPrepareStackTrace(err, callsites) : err.stack;
};
```

The SDK can then feed the script hashes into the sent image list as specified
above.

## Toolchain / Source Map Changes

For establishing the mapping from minified JavaScript to the source map, the
source map needs to gain a content hash entry. As there is currently no
specification for how that mapping is supposed to be established, we would
reserve `x-sentry-file-hash-sha256` (header) and `x_sentry_fileHashSha256` (key):

```json
{
"file": "foo.min.js",
"x_sentry_fileHashSha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
```

Note that they use of the embedded key for now is likely discouraged for the
made up header as we do not plan on standardizing the file hashes.

To avoid re-calculating the SHA 256 hashes of the minified files constantly we
also want to add them to the manifest. For this we could leverage the digest
header, with the small suboptimal situation that the HTTP digest spec uses
a base64 hash:
Comment on lines +107 to +109

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just embrace the reality that the headers have nothing to do with HTTP anymore (even though they originally were inspired by that).

This is all a bit bikesheddy, but we should strive for consistency here between the protocol, the sourcemap JSON, and these headers. We use a mix of different names and string/objects all over the place.
I believe we are currently limited by headers being a String->String mapping. But we could relax that as well if needed.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would honestly just move all this stuff out of headers and keep toplevel, but as it stands these are actually still http headers in the current implementation. It really just overlays the HTTP fetch.


```json
{
"files/_/_/foo.min.js": {
"url": "~/foo.min.js",
"type": "minified_source",
"headers": {
"digest": "sha-256=LXEWQrcmsEQBYnyp+6wy9chTD7GQPMTbAiWHF5IaSIE=",
"sourcemap": "foo.min.map"
}
},
"files/_/_/foo.min.map": {
"url": "~/foo.min.map",
"type": "sourcemap",
"headers": {
"x-sentry-file-hash-sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
}
```

Upon uploading `sentry-cli` would establish the associations as today, but in addition
it would hash the file contents and add the necessary cross references into the
manifest. In this case we can get away without any file rewriting and just associate
the hashes with the manifest instead. Like with debug IDs we would require embedding
of sources into the source map.

## Sentry Mapping

On the Sentry side in addition to indexing by `debug_id` we add an additional database
index to record files indexed by it's SHA-256 content hash. For potential future proofing
this table should allow definiting a different hash than SHA-256 in the future.

# Drawbacks

A drawback of this solution is that non-code changes in source files
will generate files with identical hashes. Non-code changes for instance
can be whitespace or, more problematic, comments. The risk of this happening
is assumed to be pretty low by the original spec authors.

This also relies on `Error.prepareStackTrace` being under Sentry's control
to capture the script hashes which might not always be possible.

# Future Direction

This proposal leverages existing developments in the source map ecosystem and some potential
improvements might be added at one point.

## Source File Hashes

There is [a pending proposal](https://github.com/source-map/source-map-rfc/issues/21) to add a
`sourcesHash` array to the source map to add SHA-256 hashes of source files to the source map
which could then be used to access sources.
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions text/0085-source-hash-sourcemap-keying.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
- Start Date: 2023-04-06
- RFC Type: feature
- RFC PR: <link>
- RFC Status: draft

# Summary

This specification proposes the keying / discovering of source maps via
the source hashes of the generated JavaScript files. This is an alternative
lookup method to the proposed debug ID lookup. It's there to support the
`CallSite.getScriptHash` browser API that Chrome and Edge already ship.

# Motivation

We want to make processing / SourceMap-ing of JavaScript stack traces more
reliable. Our preferred solution for this involes unique Debug IDs but
supporting this requires bundler support as well. On the other hand source
hashes for minified files are now possible to support in Chrome and Edge
without toolchain changes.

# Background

Looking up source maps and minified files by source hash is possible because Chrome and Edge support a [callsite script hash lookup](https://github.com/MicrosoftEdge/DevTools/blob/main/explainers/SourceHash/explainer.md).
For at least those browsers it would be possible to associate source maps
purely by that content hash in recent versions.

The hashes are available via the `Error.prepareStackTrace` API by invoking
the `getScriptHash` method on a call site object. There is no fallback
possible for browsers not supporting script hashes.

The script hashes are documented to be SHA-256.

# Implementation

To support this the changes in our pipeline are limited.

## Protocol Changes

The `sourcemap` image type gains an additional key called `hashes` which
holds an object with hashes where each key corresponds to the name of a
hash algorithm and the value to the checksum in a format appropriate for
that hash. Today only `sha256` is specified where the value is a SHA-256
hash in hexadecimal format:

```json
{
"images": [
{
"type": "sourcemap",
"code_file": "http://example.com/file.min.js",
"hashes": {
"sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
]
}
```

## SDK Changes

On the SDK side we need to capture the content hashes of the referenced
frames and build up a map over time. The easiest solution for this is to
maintain a growing list of hashes and to rely on `Error.prepareStackTrace`
to capture this:

```javascript
const scriptHashes = {};
const originalPrepareStackTrace = Error.prepareStackTrace;

// we use `Error.prepareStackTrace` exclusively for capturing the script hashes
// and we rely on `err.stack` parsing for the actual stack trace. As documented
// getScriptHash() always returns a SHA-256 hash.
Error.prepareStackTrace = (err, callsites) => {
for (var callsite of callsites) {
if (!callsite.getScriptHash) {
const filename = callsite.getFileName();
if (!scriptHashes[filename]) {
scriptHashes[filename] = callsite.getScriptHash();
}
}
}
return originalPrepareStackTrace ? originalPrepareStackTrace(err, callsites) : err.stack;
};
```

The SDK can then feed the script hashes into the sent image list as specified
above.

## Toolchain / Source Map Changes

For establishing the mapping from minified JavaScript to the source map, the
source map needs to gain a content hash entry. As there is currently no
specification for how that mapping is supposed to be established, we would
reserve `x-sentry-file-hash-sha256` (header) and `x_sentry_fileHashSha256` (key):

```json
{
"file": "foo.min.js",
"x_sentry_fileHashSha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
```

Note that they use of the embedded key for now is likely discouraged for the
made up header as we do not plan on standardizing the file hashes.

To avoid re-calculating the SHA 256 hashes of the minified files constantly we
also want to add them to the manifest. For this we could leverage the digest
header, with the small suboptimal situation that the HTTP digest spec uses
a base64 hash:
Comment on lines +107 to +109

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just embrace the reality that the headers have nothing to do with HTTP anymore (even though they originally were inspired by that).

This is all a bit bikesheddy, but we should strive for consistency here between the protocol, the sourcemap JSON, and these headers. We use a mix of different names and string/objects all over the place.
I believe we are currently limited by headers being a String->String mapping. But we could relax that as well if needed.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would honestly just move all this stuff out of headers and keep toplevel, but as it stands these are actually still http headers in the current implementation. It really just overlays the HTTP fetch.


```json
{
"files/_/_/foo.min.js": {
"url": "~/foo.min.js",
"type": "minified_source",
"headers": {
"digest": "sha-256=LXEWQrcmsEQBYnyp+6wy9chTD7GQPMTbAiWHF5IaSIE=",
"sourcemap": "foo.min.map"
}
},
"files/_/_/foo.min.map": {
"url": "~/foo.min.map",
"type": "sourcemap",
"headers": {
"x-sentry-file-hash-sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
}
```

Upon uploading `sentry-cli` would establish the associations as today, but in addition
it would hash the file contents and add the necessary cross references into the
manifest. In this case we can get away without any file rewriting and just associate
the hashes with the manifest instead. Like with debug IDs we would require embedding
of sources into the source map.

## Sentry Mapping

On the Sentry side in addition to indexing by `debug_id` we add an additional database
index to record files indexed by it's SHA-256 content hash. For potential future proofing
this table should allow definiting a different hash than SHA-256 in the future.

# Drawbacks

A drawback of this solution is that non-code changes in source files
will generate files with identical hashes. Non-code changes for instance
can be whitespace or, more problematic, comments. The risk of this happening
is assumed to be pretty low by the original spec authors.

This also relies on `Error.prepareStackTrace` being under Sentry's control
to capture the script hashes which might not always be possible.

# Future Direction

This proposal leverages existing developments in the source map ecosystem and some potential
improvements might be added at one point.

## Source File Hashes

There is [a pending proposal](https://github.com/source-map/source-map-rfc/issues/21) to add a
`sourcesHash` array to the source map to add SHA-256 hashes of source files to the source map
which could then be used to access sources.
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions text/0085-source-hash-sourcemap-keying.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
- Start Date: 2023-04-06
- RFC Type: feature
- RFC PR: <link>
- RFC Status: draft

# Summary

This specification proposes the keying / discovering of source maps via
the source hashes of the generated JavaScript files. This is an alternative
lookup method to the proposed debug ID lookup. It's there to support the
`CallSite.getScriptHash` browser API that Chrome and Edge already ship.

# Motivation

We want to make processing / SourceMap-ing of JavaScript stack traces more
reliable. Our preferred solution for this involes unique Debug IDs but
supporting this requires bundler support as well. On the other hand source
hashes for minified files are now possible to support in Chrome and Edge
without toolchain changes.

# Background

Looking up source maps and minified files by source hash is possible because Chrome and Edge support a [callsite script hash lookup](https://github.com/MicrosoftEdge/DevTools/blob/main/explainers/SourceHash/explainer.md).
For at least those browsers it would be possible to associate source maps
purely by that content hash in recent versions.

The hashes are available via the `Error.prepareStackTrace` API by invoking
the `getScriptHash` method on a call site object. There is no fallback
possible for browsers not supporting script hashes.

The script hashes are documented to be SHA-256.

# Implementation

To support this the changes in our pipeline are limited.

## Protocol Changes

The `sourcemap` image type gains an additional key called `hashes` which
holds an object with hashes where each key corresponds to the name of a
hash algorithm and the value to the checksum in a format appropriate for
that hash. Today only `sha256` is specified where the value is a SHA-256
hash in hexadecimal format:

```json
{
"images": [
{
"type": "sourcemap",
"code_file": "http://example.com/file.min.js",
"hashes": {
"sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
]
}
```

## SDK Changes

On the SDK side we need to capture the content hashes of the referenced
frames and build up a map over time. The easiest solution for this is to
maintain a growing list of hashes and to rely on `Error.prepareStackTrace`
to capture this:

```javascript
const scriptHashes = {};
const originalPrepareStackTrace = Error.prepareStackTrace;

// we use `Error.prepareStackTrace` exclusively for capturing the script hashes
// and we rely on `err.stack` parsing for the actual stack trace. As documented
// getScriptHash() always returns a SHA-256 hash.
Error.prepareStackTrace = (err, callsites) => {
for (var callsite of callsites) {
if (!callsite.getScriptHash) {
const filename = callsite.getFileName();
if (!scriptHashes[filename]) {
scriptHashes[filename] = callsite.getScriptHash();
}
}
}
return originalPrepareStackTrace ? originalPrepareStackTrace(err, callsites) : err.stack;
};
```

The SDK can then feed the script hashes into the sent image list as specified
above.

## Toolchain / Source Map Changes

For establishing the mapping from minified JavaScript to the source map, the
source map needs to gain a content hash entry. As there is currently no
specification for how that mapping is supposed to be established, we would
reserve `x-sentry-file-hash-sha256` (header) and `x_sentry_fileHashSha256` (key):

```json
{
"file": "foo.min.js",
"x_sentry_fileHashSha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
```

Note that they use of the embedded key for now is likely discouraged for the
made up header as we do not plan on standardizing the file hashes.

To avoid re-calculating the SHA 256 hashes of the minified files constantly we
also want to add them to the manifest. For this we could leverage the digest
header, with the small suboptimal situation that the HTTP digest spec uses
a base64 hash:
Comment on lines +107 to +109

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just embrace the reality that the headers have nothing to do with HTTP anymore (even though they originally were inspired by that).

This is all a bit bikesheddy, but we should strive for consistency here between the protocol, the sourcemap JSON, and these headers. We use a mix of different names and string/objects all over the place.
I believe we are currently limited by headers being a String->String mapping. But we could relax that as well if needed.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would honestly just move all this stuff out of headers and keep toplevel, but as it stands these are actually still http headers in the current implementation. It really just overlays the HTTP fetch.


```json
{
"files/_/_/foo.min.js": {
"url": "~/foo.min.js",
"type": "minified_source",
"headers": {
"digest": "sha-256=LXEWQrcmsEQBYnyp+6wy9chTD7GQPMTbAiWHF5IaSIE=",
"sourcemap": "foo.min.map"
}
},
"files/_/_/foo.min.map": {
"url": "~/foo.min.map",
"type": "sourcemap",
"headers": {
"x-sentry-file-hash-sha256": "2d711642b726b04401627ca9fbac32f5c8530fb1903cc4db02258717921a4881"
}
}
}
```

Upon uploading `sentry-cli` would establish the associations as today, but in addition
it would hash the file contents and add the necessary cross references into the
manifest. In this case we can get away without any file rewriting and just associate
the hashes with the manifest instead. Like with debug IDs we would require embedding
of sources into the source map.

## Sentry Mapping

On the Sentry side in addition to indexing by `debug_id` we add an additional database
index to record files indexed by it's SHA-256 content hash. For potential future proofing
this table should allow definiting a different hash than SHA-256 in the future.

# Drawbacks

A drawback of this solution is that non-code changes in source files
will generate files with identical hashes. Non-code changes for instance
can be whitespace or, more problematic, comments. The risk of this happening
is assumed to be pretty low by the original spec authors.

This also relies on `Error.prepareStackTrace` being under Sentry's control
to capture the script hashes which might not always be possible.

# Future Direction

This proposal leverages existing developments in the source map ecosystem and some potential
improvements might be added at one point.

## Source File Hashes

There is [a pending proposal](https://github.com/source-map/source-map-rfc/issues/21) to add a
`sourcesHash` array to the source map to add SHA-256 hashes of source files to the source map
which could then be used to access sources.