GmailElementReplacer refactoring

[rrrooommmaaa]

This PR is WIP

close#5022
close#3944
close#4164
close#4810

Tests(delete all except exactly one):

• Tests added or updated

---

To be filled by reviewers

I have reviewed that this PR... (tick whichever items you personally focused on during this review):

• addresses the issue it closes (if any)
• code is readable and understandable
• is accompanied with tests, or tests are not needed
• is free of vulnerabilities
• is documented clearly and usefully, or doesn't need documentation

[sosnovsky]

@rrrooommmaaa I built extension and now can't open any encrypted message, got such errors:

Error: BrowserMsg(db) sendRawResponse::TypeError: Cannot read properties of undefined (reading 'readKeys')`

Could not open this message with FlowCrypt.
other: TypeCannot read properties of undefined (reading 'readMessage')

Error: BrowserMsg(db) sendRawResponse::TypeError: undefined has no properties

Could not open this message with FlowCrypt.
other: Typeopgp is undefined

[sosnovsky]

BrowserMsg(db) sendRawResponse::TypeError: undefinedhasnopropertiesBrowserMsg.jsonToErr@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:8805:15processRawMsgResponse@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:8720:35[callerStack]BrowserMsg.sendAwait@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:8681:18db@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:8435:36ContactStore.getOneWithAllPubkeys@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:1478:47MessageRenderer.getVerificationPubs@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:11067:44MessageRenderer/this.renderCryptoMessage@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10809:74
url: https://mail.google.com/mail/u/0/#inbox/FMfcgzGsmrCDZttTrQKbnKckXSXKZLzn[/callerStack][responderStack]Catch.rewrapErr@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/platform/catch.js:35:20BrowserMsg.errToJson@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/browser/browser-msg.js:397:38BrowserMsg.sendRawResponse/<@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/browser/browser-msg.js:418:67[typeof:Error:TypeError]undefinedhasnopropertiesOpenPGPKey.parseMany@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/core/crypto/pgp/openpgp-key.js:23:24KeyUtil.parseMany@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/core/crypto/key.js:99:33KeyUtil.parse@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/core/crypto/key.js:89:32ContactStore.sortKeys/pubkeyInfos<@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/platform/store/contact-store.js:716:34ContactStore.sortKeys@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/platform/store/contact-store.js:715:51ContactStore.getOneWithAllPubkeys@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/platform/store/contact-store.js:563:46[/responderStack]

[sosnovsky]

webmail_bundle.js:68 Error: BrowserMsg(db) sendRawResponse::TypeError: Cannotreadpropertiesofundefined(reading'readKeys')atBrowserMsg.jsonToErr(chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:10002:15)atprocessRawMsgResponse(chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:9908:39)[callerStack]atchrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:9877:30atnewPromise(<anonymous>)
at BrowserMsg.sendAwait (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:9869:22)
at Object.db (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:9612:36)
at ContactStore.getOneWithAllPubkeys (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:1549:51)
at MessageRenderer.getVerificationPubs (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:12567:48)
at MessageRenderer.renderCryptoMessage (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:12269:78)
at async MessageRenderer.renderMsgBlock (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:12197:24)
url: https://mail.google.com/mail/u/0/#inbox/FMfcgzGsmDrQHdnkVcnWSFlhJnhfzJnD
[/callerStack]
[responderStack]
Error: sendRawResponse::TypeError: Cannot read properties of undefined (reading 'readKeys')
at Catch.rewrapErr (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/platform/catch.js:35:20)
at BrowserMsg.errToJson (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/browser/browser-msg.js:417:38)
at chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/browser/browser-msg.js:438:67
[typeof:Error:TypeError] Cannot read properties of undefined (reading 'readKeys')
TypeError: Cannot read properties of undefined (reading 'readKeys')
at OpenPGPKey.parseMany (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/core/crypto/pgp/openpgp-key.js:29:39)
at KeyUtil.parseMany (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/core/crypto/key.js:120:37)
at KeyUtil.parse (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/core/crypto/key.js:105:36)
at chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/platform/store/contact-store.js:780:42
at Array.map (<anonymous>)
at ContactStore.sortKeys (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/platform/store/contact-store.js:778:55)
at ContactStore.getOneWithAllPubkeys (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/platform/store/contact-store.js:594:50)
at async BgHandlers.dbOperationHandler (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/background_page/bg-handlers.js:34:16)
at <async> parseMany (/extension/js/common/core/crypto/pgp/openpgp-key.ts:38:28)
at <async> parseMany (/extension/js/common/core/crypto/key.ts:191:28)
at <async> parse (/extension/js/common/core/crypto/key.ts:182:24)
at <async><anonymous> (/extension/js/common/platform/store/contact-store.ts:768:19)
at <async> sortKeys (/extension/js/common/platform/store/contact-store.ts:765:28)
at <async> getOneWithAllPubkeys (/extension/js/common/platform/store/contact-store.ts:549:39)
at <async> dbOperationHandler (/extension/js/background_page/bg-handlers.ts:17:37)
[/responderStack]
at <async> sendAwait (/extension/js/common/browser/browser-msg.ts:456:29)
at <async> getOneWithAllPubkeys (/extension/js/common/platform/store/contact-store.ts:549:39)
at <async> getVerificationPubs (/extension/js/common/message-renderer.ts:215:39)
at <async> renderCryptoMessage (/extension/js/common/message-renderer.ts:686:32)
at <async> renderMsgBlock (/extension/js/common/message-renderer.ts:613:27)
### Catch.reportErr calling stack ###
# at Catch.formatExceptionForReport (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:277:90)
# at Catch.onErrorInternalHandler (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:85:29)
# at Catch.reportErr (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:101:18)
# at chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:12185:27
# # url: https://mail.google.com/mail/u/0/#inbox/FMfcgzGsmDrQHdnkVcnWSFlhJnhfzJnD
# ######################

[rrrooommmaaa]

Could not open this message with FlowCrypt.
other: Typeopgp is undefined

What browser is it?
Have you re-built in full? (with npm install)?

[sosnovsky]

Could not open this message with FlowCrypt.
other: Typeopgp is undefined

This error happened in Firefox.
I tried in both - Chrome and Firefox, there were different errors for different messages.
You can check attached logs from console - #5069 (comment) from Firefox, #5069 (comment) from Chrome

What browser is it? Have you re-built in full? (with npm install)?

Yes, I ran npm install and then also tried to re-build with npm run build

[sosnovsky]

I checked console and there is error even without opening any email, just after loading gmail page:

SyntaxError: expectedexpression,gotkeyword 'return' [openpgp.js:4378:7](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/lib/openpgp.js)
LoadingFlowCrypt8.4.8[webmail_bundle.js:13150:29](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js)
Error: SomekeyscouldnotbeparsedparseAndCheckPrivateKeysmoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10025
# Catch.onErrorInternalHandlermoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:85
# Catch.reportErrmoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:101
# processKeysFromEkmmoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:13364[webmail_bundle.js:68:13](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js)SomekeyscouldnotbeparsedparseAndCheckPrivateKeys@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10025:19[webmail_bundle.js:72:13](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js)
TypeError: opgpisundefinedparseManymoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:4561parseManymoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:3903parsemoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:3893expirationsmoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10126getLocalKeyExpirationmoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10126
# Catch.onErrorInternalHandlermoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:85
# Catch.reportErrmoz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:101[webmail_bundle.js:68:13](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js)opgpisundefinedOpenPGPKey.parseMany@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:4561:24KeyUtil.parseMany@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:3903:33KeyUtil.parse@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:3893:32getLocalKeyExpiration/expirations<@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10126:80getLocalKeyExpiration@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10126:47

[sosnovsky]

Line with error in openpgp.js

4377functionis_bytes(a){4378(returnainstanceofUint8Array||returnainstanceofglobalThis.Uint8Array);4379}

[rrrooommmaaa]

I tried in both - Chrome and Firefox, there were different errors for different messages.

So these are both on Mac?

function is_bytes(a)

I see something wrong with regexes again...

[sosnovsky]

So these are both on Mac?

Yes, that's on Mac, but there wasn't any errors during build. I can re-check later today which regexes from build script may fail on Mac

[rrrooommmaaa]

That's because of greediness.

Should be return (a instanceof Uint8Array || a instanceof globalThis.Uint8Array); instead of ( return a instanceof Uint8Array || return a instanceof globalThis.Uint8Array);

[rrrooommmaaa]

That's because of greediness.

I pushed the lazy-matching fix. Can you build it again on Mac?

[sosnovsky]

That's because of greediness.

I pushed the lazy-matching fix. Can you build it again on Mac?

No, currently I get error:

+ backup_ext=''\'''\'''
+ forfilein"$@"
+ sed '-i'\'''\''' -E 's/\(([^\(\)\x20]+?)\x20instanceof\x20Uint8Array\)/\(\1\x20instanceof\x20Uint8Array\x20\|\|\x20\1\x20instanceof\x20globalThis\.Uint8Array\)/g' ./build/generic-extension-wip/lib/openpgp.js
sed: 1: "s/\(([^\(\)\x20]+?)\x20 ...": RE error: repetition-operator operand invalid

[rrrooommmaaa]

This means the lazy modificator isn't supported. I guess, /U isn't supported either.
I'm a bit confused by this behavior as I specified the space character to be excluded with [^\(\)\x20].
How could return a have been captured?
We can try to replace \x20 with \s if it helps.
But wouldn't be better to use a package with the same behavior on all platforms,
some package installable with npm? e.g. https://www.npmjs.com/package/replace

[sosnovsky]

I updated ISUINT8ARRAY_REGEX3 to be

ISUINT8ARRAY_REGEX3="s/return\x20([^\(\)\x20]+)\x20instanceof\x20Uint8Array;/return\x20\(\1\x20instanceof\x20Uint8Array\x20\|\|\x20\1\x20instanceof\x20globalThis\.Uint8Array\);/g"

and it worked well, all occurrences in openpgp.js were replaced. Maybe let's just use this regex?

[rrrooommmaaa]

Maybe let's just use this regex?

Yes, we can make do with this regex.

[sosnovsky]

Great job, I haven't found any issues - everything works well. Maybe I missed something, as list of changes is very big, but if anything will be found - it can be fixed in another PR, this one can be merged so there won't be merge conflicts with other PRs.