Fix HTTP header parsing in SubmitBatch request - #2302

Merged
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom
kou:submit-batch-parse-header
Dec 26, 2023
Merged

Fix HTTP header parsing in SubmitBatch request#2302
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom
kou:submit-batch-parse-header

Conversation

@kou

@kouSutou Kouhei (kou) commented Nov 22, 2023

Copy link
Copy Markdown
Contributor

If a SubmitBatch request has a HTTP header that has a HTTP header
delimiter (":") in its value, the SubmitBatch request is failed as
"400 One of the request inputs is not valid."

For example, if user-agent header is azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103), all SubmitBatch() requests are failed.

@kou
Sutou Kouhei (kou)force-pushed the submit-batch-parse-header branch 2 times, most recently from a7122fa to 7c2d4ffCompareNovember 22, 2023 05:00
kou
Sutou Kouhei (kou) commented Nov 22, 2023
View reviewed changes
Comment threadtests/blob/apis/blobbatch.test.ts Outdated
keepAliveOptions: { enable: false },
// Make sure HTTP headers are parsed properly with many HTTP
// header delimiters.
userAgentOptions: { userAgentPrefix: "include:HTTP:header:delimiter " }

@kouSutou Kouhei (kou)Nov 22, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... It seems that this doesn't change user-agent header in SubmitBatch request...

Could someone advice me how to write a test for this case? I want to add header-name: value:value header to SubmitBatch request...

@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Nov 23, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JS SDK doesn't support to set headers in sub requests.
Seems you encountered the issue when you are using CPP sdk?
It should be fine if azurite can pass manual test. Could you share your repro project to me? Then I can run a manual test.

@kouSutou Kouhei (kou)Nov 23, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.
Yes, we're using azure-sdk-for-cpp.
apache/arrow#38793 is the project (and the change) that encounters this problem.
But I disabled the test that encountered this problem for now: https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R601-R604

Here is the test function that encountered this problem:
https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R600-R624

Here is the code that creates a SubmitBatch request:
https://github.com/apache/arrow/pull/38793/files#diff-3bd20682a329dbf3fde0bed6310f07f71cd25312b24d4baa1de5764ebdc8c0acR1020-R1081

If you need more information, please let me know.

Anyway, I'll revert the blobbatch.test.ts change.

@kouSutou Kouhei (kou) mentioned this pull request Nov 22, 2023
Merged
@blueww

Wei Wei (blueww) commented Nov 22, 2023

Copy link
Copy Markdown
Contributor

EmmaZhu-MSFT (@EmmaZhu)

Would you please help to look at the fix for blob batch code?

@kou
Fix HTTP header parsing in SubmitBatch request
15128c7
If a SubmitBatch request has a HTTP header that has a HTTP header
delimiter (":") in its value, the SubmitBatch request is failed as
"400 One of the request inputs is not valid."
For example, if `user-agent` header is
`azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin
Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023;
root:xnu-10002.41.9~6/RELEASE_ARM64_T8103)`, all `SubmitBatch()`
requests are failed.
@EmmaZhu

EmmaZhu-MSFT (EmmaZhu) commented Dec 19, 2023

Copy link
Copy Markdown
Contributor

/check-enforcer reset

@EmmaZhu

EmmaZhu-MSFT (EmmaZhu) commented Dec 19, 2023

Copy link
Copy Markdown
Contributor

I've manually tested the fixed. It works well with a header like testheader: testvalue:test:value in sub requests. Will approve the PR.
Wei Wei (@blueww) , do you have other concerns?

@EmmaZhu

EmmaZhu-MSFT (EmmaZhu) commented Dec 19, 2023

Copy link
Copy Markdown
Contributor

/check-enforcer evaluate

@EmmaZhu

EmmaZhu-MSFT (EmmaZhu) commented Dec 19, 2023

Copy link
Copy Markdown
Contributor

/check-enforcer reset

@EmmaZhu

EmmaZhu-MSFT (EmmaZhu) commented Dec 20, 2023

Copy link
Copy Markdown
Contributor

/check-enforcer evaluate

@felipecrv

Felipe Oliveira Carvalho (felipecrv) commented Dec 21, 2023

Copy link
Copy Markdown

I started hitting this in my unit tests as well now.

@EmmaZhu
EmmaZhu-MSFT (EmmaZhu) merged commit f005399 into Azure:mainDec 26, 2023
@kou
Sutou Kouhei (kou) deleted the submit-batch-parse-header branch December 27, 2023 02:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Awaiting requested review from EmmaZhu
1 more reviewer
@felipecrvFelipe Oliveira Carvalho (felipecrv)felipecrv approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

blob-storage

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kou@blueww@EmmaZhu@felipecrv
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Fix HTTP header parsing in SubmitBatch request - #2302
Merged
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Dec 26, 2023
Merged
Fix HTTP header parsing in SubmitBatch request#2302
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Conversation
@kou
@kouSutou Kouhei (kou)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
If a SubmitBatch request has a HTTP header that has a HTTP header

delimiter (":") in its value, the SubmitBatch request is failed as

"400 One of the request inputs is not valid."
For example, if user-agent header is azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103), all SubmitBatch() requests are failed.
@kou
Sutou Kouhei (kou)force-pushed
the
submit-batch-parse-header
branch
2 times, most recently
from
a7122fa to
7c2d4ffCompareNovember 22, 2023 05:00
kou
Sutou Kouhei (kou)
commented
Nov 22, 2023
View reviewed changes
Comment threadtests/blob/apis/blobbatch.test.ts
Outdated

 keepAliveOptions: { enable: false },
 // Make sure HTTP headers are parsed properly with many HTTP
 // header delimiters.
 userAgentOptions: { userAgentPrefix: "include:HTTP:header:delimiter " }
@kouSutou Kouhei (kou)Nov 22, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... It seems that this doesn't change user-agent header in SubmitBatch request...
Could someone advice me how to write a test for this case? I want to add header-name: value:value header to SubmitBatch request...
@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JS SDK doesn't support to set headers in sub requests.

Seems you encountered the issue when you are using CPP sdk?

It should be fine if azurite can pass manual test. Could you share your repro project to me? Then I can run a manual test.
@kouSutou Kouhei (kou)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

Yes, we're using azure-sdk-for-cpp.
apache/arrow#38793 is the project (and the change) that encounters this problem.

But I disabled the test that encountered this problem for now: https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R601-R604
Here is the test function that encountered this problem:
https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R600-R624
Here is the code that creates a SubmitBatch request:
https://github.com/apache/arrow/pull/38793/files#diff-3bd20682a329dbf3fde0bed6310f07f71cd25312b24d4baa1de5764ebdc8c0acR1020-R1081
If you need more information, please let me know.
Anyway, I'll revert the blobbatch.test.ts change.
@kouSutou Kouhei (kou)
mentioned this pull request
Nov 22, 2023
 Merged

@blueww
Wei Wei (blueww)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
EmmaZhu-MSFT (@EmmaZhu)
Would you please help to look at the fix for blob batch code?
@kou
Fix HTTP header parsing in SubmitBatch request
15128c7
If a SubmitBatch request has a HTTP header that has a HTTP header
delimiter (":") in its value, the SubmitBatch request is failed as
"400 One of the request inputs is not valid."
For example, if `user-agent` header is
`azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin
Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023;
root:xnu-10002.41.9~6/RELEASE_ARM64_T8103)`, all `SubmitBatch()`
requests are failed.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
I've manually tested the fixed. It works well with a header like testheader: testvalue:test:value in sub requests. Will approve the PR.
Wei Wei (@blueww) , do you have other concerns?
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 20, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@felipecrv
Felipe Oliveira Carvalho (felipecrv)
commented
Dec 21, 2023

Copy link


Copy Markdown

I started hitting this in my unit tests as well now.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
merged commit f005399
into
Azure:mainDec 26, 2023
@kou
Sutou Kouhei (kou)
deleted the

submit-batch-parse-header

branch
December 27, 2023 02:06
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Awaiting requested review from EmmaZhu

1 more reviewer

@felipecrvFelipe Oliveira Carvalho (felipecrv)felipecrv approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

blob-storage

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kou@blueww@EmmaZhu@felipecrv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix HTTP header parsing in SubmitBatch request - #2302
Merged
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Dec 26, 2023
Merged
Fix HTTP header parsing in SubmitBatch request#2302
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Conversation
@kou
@kouSutou Kouhei (kou)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
If a SubmitBatch request has a HTTP header that has a HTTP header

delimiter (":") in its value, the SubmitBatch request is failed as

"400 One of the request inputs is not valid."
For example, if user-agent header is azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103), all SubmitBatch() requests are failed.
@kou
Sutou Kouhei (kou)force-pushed
the
submit-batch-parse-header
branch
2 times, most recently
from
a7122fa to
7c2d4ffCompareNovember 22, 2023 05:00
kou
Sutou Kouhei (kou)
commented
Nov 22, 2023
View reviewed changes
Comment threadtests/blob/apis/blobbatch.test.ts
Outdated

 keepAliveOptions: { enable: false },
 // Make sure HTTP headers are parsed properly with many HTTP
 // header delimiters.
 userAgentOptions: { userAgentPrefix: "include:HTTP:header:delimiter " }
@kouSutou Kouhei (kou)Nov 22, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... It seems that this doesn't change user-agent header in SubmitBatch request...
Could someone advice me how to write a test for this case? I want to add header-name: value:value header to SubmitBatch request...
@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JS SDK doesn't support to set headers in sub requests.

Seems you encountered the issue when you are using CPP sdk?

It should be fine if azurite can pass manual test. Could you share your repro project to me? Then I can run a manual test.
@kouSutou Kouhei (kou)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

Yes, we're using azure-sdk-for-cpp.
apache/arrow#38793 is the project (and the change) that encounters this problem.

But I disabled the test that encountered this problem for now: https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R601-R604
Here is the test function that encountered this problem:
https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R600-R624
Here is the code that creates a SubmitBatch request:
https://github.com/apache/arrow/pull/38793/files#diff-3bd20682a329dbf3fde0bed6310f07f71cd25312b24d4baa1de5764ebdc8c0acR1020-R1081
If you need more information, please let me know.
Anyway, I'll revert the blobbatch.test.ts change.
@kouSutou Kouhei (kou)
mentioned this pull request
Nov 22, 2023
 Merged

@blueww
Wei Wei (blueww)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
EmmaZhu-MSFT (@EmmaZhu)
Would you please help to look at the fix for blob batch code?
@kou
Fix HTTP header parsing in SubmitBatch request
15128c7
If a SubmitBatch request has a HTTP header that has a HTTP header
delimiter (":") in its value, the SubmitBatch request is failed as
"400 One of the request inputs is not valid."
For example, if `user-agent` header is
`azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin
Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023;
root:xnu-10002.41.9~6/RELEASE_ARM64_T8103)`, all `SubmitBatch()`
requests are failed.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
I've manually tested the fixed. It works well with a header like testheader: testvalue:test:value in sub requests. Will approve the PR.
Wei Wei (@blueww) , do you have other concerns?
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 20, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@felipecrv
Felipe Oliveira Carvalho (felipecrv)
commented
Dec 21, 2023

Copy link


Copy Markdown

I started hitting this in my unit tests as well now.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
merged commit f005399
into
Azure:mainDec 26, 2023
@kou
Sutou Kouhei (kou)
deleted the

submit-batch-parse-header

branch
December 27, 2023 02:06
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Awaiting requested review from EmmaZhu

1 more reviewer

@felipecrvFelipe Oliveira Carvalho (felipecrv)felipecrv approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

blob-storage

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kou@blueww@EmmaZhu@felipecrv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix HTTP header parsing in SubmitBatch request - #2302
Merged
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Dec 26, 2023
Merged
Fix HTTP header parsing in SubmitBatch request#2302
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Conversation
@kou
@kouSutou Kouhei (kou)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
If a SubmitBatch request has a HTTP header that has a HTTP header

delimiter (":") in its value, the SubmitBatch request is failed as

"400 One of the request inputs is not valid."
For example, if user-agent header is azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103), all SubmitBatch() requests are failed.
@kou
Sutou Kouhei (kou)force-pushed
the
submit-batch-parse-header
branch
2 times, most recently
from
a7122fa to
7c2d4ffCompareNovember 22, 2023 05:00
kou
Sutou Kouhei (kou)
commented
Nov 22, 2023
View reviewed changes
Comment threadtests/blob/apis/blobbatch.test.ts
Outdated

 keepAliveOptions: { enable: false },
 // Make sure HTTP headers are parsed properly with many HTTP
 // header delimiters.
 userAgentOptions: { userAgentPrefix: "include:HTTP:header:delimiter " }
@kouSutou Kouhei (kou)Nov 22, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... It seems that this doesn't change user-agent header in SubmitBatch request...
Could someone advice me how to write a test for this case? I want to add header-name: value:value header to SubmitBatch request...
@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JS SDK doesn't support to set headers in sub requests.

Seems you encountered the issue when you are using CPP sdk?

It should be fine if azurite can pass manual test. Could you share your repro project to me? Then I can run a manual test.
@kouSutou Kouhei (kou)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

Yes, we're using azure-sdk-for-cpp.
apache/arrow#38793 is the project (and the change) that encounters this problem.

But I disabled the test that encountered this problem for now: https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R601-R604
Here is the test function that encountered this problem:
https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R600-R624
Here is the code that creates a SubmitBatch request:
https://github.com/apache/arrow/pull/38793/files#diff-3bd20682a329dbf3fde0bed6310f07f71cd25312b24d4baa1de5764ebdc8c0acR1020-R1081
If you need more information, please let me know.
Anyway, I'll revert the blobbatch.test.ts change.
@kouSutou Kouhei (kou)
mentioned this pull request
Nov 22, 2023
 Merged

@blueww
Wei Wei (blueww)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
EmmaZhu-MSFT (@EmmaZhu)
Would you please help to look at the fix for blob batch code?
@kou
Fix HTTP header parsing in SubmitBatch request
15128c7
If a SubmitBatch request has a HTTP header that has a HTTP header
delimiter (":") in its value, the SubmitBatch request is failed as
"400 One of the request inputs is not valid."
For example, if `user-agent` header is
`azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin
Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023;
root:xnu-10002.41.9~6/RELEASE_ARM64_T8103)`, all `SubmitBatch()`
requests are failed.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
I've manually tested the fixed. It works well with a header like testheader: testvalue:test:value in sub requests. Will approve the PR.
Wei Wei (@blueww) , do you have other concerns?
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 20, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@felipecrv
Felipe Oliveira Carvalho (felipecrv)
commented
Dec 21, 2023

Copy link


Copy Markdown

I started hitting this in my unit tests as well now.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
merged commit f005399
into
Azure:mainDec 26, 2023
@kou
Sutou Kouhei (kou)
deleted the

submit-batch-parse-header

branch
December 27, 2023 02:06
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Awaiting requested review from EmmaZhu

1 more reviewer

@felipecrvFelipe Oliveira Carvalho (felipecrv)felipecrv approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

blob-storage

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kou@blueww@EmmaZhu@felipecrv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Fix HTTP header parsing in SubmitBatch request - #2302
Merged
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Dec 26, 2023
Merged
Fix HTTP header parsing in SubmitBatch request#2302
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Conversation
@kou
@kouSutou Kouhei (kou)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
If a SubmitBatch request has a HTTP header that has a HTTP header

delimiter (":") in its value, the SubmitBatch request is failed as

"400 One of the request inputs is not valid."
For example, if user-agent header is azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103), all SubmitBatch() requests are failed.
@kou
Sutou Kouhei (kou)force-pushed
the
submit-batch-parse-header
branch
2 times, most recently
from
a7122fa to
7c2d4ffCompareNovember 22, 2023 05:00
kou
Sutou Kouhei (kou)
commented
Nov 22, 2023
View reviewed changes
Comment threadtests/blob/apis/blobbatch.test.ts
Outdated

 keepAliveOptions: { enable: false },
 // Make sure HTTP headers are parsed properly with many HTTP
 // header delimiters.
 userAgentOptions: { userAgentPrefix: "include:HTTP:header:delimiter " }
@kouSutou Kouhei (kou)Nov 22, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... It seems that this doesn't change user-agent header in SubmitBatch request...
Could someone advice me how to write a test for this case? I want to add header-name: value:value header to SubmitBatch request...
@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JS SDK doesn't support to set headers in sub requests.

Seems you encountered the issue when you are using CPP sdk?

It should be fine if azurite can pass manual test. Could you share your repro project to me? Then I can run a manual test.
@kouSutou Kouhei (kou)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

Yes, we're using azure-sdk-for-cpp.
apache/arrow#38793 is the project (and the change) that encounters this problem.

But I disabled the test that encountered this problem for now: https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R601-R604
Here is the test function that encountered this problem:
https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R600-R624
Here is the code that creates a SubmitBatch request:
https://github.com/apache/arrow/pull/38793/files#diff-3bd20682a329dbf3fde0bed6310f07f71cd25312b24d4baa1de5764ebdc8c0acR1020-R1081
If you need more information, please let me know.
Anyway, I'll revert the blobbatch.test.ts change.
@kouSutou Kouhei (kou)
mentioned this pull request
Nov 22, 2023
 Merged

@blueww
Wei Wei (blueww)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
EmmaZhu-MSFT (@EmmaZhu)
Would you please help to look at the fix for blob batch code?
@kou
Fix HTTP header parsing in SubmitBatch request
15128c7
If a SubmitBatch request has a HTTP header that has a HTTP header
delimiter (":") in its value, the SubmitBatch request is failed as
"400 One of the request inputs is not valid."
For example, if `user-agent` header is
`azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin
Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023;
root:xnu-10002.41.9~6/RELEASE_ARM64_T8103)`, all `SubmitBatch()`
requests are failed.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
I've manually tested the fixed. It works well with a header like testheader: testvalue:test:value in sub requests. Will approve the PR.
Wei Wei (@blueww) , do you have other concerns?
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 20, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@felipecrv
Felipe Oliveira Carvalho (felipecrv)
commented
Dec 21, 2023

Copy link


Copy Markdown

I started hitting this in my unit tests as well now.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
merged commit f005399
into
Azure:mainDec 26, 2023
@kou
Sutou Kouhei (kou)
deleted the

submit-batch-parse-header

branch
December 27, 2023 02:06
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Awaiting requested review from EmmaZhu

1 more reviewer

@felipecrvFelipe Oliveira Carvalho (felipecrv)felipecrv approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

blob-storage

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kou@blueww@EmmaZhu@felipecrv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix HTTP header parsing in SubmitBatch request - #2302
Merged
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Dec 26, 2023
Merged
Fix HTTP header parsing in SubmitBatch request#2302
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Conversation
@kou
@kouSutou Kouhei (kou)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
If a SubmitBatch request has a HTTP header that has a HTTP header

delimiter (":") in its value, the SubmitBatch request is failed as

"400 One of the request inputs is not valid."
For example, if user-agent header is azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103), all SubmitBatch() requests are failed.
@kou
Sutou Kouhei (kou)force-pushed
the
submit-batch-parse-header
branch
2 times, most recently
from
a7122fa to
7c2d4ffCompareNovember 22, 2023 05:00
kou
Sutou Kouhei (kou)
commented
Nov 22, 2023
View reviewed changes
Comment threadtests/blob/apis/blobbatch.test.ts
Outdated

 keepAliveOptions: { enable: false },
 // Make sure HTTP headers are parsed properly with many HTTP
 // header delimiters.
 userAgentOptions: { userAgentPrefix: "include:HTTP:header:delimiter " }
@kouSutou Kouhei (kou)Nov 22, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... It seems that this doesn't change user-agent header in SubmitBatch request...
Could someone advice me how to write a test for this case? I want to add header-name: value:value header to SubmitBatch request...
@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JS SDK doesn't support to set headers in sub requests.

Seems you encountered the issue when you are using CPP sdk?

It should be fine if azurite can pass manual test. Could you share your repro project to me? Then I can run a manual test.
@kouSutou Kouhei (kou)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

Yes, we're using azure-sdk-for-cpp.
apache/arrow#38793 is the project (and the change) that encounters this problem.

But I disabled the test that encountered this problem for now: https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R601-R604
Here is the test function that encountered this problem:
https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R600-R624
Here is the code that creates a SubmitBatch request:
https://github.com/apache/arrow/pull/38793/files#diff-3bd20682a329dbf3fde0bed6310f07f71cd25312b24d4baa1de5764ebdc8c0acR1020-R1081
If you need more information, please let me know.
Anyway, I'll revert the blobbatch.test.ts change.
@kouSutou Kouhei (kou)
mentioned this pull request
Nov 22, 2023
 Merged

@blueww
Wei Wei (blueww)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
EmmaZhu-MSFT (@EmmaZhu)
Would you please help to look at the fix for blob batch code?
@kou
Fix HTTP header parsing in SubmitBatch request
15128c7
If a SubmitBatch request has a HTTP header that has a HTTP header
delimiter (":") in its value, the SubmitBatch request is failed as
"400 One of the request inputs is not valid."
For example, if `user-agent` header is
`azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin
Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023;
root:xnu-10002.41.9~6/RELEASE_ARM64_T8103)`, all `SubmitBatch()`
requests are failed.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
I've manually tested the fixed. It works well with a header like testheader: testvalue:test:value in sub requests. Will approve the PR.
Wei Wei (@blueww) , do you have other concerns?
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 20, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@felipecrv
Felipe Oliveira Carvalho (felipecrv)
commented
Dec 21, 2023

Copy link


Copy Markdown

I started hitting this in my unit tests as well now.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
merged commit f005399
into
Azure:mainDec 26, 2023
@kou
Sutou Kouhei (kou)
deleted the

submit-batch-parse-header

branch
December 27, 2023 02:06
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Awaiting requested review from EmmaZhu

1 more reviewer

@felipecrvFelipe Oliveira Carvalho (felipecrv)felipecrv approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

blob-storage

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kou@blueww@EmmaZhu@felipecrv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix HTTP header parsing in SubmitBatch request - #2302
Merged
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Dec 26, 2023
Merged
Fix HTTP header parsing in SubmitBatch request#2302
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Conversation
@kou
@kouSutou Kouhei (kou)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
If a SubmitBatch request has a HTTP header that has a HTTP header

delimiter (":") in its value, the SubmitBatch request is failed as

"400 One of the request inputs is not valid."
For example, if user-agent header is azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103), all SubmitBatch() requests are failed.
@kou
Sutou Kouhei (kou)force-pushed
the
submit-batch-parse-header
branch
2 times, most recently
from
a7122fa to
7c2d4ffCompareNovember 22, 2023 05:00
kou
Sutou Kouhei (kou)
commented
Nov 22, 2023
View reviewed changes
Comment threadtests/blob/apis/blobbatch.test.ts
Outdated

 keepAliveOptions: { enable: false },
 // Make sure HTTP headers are parsed properly with many HTTP
 // header delimiters.
 userAgentOptions: { userAgentPrefix: "include:HTTP:header:delimiter " }
@kouSutou Kouhei (kou)Nov 22, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... It seems that this doesn't change user-agent header in SubmitBatch request...
Could someone advice me how to write a test for this case? I want to add header-name: value:value header to SubmitBatch request...
@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JS SDK doesn't support to set headers in sub requests.

Seems you encountered the issue when you are using CPP sdk?

It should be fine if azurite can pass manual test. Could you share your repro project to me? Then I can run a manual test.
@kouSutou Kouhei (kou)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

Yes, we're using azure-sdk-for-cpp.
apache/arrow#38793 is the project (and the change) that encounters this problem.

But I disabled the test that encountered this problem for now: https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R601-R604
Here is the test function that encountered this problem:
https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R600-R624
Here is the code that creates a SubmitBatch request:
https://github.com/apache/arrow/pull/38793/files#diff-3bd20682a329dbf3fde0bed6310f07f71cd25312b24d4baa1de5764ebdc8c0acR1020-R1081
If you need more information, please let me know.
Anyway, I'll revert the blobbatch.test.ts change.
@kouSutou Kouhei (kou)
mentioned this pull request
Nov 22, 2023
 Merged

@blueww
Wei Wei (blueww)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
EmmaZhu-MSFT (@EmmaZhu)
Would you please help to look at the fix for blob batch code?
@kou
Fix HTTP header parsing in SubmitBatch request
15128c7
If a SubmitBatch request has a HTTP header that has a HTTP header
delimiter (":") in its value, the SubmitBatch request is failed as
"400 One of the request inputs is not valid."
For example, if `user-agent` header is
`azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin
Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023;
root:xnu-10002.41.9~6/RELEASE_ARM64_T8103)`, all `SubmitBatch()`
requests are failed.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
I've manually tested the fixed. It works well with a header like testheader: testvalue:test:value in sub requests. Will approve the PR.
Wei Wei (@blueww) , do you have other concerns?
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 20, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@felipecrv
Felipe Oliveira Carvalho (felipecrv)
commented
Dec 21, 2023

Copy link


Copy Markdown

I started hitting this in my unit tests as well now.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
merged commit f005399
into
Azure:mainDec 26, 2023
@kou
Sutou Kouhei (kou)
deleted the

submit-batch-parse-header

branch
December 27, 2023 02:06
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Awaiting requested review from EmmaZhu

1 more reviewer

@felipecrvFelipe Oliveira Carvalho (felipecrv)felipecrv approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

blob-storage

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kou@blueww@EmmaZhu@felipecrv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Fix HTTP header parsing in SubmitBatch request - #2302
Merged
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Dec 26, 2023
Merged
Fix HTTP header parsing in SubmitBatch request#2302
EmmaZhu-MSFT (EmmaZhu) merged 2 commits into
Azure:mainfrom 
kou:submit-batch-parse-header
Conversation
@kou
@kouSutou Kouhei (kou)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
If a SubmitBatch request has a HTTP header that has a HTTP header

delimiter (":") in its value, the SubmitBatch request is failed as

"400 One of the request inputs is not valid."
For example, if user-agent header is azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023; root:xnu-10002.41.9~6/RELEASE_ARM64_T8103), all SubmitBatch() requests are failed.
@kou
Sutou Kouhei (kou)force-pushed
the
submit-batch-parse-header
branch
2 times, most recently
from
a7122fa to
7c2d4ffCompareNovember 22, 2023 05:00
kou
Sutou Kouhei (kou)
commented
Nov 22, 2023
View reviewed changes
Comment threadtests/blob/apis/blobbatch.test.ts
Outdated

 keepAliveOptions: { enable: false },
 // Make sure HTTP headers are parsed properly with many HTTP
 // header delimiters.
 userAgentOptions: { userAgentPrefix: "include:HTTP:header:delimiter " }
@kouSutou Kouhei (kou)Nov 22, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... It seems that this doesn't change user-agent header in SubmitBatch request...
Could someone advice me how to write a test for this case? I want to add header-name: value:value header to SubmitBatch request...
@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JS SDK doesn't support to set headers in sub requests.

Seems you encountered the issue when you are using CPP sdk?

It should be fine if azurite can pass manual test. Could you share your repro project to me? Then I can run a manual test.
@kouSutou Kouhei (kou)Nov 23, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

Yes, we're using azure-sdk-for-cpp.
apache/arrow#38793 is the project (and the change) that encounters this problem.

But I disabled the test that encountered this problem for now: https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R601-R604
Here is the test function that encountered this problem:
https://github.com/apache/arrow/pull/38793/files#diff-7d2cbf9c6abf1ee980b8cf5a79e222543c17b3d93b12c4c0c9fa123b1bb200b6R600-R624
Here is the code that creates a SubmitBatch request:
https://github.com/apache/arrow/pull/38793/files#diff-3bd20682a329dbf3fde0bed6310f07f71cd25312b24d4baa1de5764ebdc8c0acR1020-R1081
If you need more information, please let me know.
Anyway, I'll revert the blobbatch.test.ts change.
@kouSutou Kouhei (kou)
mentioned this pull request
Nov 22, 2023
 Merged

@blueww
Wei Wei (blueww)
commented
Nov 22, 2023

Copy link


Copy Markdown

Contributor
EmmaZhu-MSFT (@EmmaZhu)
Would you please help to look at the fix for blob batch code?
@kou
Fix HTTP header parsing in SubmitBatch request
15128c7
If a SubmitBatch request has a HTTP header that has a HTTP header
delimiter (":") in its value, the SubmitBatch request is failed as
"400 One of the request inputs is not valid."
For example, if `user-agent` header is
`azsdk-cpp-storage-blobs/12.10.0-beta.1 (Darwin 23.1.0 arm64 Darwin
Kernel Version 23.1.0: Mon Oct 9 21:28:12 PDT 2023;
root:xnu-10002.41.9~6/RELEASE_ARM64_T8103)`, all `SubmitBatch()`
requests are failed.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
I've manually tested the fixed. It works well with a header like testheader: testvalue:test:value in sub requests. Will approve the PR.
Wei Wei (@blueww) , do you have other concerns?
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 19, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer reset
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
commented
Dec 20, 2023

Copy link


Copy Markdown

Contributor
/check-enforcer evaluate
@felipecrv
Felipe Oliveira Carvalho (felipecrv)
commented
Dec 21, 2023

Copy link


Copy Markdown

I started hitting this in my unit tests as well now.
@EmmaZhu
EmmaZhu-MSFT (EmmaZhu)
merged commit f005399
into
Azure:mainDec 26, 2023
@kou
Sutou Kouhei (kou)
deleted the

submit-batch-parse-header

branch
December 27, 2023 02:06
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@EmmaZhuEmmaZhu-MSFT (EmmaZhu)Awaiting requested review from EmmaZhu

1 more reviewer

@felipecrvFelipe Oliveira Carvalho (felipecrv)felipecrv approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

blob-storage

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kou@blueww@EmmaZhu@felipecrv