Skip to content

Add long sequence support for fused attention#237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ksivaman merged 27 commits into from
Jun 22, 2023
Merged

Add long sequence support for fused attention #237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
a78ed36
add long sequence support and unify three backends for fused attention
cyanguwa Jun 12, 2023
d3ceb5b
update cudnn-frontend to v0.9.1
cyanguwa Jun 13, 2023
538b67b
replace cpu_float2half_rn with __float2half_rn
cyanguwa Jun 13, 2023
236fd38
fix backend selection and NVTEDType
cyanguwa Jun 13, 2023
8567bcf
Merge branch 'main' into fused_attn/fp8_f16_c_pytorch
ksivaman Jun 13, 2023
31f0849
minor fixes
cyanguwa Jun 13, 2023
a957430
Merge branch 'fused_attn/fp8_f16_c_pytorch' of github.com:cyanguwa/Tr…
cyanguwa Jun 13, 2023
554b1a9
resolve conflicts
ksivaman Jun 13, 2023
1ca1fe0
Merge branch 'fused_attn/fp8_f16_c_pytorch' of https://github.com/cya…
ksivaman Jun 13, 2023
9db6708
fix ci
ksivaman Jun 13, 2023
e814163
make cudnn plan caches thread_local
cyanguwa Jun 14, 2023
df59f09
fix CI
cyanguwa Jun 14, 2023
2f9be4f
replace cuDNN throw with NVTE_CHECK
cyanguwa Jun 14, 2023
eba2965
fix replacement of cuDNN throw with NVTE_CHECK
cyanguwa Jun 14, 2023
8e1cd7d
Merge branch 'NVIDIA:main' into fused_attn/fp8_f16_c_pytorch
cyanguwa Jun 14, 2023
1be8292
force dropout probablity to 0 in inference mode
cyanguwa Jun 14, 2023
d306a5a
change negInfinity to be consistent with m512 fused attn
cyanguwa Jun 14, 2023
6feec47
remove float2half conversion for scale_dropout
cyanguwa Jun 14, 2023
8210e72
add back runtime api for sm detection
cyanguwa Jun 14, 2023
71a1b95
add gemm3 to enums FP8Fwd/BwdTensors
cyanguwa Jun 14, 2023
d6344c2
merge changes in main
cyanguwa Jun 20, 2023
3c62660
change dropout from no to yes for fmha_v1
cyanguwa Jun 20, 2023
cab1ad6
Merge branch 'main' into fused_attn/fp8_f16_c_pytorch
cyanguwa Jun 21, 2023
b89c567
remove output_rng_state in m512 kernels
cyanguwa Jun 21, 2023
6086419
fix elts_per_thread calculation in kvpacked fwd
cyanguwa Jun 21, 2023
868eb87
remove dropout=0.0 restriction for m512 fused attn
cyanguwa Jun 22, 2023
8056018
remove output_rng_state completely from m512 kernels
cyanguwa Jun 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Add long sequence support for fused attention by cyanguwa · Pull Request #237 · NVIDIA/TransformerEngine · GitHub
Skip to content
Add long sequence support for fused attention#237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

ksivaman
merged 27 commits into

from

Jun 22, 2023
 Merged


Add long sequence support for fused attention
#237
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


27 commits


Select commit
Hold shift + click to select a range
a78ed36

add long sequence support and unify three backends for fused attention


cyanguwa Jun 12, 2023
d3ceb5b

update cudnn-frontend to v0.9.1


cyanguwa Jun 13, 2023
538b67b

replace cpu_float2half_rn with __float2half_rn


cyanguwa Jun 13, 2023
236fd38

fix backend selection and NVTEDType


cyanguwa Jun 13, 2023
8567bcf

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


ksivaman Jun 13, 2023
31f0849

minor fixes


cyanguwa Jun 13, 2023
a957430

Merge branch 'fused_attn/fp8_f16_c_pytorch' of github.com:cyanguwa/Tr…


cyanguwa Jun 13, 2023
554b1a9

resolve conflicts


ksivaman Jun 13, 2023
1ca1fe0

Merge branch 'fused_attn/fp8_f16_c_pytorch' of https://github.com/cya…


ksivaman Jun 13, 2023
9db6708

fix ci


ksivaman Jun 13, 2023
e814163

make cudnn plan caches thread_local


cyanguwa Jun 14, 2023
df59f09

fix CI


cyanguwa Jun 14, 2023
2f9be4f

replace cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
eba2965

fix replacement of cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
8e1cd7d

Merge branch 'NVIDIA:main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 14, 2023
1be8292

force dropout probablity to 0 in inference mode


cyanguwa Jun 14, 2023
d306a5a

change negInfinity to be consistent with m512 fused attn


cyanguwa Jun 14, 2023
6feec47

remove float2half conversion for scale_dropout


cyanguwa Jun 14, 2023
8210e72

add back runtime api for sm detection


cyanguwa Jun 14, 2023
71a1b95

add gemm3 to enums FP8Fwd/BwdTensors


cyanguwa Jun 14, 2023
d6344c2

merge changes in main


cyanguwa Jun 20, 2023
3c62660

change dropout from no to yes for fmha_v1


cyanguwa Jun 20, 2023
cab1ad6

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 21, 2023
b89c567

remove output_rng_state in m512 kernels


cyanguwa Jun 21, 2023
6086419

fix elts_per_thread calculation in kvpacked fwd


cyanguwa Jun 21, 2023
868eb87

remove dropout=0.0 restriction for m512 fused attn


cyanguwa Jun 22, 2023
8056018

remove output_rng_state completely from m512 kernels


cyanguwa Jun 22, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add long sequence support for fused attention by cyanguwa · Pull Request #237 · NVIDIA/TransformerEngine · GitHub
Skip to content
Add long sequence support for fused attention#237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

ksivaman
merged 27 commits into

from

Jun 22, 2023
 Merged


Add long sequence support for fused attention
#237
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


27 commits


Select commit
Hold shift + click to select a range
a78ed36

add long sequence support and unify three backends for fused attention


cyanguwa Jun 12, 2023
d3ceb5b

update cudnn-frontend to v0.9.1


cyanguwa Jun 13, 2023
538b67b

replace cpu_float2half_rn with __float2half_rn


cyanguwa Jun 13, 2023
236fd38

fix backend selection and NVTEDType


cyanguwa Jun 13, 2023
8567bcf

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


ksivaman Jun 13, 2023
31f0849

minor fixes


cyanguwa Jun 13, 2023
a957430

Merge branch 'fused_attn/fp8_f16_c_pytorch' of github.com:cyanguwa/Tr…


cyanguwa Jun 13, 2023
554b1a9

resolve conflicts


ksivaman Jun 13, 2023
1ca1fe0

Merge branch 'fused_attn/fp8_f16_c_pytorch' of https://github.com/cya…


ksivaman Jun 13, 2023
9db6708

fix ci


ksivaman Jun 13, 2023
e814163

make cudnn plan caches thread_local


cyanguwa Jun 14, 2023
df59f09

fix CI


cyanguwa Jun 14, 2023
2f9be4f

replace cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
eba2965

fix replacement of cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
8e1cd7d

Merge branch 'NVIDIA:main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 14, 2023
1be8292

force dropout probablity to 0 in inference mode


cyanguwa Jun 14, 2023
d306a5a

change negInfinity to be consistent with m512 fused attn


cyanguwa Jun 14, 2023
6feec47

remove float2half conversion for scale_dropout


cyanguwa Jun 14, 2023
8210e72

add back runtime api for sm detection


cyanguwa Jun 14, 2023
71a1b95

add gemm3 to enums FP8Fwd/BwdTensors


cyanguwa Jun 14, 2023
d6344c2

merge changes in main


cyanguwa Jun 20, 2023
3c62660

change dropout from no to yes for fmha_v1


cyanguwa Jun 20, 2023
cab1ad6

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 21, 2023
b89c567

remove output_rng_state in m512 kernels


cyanguwa Jun 21, 2023
6086419

fix elts_per_thread calculation in kvpacked fwd


cyanguwa Jun 21, 2023
868eb87

remove dropout=0.0 restriction for m512 fused attn


cyanguwa Jun 22, 2023
8056018

remove output_rng_state completely from m512 kernels


cyanguwa Jun 22, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add long sequence support for fused attention by cyanguwa · Pull Request #237 · NVIDIA/TransformerEngine · GitHub
Skip to content
Add long sequence support for fused attention#237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

ksivaman
merged 27 commits into

from

Jun 22, 2023
 Merged


Add long sequence support for fused attention
#237
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


27 commits


Select commit
Hold shift + click to select a range
a78ed36

add long sequence support and unify three backends for fused attention


cyanguwa Jun 12, 2023
d3ceb5b

update cudnn-frontend to v0.9.1


cyanguwa Jun 13, 2023
538b67b

replace cpu_float2half_rn with __float2half_rn


cyanguwa Jun 13, 2023
236fd38

fix backend selection and NVTEDType


cyanguwa Jun 13, 2023
8567bcf

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


ksivaman Jun 13, 2023
31f0849

minor fixes


cyanguwa Jun 13, 2023
a957430

Merge branch 'fused_attn/fp8_f16_c_pytorch' of github.com:cyanguwa/Tr…


cyanguwa Jun 13, 2023
554b1a9

resolve conflicts


ksivaman Jun 13, 2023
1ca1fe0

Merge branch 'fused_attn/fp8_f16_c_pytorch' of https://github.com/cya…


ksivaman Jun 13, 2023
9db6708

fix ci


ksivaman Jun 13, 2023
e814163

make cudnn plan caches thread_local


cyanguwa Jun 14, 2023
df59f09

fix CI


cyanguwa Jun 14, 2023
2f9be4f

replace cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
eba2965

fix replacement of cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
8e1cd7d

Merge branch 'NVIDIA:main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 14, 2023
1be8292

force dropout probablity to 0 in inference mode


cyanguwa Jun 14, 2023
d306a5a

change negInfinity to be consistent with m512 fused attn


cyanguwa Jun 14, 2023
6feec47

remove float2half conversion for scale_dropout


cyanguwa Jun 14, 2023
8210e72

add back runtime api for sm detection


cyanguwa Jun 14, 2023
71a1b95

add gemm3 to enums FP8Fwd/BwdTensors


cyanguwa Jun 14, 2023
d6344c2

merge changes in main


cyanguwa Jun 20, 2023
3c62660

change dropout from no to yes for fmha_v1


cyanguwa Jun 20, 2023
cab1ad6

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 21, 2023
b89c567

remove output_rng_state in m512 kernels


cyanguwa Jun 21, 2023
6086419

fix elts_per_thread calculation in kvpacked fwd


cyanguwa Jun 21, 2023
868eb87

remove dropout=0.0 restriction for m512 fused attn


cyanguwa Jun 22, 2023
8056018

remove output_rng_state completely from m512 kernels


cyanguwa Jun 22, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Add long sequence support for fused attention by cyanguwa · Pull Request #237 · NVIDIA/TransformerEngine · GitHub
Skip to content
Add long sequence support for fused attention#237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

ksivaman
merged 27 commits into

from

Jun 22, 2023
 Merged


Add long sequence support for fused attention
#237
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


27 commits


Select commit
Hold shift + click to select a range
a78ed36

add long sequence support and unify three backends for fused attention


cyanguwa Jun 12, 2023
d3ceb5b

update cudnn-frontend to v0.9.1


cyanguwa Jun 13, 2023
538b67b

replace cpu_float2half_rn with __float2half_rn


cyanguwa Jun 13, 2023
236fd38

fix backend selection and NVTEDType


cyanguwa Jun 13, 2023
8567bcf

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


ksivaman Jun 13, 2023
31f0849

minor fixes


cyanguwa Jun 13, 2023
a957430

Merge branch 'fused_attn/fp8_f16_c_pytorch' of github.com:cyanguwa/Tr…


cyanguwa Jun 13, 2023
554b1a9

resolve conflicts


ksivaman Jun 13, 2023
1ca1fe0

Merge branch 'fused_attn/fp8_f16_c_pytorch' of https://github.com/cya…


ksivaman Jun 13, 2023
9db6708

fix ci


ksivaman Jun 13, 2023
e814163

make cudnn plan caches thread_local


cyanguwa Jun 14, 2023
df59f09

fix CI


cyanguwa Jun 14, 2023
2f9be4f

replace cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
eba2965

fix replacement of cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
8e1cd7d

Merge branch 'NVIDIA:main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 14, 2023
1be8292

force dropout probablity to 0 in inference mode


cyanguwa Jun 14, 2023
d306a5a

change negInfinity to be consistent with m512 fused attn


cyanguwa Jun 14, 2023
6feec47

remove float2half conversion for scale_dropout


cyanguwa Jun 14, 2023
8210e72

add back runtime api for sm detection


cyanguwa Jun 14, 2023
71a1b95

add gemm3 to enums FP8Fwd/BwdTensors


cyanguwa Jun 14, 2023
d6344c2

merge changes in main


cyanguwa Jun 20, 2023
3c62660

change dropout from no to yes for fmha_v1


cyanguwa Jun 20, 2023
cab1ad6

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 21, 2023
b89c567

remove output_rng_state in m512 kernels


cyanguwa Jun 21, 2023
6086419

fix elts_per_thread calculation in kvpacked fwd


cyanguwa Jun 21, 2023
868eb87

remove dropout=0.0 restriction for m512 fused attn


cyanguwa Jun 22, 2023
8056018

remove output_rng_state completely from m512 kernels


cyanguwa Jun 22, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add long sequence support for fused attention by cyanguwa · Pull Request #237 · NVIDIA/TransformerEngine · GitHub
Skip to content
Add long sequence support for fused attention#237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

ksivaman
merged 27 commits into

from

Jun 22, 2023
 Merged


Add long sequence support for fused attention
#237
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


27 commits


Select commit
Hold shift + click to select a range
a78ed36

add long sequence support and unify three backends for fused attention


cyanguwa Jun 12, 2023
d3ceb5b

update cudnn-frontend to v0.9.1


cyanguwa Jun 13, 2023
538b67b

replace cpu_float2half_rn with __float2half_rn


cyanguwa Jun 13, 2023
236fd38

fix backend selection and NVTEDType


cyanguwa Jun 13, 2023
8567bcf

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


ksivaman Jun 13, 2023
31f0849

minor fixes


cyanguwa Jun 13, 2023
a957430

Merge branch 'fused_attn/fp8_f16_c_pytorch' of github.com:cyanguwa/Tr…


cyanguwa Jun 13, 2023
554b1a9

resolve conflicts


ksivaman Jun 13, 2023
1ca1fe0

Merge branch 'fused_attn/fp8_f16_c_pytorch' of https://github.com/cya…


ksivaman Jun 13, 2023
9db6708

fix ci


ksivaman Jun 13, 2023
e814163

make cudnn plan caches thread_local


cyanguwa Jun 14, 2023
df59f09

fix CI


cyanguwa Jun 14, 2023
2f9be4f

replace cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
eba2965

fix replacement of cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
8e1cd7d

Merge branch 'NVIDIA:main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 14, 2023
1be8292

force dropout probablity to 0 in inference mode


cyanguwa Jun 14, 2023
d306a5a

change negInfinity to be consistent with m512 fused attn


cyanguwa Jun 14, 2023
6feec47

remove float2half conversion for scale_dropout


cyanguwa Jun 14, 2023
8210e72

add back runtime api for sm detection


cyanguwa Jun 14, 2023
71a1b95

add gemm3 to enums FP8Fwd/BwdTensors


cyanguwa Jun 14, 2023
d6344c2

merge changes in main


cyanguwa Jun 20, 2023
3c62660

change dropout from no to yes for fmha_v1


cyanguwa Jun 20, 2023
cab1ad6

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 21, 2023
b89c567

remove output_rng_state in m512 kernels


cyanguwa Jun 21, 2023
6086419

fix elts_per_thread calculation in kvpacked fwd


cyanguwa Jun 21, 2023
868eb87

remove dropout=0.0 restriction for m512 fused attn


cyanguwa Jun 22, 2023
8056018

remove output_rng_state completely from m512 kernels


cyanguwa Jun 22, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add long sequence support for fused attention by cyanguwa · Pull Request #237 · NVIDIA/TransformerEngine · GitHub
Skip to content
Add long sequence support for fused attention#237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

ksivaman
merged 27 commits into

from

Jun 22, 2023
 Merged


Add long sequence support for fused attention
#237
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


27 commits


Select commit
Hold shift + click to select a range
a78ed36

add long sequence support and unify three backends for fused attention


cyanguwa Jun 12, 2023
d3ceb5b

update cudnn-frontend to v0.9.1


cyanguwa Jun 13, 2023
538b67b

replace cpu_float2half_rn with __float2half_rn


cyanguwa Jun 13, 2023
236fd38

fix backend selection and NVTEDType


cyanguwa Jun 13, 2023
8567bcf

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


ksivaman Jun 13, 2023
31f0849

minor fixes


cyanguwa Jun 13, 2023
a957430

Merge branch 'fused_attn/fp8_f16_c_pytorch' of github.com:cyanguwa/Tr…


cyanguwa Jun 13, 2023
554b1a9

resolve conflicts


ksivaman Jun 13, 2023
1ca1fe0

Merge branch 'fused_attn/fp8_f16_c_pytorch' of https://github.com/cya…


ksivaman Jun 13, 2023
9db6708

fix ci


ksivaman Jun 13, 2023
e814163

make cudnn plan caches thread_local


cyanguwa Jun 14, 2023
df59f09

fix CI


cyanguwa Jun 14, 2023
2f9be4f

replace cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
eba2965

fix replacement of cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
8e1cd7d

Merge branch 'NVIDIA:main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 14, 2023
1be8292

force dropout probablity to 0 in inference mode


cyanguwa Jun 14, 2023
d306a5a

change negInfinity to be consistent with m512 fused attn


cyanguwa Jun 14, 2023
6feec47

remove float2half conversion for scale_dropout


cyanguwa Jun 14, 2023
8210e72

add back runtime api for sm detection


cyanguwa Jun 14, 2023
71a1b95

add gemm3 to enums FP8Fwd/BwdTensors


cyanguwa Jun 14, 2023
d6344c2

merge changes in main


cyanguwa Jun 20, 2023
3c62660

change dropout from no to yes for fmha_v1


cyanguwa Jun 20, 2023
cab1ad6

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 21, 2023
b89c567

remove output_rng_state in m512 kernels


cyanguwa Jun 21, 2023
6086419

fix elts_per_thread calculation in kvpacked fwd


cyanguwa Jun 21, 2023
868eb87

remove dropout=0.0 restriction for m512 fused attn


cyanguwa Jun 22, 2023
8056018

remove output_rng_state completely from m512 kernels


cyanguwa Jun 22, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
Add long sequence support for fused attention by cyanguwa · Pull Request #237 · NVIDIA/TransformerEngine · GitHub
Skip to content
Add long sequence support for fused attention#237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

ksivaman
merged 27 commits into

from

Jun 22, 2023
 Merged


Add long sequence support for fused attention
#237
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


27 commits


Select commit
Hold shift + click to select a range
a78ed36

add long sequence support and unify three backends for fused attention


cyanguwa Jun 12, 2023
d3ceb5b

update cudnn-frontend to v0.9.1


cyanguwa Jun 13, 2023
538b67b

replace cpu_float2half_rn with __float2half_rn


cyanguwa Jun 13, 2023
236fd38

fix backend selection and NVTEDType


cyanguwa Jun 13, 2023
8567bcf

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


ksivaman Jun 13, 2023
31f0849

minor fixes


cyanguwa Jun 13, 2023
a957430

Merge branch 'fused_attn/fp8_f16_c_pytorch' of github.com:cyanguwa/Tr…


cyanguwa Jun 13, 2023
554b1a9

resolve conflicts


ksivaman Jun 13, 2023
1ca1fe0

Merge branch 'fused_attn/fp8_f16_c_pytorch' of https://github.com/cya…


ksivaman Jun 13, 2023
9db6708

fix ci


ksivaman Jun 13, 2023
e814163

make cudnn plan caches thread_local


cyanguwa Jun 14, 2023
df59f09

fix CI


cyanguwa Jun 14, 2023
2f9be4f

replace cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
eba2965

fix replacement of cuDNN throw with NVTE_CHECK


cyanguwa Jun 14, 2023
8e1cd7d

Merge branch 'NVIDIA:main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 14, 2023
1be8292

force dropout probablity to 0 in inference mode


cyanguwa Jun 14, 2023
d306a5a

change negInfinity to be consistent with m512 fused attn


cyanguwa Jun 14, 2023
6feec47

remove float2half conversion for scale_dropout


cyanguwa Jun 14, 2023
8210e72

add back runtime api for sm detection


cyanguwa Jun 14, 2023
71a1b95

add gemm3 to enums FP8Fwd/BwdTensors


cyanguwa Jun 14, 2023
d6344c2

merge changes in main


cyanguwa Jun 20, 2023
3c62660

change dropout from no to yes for fmha_v1


cyanguwa Jun 20, 2023
cab1ad6

Merge branch 'main' into fused_attn/fp8_f16_c_pytorch


cyanguwa Jun 21, 2023
b89c567

remove output_rng_state in m512 kernels


cyanguwa Jun 21, 2023
6086419

fix elts_per_thread calculation in kvpacked fwd


cyanguwa Jun 21, 2023
868eb87

remove dropout=0.0 restriction for m512 fused attn


cyanguwa Jun 22, 2023
8056018

remove output_rng_state completely from m512 kernels


cyanguwa Jun 22, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view