Step by step guide to understand protocol discussion, and decisions to be made - #31

Closed
datapythonista wants to merge 1 commit into
masterfrom
datapythonista_protocol_discussion_summary
Closed

Step by step guide to understand protocol discussion, and decisions to be made#31
datapythonista wants to merge 1 commit into
masterfrom
datapythonista_protocol_discussion_summary

Conversation

@datapythonista

@datapythonistadatapythonista commented Sep 16, 2020

Copy link
Copy Markdown
Member

This document complements #30, and provides detailed information on what are the different options regarding the dataframe exchange protocol, and what are the decisions that need to be made.

Since there are different backgrounds in the group, the aim is that everybody reading this document should be in the same page understanding the different topics of the discussion.

Publishing as a PR, so people can:

  • Disagree on the topics being presented, the different options offered, propose clarifications...
  • Comment with opinions to specific points

At the end of the document I summarize some of the decisions that IMHO need to be made, and that we can discuss in the call on Thursday. Feel free to propose other points.

@maartenbreddelsmaartenbreddels mentioned this pull request Sep 17, 2020
Closed
jorisvandenbossche
jorisvandenbossche reviewed Sep 17, 2020
View reviewed changes
Comment threadworking_documents/data_exchange_protocol_summary.md
- Lack of Python object support (e.g. strings as Python objects in pandas)
- Booleans represented in a single bit, and some implementations may prefer a one byte representation
- No support for bfloat type (not sure if any dataframe implementation uses them currently)
- Limited number of units for timestamps and durations (nano/micro/milli-seconds, and seconds are

@jorisvandenbosschejorisvandenbosscheSep 17, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow supports "extension types" (consisting of one of its native types + metadata on how to interpret it), so as long as the physical storage is supported by Arrow (which I think is the case for all those examples), if needed, all those types could be supported as extension types defined by the dataframe protocol.

@rgommers
rgommers deleted the branch masterNovember 14, 2020 17:30
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@datapythonista@jorisvandenbossche@rgommers
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Step by step guide to understand protocol discussion, and decisions to be made - #31
Closed
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Closed
Step by step guide to understand protocol discussion, and decisions to be made#31
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Conversation
@datapythonista
@datapythonistadatapythonista
commented
Sep 16, 2020

Copy link


Copy Markdown

Member
This document complements #30, and provides detailed information on what are the different options regarding the dataframe exchange protocol, and what are the decisions that need to be made.
Since there are different backgrounds in the group, the aim is that everybody reading this document should be in the same page understanding the different topics of the discussion.
Publishing as a PR, so people can:
  • Disagree on the topics being presented, the different options offered, propose clarifications...
  • Comment with opinions to specific points
At the end of the document I summarize some of the decisions that IMHO need to be made, and that we can discuss in the call on Thursday. Feel free to propose other points.
@maartenbreddelsmaartenbreddels
mentioned this pull request
Sep 17, 2020

Closed

jorisvandenbossche
jorisvandenbossche
reviewed
Sep 17, 2020
View reviewed changes
Comment threadworking_documents/data_exchange_protocol_summary.md
- Lack of Python object support (e.g. strings as Python objects in pandas)
- Booleans represented in a single bit, and some implementations may prefer a one byte representation
- No support for bfloat type (not sure if any dataframe implementation uses them currently)
- Limited number of units for timestamps and durations (nano/micro/milli-seconds, and seconds are
@jorisvandenbosschejorisvandenbosscheSep 17, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow supports "extension types" (consisting of one of its native types + metadata on how to interpret it), so as long as the physical storage is supported by Arrow (which I think is the case for all those examples), if needed, all those types could be supported as extension types defined by the dataframe protocol.
@rgommers
rgommers
deleted the branch
masterNovember 14, 2020 17:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@datapythonista@jorisvandenbossche@rgommers

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Step by step guide to understand protocol discussion, and decisions to be made - #31
Closed
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Closed
Step by step guide to understand protocol discussion, and decisions to be made#31
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Conversation
@datapythonista
@datapythonistadatapythonista
commented
Sep 16, 2020

Copy link


Copy Markdown

Member
This document complements #30, and provides detailed information on what are the different options regarding the dataframe exchange protocol, and what are the decisions that need to be made.
Since there are different backgrounds in the group, the aim is that everybody reading this document should be in the same page understanding the different topics of the discussion.
Publishing as a PR, so people can:
  • Disagree on the topics being presented, the different options offered, propose clarifications...
  • Comment with opinions to specific points
At the end of the document I summarize some of the decisions that IMHO need to be made, and that we can discuss in the call on Thursday. Feel free to propose other points.
@maartenbreddelsmaartenbreddels
mentioned this pull request
Sep 17, 2020

Closed

jorisvandenbossche
jorisvandenbossche
reviewed
Sep 17, 2020
View reviewed changes
Comment threadworking_documents/data_exchange_protocol_summary.md
- Lack of Python object support (e.g. strings as Python objects in pandas)
- Booleans represented in a single bit, and some implementations may prefer a one byte representation
- No support for bfloat type (not sure if any dataframe implementation uses them currently)
- Limited number of units for timestamps and durations (nano/micro/milli-seconds, and seconds are
@jorisvandenbosschejorisvandenbosscheSep 17, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow supports "extension types" (consisting of one of its native types + metadata on how to interpret it), so as long as the physical storage is supported by Arrow (which I think is the case for all those examples), if needed, all those types could be supported as extension types defined by the dataframe protocol.
@rgommers
rgommers
deleted the branch
masterNovember 14, 2020 17:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@datapythonista@jorisvandenbossche@rgommers

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Step by step guide to understand protocol discussion, and decisions to be made - #31
Closed
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Closed
Step by step guide to understand protocol discussion, and decisions to be made#31
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Conversation
@datapythonista
@datapythonistadatapythonista
commented
Sep 16, 2020

Copy link


Copy Markdown

Member
This document complements #30, and provides detailed information on what are the different options regarding the dataframe exchange protocol, and what are the decisions that need to be made.
Since there are different backgrounds in the group, the aim is that everybody reading this document should be in the same page understanding the different topics of the discussion.
Publishing as a PR, so people can:
  • Disagree on the topics being presented, the different options offered, propose clarifications...
  • Comment with opinions to specific points
At the end of the document I summarize some of the decisions that IMHO need to be made, and that we can discuss in the call on Thursday. Feel free to propose other points.
@maartenbreddelsmaartenbreddels
mentioned this pull request
Sep 17, 2020

Closed

jorisvandenbossche
jorisvandenbossche
reviewed
Sep 17, 2020
View reviewed changes
Comment threadworking_documents/data_exchange_protocol_summary.md
- Lack of Python object support (e.g. strings as Python objects in pandas)
- Booleans represented in a single bit, and some implementations may prefer a one byte representation
- No support for bfloat type (not sure if any dataframe implementation uses them currently)
- Limited number of units for timestamps and durations (nano/micro/milli-seconds, and seconds are
@jorisvandenbosschejorisvandenbosscheSep 17, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow supports "extension types" (consisting of one of its native types + metadata on how to interpret it), so as long as the physical storage is supported by Arrow (which I think is the case for all those examples), if needed, all those types could be supported as extension types defined by the dataframe protocol.
@rgommers
rgommers
deleted the branch
masterNovember 14, 2020 17:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@datapythonista@jorisvandenbossche@rgommers

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Step by step guide to understand protocol discussion, and decisions to be made - #31
Closed
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Closed
Step by step guide to understand protocol discussion, and decisions to be made#31
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Conversation
@datapythonista
@datapythonistadatapythonista
commented
Sep 16, 2020

Copy link


Copy Markdown

Member
This document complements #30, and provides detailed information on what are the different options regarding the dataframe exchange protocol, and what are the decisions that need to be made.
Since there are different backgrounds in the group, the aim is that everybody reading this document should be in the same page understanding the different topics of the discussion.
Publishing as a PR, so people can:
  • Disagree on the topics being presented, the different options offered, propose clarifications...
  • Comment with opinions to specific points
At the end of the document I summarize some of the decisions that IMHO need to be made, and that we can discuss in the call on Thursday. Feel free to propose other points.
@maartenbreddelsmaartenbreddels
mentioned this pull request
Sep 17, 2020

Closed

jorisvandenbossche
jorisvandenbossche
reviewed
Sep 17, 2020
View reviewed changes
Comment threadworking_documents/data_exchange_protocol_summary.md
- Lack of Python object support (e.g. strings as Python objects in pandas)
- Booleans represented in a single bit, and some implementations may prefer a one byte representation
- No support for bfloat type (not sure if any dataframe implementation uses them currently)
- Limited number of units for timestamps and durations (nano/micro/milli-seconds, and seconds are
@jorisvandenbosschejorisvandenbosscheSep 17, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow supports "extension types" (consisting of one of its native types + metadata on how to interpret it), so as long as the physical storage is supported by Arrow (which I think is the case for all those examples), if needed, all those types could be supported as extension types defined by the dataframe protocol.
@rgommers
rgommers
deleted the branch
masterNovember 14, 2020 17:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@datapythonista@jorisvandenbossche@rgommers

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Step by step guide to understand protocol discussion, and decisions to be made - #31
Closed
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Closed
Step by step guide to understand protocol discussion, and decisions to be made#31
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Conversation
@datapythonista
@datapythonistadatapythonista
commented
Sep 16, 2020

Copy link


Copy Markdown

Member
This document complements #30, and provides detailed information on what are the different options regarding the dataframe exchange protocol, and what are the decisions that need to be made.
Since there are different backgrounds in the group, the aim is that everybody reading this document should be in the same page understanding the different topics of the discussion.
Publishing as a PR, so people can:
  • Disagree on the topics being presented, the different options offered, propose clarifications...
  • Comment with opinions to specific points
At the end of the document I summarize some of the decisions that IMHO need to be made, and that we can discuss in the call on Thursday. Feel free to propose other points.
@maartenbreddelsmaartenbreddels
mentioned this pull request
Sep 17, 2020

Closed

jorisvandenbossche
jorisvandenbossche
reviewed
Sep 17, 2020
View reviewed changes
Comment threadworking_documents/data_exchange_protocol_summary.md
- Lack of Python object support (e.g. strings as Python objects in pandas)
- Booleans represented in a single bit, and some implementations may prefer a one byte representation
- No support for bfloat type (not sure if any dataframe implementation uses them currently)
- Limited number of units for timestamps and durations (nano/micro/milli-seconds, and seconds are
@jorisvandenbosschejorisvandenbosscheSep 17, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow supports "extension types" (consisting of one of its native types + metadata on how to interpret it), so as long as the physical storage is supported by Arrow (which I think is the case for all those examples), if needed, all those types could be supported as extension types defined by the dataframe protocol.
@rgommers
rgommers
deleted the branch
masterNovember 14, 2020 17:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@datapythonista@jorisvandenbossche@rgommers

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Step by step guide to understand protocol discussion, and decisions to be made - #31
Closed
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Closed
Step by step guide to understand protocol discussion, and decisions to be made#31
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Conversation
@datapythonista
@datapythonistadatapythonista
commented
Sep 16, 2020

Copy link


Copy Markdown

Member
This document complements #30, and provides detailed information on what are the different options regarding the dataframe exchange protocol, and what are the decisions that need to be made.
Since there are different backgrounds in the group, the aim is that everybody reading this document should be in the same page understanding the different topics of the discussion.
Publishing as a PR, so people can:
  • Disagree on the topics being presented, the different options offered, propose clarifications...
  • Comment with opinions to specific points
At the end of the document I summarize some of the decisions that IMHO need to be made, and that we can discuss in the call on Thursday. Feel free to propose other points.
@maartenbreddelsmaartenbreddels
mentioned this pull request
Sep 17, 2020

Closed

jorisvandenbossche
jorisvandenbossche
reviewed
Sep 17, 2020
View reviewed changes
Comment threadworking_documents/data_exchange_protocol_summary.md
- Lack of Python object support (e.g. strings as Python objects in pandas)
- Booleans represented in a single bit, and some implementations may prefer a one byte representation
- No support for bfloat type (not sure if any dataframe implementation uses them currently)
- Limited number of units for timestamps and durations (nano/micro/milli-seconds, and seconds are
@jorisvandenbosschejorisvandenbosscheSep 17, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow supports "extension types" (consisting of one of its native types + metadata on how to interpret it), so as long as the physical storage is supported by Arrow (which I think is the case for all those examples), if needed, all those types could be supported as extension types defined by the dataframe protocol.
@rgommers
rgommers
deleted the branch
masterNovember 14, 2020 17:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@datapythonista@jorisvandenbossche@rgommers

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Step by step guide to understand protocol discussion, and decisions to be made - #31
Closed
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Closed
Step by step guide to understand protocol discussion, and decisions to be made#31
datapythonista wants to merge 1 commit into
masterfrom 
datapythonista_protocol_discussion_summary
Conversation
@datapythonista
@datapythonistadatapythonista
commented
Sep 16, 2020

Copy link


Copy Markdown

Member
This document complements #30, and provides detailed information on what are the different options regarding the dataframe exchange protocol, and what are the decisions that need to be made.
Since there are different backgrounds in the group, the aim is that everybody reading this document should be in the same page understanding the different topics of the discussion.
Publishing as a PR, so people can:
  • Disagree on the topics being presented, the different options offered, propose clarifications...
  • Comment with opinions to specific points
At the end of the document I summarize some of the decisions that IMHO need to be made, and that we can discuss in the call on Thursday. Feel free to propose other points.
@maartenbreddelsmaartenbreddels
mentioned this pull request
Sep 17, 2020

Closed

jorisvandenbossche
jorisvandenbossche
reviewed
Sep 17, 2020
View reviewed changes
Comment threadworking_documents/data_exchange_protocol_summary.md
- Lack of Python object support (e.g. strings as Python objects in pandas)
- Booleans represented in a single bit, and some implementations may prefer a one byte representation
- No support for bfloat type (not sure if any dataframe implementation uses them currently)
- Limited number of units for timestamps and durations (nano/micro/milli-seconds, and seconds are
@jorisvandenbosschejorisvandenbosscheSep 17, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow supports "extension types" (consisting of one of its native types + metadata on how to interpret it), so as long as the physical storage is supported by Arrow (which I think is the case for all those examples), if needed, all those types could be supported as extension types defined by the dataframe protocol.
@rgommers
rgommers
deleted the branch
masterNovember 14, 2020 17:30
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jorisvandenbosschejorisvandenbosschejorisvandenbossche left review comments

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@datapythonista@jorisvandenbossche@rgommers