[Python][Azure][Docs] Add documentation about AzureFilesystem #41496

Closed
Closed
[Python][Azure][Docs] Add documentation about AzureFilesystem#41496
Assignees
AlenkaF
Labels
Component: DocumentationComponent: Python
Milestone
21.0.0

Description

@sugibuchi
sugibuchi
opened on May 2, 2024

Describe the enhancement requested

#39968, which is included in version 16.0.0 release, added a Python binding of C++ AzureFileSystem to the PyArrow API.

However, (1) this addition of the native file system implementation for Azure has not yet been documented, and (2) this addition causes a backward compatibility issue in Pandas.

Documentation of the API and usage

We should document AzureFileSystem in the API reference and its usage in the user guide.

Note about the backward compatibility in Pandas

Pandas' read_parquet() and to_parquet() with abfs:// have stopped working in specific cases since PyArrow 16.0.0 due to the addition of AzureFileSystem in PyArrow.

Pandas implement a logic that first tries to get a PyArrow native file system implementation for a given URL and then falls back to fsspec if PyArrow does not have a native implementation for the URL.

https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/io/parquet.py#L116-L124

Due to this fallback logic, Pandas's read_parquet() and to_parquet() always use fssepc with PyArrow before 16.0.0.

With PyArrow 16.0.0, Pandas automatically uses PyArrow's native AzureFileSystem. However, this AzureFileSystem does not use authentication settings set in fsspec's global configuration. Instead, we must explicitly provide an authentication setting to read_parquet() and to_parquet() as storage_options independently of fsspec.

We need to figure out where and how we should document this backward compatibility issue.

Component(s)

Python

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: DocumentationComponent: Python

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    , 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
     blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

    Skip to content
    [Python][Azure][Docs] Add documentation about AzureFilesystem #41496
    Closed
    Closed
    [Python][Azure][Docs] Add documentation about AzureFilesystem#41496
    Assignees
    AlenkaF
    Labels
    Component: DocumentationComponent: Python
    Milestone
    21.0.0
    Description
    @sugibuchi
    sugibuchi
    opened on May 2, 2024
    Describe the enhancement requested
    #39968, which is included in version 16.0.0 release, added a Python binding of C++ AzureFileSystem to the PyArrow API.
    However, (1) this addition of the native file system implementation for Azure has not yet been documented, and (2) this addition causes a backward compatibility issue in Pandas.
    Documentation of the API and usage
    We should document AzureFileSystem in the API reference and its usage in the user guide.
    Note about the backward compatibility in Pandas
    Pandas' read_parquet() and to_parquet() with abfs:// have stopped working in specific cases since PyArrow 16.0.0 due to the addition of AzureFileSystem in PyArrow.
    Pandas implement a logic that first tries to get a PyArrow native file system implementation for a given URL and then falls back to fsspec if PyArrow does not have a native implementation for the URL.
    https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/io/parquet.py#L116-L124
    Due to this fallback logic, Pandas's read_parquet() and to_parquet() always use fssepc with PyArrow before 16.0.0.
    With PyArrow 16.0.0, Pandas automatically uses PyArrow's native AzureFileSystem. However, this AzureFileSystem does not use authentication settings set in fsspec's global configuration. Instead, we must explicitly provide an authentication setting to read_parquet() and to_parquet() as storage_options independently of fsspec.
    We need to figure out where and how we should document this backward compatibility issue.
    Component(s)
    Python
    Activity
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Metadata
    Metadata
    Assignees
    Labels
    Component: DocumentationComponent: Python
    Type
    No type
    Projects
    No projects
      Milestone
      Relationships
      None yet
      Development
      No branches or pull requests
      Issue actions
      
, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

      Skip to content
      [Python][Azure][Docs] Add documentation about AzureFilesystem #41496
      Closed
      Closed
      [Python][Azure][Docs] Add documentation about AzureFilesystem#41496
      Assignees
      AlenkaF
      Labels
      Component: DocumentationComponent: Python
      Milestone
      21.0.0
      Description
      @sugibuchi
      sugibuchi
      opened on May 2, 2024
      Describe the enhancement requested
      #39968, which is included in version 16.0.0 release, added a Python binding of C++ AzureFileSystem to the PyArrow API.
      However, (1) this addition of the native file system implementation for Azure has not yet been documented, and (2) this addition causes a backward compatibility issue in Pandas.
      Documentation of the API and usage
      We should document AzureFileSystem in the API reference and its usage in the user guide.
      Note about the backward compatibility in Pandas
      Pandas' read_parquet() and to_parquet() with abfs:// have stopped working in specific cases since PyArrow 16.0.0 due to the addition of AzureFileSystem in PyArrow.
      Pandas implement a logic that first tries to get a PyArrow native file system implementation for a given URL and then falls back to fsspec if PyArrow does not have a native implementation for the URL.
      https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/io/parquet.py#L116-L124
      Due to this fallback logic, Pandas's read_parquet() and to_parquet() always use fssepc with PyArrow before 16.0.0.
      With PyArrow 16.0.0, Pandas automatically uses PyArrow's native AzureFileSystem. However, this AzureFileSystem does not use authentication settings set in fsspec's global configuration. Instead, we must explicitly provide an authentication setting to read_parquet() and to_parquet() as storage_options independently of fsspec.
      We need to figure out where and how we should document this backward compatibility issue.
      Component(s)
      Python
      Activity
      Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
      Metadata
      Metadata
      Assignees
      Labels
      Component: DocumentationComponent: Python
      Type
      No type
      Projects
      No projects
        Milestone
        Relationships
        None yet
        Development
        No branches or pull requests
        Issue actions
        
, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

        Skip to content
        [Python][Azure][Docs] Add documentation about AzureFilesystem #41496
        Closed
        Closed
        [Python][Azure][Docs] Add documentation about AzureFilesystem#41496
        Assignees
        AlenkaF
        Labels
        Component: DocumentationComponent: Python
        Milestone
        21.0.0
        Description
        @sugibuchi
        sugibuchi
        opened on May 2, 2024
        Describe the enhancement requested
        #39968, which is included in version 16.0.0 release, added a Python binding of C++ AzureFileSystem to the PyArrow API.
        However, (1) this addition of the native file system implementation for Azure has not yet been documented, and (2) this addition causes a backward compatibility issue in Pandas.
        Documentation of the API and usage
        We should document AzureFileSystem in the API reference and its usage in the user guide.
        Note about the backward compatibility in Pandas
        Pandas' read_parquet() and to_parquet() with abfs:// have stopped working in specific cases since PyArrow 16.0.0 due to the addition of AzureFileSystem in PyArrow.
        Pandas implement a logic that first tries to get a PyArrow native file system implementation for a given URL and then falls back to fsspec if PyArrow does not have a native implementation for the URL.
        https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/io/parquet.py#L116-L124
        Due to this fallback logic, Pandas's read_parquet() and to_parquet() always use fssepc with PyArrow before 16.0.0.
        With PyArrow 16.0.0, Pandas automatically uses PyArrow's native AzureFileSystem. However, this AzureFileSystem does not use authentication settings set in fsspec's global configuration. Instead, we must explicitly provide an authentication setting to read_parquet() and to_parquet() as storage_options independently of fsspec.
        We need to figure out where and how we should document this backward compatibility issue.
        Component(s)
        Python
        Activity
        Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
        Metadata
        Metadata
        Assignees
        Labels
        Component: DocumentationComponent: Python
        Type
        No type
        Projects
        No projects
          Milestone
          Relationships
          None yet
          Development
          No branches or pull requests
          Issue actions
          
, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

          Skip to content
          [Python][Azure][Docs] Add documentation about AzureFilesystem #41496
          Closed
          Closed
          [Python][Azure][Docs] Add documentation about AzureFilesystem#41496
          Assignees
          AlenkaF
          Labels
          Component: DocumentationComponent: Python
          Milestone
          21.0.0
          Description
          @sugibuchi
          sugibuchi
          opened on May 2, 2024
          Describe the enhancement requested
          #39968, which is included in version 16.0.0 release, added a Python binding of C++ AzureFileSystem to the PyArrow API.
          However, (1) this addition of the native file system implementation for Azure has not yet been documented, and (2) this addition causes a backward compatibility issue in Pandas.
          Documentation of the API and usage
          We should document AzureFileSystem in the API reference and its usage in the user guide.
          Note about the backward compatibility in Pandas
          Pandas' read_parquet() and to_parquet() with abfs:// have stopped working in specific cases since PyArrow 16.0.0 due to the addition of AzureFileSystem in PyArrow.
          Pandas implement a logic that first tries to get a PyArrow native file system implementation for a given URL and then falls back to fsspec if PyArrow does not have a native implementation for the URL.
          https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/io/parquet.py#L116-L124
          Due to this fallback logic, Pandas's read_parquet() and to_parquet() always use fssepc with PyArrow before 16.0.0.
          With PyArrow 16.0.0, Pandas automatically uses PyArrow's native AzureFileSystem. However, this AzureFileSystem does not use authentication settings set in fsspec's global configuration. Instead, we must explicitly provide an authentication setting to read_parquet() and to_parquet() as storage_options independently of fsspec.
          We need to figure out where and how we should document this backward compatibility issue.
          Component(s)
          Python
          Activity
          Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
          Metadata
          Metadata
          Assignees
          Labels
          Component: DocumentationComponent: Python
          Type
          No type
          Projects
          No projects
            Milestone
            Relationships
            None yet
            Development
            No branches or pull requests
            Issue actions
            
, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

            Skip to content
            [Python][Azure][Docs] Add documentation about AzureFilesystem #41496
            Closed
            Closed
            [Python][Azure][Docs] Add documentation about AzureFilesystem#41496
            Assignees
            AlenkaF
            Labels
            Component: DocumentationComponent: Python
            Milestone
            21.0.0
            Description
            @sugibuchi
            sugibuchi
            opened on May 2, 2024
            Describe the enhancement requested
            #39968, which is included in version 16.0.0 release, added a Python binding of C++ AzureFileSystem to the PyArrow API.
            However, (1) this addition of the native file system implementation for Azure has not yet been documented, and (2) this addition causes a backward compatibility issue in Pandas.
            Documentation of the API and usage
            We should document AzureFileSystem in the API reference and its usage in the user guide.
            Note about the backward compatibility in Pandas
            Pandas' read_parquet() and to_parquet() with abfs:// have stopped working in specific cases since PyArrow 16.0.0 due to the addition of AzureFileSystem in PyArrow.
            Pandas implement a logic that first tries to get a PyArrow native file system implementation for a given URL and then falls back to fsspec if PyArrow does not have a native implementation for the URL.
            https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/io/parquet.py#L116-L124
            Due to this fallback logic, Pandas's read_parquet() and to_parquet() always use fssepc with PyArrow before 16.0.0.
            With PyArrow 16.0.0, Pandas automatically uses PyArrow's native AzureFileSystem. However, this AzureFileSystem does not use authentication settings set in fsspec's global configuration. Instead, we must explicitly provide an authentication setting to read_parquet() and to_parquet() as storage_options independently of fsspec.
            We need to figure out where and how we should document this backward compatibility issue.
            Component(s)
            Python
            Activity
            Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
            Metadata
            Metadata
            Assignees
            Labels
            Component: DocumentationComponent: Python
            Type
            No type
            Projects
            No projects
              Milestone
              Relationships
              None yet
              Development
              No branches or pull requests
              Issue actions
              
, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

              Skip to content
              [Python][Azure][Docs] Add documentation about AzureFilesystem #41496
              Closed
              Closed
              [Python][Azure][Docs] Add documentation about AzureFilesystem#41496
              Assignees
              AlenkaF
              Labels
              Component: DocumentationComponent: Python
              Milestone
              21.0.0
              Description
              @sugibuchi
              sugibuchi
              opened on May 2, 2024
              Describe the enhancement requested
              #39968, which is included in version 16.0.0 release, added a Python binding of C++ AzureFileSystem to the PyArrow API.
              However, (1) this addition of the native file system implementation for Azure has not yet been documented, and (2) this addition causes a backward compatibility issue in Pandas.
              Documentation of the API and usage
              We should document AzureFileSystem in the API reference and its usage in the user guide.
              Note about the backward compatibility in Pandas
              Pandas' read_parquet() and to_parquet() with abfs:// have stopped working in specific cases since PyArrow 16.0.0 due to the addition of AzureFileSystem in PyArrow.
              Pandas implement a logic that first tries to get a PyArrow native file system implementation for a given URL and then falls back to fsspec if PyArrow does not have a native implementation for the URL.
              https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/io/parquet.py#L116-L124
              Due to this fallback logic, Pandas's read_parquet() and to_parquet() always use fssepc with PyArrow before 16.0.0.
              With PyArrow 16.0.0, Pandas automatically uses PyArrow's native AzureFileSystem. However, this AzureFileSystem does not use authentication settings set in fsspec's global configuration. Instead, we must explicitly provide an authentication setting to read_parquet() and to_parquet() as storage_options independently of fsspec.
              We need to figure out where and how we should document this backward compatibility issue.
              Component(s)
              Python
              Activity
              Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
              Metadata
              Metadata
              Assignees
              Labels
              Component: DocumentationComponent: Python
              Type
              No type
              Projects
              No projects
                Milestone
                Relationships
                None yet
                Development
                No branches or pull requests
                Issue actions
                
, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

                Skip to content
                [Python][Azure][Docs] Add documentation about AzureFilesystem #41496
                Closed
                Closed
                [Python][Azure][Docs] Add documentation about AzureFilesystem#41496
                Assignees
                AlenkaF
                Labels
                Component: DocumentationComponent: Python
                Milestone
                21.0.0
                Description
                @sugibuchi
                sugibuchi
                opened on May 2, 2024
                Describe the enhancement requested
                #39968, which is included in version 16.0.0 release, added a Python binding of C++ AzureFileSystem to the PyArrow API.
                However, (1) this addition of the native file system implementation for Azure has not yet been documented, and (2) this addition causes a backward compatibility issue in Pandas.
                Documentation of the API and usage
                We should document AzureFileSystem in the API reference and its usage in the user guide.
                Note about the backward compatibility in Pandas
                Pandas' read_parquet() and to_parquet() with abfs:// have stopped working in specific cases since PyArrow 16.0.0 due to the addition of AzureFileSystem in PyArrow.
                Pandas implement a logic that first tries to get a PyArrow native file system implementation for a given URL and then falls back to fsspec if PyArrow does not have a native implementation for the URL.
                https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/io/parquet.py#L116-L124
                Due to this fallback logic, Pandas's read_parquet() and to_parquet() always use fssepc with PyArrow before 16.0.0.
                With PyArrow 16.0.0, Pandas automatically uses PyArrow's native AzureFileSystem. However, this AzureFileSystem does not use authentication settings set in fsspec's global configuration. Instead, we must explicitly provide an authentication setting to read_parquet() and to_parquet() as storage_options independently of fsspec.
                We need to figure out where and how we should document this backward compatibility issue.
                Component(s)
                Python
                Activity
                Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
                Metadata
                Metadata
                Assignees
                Labels
                Component: DocumentationComponent: Python
                Type
                No type
                Projects
                No projects
                  Milestone
                  Relationships
                  None yet
                  Development
                  No branches or pull requests
                  Issue actions