Skip to content

Repository files navigation

Numpy Data Type Serialization Using Msgpack

Package Description

This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.

Latest VersionBuild Status

Installation

msgpack-numpy requires msgpack-python and numpy.

To install the package and all dependencies. You can download the source tarball, unpack it, and run

python setup.py install

from within the source directory.

Usage

The easiest way to use msgpack-numpy is to call its monkey patching function after importing the Python msgpack package:

import msgpack
import msgpack_numpy as m
m.patch()

This will automatically force all msgpack serialization and deserialization routines (and other packages that use them) to become numpy-aware. Of course, one can also manually pass the encoder and decoder provided by msgpack-numpy to the msgpack routines:

import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_enc = msgpack.packb(x, default=m.encode)
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)

msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
If msgpack was not compiled with Cython (or if the MSGPACK_PUREPYTHON variable is set), it will fall back to using the slower pure Python msgpack implementation.

Notes

The primary design goal of msgpack-numpy is ensuring preservation of numerical data types during msgpack serialization and deserialization. Inclusion of type information in the serialized data necessarily incurs some storage overhead; if preservation of type information is not needed, one may be able to avoid some of this overhead by writing a custom encoder/decoder pair that produces more efficient serializations for those specific use cases.

Numpy arrays with a dtype of 'O' are serialized/deserialized using pickle as a fallback solution to enable msgpack-numpy to handle such arrays. As the additional overhead of pickle serialization negates one of the reasons to use msgpack, it may be advisable to either write a custom encoder/decoder to handle the specific use case efficiently or else not bother using msgpack-numpy.

Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied if they are to be modified.

Development

The latest source code can be obtained from GitHub.

msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.

Install tox to support testing across multiple python versions in your development environment. If you use conda to install python use tox-conda to automatically manage testing across all supported python versions.

# Using a system python
pip install tox
# Additionally, using a conda-provided python
pip install tox tox-conda

Execute tests across supported python versions:

tox

Authors

See the included AUTHORS.md file for more information.

License

This software is licensed under the BSD License. See the included LICENSE.md file for more information.

About

Serialize numpy arrays using msgpack

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - opentensor/msgpack-numpy: Serialize numpy arrays using msgpack · GitHub
Skip to content

Repository files navigation

Numpy Data Type Serialization Using Msgpack

Package Description

This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.

Latest VersionBuild Status

Installation

msgpack-numpy requires msgpack-python and numpy.

To install the package and all dependencies. You can download the source tarball, unpack it, and run

python setup.py install

from within the source directory.

Usage

The easiest way to use msgpack-numpy is to call its monkey patching function after importing the Python msgpack package:

import msgpack
import msgpack_numpy as m
m.patch()

This will automatically force all msgpack serialization and deserialization routines (and other packages that use them) to become numpy-aware. Of course, one can also manually pass the encoder and decoder provided by msgpack-numpy to the msgpack routines:

import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_enc = msgpack.packb(x, default=m.encode)
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)

msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
If msgpack was not compiled with Cython (or if the MSGPACK_PUREPYTHON variable is set), it will fall back to using the slower pure Python msgpack implementation.

Notes

The primary design goal of msgpack-numpy is ensuring preservation of numerical data types during msgpack serialization and deserialization. Inclusion of type information in the serialized data necessarily incurs some storage overhead; if preservation of type information is not needed, one may be able to avoid some of this overhead by writing a custom encoder/decoder pair that produces more efficient serializations for those specific use cases.

Numpy arrays with a dtype of 'O' are serialized/deserialized using pickle as a fallback solution to enable msgpack-numpy to handle such arrays. As the additional overhead of pickle serialization negates one of the reasons to use msgpack, it may be advisable to either write a custom encoder/decoder to handle the specific use case efficiently or else not bother using msgpack-numpy.

Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied if they are to be modified.

Development

The latest source code can be obtained from GitHub.

msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.

Install tox to support testing across multiple python versions in your development environment. If you use conda to install python use tox-conda to automatically manage testing across all supported python versions.

# Using a system python
pip install tox
# Additionally, using a conda-provided python
pip install tox tox-conda

Execute tests across supported python versions:

tox

Authors

See the included AUTHORS.md file for more information.

License

This software is licensed under the BSD License. See the included LICENSE.md file for more information.

About

Serialize numpy arrays using msgpack

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' GitHub - opentensor/msgpack-numpy: Serialize numpy arrays using msgpack · GitHub
Skip to content

Repository files navigation

Numpy Data Type Serialization Using Msgpack

Package Description

This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.

Latest VersionBuild Status

Installation

msgpack-numpy requires msgpack-python and numpy.

To install the package and all dependencies. You can download the source tarball, unpack it, and run

python setup.py install

from within the source directory.

Usage

The easiest way to use msgpack-numpy is to call its monkey patching function after importing the Python msgpack package:

import msgpack
import msgpack_numpy as m
m.patch()

This will automatically force all msgpack serialization and deserialization routines (and other packages that use them) to become numpy-aware. Of course, one can also manually pass the encoder and decoder provided by msgpack-numpy to the msgpack routines:

import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_enc = msgpack.packb(x, default=m.encode)
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)

msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
If msgpack was not compiled with Cython (or if the MSGPACK_PUREPYTHON variable is set), it will fall back to using the slower pure Python msgpack implementation.

Notes

The primary design goal of msgpack-numpy is ensuring preservation of numerical data types during msgpack serialization and deserialization. Inclusion of type information in the serialized data necessarily incurs some storage overhead; if preservation of type information is not needed, one may be able to avoid some of this overhead by writing a custom encoder/decoder pair that produces more efficient serializations for those specific use cases.

Numpy arrays with a dtype of 'O' are serialized/deserialized using pickle as a fallback solution to enable msgpack-numpy to handle such arrays. As the additional overhead of pickle serialization negates one of the reasons to use msgpack, it may be advisable to either write a custom encoder/decoder to handle the specific use case efficiently or else not bother using msgpack-numpy.

Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied if they are to be modified.

Development

The latest source code can be obtained from GitHub.

msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.

Install tox to support testing across multiple python versions in your development environment. If you use conda to install python use tox-conda to automatically manage testing across all supported python versions.

# Using a system python
pip install tox
# Additionally, using a conda-provided python
pip install tox tox-conda

Execute tests across supported python versions:

tox

Authors

See the included AUTHORS.md file for more information.

License

This software is licensed under the BSD License. See the included LICENSE.md file for more information.

About

Serialize numpy arrays using msgpack

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' GitHub - opentensor/msgpack-numpy: Serialize numpy arrays using msgpack · GitHub
Skip to content

Repository files navigation

Numpy Data Type Serialization Using Msgpack

Package Description

This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.

Latest VersionBuild Status

Installation

msgpack-numpy requires msgpack-python and numpy.

To install the package and all dependencies. You can download the source tarball, unpack it, and run

python setup.py install

from within the source directory.

Usage

The easiest way to use msgpack-numpy is to call its monkey patching function after importing the Python msgpack package:

import msgpack
import msgpack_numpy as m
m.patch()

This will automatically force all msgpack serialization and deserialization routines (and other packages that use them) to become numpy-aware. Of course, one can also manually pass the encoder and decoder provided by msgpack-numpy to the msgpack routines:

import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_enc = msgpack.packb(x, default=m.encode)
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)

msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
If msgpack was not compiled with Cython (or if the MSGPACK_PUREPYTHON variable is set), it will fall back to using the slower pure Python msgpack implementation.

Notes

The primary design goal of msgpack-numpy is ensuring preservation of numerical data types during msgpack serialization and deserialization. Inclusion of type information in the serialized data necessarily incurs some storage overhead; if preservation of type information is not needed, one may be able to avoid some of this overhead by writing a custom encoder/decoder pair that produces more efficient serializations for those specific use cases.

Numpy arrays with a dtype of 'O' are serialized/deserialized using pickle as a fallback solution to enable msgpack-numpy to handle such arrays. As the additional overhead of pickle serialization negates one of the reasons to use msgpack, it may be advisable to either write a custom encoder/decoder to handle the specific use case efficiently or else not bother using msgpack-numpy.

Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied if they are to be modified.

Development

The latest source code can be obtained from GitHub.

msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.

Install tox to support testing across multiple python versions in your development environment. If you use conda to install python use tox-conda to automatically manage testing across all supported python versions.

# Using a system python
pip install tox
# Additionally, using a conda-provided python
pip install tox tox-conda

Execute tests across supported python versions:

tox

Authors

See the included AUTHORS.md file for more information.

License

This software is licensed under the BSD License. See the included LICENSE.md file for more information.

About

Serialize numpy arrays using msgpack

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' GitHub - opentensor/msgpack-numpy: Serialize numpy arrays using msgpack · GitHub
Skip to content

Repository files navigation

Numpy Data Type Serialization Using Msgpack

Package Description

This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.

Latest VersionBuild Status

Installation

msgpack-numpy requires msgpack-python and numpy.

To install the package and all dependencies. You can download the source tarball, unpack it, and run

python setup.py install

from within the source directory.

Usage

The easiest way to use msgpack-numpy is to call its monkey patching function after importing the Python msgpack package:

import msgpack
import msgpack_numpy as m
m.patch()

This will automatically force all msgpack serialization and deserialization routines (and other packages that use them) to become numpy-aware. Of course, one can also manually pass the encoder and decoder provided by msgpack-numpy to the msgpack routines:

import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_enc = msgpack.packb(x, default=m.encode)
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)

msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
If msgpack was not compiled with Cython (or if the MSGPACK_PUREPYTHON variable is set), it will fall back to using the slower pure Python msgpack implementation.

Notes

The primary design goal of msgpack-numpy is ensuring preservation of numerical data types during msgpack serialization and deserialization. Inclusion of type information in the serialized data necessarily incurs some storage overhead; if preservation of type information is not needed, one may be able to avoid some of this overhead by writing a custom encoder/decoder pair that produces more efficient serializations for those specific use cases.

Numpy arrays with a dtype of 'O' are serialized/deserialized using pickle as a fallback solution to enable msgpack-numpy to handle such arrays. As the additional overhead of pickle serialization negates one of the reasons to use msgpack, it may be advisable to either write a custom encoder/decoder to handle the specific use case efficiently or else not bother using msgpack-numpy.

Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied if they are to be modified.

Development

The latest source code can be obtained from GitHub.

msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.

Install tox to support testing across multiple python versions in your development environment. If you use conda to install python use tox-conda to automatically manage testing across all supported python versions.

# Using a system python
pip install tox
# Additionally, using a conda-provided python
pip install tox tox-conda

Execute tests across supported python versions:

tox

Authors

See the included AUTHORS.md file for more information.

License

This software is licensed under the BSD License. See the included LICENSE.md file for more information.

About

Serialize numpy arrays using msgpack

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' GitHub - opentensor/msgpack-numpy: Serialize numpy arrays using msgpack · GitHub
Skip to content

Repository files navigation

Numpy Data Type Serialization Using Msgpack

Package Description

This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.

Latest VersionBuild Status

Installation

msgpack-numpy requires msgpack-python and numpy.

To install the package and all dependencies. You can download the source tarball, unpack it, and run

python setup.py install

from within the source directory.

Usage

The easiest way to use msgpack-numpy is to call its monkey patching function after importing the Python msgpack package:

import msgpack
import msgpack_numpy as m
m.patch()

This will automatically force all msgpack serialization and deserialization routines (and other packages that use them) to become numpy-aware. Of course, one can also manually pass the encoder and decoder provided by msgpack-numpy to the msgpack routines:

import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_enc = msgpack.packb(x, default=m.encode)
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)

msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
If msgpack was not compiled with Cython (or if the MSGPACK_PUREPYTHON variable is set), it will fall back to using the slower pure Python msgpack implementation.

Notes

The primary design goal of msgpack-numpy is ensuring preservation of numerical data types during msgpack serialization and deserialization. Inclusion of type information in the serialized data necessarily incurs some storage overhead; if preservation of type information is not needed, one may be able to avoid some of this overhead by writing a custom encoder/decoder pair that produces more efficient serializations for those specific use cases.

Numpy arrays with a dtype of 'O' are serialized/deserialized using pickle as a fallback solution to enable msgpack-numpy to handle such arrays. As the additional overhead of pickle serialization negates one of the reasons to use msgpack, it may be advisable to either write a custom encoder/decoder to handle the specific use case efficiently or else not bother using msgpack-numpy.

Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied if they are to be modified.

Development

The latest source code can be obtained from GitHub.

msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.

Install tox to support testing across multiple python versions in your development environment. If you use conda to install python use tox-conda to automatically manage testing across all supported python versions.

# Using a system python
pip install tox
# Additionally, using a conda-provided python
pip install tox tox-conda

Execute tests across supported python versions:

tox

Authors

See the included AUTHORS.md file for more information.

License

This software is licensed under the BSD License. See the included LICENSE.md file for more information.

About

Serialize numpy arrays using msgpack

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' GitHub - opentensor/msgpack-numpy: Serialize numpy arrays using msgpack · GitHub
Skip to content

Repository files navigation

Numpy Data Type Serialization Using Msgpack

Package Description

This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.

Latest VersionBuild Status

Installation

msgpack-numpy requires msgpack-python and numpy.

To install the package and all dependencies. You can download the source tarball, unpack it, and run

python setup.py install

from within the source directory.

Usage

The easiest way to use msgpack-numpy is to call its monkey patching function after importing the Python msgpack package:

import msgpack
import msgpack_numpy as m
m.patch()

This will automatically force all msgpack serialization and deserialization routines (and other packages that use them) to become numpy-aware. Of course, one can also manually pass the encoder and decoder provided by msgpack-numpy to the msgpack routines:

import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_enc = msgpack.packb(x, default=m.encode)
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)

msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
If msgpack was not compiled with Cython (or if the MSGPACK_PUREPYTHON variable is set), it will fall back to using the slower pure Python msgpack implementation.

Notes

The primary design goal of msgpack-numpy is ensuring preservation of numerical data types during msgpack serialization and deserialization. Inclusion of type information in the serialized data necessarily incurs some storage overhead; if preservation of type information is not needed, one may be able to avoid some of this overhead by writing a custom encoder/decoder pair that produces more efficient serializations for those specific use cases.

Numpy arrays with a dtype of 'O' are serialized/deserialized using pickle as a fallback solution to enable msgpack-numpy to handle such arrays. As the additional overhead of pickle serialization negates one of the reasons to use msgpack, it may be advisable to either write a custom encoder/decoder to handle the specific use case efficiently or else not bother using msgpack-numpy.

Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied if they are to be modified.

Development

The latest source code can be obtained from GitHub.

msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.

Install tox to support testing across multiple python versions in your development environment. If you use conda to install python use tox-conda to automatically manage testing across all supported python versions.

# Using a system python
pip install tox
# Additionally, using a conda-provided python
pip install tox tox-conda

Execute tests across supported python versions:

tox

Authors

See the included AUTHORS.md file for more information.

License

This software is licensed under the BSD License. See the included LICENSE.md file for more information.

About

Serialize numpy arrays using msgpack

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); GitHub - opentensor/msgpack-numpy: Serialize numpy arrays using msgpack · GitHub
Skip to content

Repository files navigation

Numpy Data Type Serialization Using Msgpack

Package Description

This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.

Latest VersionBuild Status

Installation

msgpack-numpy requires msgpack-python and numpy.

To install the package and all dependencies. You can download the source tarball, unpack it, and run

python setup.py install

from within the source directory.

Usage

The easiest way to use msgpack-numpy is to call its monkey patching function after importing the Python msgpack package:

import msgpack
import msgpack_numpy as m
m.patch()

This will automatically force all msgpack serialization and deserialization routines (and other packages that use them) to become numpy-aware. Of course, one can also manually pass the encoder and decoder provided by msgpack-numpy to the msgpack routines:

import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_enc = msgpack.packb(x, default=m.encode)
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)

msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
If msgpack was not compiled with Cython (or if the MSGPACK_PUREPYTHON variable is set), it will fall back to using the slower pure Python msgpack implementation.

Notes

The primary design goal of msgpack-numpy is ensuring preservation of numerical data types during msgpack serialization and deserialization. Inclusion of type information in the serialized data necessarily incurs some storage overhead; if preservation of type information is not needed, one may be able to avoid some of this overhead by writing a custom encoder/decoder pair that produces more efficient serializations for those specific use cases.

Numpy arrays with a dtype of 'O' are serialized/deserialized using pickle as a fallback solution to enable msgpack-numpy to handle such arrays. As the additional overhead of pickle serialization negates one of the reasons to use msgpack, it may be advisable to either write a custom encoder/decoder to handle the specific use case efficiently or else not bother using msgpack-numpy.

Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied if they are to be modified.

Development

The latest source code can be obtained from GitHub.

msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.

Install tox to support testing across multiple python versions in your development environment. If you use conda to install python use tox-conda to automatically manage testing across all supported python versions.

# Using a system python
pip install tox
# Additionally, using a conda-provided python
pip install tox tox-conda

Execute tests across supported python versions:

tox

Authors

See the included AUTHORS.md file for more information.

License

This software is licensed under the BSD License. See the included LICENSE.md file for more information.

About

Serialize numpy arrays using msgpack

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages