ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream#690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wesm wants to merge 3 commits into from
Closed

ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream #690

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a15910a
Rename InMemoryOutputStream to BufferOutputStream
wesm May 15, 2017
69a99cd
Fix Cython compilation
wesm May 15, 2017
85b352c
Add more graceful deprecation warnings for renamed classes, test suite
wesm May 15, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion python/doc/source/api.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -162,7 +162,7 @@ Input / Output and Shared Memory

Buffer
BufferReader
InMemoryOutputStream
BufferOutputStream
NativeFile
MemoryMappedFile
memory_map
Expand Down
4 changes: 2 additions & 2 deletions python/doc/source/ipc.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ this we use :class:`~pyarrow.RecordBatchStreamWriter`, which can write to a writ

.. ipython:: python

sink = pa.InMemoryOutputStream()
sink = pa.BufferOutputStream()
writer = pa.RecordBatchStreamWriter(sink, batch.schema)

Here we used an in-memory Arrow buffer stream, but this could have been a
Expand DownExpand Up@@ -109,7 +109,7 @@ The :class:`~pyarrow.RecordBatchFileWriter` has the same API as

.. ipython:: python

sink = pa.InMemoryOutputStream()
sink = pa.BufferOutputStream()
writer = pa.RecordBatchFileWriter(sink, batch.schema)

for i in range(10):
Expand Down
12 changes: 6 additions & 6 deletions python/doc/source/memory.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -104,8 +104,8 @@ There are several kinds of :class:`~pyarrow.NativeFile` options available:
memory maps
* :class:`~pyarrow.BufferReader`, for reading :class:`~pyarrow.Buffer` objects
as a file
* :class:`~pyarrow.InMemoryOutputStream`, for writing data in-memory, producing
a Buffer at the end
* :class:`~pyarrow.BufferOutputStream`, for writing data in-memory, producing a
Buffer at the end
* :class:`~pyarrow.HdfsFile`, for reading and writing data to the Hadoop Filesystem
* :class:`~pyarrow.PythonFile`, for interfacing with Python file objects in C++

Expand All@@ -124,11 +124,11 @@ then precisely track amount of memory that has been allocated:

PyArrow uses a default built-in memory pool, but in the future there may be
additional memory pools (and subpools) to choose from. Let's consider an
``InMemoryOutputStream``, which is like a ``BytesIO``:
``BufferOutputStream``, which is like a ``BytesIO``:

.. ipython:: python

stream = pa.InMemoryOutputStream()
stream = pa.BufferOutputStream()
stream.write(b'foo')
pa.total_allocated_bytes()
for i in range(1024): stream.write(b'foo')
Expand All@@ -150,7 +150,7 @@ pass in a custom memory pool:
my_pool = pa.jemalloc_memory_pool()
my_pool
my_pool.bytes_allocated()
stream = pa.InMemoryOutputStream(my_pool)
stream = pa.BufferOutputStream(my_pool)
stream.write(b'foo')
my_pool.bytes_allocated()

Expand DownExpand Up@@ -215,7 +215,7 @@ file interfaces that can read and write to Arrow Buffers.

.. ipython:: python

writer = pa.InMemoryOutputStream()
writer = pa.BufferOutputStream()
writer.write(b'hello, friends')

buf = writer.get_result()
Expand Down
37 changes: 36 additions & 1 deletion python/pyarrow/__init__.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@
Date32Value, Date64Value, TimestampValue)

from pyarrow.lib import (HdfsFile, NativeFile, PythonFile,
Buffer, BufferReader, InMemoryOutputStream,
Buffer, BufferReader, BufferOutputStream,
OSFile, MemoryMappedFile, memory_map,
frombuffer, read_tensor, write_tensor,
memory_map, create_memory_map,
Expand DownExpand Up@@ -108,3 +108,38 @@ def jemalloc_memory_pool():


localfs = LocalFilesystem.get_instance()


# ----------------------------------------------------------------------
# 0.4.0 deprecations

import warnings

def _deprecate_class(old_name, new_name, klass, next_version='0.5.0'):
msg = ('pyarrow.{0} has been renamed to '
'{1}, will be removed in {2}'
.format(old_name, new_name, next_version))
def deprecated_factory(*args, **kwargs):
warnings.warn(msg, FutureWarning)
return klass(*args)
return deprecated_factory

FileReader = _deprecate_class('FileReader',
'RecordBatchFileReader',
RecordBatchFileReader, '0.5.0')

FileWriter = _deprecate_class('FileWriter',
'RecordBatchFileWriter',
RecordBatchFileWriter, '0.5.0')

StreamReader = _deprecate_class('StreamReader',
'RecordBatchStreamReader',
RecordBatchStreamReader, '0.5.0')

StreamWriter = _deprecate_class('StreamWriter',
'RecordBatchStreamWriter',
RecordBatchStreamWriter, '0.5.0')

InMemoryOutputStream = _deprecate_class('InMemoryOutputStream',
'BufferOutputStream',
BufferOutputStream, '0.5.0')
5 changes: 3 additions & 2 deletions python/pyarrow/includes/libarrow.pxd
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -507,8 +507,9 @@ cdef extern from "arrow/io/memory.h" namespace "arrow::io" nogil:
CBufferReader(const shared_ptr[CBuffer]& buffer)
CBufferReader(const uint8_t* data, int64_t nbytes)

cdef cppclass BufferOutputStream(OutputStream):
BufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)
cdef cppclass CBufferOutputStream" arrow::io::BufferOutputStream"\
(OutputStream):
CBufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)


cdef extern from "arrow/ipc/metadata.h" namespace "arrow::ipc" nogil:
Expand Down
4 changes: 2 additions & 2 deletions python/pyarrow/io.pxi
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -515,14 +515,14 @@ cdef shared_ptr[PoolBuffer] allocate_buffer(CMemoryPool* pool):
return result


cdef class InMemoryOutputStream(NativeFile):
cdef class BufferOutputStream(NativeFile):

cdef:
shared_ptr[PoolBuffer] buffer

def __cinit__(self, MemoryPool memory_pool=None):
self.buffer = allocate_buffer(maybe_unbox_memory_pool(memory_pool))
self.wr_file.reset(new BufferOutputStream(
self.wr_file.reset(new CBufferOutputStream(
<shared_ptr[ResizableBuffer]> self.buffer))
self.is_readable = 0
self.is_writeable = 1
Expand Down
64 changes: 64 additions & 0 deletions python/pyarrow/tests/test_deprecations.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Check that various deprecation warnings are raised

import pyarrow as pa
import pytest


def test_inmemory_output_stream():
with pytest.warns(FutureWarning):
stream = pa.InMemoryOutputStream()
assert isinstance(stream, pa.BufferOutputStream)


def test_file_reader_writer():
data = [
pa.array([1, 2, 3, 4]),
pa.array(['foo', 'bar', 'baz', None]),
pa.array([True, None, False, True])
]
batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])

sink = pa.BufferOutputStream()

with pytest.warns(FutureWarning):
stream_writer = pa.StreamWriter(sink, batch.schema)
assert isinstance(stream_writer, pa.RecordBatchStreamWriter)

sink2 = pa.BufferOutputStream()
with pytest.warns(FutureWarning):
file_writer = pa.FileWriter(sink2, batch.schema)
assert isinstance(file_writer, pa.RecordBatchFileWriter)

file_writer.write_batch(batch)
stream_writer.write_batch(batch)

file_writer.close()
stream_writer.close()

buf = sink.get_result()
buf2 = sink2.get_result()

with pytest.warns(FutureWarning):
stream_reader = pa.StreamReader(buf)
assert isinstance(stream_reader, pa.RecordBatchStreamReader)

with pytest.warns(FutureWarning):
file_reader = pa.FileReader(buf2)
assert isinstance(file_reader, pa.RecordBatchFileReader)
6 changes: 3 additions & 3 deletions python/pyarrow/tests/test_io.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def test_memory_output_stream():
# 10 bytes
val = b'dataabcdef'

f = pa.InMemoryOutputStream()
f = pa.BufferOutputStream()

K = 1000
for i in range(K):
Expand All@@ -192,7 +192,7 @@ def test_memory_output_stream():


def test_inmemory_write_after_closed():
f = pa.InMemoryOutputStream()
f = pa.BufferOutputStream()
f.write(b'ok')
f.get_result()

Expand All@@ -212,7 +212,7 @@ def make_buffer(bytes_obj):


def test_nativefile_write_memoryview():
f = pa.InMemoryOutputStream()
f = pa.BufferOutputStream()
data = b'ok'

arr = np.frombuffer(data, dtype='S1')
Expand Down
4 changes: 2 additions & 2 deletions python/pyarrow/tests/test_ipc.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -209,7 +209,7 @@ def test_read_all(self):
class TestInMemoryFile(TestFile):

def _get_sink(self):
return pa.InMemoryOutputStream()
return pa.BufferOutputStream()

def _get_source(self):
return self.sink.get_result()
Expand All@@ -219,7 +219,7 @@ def test_ipc_zero_copy_numpy():
df = pd.DataFrame({'foo': [1.5]})

batch = pa.RecordBatch.from_pandas(df)
sink = pa.InMemoryOutputStream()
sink = pa.BufferOutputStream()
write_file(batch, sink)
buffer = sink.get_result()
reader = pa.BufferReader(buffer)
Expand Down
2 changes: 1 addition & 1 deletion python/pyarrow/tests/test_parquet.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def _test_dataframe(size=10000, seed=0):
def test_pandas_parquet_native_file_roundtrip(tmpdir):
df = _test_dataframe(10000)
arrow_table = pa.Table.from_pandas(df)
imos = pa.InMemoryOutputStream()
imos = pa.BufferOutputStream()
pq.write_table(arrow_table, imos, version="2.0")
buf = imos.get_result()
reader = pa.BufferReader(buf)
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream#690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

wesm
wants to merge
3
commits into

from

 Closed


ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream
#690
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
a15910a

Rename InMemoryOutputStream to BufferOutputStream


wesm May 15, 2017
69a99cd

Fix Cython compilation


wesm May 15, 2017
85b352c

Add more graceful deprecation warnings for renamed classes, test suite


wesm May 15, 2017

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
python/doc/source/api.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -162,7 +162,7 @@ Input / Output and Shared Memory

 Buffer
 BufferReader
InMemoryOutputStream
BufferOutputStream
 NativeFile
 MemoryMappedFile
 memory_map
Expand Down

4 changes: 2 additions & 2 deletions
python/doc/source/ipc.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ this we use :class:`~pyarrow.RecordBatchStreamWriter`, which can write to a writ

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchStreamWriter(sink, batch.schema)

Here we used an in-memory Arrow buffer stream, but this could have been a
Expand DownExpand Up@@ -109,7 +109,7 @@ The :class:`~pyarrow.RecordBatchFileWriter` has the same API as

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchFileWriter(sink, batch.schema)

 for i in range(10):
Expand Down

12 changes: 6 additions & 6 deletions
python/doc/source/memory.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -104,8 +104,8 @@ There are several kinds of :class:`~pyarrow.NativeFile` options available:
 memory maps
* :class:`~pyarrow.BufferReader`, for reading :class:`~pyarrow.Buffer` objects
 as a file
* :class:`~pyarrow.InMemoryOutputStream`, for writing data in-memory, producing
a Buffer at the end
* :class:`~pyarrow.BufferOutputStream`, for writing data in-memory, producing a
 Buffer at the end
* :class:`~pyarrow.HdfsFile`, for reading and writing data to the Hadoop Filesystem
* :class:`~pyarrow.PythonFile`, for interfacing with Python file objects in C++

Expand All@@ -124,11 +124,11 @@ then precisely track amount of memory that has been allocated:

PyArrow uses a default built-in memory pool, but in the future there may be
additional memory pools (and subpools) to choose from. Let's consider an
``InMemoryOutputStream``, which is like a ``BytesIO``:
``BufferOutputStream``, which is like a ``BytesIO``:

.. ipython:: python

 stream = pa.InMemoryOutputStream()
 stream = pa.BufferOutputStream()
 stream.write(b'foo')
 pa.total_allocated_bytes()
 for i in range(1024): stream.write(b'foo')
Expand All@@ -150,7 +150,7 @@ pass in a custom memory pool:
 my_pool = pa.jemalloc_memory_pool()
 my_pool
 my_pool.bytes_allocated()
 stream = pa.InMemoryOutputStream(my_pool)
 stream = pa.BufferOutputStream(my_pool)
 stream.write(b'foo')
 my_pool.bytes_allocated()

Expand DownExpand Up@@ -215,7 +215,7 @@ file interfaces that can read and write to Arrow Buffers.

.. ipython:: python

 writer = pa.InMemoryOutputStream()
 writer = pa.BufferOutputStream()
 writer.write(b'hello, friends')

 buf = writer.get_result()
Expand Down

37 changes: 36 additions & 1 deletion
python/pyarrow/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@
 Date32Value, Date64Value, TimestampValue)

from pyarrow.lib import (HdfsFile, NativeFile, PythonFile,
 Buffer, BufferReader, InMemoryOutputStream,
 Buffer, BufferReader, BufferOutputStream,
 OSFile, MemoryMappedFile, memory_map,
 frombuffer, read_tensor, write_tensor,
 memory_map, create_memory_map,
Expand DownExpand Up@@ -108,3 +108,38 @@ def jemalloc_memory_pool():


localfs = LocalFilesystem.get_instance()


# ----------------------------------------------------------------------
# 0.4.0 deprecations

import warnings

def _deprecate_class(old_name, new_name, klass, next_version='0.5.0'):
 msg = ('pyarrow.{0} has been renamed to '
 '{1}, will be removed in {2}'
 .format(old_name, new_name, next_version))
 def deprecated_factory(*args, **kwargs):
 warnings.warn(msg, FutureWarning)
 return klass(*args)
 return deprecated_factory

FileReader = _deprecate_class('FileReader',
 'RecordBatchFileReader',
 RecordBatchFileReader, '0.5.0')

FileWriter = _deprecate_class('FileWriter',
 'RecordBatchFileWriter',
 RecordBatchFileWriter, '0.5.0')

StreamReader = _deprecate_class('StreamReader',
 'RecordBatchStreamReader',
 RecordBatchStreamReader, '0.5.0')

StreamWriter = _deprecate_class('StreamWriter',
 'RecordBatchStreamWriter',
 RecordBatchStreamWriter, '0.5.0')

InMemoryOutputStream = _deprecate_class('InMemoryOutputStream',
 'BufferOutputStream',
 BufferOutputStream, '0.5.0')

5 changes: 3 additions & 2 deletions
python/pyarrow/includes/libarrow.pxd

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -507,8 +507,9 @@ cdef extern from "arrow/io/memory.h" namespace "arrow::io" nogil:
 CBufferReader(const shared_ptr[CBuffer]& buffer)
 CBufferReader(const uint8_t* data, int64_t nbytes)

 cdef cppclass BufferOutputStream(OutputStream):
 BufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)
 cdef cppclass CBufferOutputStream" arrow::io::BufferOutputStream"\
 (OutputStream):
 CBufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)


cdef extern from "arrow/ipc/metadata.h" namespace "arrow::ipc" nogil:
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/io.pxi

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -515,14 +515,14 @@ cdef shared_ptr[PoolBuffer] allocate_buffer(CMemoryPool* pool):
 return result


cdef class InMemoryOutputStream(NativeFile):
cdef class BufferOutputStream(NativeFile):

 cdef:
 shared_ptr[PoolBuffer] buffer

 def __cinit__(self, MemoryPool memory_pool=None):
 self.buffer = allocate_buffer(maybe_unbox_memory_pool(memory_pool))
 self.wr_file.reset(new BufferOutputStream(
 self.wr_file.reset(new CBufferOutputStream(
 <shared_ptr[ResizableBuffer]> self.buffer))
 self.is_readable = 0
 self.is_writeable = 1
Expand Down

64 changes: 64 additions & 0 deletions
python/pyarrow/tests/test_deprecations.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Check that various deprecation warnings are raised

import pyarrow as pa
import pytest


def test_inmemory_output_stream():
 with pytest.warns(FutureWarning):
 stream = pa.InMemoryOutputStream()
 assert isinstance(stream, pa.BufferOutputStream)


def test_file_reader_writer():
 data = [
 pa.array([1, 2, 3, 4]),
 pa.array(['foo', 'bar', 'baz', None]),
 pa.array([True, None, False, True])
 ]
 batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])

 sink = pa.BufferOutputStream()

 with pytest.warns(FutureWarning):
 stream_writer = pa.StreamWriter(sink, batch.schema)
 assert isinstance(stream_writer, pa.RecordBatchStreamWriter)

 sink2 = pa.BufferOutputStream()
 with pytest.warns(FutureWarning):
 file_writer = pa.FileWriter(sink2, batch.schema)
 assert isinstance(file_writer, pa.RecordBatchFileWriter)

 file_writer.write_batch(batch)
 stream_writer.write_batch(batch)

 file_writer.close()
 stream_writer.close()

 buf = sink.get_result()
 buf2 = sink2.get_result()

 with pytest.warns(FutureWarning):
 stream_reader = pa.StreamReader(buf)
 assert isinstance(stream_reader, pa.RecordBatchStreamReader)

 with pytest.warns(FutureWarning):
 file_reader = pa.FileReader(buf2)
 assert isinstance(file_reader, pa.RecordBatchFileReader)

6 changes: 3 additions & 3 deletions
python/pyarrow/tests/test_io.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def test_memory_output_stream():
 # 10 bytes
 val = b'dataabcdef'

 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()

 K = 1000
 for i in range(K):
Expand All@@ -192,7 +192,7 @@ def test_memory_output_stream():


def test_inmemory_write_after_closed():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 f.write(b'ok')
 f.get_result()

Expand All@@ -212,7 +212,7 @@ def make_buffer(bytes_obj):


def test_nativefile_write_memoryview():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 data = b'ok'

 arr = np.frombuffer(data, dtype='S1')
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/tests/test_ipc.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -209,7 +209,7 @@ def test_read_all(self):
class TestInMemoryFile(TestFile):

 def _get_sink(self):
 return pa.InMemoryOutputStream()
 return pa.BufferOutputStream()

 def _get_source(self):
 return self.sink.get_result()
Expand All@@ -219,7 +219,7 @@ def test_ipc_zero_copy_numpy():
 df = pd.DataFrame({'foo': [1.5]})

 batch = pa.RecordBatch.from_pandas(df)
 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 write_file(batch, sink)
 buffer = sink.get_result()
 reader = pa.BufferReader(buffer)
Expand Down

2 changes: 1 addition & 1 deletion
python/pyarrow/tests/test_parquet.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def _test_dataframe(size=10000, seed=0):
def test_pandas_parquet_native_file_roundtrip(tmpdir):
 df = _test_dataframe(10000)
 arrow_table = pa.Table.from_pandas(df)
 imos = pa.InMemoryOutputStream()
 imos = pa.BufferOutputStream()
 pq.write_table(arrow_table, imos, version="2.0")
 buf = imos.get_result()
 reader = pa.BufferReader(buf)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream#690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

wesm
wants to merge
3
commits into

from

 Closed


ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream
#690
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
a15910a

Rename InMemoryOutputStream to BufferOutputStream


wesm May 15, 2017
69a99cd

Fix Cython compilation


wesm May 15, 2017
85b352c

Add more graceful deprecation warnings for renamed classes, test suite


wesm May 15, 2017

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
python/doc/source/api.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -162,7 +162,7 @@ Input / Output and Shared Memory

 Buffer
 BufferReader
InMemoryOutputStream
BufferOutputStream
 NativeFile
 MemoryMappedFile
 memory_map
Expand Down

4 changes: 2 additions & 2 deletions
python/doc/source/ipc.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ this we use :class:`~pyarrow.RecordBatchStreamWriter`, which can write to a writ

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchStreamWriter(sink, batch.schema)

Here we used an in-memory Arrow buffer stream, but this could have been a
Expand DownExpand Up@@ -109,7 +109,7 @@ The :class:`~pyarrow.RecordBatchFileWriter` has the same API as

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchFileWriter(sink, batch.schema)

 for i in range(10):
Expand Down

12 changes: 6 additions & 6 deletions
python/doc/source/memory.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -104,8 +104,8 @@ There are several kinds of :class:`~pyarrow.NativeFile` options available:
 memory maps
* :class:`~pyarrow.BufferReader`, for reading :class:`~pyarrow.Buffer` objects
 as a file
* :class:`~pyarrow.InMemoryOutputStream`, for writing data in-memory, producing
a Buffer at the end
* :class:`~pyarrow.BufferOutputStream`, for writing data in-memory, producing a
 Buffer at the end
* :class:`~pyarrow.HdfsFile`, for reading and writing data to the Hadoop Filesystem
* :class:`~pyarrow.PythonFile`, for interfacing with Python file objects in C++

Expand All@@ -124,11 +124,11 @@ then precisely track amount of memory that has been allocated:

PyArrow uses a default built-in memory pool, but in the future there may be
additional memory pools (and subpools) to choose from. Let's consider an
``InMemoryOutputStream``, which is like a ``BytesIO``:
``BufferOutputStream``, which is like a ``BytesIO``:

.. ipython:: python

 stream = pa.InMemoryOutputStream()
 stream = pa.BufferOutputStream()
 stream.write(b'foo')
 pa.total_allocated_bytes()
 for i in range(1024): stream.write(b'foo')
Expand All@@ -150,7 +150,7 @@ pass in a custom memory pool:
 my_pool = pa.jemalloc_memory_pool()
 my_pool
 my_pool.bytes_allocated()
 stream = pa.InMemoryOutputStream(my_pool)
 stream = pa.BufferOutputStream(my_pool)
 stream.write(b'foo')
 my_pool.bytes_allocated()

Expand DownExpand Up@@ -215,7 +215,7 @@ file interfaces that can read and write to Arrow Buffers.

.. ipython:: python

 writer = pa.InMemoryOutputStream()
 writer = pa.BufferOutputStream()
 writer.write(b'hello, friends')

 buf = writer.get_result()
Expand Down

37 changes: 36 additions & 1 deletion
python/pyarrow/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@
 Date32Value, Date64Value, TimestampValue)

from pyarrow.lib import (HdfsFile, NativeFile, PythonFile,
 Buffer, BufferReader, InMemoryOutputStream,
 Buffer, BufferReader, BufferOutputStream,
 OSFile, MemoryMappedFile, memory_map,
 frombuffer, read_tensor, write_tensor,
 memory_map, create_memory_map,
Expand DownExpand Up@@ -108,3 +108,38 @@ def jemalloc_memory_pool():


localfs = LocalFilesystem.get_instance()


# ----------------------------------------------------------------------
# 0.4.0 deprecations

import warnings

def _deprecate_class(old_name, new_name, klass, next_version='0.5.0'):
 msg = ('pyarrow.{0} has been renamed to '
 '{1}, will be removed in {2}'
 .format(old_name, new_name, next_version))
 def deprecated_factory(*args, **kwargs):
 warnings.warn(msg, FutureWarning)
 return klass(*args)
 return deprecated_factory

FileReader = _deprecate_class('FileReader',
 'RecordBatchFileReader',
 RecordBatchFileReader, '0.5.0')

FileWriter = _deprecate_class('FileWriter',
 'RecordBatchFileWriter',
 RecordBatchFileWriter, '0.5.0')

StreamReader = _deprecate_class('StreamReader',
 'RecordBatchStreamReader',
 RecordBatchStreamReader, '0.5.0')

StreamWriter = _deprecate_class('StreamWriter',
 'RecordBatchStreamWriter',
 RecordBatchStreamWriter, '0.5.0')

InMemoryOutputStream = _deprecate_class('InMemoryOutputStream',
 'BufferOutputStream',
 BufferOutputStream, '0.5.0')

5 changes: 3 additions & 2 deletions
python/pyarrow/includes/libarrow.pxd

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -507,8 +507,9 @@ cdef extern from "arrow/io/memory.h" namespace "arrow::io" nogil:
 CBufferReader(const shared_ptr[CBuffer]& buffer)
 CBufferReader(const uint8_t* data, int64_t nbytes)

 cdef cppclass BufferOutputStream(OutputStream):
 BufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)
 cdef cppclass CBufferOutputStream" arrow::io::BufferOutputStream"\
 (OutputStream):
 CBufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)


cdef extern from "arrow/ipc/metadata.h" namespace "arrow::ipc" nogil:
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/io.pxi

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -515,14 +515,14 @@ cdef shared_ptr[PoolBuffer] allocate_buffer(CMemoryPool* pool):
 return result


cdef class InMemoryOutputStream(NativeFile):
cdef class BufferOutputStream(NativeFile):

 cdef:
 shared_ptr[PoolBuffer] buffer

 def __cinit__(self, MemoryPool memory_pool=None):
 self.buffer = allocate_buffer(maybe_unbox_memory_pool(memory_pool))
 self.wr_file.reset(new BufferOutputStream(
 self.wr_file.reset(new CBufferOutputStream(
 <shared_ptr[ResizableBuffer]> self.buffer))
 self.is_readable = 0
 self.is_writeable = 1
Expand Down

64 changes: 64 additions & 0 deletions
python/pyarrow/tests/test_deprecations.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Check that various deprecation warnings are raised

import pyarrow as pa
import pytest


def test_inmemory_output_stream():
 with pytest.warns(FutureWarning):
 stream = pa.InMemoryOutputStream()
 assert isinstance(stream, pa.BufferOutputStream)


def test_file_reader_writer():
 data = [
 pa.array([1, 2, 3, 4]),
 pa.array(['foo', 'bar', 'baz', None]),
 pa.array([True, None, False, True])
 ]
 batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])

 sink = pa.BufferOutputStream()

 with pytest.warns(FutureWarning):
 stream_writer = pa.StreamWriter(sink, batch.schema)
 assert isinstance(stream_writer, pa.RecordBatchStreamWriter)

 sink2 = pa.BufferOutputStream()
 with pytest.warns(FutureWarning):
 file_writer = pa.FileWriter(sink2, batch.schema)
 assert isinstance(file_writer, pa.RecordBatchFileWriter)

 file_writer.write_batch(batch)
 stream_writer.write_batch(batch)

 file_writer.close()
 stream_writer.close()

 buf = sink.get_result()
 buf2 = sink2.get_result()

 with pytest.warns(FutureWarning):
 stream_reader = pa.StreamReader(buf)
 assert isinstance(stream_reader, pa.RecordBatchStreamReader)

 with pytest.warns(FutureWarning):
 file_reader = pa.FileReader(buf2)
 assert isinstance(file_reader, pa.RecordBatchFileReader)

6 changes: 3 additions & 3 deletions
python/pyarrow/tests/test_io.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def test_memory_output_stream():
 # 10 bytes
 val = b'dataabcdef'

 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()

 K = 1000
 for i in range(K):
Expand All@@ -192,7 +192,7 @@ def test_memory_output_stream():


def test_inmemory_write_after_closed():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 f.write(b'ok')
 f.get_result()

Expand All@@ -212,7 +212,7 @@ def make_buffer(bytes_obj):


def test_nativefile_write_memoryview():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 data = b'ok'

 arr = np.frombuffer(data, dtype='S1')
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/tests/test_ipc.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -209,7 +209,7 @@ def test_read_all(self):
class TestInMemoryFile(TestFile):

 def _get_sink(self):
 return pa.InMemoryOutputStream()
 return pa.BufferOutputStream()

 def _get_source(self):
 return self.sink.get_result()
Expand All@@ -219,7 +219,7 @@ def test_ipc_zero_copy_numpy():
 df = pd.DataFrame({'foo': [1.5]})

 batch = pa.RecordBatch.from_pandas(df)
 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 write_file(batch, sink)
 buffer = sink.get_result()
 reader = pa.BufferReader(buffer)
Expand Down

2 changes: 1 addition & 1 deletion
python/pyarrow/tests/test_parquet.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def _test_dataframe(size=10000, seed=0):
def test_pandas_parquet_native_file_roundtrip(tmpdir):
 df = _test_dataframe(10000)
 arrow_table = pa.Table.from_pandas(df)
 imos = pa.InMemoryOutputStream()
 imos = pa.BufferOutputStream()
 pq.write_table(arrow_table, imos, version="2.0")
 buf = imos.get_result()
 reader = pa.BufferReader(buf)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream#690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

wesm
wants to merge
3
commits into

from

 Closed


ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream
#690
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
a15910a

Rename InMemoryOutputStream to BufferOutputStream


wesm May 15, 2017
69a99cd

Fix Cython compilation


wesm May 15, 2017
85b352c

Add more graceful deprecation warnings for renamed classes, test suite


wesm May 15, 2017

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
python/doc/source/api.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -162,7 +162,7 @@ Input / Output and Shared Memory

 Buffer
 BufferReader
InMemoryOutputStream
BufferOutputStream
 NativeFile
 MemoryMappedFile
 memory_map
Expand Down

4 changes: 2 additions & 2 deletions
python/doc/source/ipc.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ this we use :class:`~pyarrow.RecordBatchStreamWriter`, which can write to a writ

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchStreamWriter(sink, batch.schema)

Here we used an in-memory Arrow buffer stream, but this could have been a
Expand DownExpand Up@@ -109,7 +109,7 @@ The :class:`~pyarrow.RecordBatchFileWriter` has the same API as

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchFileWriter(sink, batch.schema)

 for i in range(10):
Expand Down

12 changes: 6 additions & 6 deletions
python/doc/source/memory.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -104,8 +104,8 @@ There are several kinds of :class:`~pyarrow.NativeFile` options available:
 memory maps
* :class:`~pyarrow.BufferReader`, for reading :class:`~pyarrow.Buffer` objects
 as a file
* :class:`~pyarrow.InMemoryOutputStream`, for writing data in-memory, producing
a Buffer at the end
* :class:`~pyarrow.BufferOutputStream`, for writing data in-memory, producing a
 Buffer at the end
* :class:`~pyarrow.HdfsFile`, for reading and writing data to the Hadoop Filesystem
* :class:`~pyarrow.PythonFile`, for interfacing with Python file objects in C++

Expand All@@ -124,11 +124,11 @@ then precisely track amount of memory that has been allocated:

PyArrow uses a default built-in memory pool, but in the future there may be
additional memory pools (and subpools) to choose from. Let's consider an
``InMemoryOutputStream``, which is like a ``BytesIO``:
``BufferOutputStream``, which is like a ``BytesIO``:

.. ipython:: python

 stream = pa.InMemoryOutputStream()
 stream = pa.BufferOutputStream()
 stream.write(b'foo')
 pa.total_allocated_bytes()
 for i in range(1024): stream.write(b'foo')
Expand All@@ -150,7 +150,7 @@ pass in a custom memory pool:
 my_pool = pa.jemalloc_memory_pool()
 my_pool
 my_pool.bytes_allocated()
 stream = pa.InMemoryOutputStream(my_pool)
 stream = pa.BufferOutputStream(my_pool)
 stream.write(b'foo')
 my_pool.bytes_allocated()

Expand DownExpand Up@@ -215,7 +215,7 @@ file interfaces that can read and write to Arrow Buffers.

.. ipython:: python

 writer = pa.InMemoryOutputStream()
 writer = pa.BufferOutputStream()
 writer.write(b'hello, friends')

 buf = writer.get_result()
Expand Down

37 changes: 36 additions & 1 deletion
python/pyarrow/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@
 Date32Value, Date64Value, TimestampValue)

from pyarrow.lib import (HdfsFile, NativeFile, PythonFile,
 Buffer, BufferReader, InMemoryOutputStream,
 Buffer, BufferReader, BufferOutputStream,
 OSFile, MemoryMappedFile, memory_map,
 frombuffer, read_tensor, write_tensor,
 memory_map, create_memory_map,
Expand DownExpand Up@@ -108,3 +108,38 @@ def jemalloc_memory_pool():


localfs = LocalFilesystem.get_instance()


# ----------------------------------------------------------------------
# 0.4.0 deprecations

import warnings

def _deprecate_class(old_name, new_name, klass, next_version='0.5.0'):
 msg = ('pyarrow.{0} has been renamed to '
 '{1}, will be removed in {2}'
 .format(old_name, new_name, next_version))
 def deprecated_factory(*args, **kwargs):
 warnings.warn(msg, FutureWarning)
 return klass(*args)
 return deprecated_factory

FileReader = _deprecate_class('FileReader',
 'RecordBatchFileReader',
 RecordBatchFileReader, '0.5.0')

FileWriter = _deprecate_class('FileWriter',
 'RecordBatchFileWriter',
 RecordBatchFileWriter, '0.5.0')

StreamReader = _deprecate_class('StreamReader',
 'RecordBatchStreamReader',
 RecordBatchStreamReader, '0.5.0')

StreamWriter = _deprecate_class('StreamWriter',
 'RecordBatchStreamWriter',
 RecordBatchStreamWriter, '0.5.0')

InMemoryOutputStream = _deprecate_class('InMemoryOutputStream',
 'BufferOutputStream',
 BufferOutputStream, '0.5.0')

5 changes: 3 additions & 2 deletions
python/pyarrow/includes/libarrow.pxd

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -507,8 +507,9 @@ cdef extern from "arrow/io/memory.h" namespace "arrow::io" nogil:
 CBufferReader(const shared_ptr[CBuffer]& buffer)
 CBufferReader(const uint8_t* data, int64_t nbytes)

 cdef cppclass BufferOutputStream(OutputStream):
 BufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)
 cdef cppclass CBufferOutputStream" arrow::io::BufferOutputStream"\
 (OutputStream):
 CBufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)


cdef extern from "arrow/ipc/metadata.h" namespace "arrow::ipc" nogil:
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/io.pxi

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -515,14 +515,14 @@ cdef shared_ptr[PoolBuffer] allocate_buffer(CMemoryPool* pool):
 return result


cdef class InMemoryOutputStream(NativeFile):
cdef class BufferOutputStream(NativeFile):

 cdef:
 shared_ptr[PoolBuffer] buffer

 def __cinit__(self, MemoryPool memory_pool=None):
 self.buffer = allocate_buffer(maybe_unbox_memory_pool(memory_pool))
 self.wr_file.reset(new BufferOutputStream(
 self.wr_file.reset(new CBufferOutputStream(
 <shared_ptr[ResizableBuffer]> self.buffer))
 self.is_readable = 0
 self.is_writeable = 1
Expand Down

64 changes: 64 additions & 0 deletions
python/pyarrow/tests/test_deprecations.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Check that various deprecation warnings are raised

import pyarrow as pa
import pytest


def test_inmemory_output_stream():
 with pytest.warns(FutureWarning):
 stream = pa.InMemoryOutputStream()
 assert isinstance(stream, pa.BufferOutputStream)


def test_file_reader_writer():
 data = [
 pa.array([1, 2, 3, 4]),
 pa.array(['foo', 'bar', 'baz', None]),
 pa.array([True, None, False, True])
 ]
 batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])

 sink = pa.BufferOutputStream()

 with pytest.warns(FutureWarning):
 stream_writer = pa.StreamWriter(sink, batch.schema)
 assert isinstance(stream_writer, pa.RecordBatchStreamWriter)

 sink2 = pa.BufferOutputStream()
 with pytest.warns(FutureWarning):
 file_writer = pa.FileWriter(sink2, batch.schema)
 assert isinstance(file_writer, pa.RecordBatchFileWriter)

 file_writer.write_batch(batch)
 stream_writer.write_batch(batch)

 file_writer.close()
 stream_writer.close()

 buf = sink.get_result()
 buf2 = sink2.get_result()

 with pytest.warns(FutureWarning):
 stream_reader = pa.StreamReader(buf)
 assert isinstance(stream_reader, pa.RecordBatchStreamReader)

 with pytest.warns(FutureWarning):
 file_reader = pa.FileReader(buf2)
 assert isinstance(file_reader, pa.RecordBatchFileReader)

6 changes: 3 additions & 3 deletions
python/pyarrow/tests/test_io.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def test_memory_output_stream():
 # 10 bytes
 val = b'dataabcdef'

 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()

 K = 1000
 for i in range(K):
Expand All@@ -192,7 +192,7 @@ def test_memory_output_stream():


def test_inmemory_write_after_closed():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 f.write(b'ok')
 f.get_result()

Expand All@@ -212,7 +212,7 @@ def make_buffer(bytes_obj):


def test_nativefile_write_memoryview():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 data = b'ok'

 arr = np.frombuffer(data, dtype='S1')
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/tests/test_ipc.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -209,7 +209,7 @@ def test_read_all(self):
class TestInMemoryFile(TestFile):

 def _get_sink(self):
 return pa.InMemoryOutputStream()
 return pa.BufferOutputStream()

 def _get_source(self):
 return self.sink.get_result()
Expand All@@ -219,7 +219,7 @@ def test_ipc_zero_copy_numpy():
 df = pd.DataFrame({'foo': [1.5]})

 batch = pa.RecordBatch.from_pandas(df)
 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 write_file(batch, sink)
 buffer = sink.get_result()
 reader = pa.BufferReader(buffer)
Expand Down

2 changes: 1 addition & 1 deletion
python/pyarrow/tests/test_parquet.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def _test_dataframe(size=10000, seed=0):
def test_pandas_parquet_native_file_roundtrip(tmpdir):
 df = _test_dataframe(10000)
 arrow_table = pa.Table.from_pandas(df)
 imos = pa.InMemoryOutputStream()
 imos = pa.BufferOutputStream()
 pq.write_table(arrow_table, imos, version="2.0")
 buf = imos.get_result()
 reader = pa.BufferReader(buf)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream#690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

wesm
wants to merge
3
commits into

from

 Closed


ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream
#690
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
a15910a

Rename InMemoryOutputStream to BufferOutputStream


wesm May 15, 2017
69a99cd

Fix Cython compilation


wesm May 15, 2017
85b352c

Add more graceful deprecation warnings for renamed classes, test suite


wesm May 15, 2017

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
python/doc/source/api.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -162,7 +162,7 @@ Input / Output and Shared Memory

 Buffer
 BufferReader
InMemoryOutputStream
BufferOutputStream
 NativeFile
 MemoryMappedFile
 memory_map
Expand Down

4 changes: 2 additions & 2 deletions
python/doc/source/ipc.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ this we use :class:`~pyarrow.RecordBatchStreamWriter`, which can write to a writ

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchStreamWriter(sink, batch.schema)

Here we used an in-memory Arrow buffer stream, but this could have been a
Expand DownExpand Up@@ -109,7 +109,7 @@ The :class:`~pyarrow.RecordBatchFileWriter` has the same API as

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchFileWriter(sink, batch.schema)

 for i in range(10):
Expand Down

12 changes: 6 additions & 6 deletions
python/doc/source/memory.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -104,8 +104,8 @@ There are several kinds of :class:`~pyarrow.NativeFile` options available:
 memory maps
* :class:`~pyarrow.BufferReader`, for reading :class:`~pyarrow.Buffer` objects
 as a file
* :class:`~pyarrow.InMemoryOutputStream`, for writing data in-memory, producing
a Buffer at the end
* :class:`~pyarrow.BufferOutputStream`, for writing data in-memory, producing a
 Buffer at the end
* :class:`~pyarrow.HdfsFile`, for reading and writing data to the Hadoop Filesystem
* :class:`~pyarrow.PythonFile`, for interfacing with Python file objects in C++

Expand All@@ -124,11 +124,11 @@ then precisely track amount of memory that has been allocated:

PyArrow uses a default built-in memory pool, but in the future there may be
additional memory pools (and subpools) to choose from. Let's consider an
``InMemoryOutputStream``, which is like a ``BytesIO``:
``BufferOutputStream``, which is like a ``BytesIO``:

.. ipython:: python

 stream = pa.InMemoryOutputStream()
 stream = pa.BufferOutputStream()
 stream.write(b'foo')
 pa.total_allocated_bytes()
 for i in range(1024): stream.write(b'foo')
Expand All@@ -150,7 +150,7 @@ pass in a custom memory pool:
 my_pool = pa.jemalloc_memory_pool()
 my_pool
 my_pool.bytes_allocated()
 stream = pa.InMemoryOutputStream(my_pool)
 stream = pa.BufferOutputStream(my_pool)
 stream.write(b'foo')
 my_pool.bytes_allocated()

Expand DownExpand Up@@ -215,7 +215,7 @@ file interfaces that can read and write to Arrow Buffers.

.. ipython:: python

 writer = pa.InMemoryOutputStream()
 writer = pa.BufferOutputStream()
 writer.write(b'hello, friends')

 buf = writer.get_result()
Expand Down

37 changes: 36 additions & 1 deletion
python/pyarrow/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@
 Date32Value, Date64Value, TimestampValue)

from pyarrow.lib import (HdfsFile, NativeFile, PythonFile,
 Buffer, BufferReader, InMemoryOutputStream,
 Buffer, BufferReader, BufferOutputStream,
 OSFile, MemoryMappedFile, memory_map,
 frombuffer, read_tensor, write_tensor,
 memory_map, create_memory_map,
Expand DownExpand Up@@ -108,3 +108,38 @@ def jemalloc_memory_pool():


localfs = LocalFilesystem.get_instance()


# ----------------------------------------------------------------------
# 0.4.0 deprecations

import warnings

def _deprecate_class(old_name, new_name, klass, next_version='0.5.0'):
 msg = ('pyarrow.{0} has been renamed to '
 '{1}, will be removed in {2}'
 .format(old_name, new_name, next_version))
 def deprecated_factory(*args, **kwargs):
 warnings.warn(msg, FutureWarning)
 return klass(*args)
 return deprecated_factory

FileReader = _deprecate_class('FileReader',
 'RecordBatchFileReader',
 RecordBatchFileReader, '0.5.0')

FileWriter = _deprecate_class('FileWriter',
 'RecordBatchFileWriter',
 RecordBatchFileWriter, '0.5.0')

StreamReader = _deprecate_class('StreamReader',
 'RecordBatchStreamReader',
 RecordBatchStreamReader, '0.5.0')

StreamWriter = _deprecate_class('StreamWriter',
 'RecordBatchStreamWriter',
 RecordBatchStreamWriter, '0.5.0')

InMemoryOutputStream = _deprecate_class('InMemoryOutputStream',
 'BufferOutputStream',
 BufferOutputStream, '0.5.0')

5 changes: 3 additions & 2 deletions
python/pyarrow/includes/libarrow.pxd

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -507,8 +507,9 @@ cdef extern from "arrow/io/memory.h" namespace "arrow::io" nogil:
 CBufferReader(const shared_ptr[CBuffer]& buffer)
 CBufferReader(const uint8_t* data, int64_t nbytes)

 cdef cppclass BufferOutputStream(OutputStream):
 BufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)
 cdef cppclass CBufferOutputStream" arrow::io::BufferOutputStream"\
 (OutputStream):
 CBufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)


cdef extern from "arrow/ipc/metadata.h" namespace "arrow::ipc" nogil:
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/io.pxi

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -515,14 +515,14 @@ cdef shared_ptr[PoolBuffer] allocate_buffer(CMemoryPool* pool):
 return result


cdef class InMemoryOutputStream(NativeFile):
cdef class BufferOutputStream(NativeFile):

 cdef:
 shared_ptr[PoolBuffer] buffer

 def __cinit__(self, MemoryPool memory_pool=None):
 self.buffer = allocate_buffer(maybe_unbox_memory_pool(memory_pool))
 self.wr_file.reset(new BufferOutputStream(
 self.wr_file.reset(new CBufferOutputStream(
 <shared_ptr[ResizableBuffer]> self.buffer))
 self.is_readable = 0
 self.is_writeable = 1
Expand Down

64 changes: 64 additions & 0 deletions
python/pyarrow/tests/test_deprecations.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Check that various deprecation warnings are raised

import pyarrow as pa
import pytest


def test_inmemory_output_stream():
 with pytest.warns(FutureWarning):
 stream = pa.InMemoryOutputStream()
 assert isinstance(stream, pa.BufferOutputStream)


def test_file_reader_writer():
 data = [
 pa.array([1, 2, 3, 4]),
 pa.array(['foo', 'bar', 'baz', None]),
 pa.array([True, None, False, True])
 ]
 batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])

 sink = pa.BufferOutputStream()

 with pytest.warns(FutureWarning):
 stream_writer = pa.StreamWriter(sink, batch.schema)
 assert isinstance(stream_writer, pa.RecordBatchStreamWriter)

 sink2 = pa.BufferOutputStream()
 with pytest.warns(FutureWarning):
 file_writer = pa.FileWriter(sink2, batch.schema)
 assert isinstance(file_writer, pa.RecordBatchFileWriter)

 file_writer.write_batch(batch)
 stream_writer.write_batch(batch)

 file_writer.close()
 stream_writer.close()

 buf = sink.get_result()
 buf2 = sink2.get_result()

 with pytest.warns(FutureWarning):
 stream_reader = pa.StreamReader(buf)
 assert isinstance(stream_reader, pa.RecordBatchStreamReader)

 with pytest.warns(FutureWarning):
 file_reader = pa.FileReader(buf2)
 assert isinstance(file_reader, pa.RecordBatchFileReader)

6 changes: 3 additions & 3 deletions
python/pyarrow/tests/test_io.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def test_memory_output_stream():
 # 10 bytes
 val = b'dataabcdef'

 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()

 K = 1000
 for i in range(K):
Expand All@@ -192,7 +192,7 @@ def test_memory_output_stream():


def test_inmemory_write_after_closed():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 f.write(b'ok')
 f.get_result()

Expand All@@ -212,7 +212,7 @@ def make_buffer(bytes_obj):


def test_nativefile_write_memoryview():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 data = b'ok'

 arr = np.frombuffer(data, dtype='S1')
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/tests/test_ipc.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -209,7 +209,7 @@ def test_read_all(self):
class TestInMemoryFile(TestFile):

 def _get_sink(self):
 return pa.InMemoryOutputStream()
 return pa.BufferOutputStream()

 def _get_source(self):
 return self.sink.get_result()
Expand All@@ -219,7 +219,7 @@ def test_ipc_zero_copy_numpy():
 df = pd.DataFrame({'foo': [1.5]})

 batch = pa.RecordBatch.from_pandas(df)
 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 write_file(batch, sink)
 buffer = sink.get_result()
 reader = pa.BufferReader(buffer)
Expand Down

2 changes: 1 addition & 1 deletion
python/pyarrow/tests/test_parquet.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def _test_dataframe(size=10000, seed=0):
def test_pandas_parquet_native_file_roundtrip(tmpdir):
 df = _test_dataframe(10000)
 arrow_table = pa.Table.from_pandas(df)
 imos = pa.InMemoryOutputStream()
 imos = pa.BufferOutputStream()
 pq.write_table(arrow_table, imos, version="2.0")
 buf = imos.get_result()
 reader = pa.BufferReader(buf)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream#690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

wesm
wants to merge
3
commits into

from

 Closed


ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream
#690
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
a15910a

Rename InMemoryOutputStream to BufferOutputStream


wesm May 15, 2017
69a99cd

Fix Cython compilation


wesm May 15, 2017
85b352c

Add more graceful deprecation warnings for renamed classes, test suite


wesm May 15, 2017

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
python/doc/source/api.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -162,7 +162,7 @@ Input / Output and Shared Memory

 Buffer
 BufferReader
InMemoryOutputStream
BufferOutputStream
 NativeFile
 MemoryMappedFile
 memory_map
Expand Down

4 changes: 2 additions & 2 deletions
python/doc/source/ipc.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ this we use :class:`~pyarrow.RecordBatchStreamWriter`, which can write to a writ

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchStreamWriter(sink, batch.schema)

Here we used an in-memory Arrow buffer stream, but this could have been a
Expand DownExpand Up@@ -109,7 +109,7 @@ The :class:`~pyarrow.RecordBatchFileWriter` has the same API as

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchFileWriter(sink, batch.schema)

 for i in range(10):
Expand Down

12 changes: 6 additions & 6 deletions
python/doc/source/memory.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -104,8 +104,8 @@ There are several kinds of :class:`~pyarrow.NativeFile` options available:
 memory maps
* :class:`~pyarrow.BufferReader`, for reading :class:`~pyarrow.Buffer` objects
 as a file
* :class:`~pyarrow.InMemoryOutputStream`, for writing data in-memory, producing
a Buffer at the end
* :class:`~pyarrow.BufferOutputStream`, for writing data in-memory, producing a
 Buffer at the end
* :class:`~pyarrow.HdfsFile`, for reading and writing data to the Hadoop Filesystem
* :class:`~pyarrow.PythonFile`, for interfacing with Python file objects in C++

Expand All@@ -124,11 +124,11 @@ then precisely track amount of memory that has been allocated:

PyArrow uses a default built-in memory pool, but in the future there may be
additional memory pools (and subpools) to choose from. Let's consider an
``InMemoryOutputStream``, which is like a ``BytesIO``:
``BufferOutputStream``, which is like a ``BytesIO``:

.. ipython:: python

 stream = pa.InMemoryOutputStream()
 stream = pa.BufferOutputStream()
 stream.write(b'foo')
 pa.total_allocated_bytes()
 for i in range(1024): stream.write(b'foo')
Expand All@@ -150,7 +150,7 @@ pass in a custom memory pool:
 my_pool = pa.jemalloc_memory_pool()
 my_pool
 my_pool.bytes_allocated()
 stream = pa.InMemoryOutputStream(my_pool)
 stream = pa.BufferOutputStream(my_pool)
 stream.write(b'foo')
 my_pool.bytes_allocated()

Expand DownExpand Up@@ -215,7 +215,7 @@ file interfaces that can read and write to Arrow Buffers.

.. ipython:: python

 writer = pa.InMemoryOutputStream()
 writer = pa.BufferOutputStream()
 writer.write(b'hello, friends')

 buf = writer.get_result()
Expand Down

37 changes: 36 additions & 1 deletion
python/pyarrow/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@
 Date32Value, Date64Value, TimestampValue)

from pyarrow.lib import (HdfsFile, NativeFile, PythonFile,
 Buffer, BufferReader, InMemoryOutputStream,
 Buffer, BufferReader, BufferOutputStream,
 OSFile, MemoryMappedFile, memory_map,
 frombuffer, read_tensor, write_tensor,
 memory_map, create_memory_map,
Expand DownExpand Up@@ -108,3 +108,38 @@ def jemalloc_memory_pool():


localfs = LocalFilesystem.get_instance()


# ----------------------------------------------------------------------
# 0.4.0 deprecations

import warnings

def _deprecate_class(old_name, new_name, klass, next_version='0.5.0'):
 msg = ('pyarrow.{0} has been renamed to '
 '{1}, will be removed in {2}'
 .format(old_name, new_name, next_version))
 def deprecated_factory(*args, **kwargs):
 warnings.warn(msg, FutureWarning)
 return klass(*args)
 return deprecated_factory

FileReader = _deprecate_class('FileReader',
 'RecordBatchFileReader',
 RecordBatchFileReader, '0.5.0')

FileWriter = _deprecate_class('FileWriter',
 'RecordBatchFileWriter',
 RecordBatchFileWriter, '0.5.0')

StreamReader = _deprecate_class('StreamReader',
 'RecordBatchStreamReader',
 RecordBatchStreamReader, '0.5.0')

StreamWriter = _deprecate_class('StreamWriter',
 'RecordBatchStreamWriter',
 RecordBatchStreamWriter, '0.5.0')

InMemoryOutputStream = _deprecate_class('InMemoryOutputStream',
 'BufferOutputStream',
 BufferOutputStream, '0.5.0')

5 changes: 3 additions & 2 deletions
python/pyarrow/includes/libarrow.pxd

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -507,8 +507,9 @@ cdef extern from "arrow/io/memory.h" namespace "arrow::io" nogil:
 CBufferReader(const shared_ptr[CBuffer]& buffer)
 CBufferReader(const uint8_t* data, int64_t nbytes)

 cdef cppclass BufferOutputStream(OutputStream):
 BufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)
 cdef cppclass CBufferOutputStream" arrow::io::BufferOutputStream"\
 (OutputStream):
 CBufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)


cdef extern from "arrow/ipc/metadata.h" namespace "arrow::ipc" nogil:
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/io.pxi

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -515,14 +515,14 @@ cdef shared_ptr[PoolBuffer] allocate_buffer(CMemoryPool* pool):
 return result


cdef class InMemoryOutputStream(NativeFile):
cdef class BufferOutputStream(NativeFile):

 cdef:
 shared_ptr[PoolBuffer] buffer

 def __cinit__(self, MemoryPool memory_pool=None):
 self.buffer = allocate_buffer(maybe_unbox_memory_pool(memory_pool))
 self.wr_file.reset(new BufferOutputStream(
 self.wr_file.reset(new CBufferOutputStream(
 <shared_ptr[ResizableBuffer]> self.buffer))
 self.is_readable = 0
 self.is_writeable = 1
Expand Down

64 changes: 64 additions & 0 deletions
python/pyarrow/tests/test_deprecations.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Check that various deprecation warnings are raised

import pyarrow as pa
import pytest


def test_inmemory_output_stream():
 with pytest.warns(FutureWarning):
 stream = pa.InMemoryOutputStream()
 assert isinstance(stream, pa.BufferOutputStream)


def test_file_reader_writer():
 data = [
 pa.array([1, 2, 3, 4]),
 pa.array(['foo', 'bar', 'baz', None]),
 pa.array([True, None, False, True])
 ]
 batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])

 sink = pa.BufferOutputStream()

 with pytest.warns(FutureWarning):
 stream_writer = pa.StreamWriter(sink, batch.schema)
 assert isinstance(stream_writer, pa.RecordBatchStreamWriter)

 sink2 = pa.BufferOutputStream()
 with pytest.warns(FutureWarning):
 file_writer = pa.FileWriter(sink2, batch.schema)
 assert isinstance(file_writer, pa.RecordBatchFileWriter)

 file_writer.write_batch(batch)
 stream_writer.write_batch(batch)

 file_writer.close()
 stream_writer.close()

 buf = sink.get_result()
 buf2 = sink2.get_result()

 with pytest.warns(FutureWarning):
 stream_reader = pa.StreamReader(buf)
 assert isinstance(stream_reader, pa.RecordBatchStreamReader)

 with pytest.warns(FutureWarning):
 file_reader = pa.FileReader(buf2)
 assert isinstance(file_reader, pa.RecordBatchFileReader)

6 changes: 3 additions & 3 deletions
python/pyarrow/tests/test_io.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def test_memory_output_stream():
 # 10 bytes
 val = b'dataabcdef'

 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()

 K = 1000
 for i in range(K):
Expand All@@ -192,7 +192,7 @@ def test_memory_output_stream():


def test_inmemory_write_after_closed():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 f.write(b'ok')
 f.get_result()

Expand All@@ -212,7 +212,7 @@ def make_buffer(bytes_obj):


def test_nativefile_write_memoryview():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 data = b'ok'

 arr = np.frombuffer(data, dtype='S1')
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/tests/test_ipc.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -209,7 +209,7 @@ def test_read_all(self):
class TestInMemoryFile(TestFile):

 def _get_sink(self):
 return pa.InMemoryOutputStream()
 return pa.BufferOutputStream()

 def _get_source(self):
 return self.sink.get_result()
Expand All@@ -219,7 +219,7 @@ def test_ipc_zero_copy_numpy():
 df = pd.DataFrame({'foo': [1.5]})

 batch = pa.RecordBatch.from_pandas(df)
 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 write_file(batch, sink)
 buffer = sink.get_result()
 reader = pa.BufferReader(buffer)
Expand Down

2 changes: 1 addition & 1 deletion
python/pyarrow/tests/test_parquet.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def _test_dataframe(size=10000, seed=0):
def test_pandas_parquet_native_file_roundtrip(tmpdir):
 df = _test_dataframe(10000)
 arrow_table = pa.Table.from_pandas(df)
 imos = pa.InMemoryOutputStream()
 imos = pa.BufferOutputStream()
 pq.write_table(arrow_table, imos, version="2.0")
 buf = imos.get_result()
 reader = pa.BufferReader(buf)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream#690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

wesm
wants to merge
3
commits into

from

 Closed


ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream
#690
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
a15910a

Rename InMemoryOutputStream to BufferOutputStream


wesm May 15, 2017
69a99cd

Fix Cython compilation


wesm May 15, 2017
85b352c

Add more graceful deprecation warnings for renamed classes, test suite


wesm May 15, 2017

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
python/doc/source/api.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -162,7 +162,7 @@ Input / Output and Shared Memory

 Buffer
 BufferReader
InMemoryOutputStream
BufferOutputStream
 NativeFile
 MemoryMappedFile
 memory_map
Expand Down

4 changes: 2 additions & 2 deletions
python/doc/source/ipc.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ this we use :class:`~pyarrow.RecordBatchStreamWriter`, which can write to a writ

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchStreamWriter(sink, batch.schema)

Here we used an in-memory Arrow buffer stream, but this could have been a
Expand DownExpand Up@@ -109,7 +109,7 @@ The :class:`~pyarrow.RecordBatchFileWriter` has the same API as

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchFileWriter(sink, batch.schema)

 for i in range(10):
Expand Down

12 changes: 6 additions & 6 deletions
python/doc/source/memory.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -104,8 +104,8 @@ There are several kinds of :class:`~pyarrow.NativeFile` options available:
 memory maps
* :class:`~pyarrow.BufferReader`, for reading :class:`~pyarrow.Buffer` objects
 as a file
* :class:`~pyarrow.InMemoryOutputStream`, for writing data in-memory, producing
a Buffer at the end
* :class:`~pyarrow.BufferOutputStream`, for writing data in-memory, producing a
 Buffer at the end
* :class:`~pyarrow.HdfsFile`, for reading and writing data to the Hadoop Filesystem
* :class:`~pyarrow.PythonFile`, for interfacing with Python file objects in C++

Expand All@@ -124,11 +124,11 @@ then precisely track amount of memory that has been allocated:

PyArrow uses a default built-in memory pool, but in the future there may be
additional memory pools (and subpools) to choose from. Let's consider an
``InMemoryOutputStream``, which is like a ``BytesIO``:
``BufferOutputStream``, which is like a ``BytesIO``:

.. ipython:: python

 stream = pa.InMemoryOutputStream()
 stream = pa.BufferOutputStream()
 stream.write(b'foo')
 pa.total_allocated_bytes()
 for i in range(1024): stream.write(b'foo')
Expand All@@ -150,7 +150,7 @@ pass in a custom memory pool:
 my_pool = pa.jemalloc_memory_pool()
 my_pool
 my_pool.bytes_allocated()
 stream = pa.InMemoryOutputStream(my_pool)
 stream = pa.BufferOutputStream(my_pool)
 stream.write(b'foo')
 my_pool.bytes_allocated()

Expand DownExpand Up@@ -215,7 +215,7 @@ file interfaces that can read and write to Arrow Buffers.

.. ipython:: python

 writer = pa.InMemoryOutputStream()
 writer = pa.BufferOutputStream()
 writer.write(b'hello, friends')

 buf = writer.get_result()
Expand Down

37 changes: 36 additions & 1 deletion
python/pyarrow/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@
 Date32Value, Date64Value, TimestampValue)

from pyarrow.lib import (HdfsFile, NativeFile, PythonFile,
 Buffer, BufferReader, InMemoryOutputStream,
 Buffer, BufferReader, BufferOutputStream,
 OSFile, MemoryMappedFile, memory_map,
 frombuffer, read_tensor, write_tensor,
 memory_map, create_memory_map,
Expand DownExpand Up@@ -108,3 +108,38 @@ def jemalloc_memory_pool():


localfs = LocalFilesystem.get_instance()


# ----------------------------------------------------------------------
# 0.4.0 deprecations

import warnings

def _deprecate_class(old_name, new_name, klass, next_version='0.5.0'):
 msg = ('pyarrow.{0} has been renamed to '
 '{1}, will be removed in {2}'
 .format(old_name, new_name, next_version))
 def deprecated_factory(*args, **kwargs):
 warnings.warn(msg, FutureWarning)
 return klass(*args)
 return deprecated_factory

FileReader = _deprecate_class('FileReader',
 'RecordBatchFileReader',
 RecordBatchFileReader, '0.5.0')

FileWriter = _deprecate_class('FileWriter',
 'RecordBatchFileWriter',
 RecordBatchFileWriter, '0.5.0')

StreamReader = _deprecate_class('StreamReader',
 'RecordBatchStreamReader',
 RecordBatchStreamReader, '0.5.0')

StreamWriter = _deprecate_class('StreamWriter',
 'RecordBatchStreamWriter',
 RecordBatchStreamWriter, '0.5.0')

InMemoryOutputStream = _deprecate_class('InMemoryOutputStream',
 'BufferOutputStream',
 BufferOutputStream, '0.5.0')

5 changes: 3 additions & 2 deletions
python/pyarrow/includes/libarrow.pxd

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -507,8 +507,9 @@ cdef extern from "arrow/io/memory.h" namespace "arrow::io" nogil:
 CBufferReader(const shared_ptr[CBuffer]& buffer)
 CBufferReader(const uint8_t* data, int64_t nbytes)

 cdef cppclass BufferOutputStream(OutputStream):
 BufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)
 cdef cppclass CBufferOutputStream" arrow::io::BufferOutputStream"\
 (OutputStream):
 CBufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)


cdef extern from "arrow/ipc/metadata.h" namespace "arrow::ipc" nogil:
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/io.pxi

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -515,14 +515,14 @@ cdef shared_ptr[PoolBuffer] allocate_buffer(CMemoryPool* pool):
 return result


cdef class InMemoryOutputStream(NativeFile):
cdef class BufferOutputStream(NativeFile):

 cdef:
 shared_ptr[PoolBuffer] buffer

 def __cinit__(self, MemoryPool memory_pool=None):
 self.buffer = allocate_buffer(maybe_unbox_memory_pool(memory_pool))
 self.wr_file.reset(new BufferOutputStream(
 self.wr_file.reset(new CBufferOutputStream(
 <shared_ptr[ResizableBuffer]> self.buffer))
 self.is_readable = 0
 self.is_writeable = 1
Expand Down

64 changes: 64 additions & 0 deletions
python/pyarrow/tests/test_deprecations.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Check that various deprecation warnings are raised

import pyarrow as pa
import pytest


def test_inmemory_output_stream():
 with pytest.warns(FutureWarning):
 stream = pa.InMemoryOutputStream()
 assert isinstance(stream, pa.BufferOutputStream)


def test_file_reader_writer():
 data = [
 pa.array([1, 2, 3, 4]),
 pa.array(['foo', 'bar', 'baz', None]),
 pa.array([True, None, False, True])
 ]
 batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])

 sink = pa.BufferOutputStream()

 with pytest.warns(FutureWarning):
 stream_writer = pa.StreamWriter(sink, batch.schema)
 assert isinstance(stream_writer, pa.RecordBatchStreamWriter)

 sink2 = pa.BufferOutputStream()
 with pytest.warns(FutureWarning):
 file_writer = pa.FileWriter(sink2, batch.schema)
 assert isinstance(file_writer, pa.RecordBatchFileWriter)

 file_writer.write_batch(batch)
 stream_writer.write_batch(batch)

 file_writer.close()
 stream_writer.close()

 buf = sink.get_result()
 buf2 = sink2.get_result()

 with pytest.warns(FutureWarning):
 stream_reader = pa.StreamReader(buf)
 assert isinstance(stream_reader, pa.RecordBatchStreamReader)

 with pytest.warns(FutureWarning):
 file_reader = pa.FileReader(buf2)
 assert isinstance(file_reader, pa.RecordBatchFileReader)

6 changes: 3 additions & 3 deletions
python/pyarrow/tests/test_io.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def test_memory_output_stream():
 # 10 bytes
 val = b'dataabcdef'

 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()

 K = 1000
 for i in range(K):
Expand All@@ -192,7 +192,7 @@ def test_memory_output_stream():


def test_inmemory_write_after_closed():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 f.write(b'ok')
 f.get_result()

Expand All@@ -212,7 +212,7 @@ def make_buffer(bytes_obj):


def test_nativefile_write_memoryview():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 data = b'ok'

 arr = np.frombuffer(data, dtype='S1')
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/tests/test_ipc.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -209,7 +209,7 @@ def test_read_all(self):
class TestInMemoryFile(TestFile):

 def _get_sink(self):
 return pa.InMemoryOutputStream()
 return pa.BufferOutputStream()

 def _get_source(self):
 return self.sink.get_result()
Expand All@@ -219,7 +219,7 @@ def test_ipc_zero_copy_numpy():
 df = pd.DataFrame({'foo': [1.5]})

 batch = pa.RecordBatch.from_pandas(df)
 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 write_file(batch, sink)
 buffer = sink.get_result()
 reader = pa.BufferReader(buffer)
Expand Down

2 changes: 1 addition & 1 deletion
python/pyarrow/tests/test_parquet.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def _test_dataframe(size=10000, seed=0):
def test_pandas_parquet_native_file_roundtrip(tmpdir):
 df = _test_dataframe(10000)
 arrow_table = pa.Table.from_pandas(df)
 imos = pa.InMemoryOutputStream()
 imos = pa.BufferOutputStream()
 pq.write_table(arrow_table, imos, version="2.0")
 buf = imos.get_result()
 reader = pa.BufferReader(buf)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream#690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

wesm
wants to merge
3
commits into

from

 Closed


ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream
#690
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
a15910a

Rename InMemoryOutputStream to BufferOutputStream


wesm May 15, 2017
69a99cd

Fix Cython compilation


wesm May 15, 2017
85b352c

Add more graceful deprecation warnings for renamed classes, test suite


wesm May 15, 2017

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


2 changes: 1 addition & 1 deletion
python/doc/source/api.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -162,7 +162,7 @@ Input / Output and Shared Memory

 Buffer
 BufferReader
InMemoryOutputStream
BufferOutputStream
 NativeFile
 MemoryMappedFile
 memory_map
Expand Down

4 changes: 2 additions & 2 deletions
python/doc/source/ipc.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ this we use :class:`~pyarrow.RecordBatchStreamWriter`, which can write to a writ

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchStreamWriter(sink, batch.schema)

Here we used an in-memory Arrow buffer stream, but this could have been a
Expand DownExpand Up@@ -109,7 +109,7 @@ The :class:`~pyarrow.RecordBatchFileWriter` has the same API as

.. ipython:: python

 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 writer = pa.RecordBatchFileWriter(sink, batch.schema)

 for i in range(10):
Expand Down

12 changes: 6 additions & 6 deletions
python/doc/source/memory.rst

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -104,8 +104,8 @@ There are several kinds of :class:`~pyarrow.NativeFile` options available:
 memory maps
* :class:`~pyarrow.BufferReader`, for reading :class:`~pyarrow.Buffer` objects
 as a file
* :class:`~pyarrow.InMemoryOutputStream`, for writing data in-memory, producing
a Buffer at the end
* :class:`~pyarrow.BufferOutputStream`, for writing data in-memory, producing a
 Buffer at the end
* :class:`~pyarrow.HdfsFile`, for reading and writing data to the Hadoop Filesystem
* :class:`~pyarrow.PythonFile`, for interfacing with Python file objects in C++

Expand All@@ -124,11 +124,11 @@ then precisely track amount of memory that has been allocated:

PyArrow uses a default built-in memory pool, but in the future there may be
additional memory pools (and subpools) to choose from. Let's consider an
``InMemoryOutputStream``, which is like a ``BytesIO``:
``BufferOutputStream``, which is like a ``BytesIO``:

.. ipython:: python

 stream = pa.InMemoryOutputStream()
 stream = pa.BufferOutputStream()
 stream.write(b'foo')
 pa.total_allocated_bytes()
 for i in range(1024): stream.write(b'foo')
Expand All@@ -150,7 +150,7 @@ pass in a custom memory pool:
 my_pool = pa.jemalloc_memory_pool()
 my_pool
 my_pool.bytes_allocated()
 stream = pa.InMemoryOutputStream(my_pool)
 stream = pa.BufferOutputStream(my_pool)
 stream.write(b'foo')
 my_pool.bytes_allocated()

Expand DownExpand Up@@ -215,7 +215,7 @@ file interfaces that can read and write to Arrow Buffers.

.. ipython:: python

 writer = pa.InMemoryOutputStream()
 writer = pa.BufferOutputStream()
 writer.write(b'hello, friends')

 buf = writer.get_result()
Expand Down

37 changes: 36 additions & 1 deletion
python/pyarrow/__init__.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@
 Date32Value, Date64Value, TimestampValue)

from pyarrow.lib import (HdfsFile, NativeFile, PythonFile,
 Buffer, BufferReader, InMemoryOutputStream,
 Buffer, BufferReader, BufferOutputStream,
 OSFile, MemoryMappedFile, memory_map,
 frombuffer, read_tensor, write_tensor,
 memory_map, create_memory_map,
Expand DownExpand Up@@ -108,3 +108,38 @@ def jemalloc_memory_pool():


localfs = LocalFilesystem.get_instance()


# ----------------------------------------------------------------------
# 0.4.0 deprecations

import warnings

def _deprecate_class(old_name, new_name, klass, next_version='0.5.0'):
 msg = ('pyarrow.{0} has been renamed to '
 '{1}, will be removed in {2}'
 .format(old_name, new_name, next_version))
 def deprecated_factory(*args, **kwargs):
 warnings.warn(msg, FutureWarning)
 return klass(*args)
 return deprecated_factory

FileReader = _deprecate_class('FileReader',
 'RecordBatchFileReader',
 RecordBatchFileReader, '0.5.0')

FileWriter = _deprecate_class('FileWriter',
 'RecordBatchFileWriter',
 RecordBatchFileWriter, '0.5.0')

StreamReader = _deprecate_class('StreamReader',
 'RecordBatchStreamReader',
 RecordBatchStreamReader, '0.5.0')

StreamWriter = _deprecate_class('StreamWriter',
 'RecordBatchStreamWriter',
 RecordBatchStreamWriter, '0.5.0')

InMemoryOutputStream = _deprecate_class('InMemoryOutputStream',
 'BufferOutputStream',
 BufferOutputStream, '0.5.0')

5 changes: 3 additions & 2 deletions
python/pyarrow/includes/libarrow.pxd

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -507,8 +507,9 @@ cdef extern from "arrow/io/memory.h" namespace "arrow::io" nogil:
 CBufferReader(const shared_ptr[CBuffer]& buffer)
 CBufferReader(const uint8_t* data, int64_t nbytes)

 cdef cppclass BufferOutputStream(OutputStream):
 BufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)
 cdef cppclass CBufferOutputStream" arrow::io::BufferOutputStream"\
 (OutputStream):
 CBufferOutputStream(const shared_ptr[ResizableBuffer]& buffer)


cdef extern from "arrow/ipc/metadata.h" namespace "arrow::ipc" nogil:
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/io.pxi

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -515,14 +515,14 @@ cdef shared_ptr[PoolBuffer] allocate_buffer(CMemoryPool* pool):
 return result


cdef class InMemoryOutputStream(NativeFile):
cdef class BufferOutputStream(NativeFile):

 cdef:
 shared_ptr[PoolBuffer] buffer

 def __cinit__(self, MemoryPool memory_pool=None):
 self.buffer = allocate_buffer(maybe_unbox_memory_pool(memory_pool))
 self.wr_file.reset(new BufferOutputStream(
 self.wr_file.reset(new CBufferOutputStream(
 <shared_ptr[ResizableBuffer]> self.buffer))
 self.is_readable = 0
 self.is_writeable = 1
Expand Down

64 changes: 64 additions & 0 deletions
python/pyarrow/tests/test_deprecations.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Check that various deprecation warnings are raised

import pyarrow as pa
import pytest


def test_inmemory_output_stream():
 with pytest.warns(FutureWarning):
 stream = pa.InMemoryOutputStream()
 assert isinstance(stream, pa.BufferOutputStream)


def test_file_reader_writer():
 data = [
 pa.array([1, 2, 3, 4]),
 pa.array(['foo', 'bar', 'baz', None]),
 pa.array([True, None, False, True])
 ]
 batch = pa.RecordBatch.from_arrays(data, ['f0', 'f1', 'f2'])

 sink = pa.BufferOutputStream()

 with pytest.warns(FutureWarning):
 stream_writer = pa.StreamWriter(sink, batch.schema)
 assert isinstance(stream_writer, pa.RecordBatchStreamWriter)

 sink2 = pa.BufferOutputStream()
 with pytest.warns(FutureWarning):
 file_writer = pa.FileWriter(sink2, batch.schema)
 assert isinstance(file_writer, pa.RecordBatchFileWriter)

 file_writer.write_batch(batch)
 stream_writer.write_batch(batch)

 file_writer.close()
 stream_writer.close()

 buf = sink.get_result()
 buf2 = sink2.get_result()

 with pytest.warns(FutureWarning):
 stream_reader = pa.StreamReader(buf)
 assert isinstance(stream_reader, pa.RecordBatchStreamReader)

 with pytest.warns(FutureWarning):
 file_reader = pa.FileReader(buf2)
 assert isinstance(file_reader, pa.RecordBatchFileReader)

6 changes: 3 additions & 3 deletions
python/pyarrow/tests/test_io.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def test_memory_output_stream():
 # 10 bytes
 val = b'dataabcdef'

 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()

 K = 1000
 for i in range(K):
Expand All@@ -192,7 +192,7 @@ def test_memory_output_stream():


def test_inmemory_write_after_closed():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 f.write(b'ok')
 f.get_result()

Expand All@@ -212,7 +212,7 @@ def make_buffer(bytes_obj):


def test_nativefile_write_memoryview():
 f = pa.InMemoryOutputStream()
 f = pa.BufferOutputStream()
 data = b'ok'

 arr = np.frombuffer(data, dtype='S1')
Expand Down

4 changes: 2 additions & 2 deletions
python/pyarrow/tests/test_ipc.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -209,7 +209,7 @@ def test_read_all(self):
class TestInMemoryFile(TestFile):

 def _get_sink(self):
 return pa.InMemoryOutputStream()
 return pa.BufferOutputStream()

 def _get_source(self):
 return self.sink.get_result()
Expand All@@ -219,7 +219,7 @@ def test_ipc_zero_copy_numpy():
 df = pd.DataFrame({'foo': [1.5]})

 batch = pa.RecordBatch.from_pandas(df)
 sink = pa.InMemoryOutputStream()
 sink = pa.BufferOutputStream()
 write_file(batch, sink)
 buffer = sink.get_result()
 reader = pa.BufferReader(buffer)
Expand Down

2 changes: 1 addition & 1 deletion
python/pyarrow/tests/test_parquet.py

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -179,7 +179,7 @@ def _test_dataframe(size=10000, seed=0):
def test_pandas_parquet_native_file_roundtrip(tmpdir):
 df = _test_dataframe(10000)
 arrow_table = pa.Table.from_pandas(df)
 imos = pa.InMemoryOutputStream()
 imos = pa.BufferOutputStream()
 pq.write_table(arrow_table, imos, version="2.0")
 buf = imos.get_result()
 reader = pa.BufferReader(buf)
Expand Down