ARROW-2229: [C++][Python] Add WriteCsv functionality.#9504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
emkornfield wants to merge 18 commits into from
Closed

ARROW-2229: [C++][Python] Add WriteCsv functionality. #9504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
7eaab50
ARROW-2229: [C++][Python] Add WriteCsv functionality.
emkornfield Feb 3, 2021
ecc508b
Update cpp/src/arrow/csv/writer.cc
Feb 16, 2021
ac7b3f4
fix lint
emkornfield Feb 16, 2021
8dcef6a
add missing test
emkornfield Feb 16, 2021
efffe81
lint and appveyor fixes
emkornfield Feb 16, 2021
7794307
start addressing feedback
emkornfield Feb 20, 2021
4deb8a8
add cmake configuration
emkornfield Feb 20, 2021
8b40ed7
update python
emkornfield Feb 21, 2021
61856a4
refactor c++ tests
emkornfield Feb 22, 2021
5c833a8
hopefully fix lint
emkornfield Feb 22, 2021
73826b8
hopefully make windows builds happy
emkornfield Feb 22, 2021
c5d76eb
more lint and windows
emkornfield Feb 22, 2021
2695891
address comments
emkornfield Feb 24, 2021
da03cd6
add back options
emkornfield Feb 24, 2021
2eb2251
Try uint8_t instead of bool.
Feb 24, 2021
23a2b0c
change back to bool and remove custom allocator
Feb 24, 2021
7e33776
remove from initialize list
Feb 25, 2021
a5ee563
Improve docstrings
pitrou Feb 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cpp/src/arrow/CMakeLists.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -349,6 +349,9 @@ if(ARROW_CSV)
csv/options.cc
csv/parser.cc
csv/reader.cc)
if(ARROW_COMPUTE)
list(APPEND ARROW_SRCS csv/writer.cc)
endif()

list(APPEND ARROW_TESTING_SRCS csv/test_common.cc)
endif()
Expand Down
22 changes: 14 additions & 8 deletions cpp/src/arrow/csv/CMakeLists.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -15,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.

add_arrow_test(csv-test
SOURCES
chunker_test.cc
column_builder_test.cc
column_decoder_test.cc
converter_test.cc
parser_test.cc
reader_test.cc)
set(CSV_TEST_SRCS
chunker_test.cc
column_builder_test.cc
column_decoder_test.cc
converter_test.cc
parser_test.cc
reader_test.cc)

# Writer depends on compute's cast functionality
if(ARROW_COMPUTE)
list(APPEND CSV_TEST_SRCS writer_test.cc)
endif()

add_arrow_test(csv-test SOURCES ${CSV_TEST_SRCS})

add_arrow_benchmark(converter_benchmark PREFIX "arrow-csv")
add_arrow_benchmark(parser_benchmark PREFIX "arrow-csv")
Expand Down
5 changes: 5 additions & 0 deletions cpp/src/arrow/csv/api.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -19,3 +19,8 @@

#include "arrow/csv/options.h"
#include "arrow/csv/reader.h"

// The writer depends on compute module for casting.
#ifdef ARROW_COMPUTE
#include "arrow/csv/writer.h"
#endif
1 change: 1 addition & 0 deletions cpp/src/arrow/csv/options.cc
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -34,6 +34,7 @@ ConvertOptions ConvertOptions::Defaults() {
}

ReadOptions ReadOptions::Defaults() { return ReadOptions(); }
WriteOptions WriteOptions::Defaults() { return WriteOptions(); }

} // namespace csv
} // namespace arrow
15 changes: 15 additions & 0 deletions cpp/src/arrow/csv/options.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -137,5 +137,20 @@ struct ARROW_EXPORT ReadOptions {
static ReadOptions Defaults();
};

/// Experimental
struct ARROW_EXPORT WriteOptions {
/// Whether to write an initial header line with column names
bool include_header = true;

/// \brief Maximum number of rows processed at a time
///
/// The CSV writer converts and writes data in batches of N rows.
/// This number can impact performance.
int32_t batch_size = 1024;

@pitroupitrouFeb 16, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intuitively, this seems a bit low, but we can tune it later.


/// Create write options with default values
static WriteOptions Defaults();
};

} // namespace csv
} // namespace arrow
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-2229: [C++][Python] Add WriteCsv functionality.#9504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
18
commits into

from

 Closed


ARROW-2229: [C++][Python] Add WriteCsv functionality.
#9504
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
7eaab50

ARROW-2229: [C++][Python] Add WriteCsv functionality.


emkornfield Feb 3, 2021
ecc508b

Update cpp/src/arrow/csv/writer.cc

Feb 16, 2021
ac7b3f4

fix lint


emkornfield Feb 16, 2021
8dcef6a

add missing test


emkornfield Feb 16, 2021
efffe81

lint and appveyor fixes


emkornfield Feb 16, 2021
7794307

start addressing feedback


emkornfield Feb 20, 2021
4deb8a8

add cmake configuration


emkornfield Feb 20, 2021
8b40ed7

update python


emkornfield Feb 21, 2021
61856a4

refactor c++ tests


emkornfield Feb 22, 2021
5c833a8

hopefully fix lint


emkornfield Feb 22, 2021
73826b8

hopefully make windows builds happy


emkornfield Feb 22, 2021
c5d76eb

more lint and windows


emkornfield Feb 22, 2021
2695891

address comments


emkornfield Feb 24, 2021
da03cd6

add back options


emkornfield Feb 24, 2021
2eb2251

Try uint8_t instead of bool.

Feb 24, 2021
23a2b0c

change back to bool and remove custom allocator

Feb 24, 2021
7e33776

remove from initialize list

Feb 25, 2021
a5ee563

Improve docstrings


pitrou Feb 25, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


3 changes: 3 additions & 0 deletions
cpp/src/arrow/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -349,6 +349,9 @@ if(ARROW_CSV)
 csv/options.cc
 csv/parser.cc
 csv/reader.cc)
 if(ARROW_COMPUTE)
 list(APPEND ARROW_SRCS csv/writer.cc)
 endif()

 list(APPEND ARROW_TESTING_SRCS csv/test_common.cc)
endif()
Expand Down

22 changes: 14 additions & 8 deletions
cpp/src/arrow/csv/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.

add_arrow_test(csv-test
 SOURCES
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)
set(CSV_TEST_SRCS
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)

# Writer depends on compute's cast functionality
if(ARROW_COMPUTE)
 list(APPEND CSV_TEST_SRCS writer_test.cc)
endif()

add_arrow_test(csv-test SOURCES ${CSV_TEST_SRCS})

add_arrow_benchmark(converter_benchmark PREFIX "arrow-csv")
add_arrow_benchmark(parser_benchmark PREFIX "arrow-csv")
Expand Down

5 changes: 5 additions & 0 deletions
cpp/src/arrow/csv/api.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,3 +19,8 @@

#include "arrow/csv/options.h"
#include "arrow/csv/reader.h"

// The writer depends on compute module for casting.
#ifdef ARROW_COMPUTE
#include "arrow/csv/writer.h"
#endif

1 change: 1 addition & 0 deletions
cpp/src/arrow/csv/options.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,6 +34,7 @@ ConvertOptions ConvertOptions::Defaults() {
}

ReadOptions ReadOptions::Defaults() { return ReadOptions(); }
WriteOptions WriteOptions::Defaults() { return WriteOptions(); }

} // namespace csv
} // namespace arrow

15 changes: 15 additions & 0 deletions
cpp/src/arrow/csv/options.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,5 +137,20 @@ struct ARROW_EXPORT ReadOptions {
 static ReadOptions Defaults();
};

/// Experimental
struct ARROW_EXPORT WriteOptions {
 /// Whether to write an initial header line with column names
 bool include_header = true;

 /// \brief Maximum number of rows processed at a time
 ///
 /// The CSV writer converts and writes data in batches of N rows.
 /// This number can impact performance.
 int32_t batch_size = 1024;
@pitroupitrouFeb 16, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intuitively, this seems a bit low, but we can tune it later.

 /// Create write options with default values
 static WriteOptions Defaults();
};

} // namespace csv
} // namespace arrow
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-2229: [C++][Python] Add WriteCsv functionality.#9504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
18
commits into

from

 Closed


ARROW-2229: [C++][Python] Add WriteCsv functionality.
#9504
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
7eaab50

ARROW-2229: [C++][Python] Add WriteCsv functionality.


emkornfield Feb 3, 2021
ecc508b

Update cpp/src/arrow/csv/writer.cc

Feb 16, 2021
ac7b3f4

fix lint


emkornfield Feb 16, 2021
8dcef6a

add missing test


emkornfield Feb 16, 2021
efffe81

lint and appveyor fixes


emkornfield Feb 16, 2021
7794307

start addressing feedback


emkornfield Feb 20, 2021
4deb8a8

add cmake configuration


emkornfield Feb 20, 2021
8b40ed7

update python


emkornfield Feb 21, 2021
61856a4

refactor c++ tests


emkornfield Feb 22, 2021
5c833a8

hopefully fix lint


emkornfield Feb 22, 2021
73826b8

hopefully make windows builds happy


emkornfield Feb 22, 2021
c5d76eb

more lint and windows


emkornfield Feb 22, 2021
2695891

address comments


emkornfield Feb 24, 2021
da03cd6

add back options


emkornfield Feb 24, 2021
2eb2251

Try uint8_t instead of bool.

Feb 24, 2021
23a2b0c

change back to bool and remove custom allocator

Feb 24, 2021
7e33776

remove from initialize list

Feb 25, 2021
a5ee563

Improve docstrings


pitrou Feb 25, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


3 changes: 3 additions & 0 deletions
cpp/src/arrow/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -349,6 +349,9 @@ if(ARROW_CSV)
 csv/options.cc
 csv/parser.cc
 csv/reader.cc)
 if(ARROW_COMPUTE)
 list(APPEND ARROW_SRCS csv/writer.cc)
 endif()

 list(APPEND ARROW_TESTING_SRCS csv/test_common.cc)
endif()
Expand Down

22 changes: 14 additions & 8 deletions
cpp/src/arrow/csv/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.

add_arrow_test(csv-test
 SOURCES
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)
set(CSV_TEST_SRCS
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)

# Writer depends on compute's cast functionality
if(ARROW_COMPUTE)
 list(APPEND CSV_TEST_SRCS writer_test.cc)
endif()

add_arrow_test(csv-test SOURCES ${CSV_TEST_SRCS})

add_arrow_benchmark(converter_benchmark PREFIX "arrow-csv")
add_arrow_benchmark(parser_benchmark PREFIX "arrow-csv")
Expand Down

5 changes: 5 additions & 0 deletions
cpp/src/arrow/csv/api.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,3 +19,8 @@

#include "arrow/csv/options.h"
#include "arrow/csv/reader.h"

// The writer depends on compute module for casting.
#ifdef ARROW_COMPUTE
#include "arrow/csv/writer.h"
#endif

1 change: 1 addition & 0 deletions
cpp/src/arrow/csv/options.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,6 +34,7 @@ ConvertOptions ConvertOptions::Defaults() {
}

ReadOptions ReadOptions::Defaults() { return ReadOptions(); }
WriteOptions WriteOptions::Defaults() { return WriteOptions(); }

} // namespace csv
} // namespace arrow

15 changes: 15 additions & 0 deletions
cpp/src/arrow/csv/options.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,5 +137,20 @@ struct ARROW_EXPORT ReadOptions {
 static ReadOptions Defaults();
};

/// Experimental
struct ARROW_EXPORT WriteOptions {
 /// Whether to write an initial header line with column names
 bool include_header = true;

 /// \brief Maximum number of rows processed at a time
 ///
 /// The CSV writer converts and writes data in batches of N rows.
 /// This number can impact performance.
 int32_t batch_size = 1024;
@pitroupitrouFeb 16, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intuitively, this seems a bit low, but we can tune it later.

 /// Create write options with default values
 static WriteOptions Defaults();
};

} // namespace csv
} // namespace arrow
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-2229: [C++][Python] Add WriteCsv functionality.#9504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
18
commits into

from

 Closed


ARROW-2229: [C++][Python] Add WriteCsv functionality.
#9504
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
7eaab50

ARROW-2229: [C++][Python] Add WriteCsv functionality.


emkornfield Feb 3, 2021
ecc508b

Update cpp/src/arrow/csv/writer.cc

Feb 16, 2021
ac7b3f4

fix lint


emkornfield Feb 16, 2021
8dcef6a

add missing test


emkornfield Feb 16, 2021
efffe81

lint and appveyor fixes


emkornfield Feb 16, 2021
7794307

start addressing feedback


emkornfield Feb 20, 2021
4deb8a8

add cmake configuration


emkornfield Feb 20, 2021
8b40ed7

update python


emkornfield Feb 21, 2021
61856a4

refactor c++ tests


emkornfield Feb 22, 2021
5c833a8

hopefully fix lint


emkornfield Feb 22, 2021
73826b8

hopefully make windows builds happy


emkornfield Feb 22, 2021
c5d76eb

more lint and windows


emkornfield Feb 22, 2021
2695891

address comments


emkornfield Feb 24, 2021
da03cd6

add back options


emkornfield Feb 24, 2021
2eb2251

Try uint8_t instead of bool.

Feb 24, 2021
23a2b0c

change back to bool and remove custom allocator

Feb 24, 2021
7e33776

remove from initialize list

Feb 25, 2021
a5ee563

Improve docstrings


pitrou Feb 25, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


3 changes: 3 additions & 0 deletions
cpp/src/arrow/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -349,6 +349,9 @@ if(ARROW_CSV)
 csv/options.cc
 csv/parser.cc
 csv/reader.cc)
 if(ARROW_COMPUTE)
 list(APPEND ARROW_SRCS csv/writer.cc)
 endif()

 list(APPEND ARROW_TESTING_SRCS csv/test_common.cc)
endif()
Expand Down

22 changes: 14 additions & 8 deletions
cpp/src/arrow/csv/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.

add_arrow_test(csv-test
 SOURCES
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)
set(CSV_TEST_SRCS
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)

# Writer depends on compute's cast functionality
if(ARROW_COMPUTE)
 list(APPEND CSV_TEST_SRCS writer_test.cc)
endif()

add_arrow_test(csv-test SOURCES ${CSV_TEST_SRCS})

add_arrow_benchmark(converter_benchmark PREFIX "arrow-csv")
add_arrow_benchmark(parser_benchmark PREFIX "arrow-csv")
Expand Down

5 changes: 5 additions & 0 deletions
cpp/src/arrow/csv/api.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,3 +19,8 @@

#include "arrow/csv/options.h"
#include "arrow/csv/reader.h"

// The writer depends on compute module for casting.
#ifdef ARROW_COMPUTE
#include "arrow/csv/writer.h"
#endif

1 change: 1 addition & 0 deletions
cpp/src/arrow/csv/options.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,6 +34,7 @@ ConvertOptions ConvertOptions::Defaults() {
}

ReadOptions ReadOptions::Defaults() { return ReadOptions(); }
WriteOptions WriteOptions::Defaults() { return WriteOptions(); }

} // namespace csv
} // namespace arrow

15 changes: 15 additions & 0 deletions
cpp/src/arrow/csv/options.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,5 +137,20 @@ struct ARROW_EXPORT ReadOptions {
 static ReadOptions Defaults();
};

/// Experimental
struct ARROW_EXPORT WriteOptions {
 /// Whether to write an initial header line with column names
 bool include_header = true;

 /// \brief Maximum number of rows processed at a time
 ///
 /// The CSV writer converts and writes data in batches of N rows.
 /// This number can impact performance.
 int32_t batch_size = 1024;
@pitroupitrouFeb 16, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intuitively, this seems a bit low, but we can tune it later.

 /// Create write options with default values
 static WriteOptions Defaults();
};

} // namespace csv
} // namespace arrow
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-2229: [C++][Python] Add WriteCsv functionality.#9504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
18
commits into

from

 Closed


ARROW-2229: [C++][Python] Add WriteCsv functionality.
#9504
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
7eaab50

ARROW-2229: [C++][Python] Add WriteCsv functionality.


emkornfield Feb 3, 2021
ecc508b

Update cpp/src/arrow/csv/writer.cc

Feb 16, 2021
ac7b3f4

fix lint


emkornfield Feb 16, 2021
8dcef6a

add missing test


emkornfield Feb 16, 2021
efffe81

lint and appveyor fixes


emkornfield Feb 16, 2021
7794307

start addressing feedback


emkornfield Feb 20, 2021
4deb8a8

add cmake configuration


emkornfield Feb 20, 2021
8b40ed7

update python


emkornfield Feb 21, 2021
61856a4

refactor c++ tests


emkornfield Feb 22, 2021
5c833a8

hopefully fix lint


emkornfield Feb 22, 2021
73826b8

hopefully make windows builds happy


emkornfield Feb 22, 2021
c5d76eb

more lint and windows


emkornfield Feb 22, 2021
2695891

address comments


emkornfield Feb 24, 2021
da03cd6

add back options


emkornfield Feb 24, 2021
2eb2251

Try uint8_t instead of bool.

Feb 24, 2021
23a2b0c

change back to bool and remove custom allocator

Feb 24, 2021
7e33776

remove from initialize list

Feb 25, 2021
a5ee563

Improve docstrings


pitrou Feb 25, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


3 changes: 3 additions & 0 deletions
cpp/src/arrow/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -349,6 +349,9 @@ if(ARROW_CSV)
 csv/options.cc
 csv/parser.cc
 csv/reader.cc)
 if(ARROW_COMPUTE)
 list(APPEND ARROW_SRCS csv/writer.cc)
 endif()

 list(APPEND ARROW_TESTING_SRCS csv/test_common.cc)
endif()
Expand Down

22 changes: 14 additions & 8 deletions
cpp/src/arrow/csv/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.

add_arrow_test(csv-test
 SOURCES
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)
set(CSV_TEST_SRCS
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)

# Writer depends on compute's cast functionality
if(ARROW_COMPUTE)
 list(APPEND CSV_TEST_SRCS writer_test.cc)
endif()

add_arrow_test(csv-test SOURCES ${CSV_TEST_SRCS})

add_arrow_benchmark(converter_benchmark PREFIX "arrow-csv")
add_arrow_benchmark(parser_benchmark PREFIX "arrow-csv")
Expand Down

5 changes: 5 additions & 0 deletions
cpp/src/arrow/csv/api.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,3 +19,8 @@

#include "arrow/csv/options.h"
#include "arrow/csv/reader.h"

// The writer depends on compute module for casting.
#ifdef ARROW_COMPUTE
#include "arrow/csv/writer.h"
#endif

1 change: 1 addition & 0 deletions
cpp/src/arrow/csv/options.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,6 +34,7 @@ ConvertOptions ConvertOptions::Defaults() {
}

ReadOptions ReadOptions::Defaults() { return ReadOptions(); }
WriteOptions WriteOptions::Defaults() { return WriteOptions(); }

} // namespace csv
} // namespace arrow

15 changes: 15 additions & 0 deletions
cpp/src/arrow/csv/options.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,5 +137,20 @@ struct ARROW_EXPORT ReadOptions {
 static ReadOptions Defaults();
};

/// Experimental
struct ARROW_EXPORT WriteOptions {
 /// Whether to write an initial header line with column names
 bool include_header = true;

 /// \brief Maximum number of rows processed at a time
 ///
 /// The CSV writer converts and writes data in batches of N rows.
 /// This number can impact performance.
 int32_t batch_size = 1024;
@pitroupitrouFeb 16, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intuitively, this seems a bit low, but we can tune it later.

 /// Create write options with default values
 static WriteOptions Defaults();
};

} // namespace csv
} // namespace arrow
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-2229: [C++][Python] Add WriteCsv functionality.#9504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
18
commits into

from

 Closed


ARROW-2229: [C++][Python] Add WriteCsv functionality.
#9504
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
7eaab50

ARROW-2229: [C++][Python] Add WriteCsv functionality.


emkornfield Feb 3, 2021
ecc508b

Update cpp/src/arrow/csv/writer.cc

Feb 16, 2021
ac7b3f4

fix lint


emkornfield Feb 16, 2021
8dcef6a

add missing test


emkornfield Feb 16, 2021
efffe81

lint and appveyor fixes


emkornfield Feb 16, 2021
7794307

start addressing feedback


emkornfield Feb 20, 2021
4deb8a8

add cmake configuration


emkornfield Feb 20, 2021
8b40ed7

update python


emkornfield Feb 21, 2021
61856a4

refactor c++ tests


emkornfield Feb 22, 2021
5c833a8

hopefully fix lint


emkornfield Feb 22, 2021
73826b8

hopefully make windows builds happy


emkornfield Feb 22, 2021
c5d76eb

more lint and windows


emkornfield Feb 22, 2021
2695891

address comments


emkornfield Feb 24, 2021
da03cd6

add back options


emkornfield Feb 24, 2021
2eb2251

Try uint8_t instead of bool.

Feb 24, 2021
23a2b0c

change back to bool and remove custom allocator

Feb 24, 2021
7e33776

remove from initialize list

Feb 25, 2021
a5ee563

Improve docstrings


pitrou Feb 25, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


3 changes: 3 additions & 0 deletions
cpp/src/arrow/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -349,6 +349,9 @@ if(ARROW_CSV)
 csv/options.cc
 csv/parser.cc
 csv/reader.cc)
 if(ARROW_COMPUTE)
 list(APPEND ARROW_SRCS csv/writer.cc)
 endif()

 list(APPEND ARROW_TESTING_SRCS csv/test_common.cc)
endif()
Expand Down

22 changes: 14 additions & 8 deletions
cpp/src/arrow/csv/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.

add_arrow_test(csv-test
 SOURCES
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)
set(CSV_TEST_SRCS
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)

# Writer depends on compute's cast functionality
if(ARROW_COMPUTE)
 list(APPEND CSV_TEST_SRCS writer_test.cc)
endif()

add_arrow_test(csv-test SOURCES ${CSV_TEST_SRCS})

add_arrow_benchmark(converter_benchmark PREFIX "arrow-csv")
add_arrow_benchmark(parser_benchmark PREFIX "arrow-csv")
Expand Down

5 changes: 5 additions & 0 deletions
cpp/src/arrow/csv/api.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,3 +19,8 @@

#include "arrow/csv/options.h"
#include "arrow/csv/reader.h"

// The writer depends on compute module for casting.
#ifdef ARROW_COMPUTE
#include "arrow/csv/writer.h"
#endif

1 change: 1 addition & 0 deletions
cpp/src/arrow/csv/options.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,6 +34,7 @@ ConvertOptions ConvertOptions::Defaults() {
}

ReadOptions ReadOptions::Defaults() { return ReadOptions(); }
WriteOptions WriteOptions::Defaults() { return WriteOptions(); }

} // namespace csv
} // namespace arrow

15 changes: 15 additions & 0 deletions
cpp/src/arrow/csv/options.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,5 +137,20 @@ struct ARROW_EXPORT ReadOptions {
 static ReadOptions Defaults();
};

/// Experimental
struct ARROW_EXPORT WriteOptions {
 /// Whether to write an initial header line with column names
 bool include_header = true;

 /// \brief Maximum number of rows processed at a time
 ///
 /// The CSV writer converts and writes data in batches of N rows.
 /// This number can impact performance.
 int32_t batch_size = 1024;
@pitroupitrouFeb 16, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intuitively, this seems a bit low, but we can tune it later.

 /// Create write options with default values
 static WriteOptions Defaults();
};

} // namespace csv
} // namespace arrow
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-2229: [C++][Python] Add WriteCsv functionality.#9504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
18
commits into

from

 Closed


ARROW-2229: [C++][Python] Add WriteCsv functionality.
#9504
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
7eaab50

ARROW-2229: [C++][Python] Add WriteCsv functionality.


emkornfield Feb 3, 2021
ecc508b

Update cpp/src/arrow/csv/writer.cc

Feb 16, 2021
ac7b3f4

fix lint


emkornfield Feb 16, 2021
8dcef6a

add missing test


emkornfield Feb 16, 2021
efffe81

lint and appveyor fixes


emkornfield Feb 16, 2021
7794307

start addressing feedback


emkornfield Feb 20, 2021
4deb8a8

add cmake configuration


emkornfield Feb 20, 2021
8b40ed7

update python


emkornfield Feb 21, 2021
61856a4

refactor c++ tests


emkornfield Feb 22, 2021
5c833a8

hopefully fix lint


emkornfield Feb 22, 2021
73826b8

hopefully make windows builds happy


emkornfield Feb 22, 2021
c5d76eb

more lint and windows


emkornfield Feb 22, 2021
2695891

address comments


emkornfield Feb 24, 2021
da03cd6

add back options


emkornfield Feb 24, 2021
2eb2251

Try uint8_t instead of bool.

Feb 24, 2021
23a2b0c

change back to bool and remove custom allocator

Feb 24, 2021
7e33776

remove from initialize list

Feb 25, 2021
a5ee563

Improve docstrings


pitrou Feb 25, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


3 changes: 3 additions & 0 deletions
cpp/src/arrow/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -349,6 +349,9 @@ if(ARROW_CSV)
 csv/options.cc
 csv/parser.cc
 csv/reader.cc)
 if(ARROW_COMPUTE)
 list(APPEND ARROW_SRCS csv/writer.cc)
 endif()

 list(APPEND ARROW_TESTING_SRCS csv/test_common.cc)
endif()
Expand Down

22 changes: 14 additions & 8 deletions
cpp/src/arrow/csv/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.

add_arrow_test(csv-test
 SOURCES
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)
set(CSV_TEST_SRCS
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)

# Writer depends on compute's cast functionality
if(ARROW_COMPUTE)
 list(APPEND CSV_TEST_SRCS writer_test.cc)
endif()

add_arrow_test(csv-test SOURCES ${CSV_TEST_SRCS})

add_arrow_benchmark(converter_benchmark PREFIX "arrow-csv")
add_arrow_benchmark(parser_benchmark PREFIX "arrow-csv")
Expand Down

5 changes: 5 additions & 0 deletions
cpp/src/arrow/csv/api.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,3 +19,8 @@

#include "arrow/csv/options.h"
#include "arrow/csv/reader.h"

// The writer depends on compute module for casting.
#ifdef ARROW_COMPUTE
#include "arrow/csv/writer.h"
#endif

1 change: 1 addition & 0 deletions
cpp/src/arrow/csv/options.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,6 +34,7 @@ ConvertOptions ConvertOptions::Defaults() {
}

ReadOptions ReadOptions::Defaults() { return ReadOptions(); }
WriteOptions WriteOptions::Defaults() { return WriteOptions(); }

} // namespace csv
} // namespace arrow

15 changes: 15 additions & 0 deletions
cpp/src/arrow/csv/options.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,5 +137,20 @@ struct ARROW_EXPORT ReadOptions {
 static ReadOptions Defaults();
};

/// Experimental
struct ARROW_EXPORT WriteOptions {
 /// Whether to write an initial header line with column names
 bool include_header = true;

 /// \brief Maximum number of rows processed at a time
 ///
 /// The CSV writer converts and writes data in batches of N rows.
 /// This number can impact performance.
 int32_t batch_size = 1024;
@pitroupitrouFeb 16, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intuitively, this seems a bit low, but we can tune it later.

 /// Create write options with default values
 static WriteOptions Defaults();
};

} // namespace csv
} // namespace arrow
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-2229: [C++][Python] Add WriteCsv functionality.#9504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Closed

emkornfield
wants to merge
18
commits into

from

 Closed


ARROW-2229: [C++][Python] Add WriteCsv functionality.
#9504
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


18 commits


Select commit
Hold shift + click to select a range
7eaab50

ARROW-2229: [C++][Python] Add WriteCsv functionality.


emkornfield Feb 3, 2021
ecc508b

Update cpp/src/arrow/csv/writer.cc

Feb 16, 2021
ac7b3f4

fix lint


emkornfield Feb 16, 2021
8dcef6a

add missing test


emkornfield Feb 16, 2021
efffe81

lint and appveyor fixes


emkornfield Feb 16, 2021
7794307

start addressing feedback


emkornfield Feb 20, 2021
4deb8a8

add cmake configuration


emkornfield Feb 20, 2021
8b40ed7

update python


emkornfield Feb 21, 2021
61856a4

refactor c++ tests


emkornfield Feb 22, 2021
5c833a8

hopefully fix lint


emkornfield Feb 22, 2021
73826b8

hopefully make windows builds happy


emkornfield Feb 22, 2021
c5d76eb

more lint and windows


emkornfield Feb 22, 2021
2695891

address comments


emkornfield Feb 24, 2021
da03cd6

add back options


emkornfield Feb 24, 2021
2eb2251

Try uint8_t instead of bool.

Feb 24, 2021
23a2b0c

change back to bool and remove custom allocator

Feb 24, 2021
7e33776

remove from initialize list

Feb 25, 2021
a5ee563

Improve docstrings


pitrou Feb 25, 2021

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


3 changes: 3 additions & 0 deletions
cpp/src/arrow/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -349,6 +349,9 @@ if(ARROW_CSV)
 csv/options.cc
 csv/parser.cc
 csv/reader.cc)
 if(ARROW_COMPUTE)
 list(APPEND ARROW_SRCS csv/writer.cc)
 endif()

 list(APPEND ARROW_TESTING_SRCS csv/test_common.cc)
endif()
Expand Down

22 changes: 14 additions & 8 deletions
cpp/src/arrow/csv/CMakeLists.txt

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -15,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.

add_arrow_test(csv-test
 SOURCES
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)
set(CSV_TEST_SRCS
 chunker_test.cc
 column_builder_test.cc
 column_decoder_test.cc
 converter_test.cc
 parser_test.cc
 reader_test.cc)

# Writer depends on compute's cast functionality
if(ARROW_COMPUTE)
 list(APPEND CSV_TEST_SRCS writer_test.cc)
endif()

add_arrow_test(csv-test SOURCES ${CSV_TEST_SRCS})

add_arrow_benchmark(converter_benchmark PREFIX "arrow-csv")
add_arrow_benchmark(parser_benchmark PREFIX "arrow-csv")
Expand Down

5 changes: 5 additions & 0 deletions
cpp/src/arrow/csv/api.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -19,3 +19,8 @@

#include "arrow/csv/options.h"
#include "arrow/csv/reader.h"

// The writer depends on compute module for casting.
#ifdef ARROW_COMPUTE
#include "arrow/csv/writer.h"
#endif

1 change: 1 addition & 0 deletions
cpp/src/arrow/csv/options.cc

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -34,6 +34,7 @@ ConvertOptions ConvertOptions::Defaults() {
}

ReadOptions ReadOptions::Defaults() { return ReadOptions(); }
WriteOptions WriteOptions::Defaults() { return WriteOptions(); }

} // namespace csv
} // namespace arrow

15 changes: 15 additions & 0 deletions
cpp/src/arrow/csv/options.h

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -137,5 +137,20 @@ struct ARROW_EXPORT ReadOptions {
 static ReadOptions Defaults();
};

/// Experimental
struct ARROW_EXPORT WriteOptions {
 /// Whether to write an initial header line with column names
 bool include_header = true;

 /// \brief Maximum number of rows processed at a time
 ///
 /// The CSV writer converts and writes data in batches of N rows.
 /// This number can impact performance.
 int32_t batch_size = 1024;
@pitroupitrouFeb 16, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intuitively, this seems a bit low, but we can tune it later.

 /// Create write options with default values
 static WriteOptions Defaults();
};

} // namespace csv
} // namespace arrow
Loading