Compressors not getting recycled while writing parquet files, causing memory leak

[asfimport]

Compressors are not getting recycled while writing parquet files. This is causing native/physical memory leak in my spark app which is parquet write intensive since its creating new compressors everytime i write parquet files.

The actual code issue is that we are creating 'codecFactory' in 'getRecordWriter' method of ParquetOutputFormat.java but not calling codecFactory.release() which is responsible for recycling compressors.

Reporter: Nitin Goyal / @nitin2goyal
Assignee: Nitin Goyal / @nitin2goyal

Related issues:

• Release Parquet-mr 1.9.0 (blocks)

PRs and other links:

• PR #282
• PR #295

_{Note: This issue was originally created as PARQUET-353. Please see the migration documentation for further details.}

[asfimport]

Nitin Goyal / @nitin2goyal:
PR - https://github.com/Parquet/parquet-mr/pull/445

[asfimport]

Ryan Blue / @rdblue:
Merged #295, which has a small change to Nitin's original fix. Thanks for contributing this fix, @nitin2goyal!