remove/delete method for zipfile objects

[rossmclendon]

BPO	6818
Nosy	@loewis, @rhettinger, @terryjreedy, @gustaebel, @tiran, @merwok, @sandrotosi, @serhiy-storchaka, @denfromufa, @yudilevi2
PRs	bpo-6818: Add remove() in ZipInfo #19336 gh-51067: Add remove() in ZipInfo #19358
Files	delete.tar.gz: Includes 2 files,the one named delete.py is the main file. zipfile.remove.patch: bugs, docs and tests zipfile.remove.2.patch: improved patch for zipfile.remove tricky.zip mywork.patch zipfile_filter.patch: renamed, fixed comment, removed some debugging code

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2009-09-02.04:42:10.018>
labels = ['type-feature', 'library']
title = 'remove/delete method for zipfile/tarfile objects'
updated_at = <Date 2020-04-04.10:40:28.415>
user = 'https://bugs.python.org/rossmclendon'

bugs.python.org fields:

activity = <Date 2020-04-04.10:40:28.415>
actor = 'yudilevi'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2009-09-02.04:42:10.018>
creator = 'rossmclendon'
dependencies = []
files = ['15199', '21063', '21188', '38231', '39061', '39063']
hgrepos = ['305']
issue_num = 6818
keywords = ['patch']
message_count = 36.0
messages = ['92154', '92155', '92156', '92158', '92164', '92172', '92177', '94475', '94478', '109158', '109160', '130299', '130388', '130463', '130938', '140680', '159418', '160533', '160534', '192574', '229801', '229893', '236565', '236568', '240336', '240345', '240394', '240422', '241181', '241192', '241201', '241346', '266374', '365755', '365756', '365757']
nosy_count = 17.0
nosy_names = ['loewis', 'rhettinger', 'terry.reedy', 'lars.gustaebel', 'christian.heimes', 'rossmclendon', 'eric.araujo', 'ubershmekel', 'victorlee129', 'sandro.tosi', 'chroipahtz', 'serhiy.storchaka', 'Arthur.Darcet', 'Dave Sawyer', 'gambl', 'denfromufa', 'yudilevi']
pr_nums = ['19336', '19358']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6818'
versions = ['Python 3.5']

Linked PRs

• gh-51067: add ZipFile.remove() #103033
• gh-51067: Add remove() and repack() to ZipFile #134627
• gh-51067: Rephrase doc for ZipFile.{remove,repack} #151932
• gh-51067: Remove trailing space in _Repacker.repack() error message #152138
• gh-51067: Extract _read_local_file_header() as a module-level function #152139
• gh-152190: Fix memory checking failure in test_zipfile64.py #152203

[rossmclendon]

It would be most helpful if a method could be included in the TarFile
class of the tarfile module and the ZipFile class of the zipfile module
that would remove a particular file (either given by a name or a
TarInfo/ZipInfo object) from the archive.

Usage to remove a single file from an archive would be as follows:

import zipfile
zipFileObject = zipfile.ZipFile(archiveName,'a')
zipFileObject.remove(fileToRemove)
zipFileObject.close()

Such a method should probably only apply to archives that are in append
mode as write mode would erase the original archive and read mode should
render the archive immutable.

One possible extra to be included is to allow a list of file names or
ZipInfo/TarInfo objects to be passed into the remove method, such that
all items in the list would be removed from the archive.

[rhettinger]

+1

[rossmclendon]

Slight change to:

"Such a method should probably only apply to archives that are in append
mode as write mode would erase the original archive and read mode should
render the archive immutable."

The method should probably still apply to an archive in write mode. It
is conceivable that one may need to delete a file from the archive after
it has been written but before the archive object has been closed.

[loewis]

-1. I don't think this can be implemented in a reasonable way, assuming
that you want the file to become smaller as a consequence of removal.

[gustaebel]

-1, although I can only speak for tarfile. Removing members from a tar
archive sounds obvious and easy but it is *not*. A file in an archive is
stored as a header block (that contains the metadata) followed by a
number of data blocks (that contain the file's data). New files are
simply appended to the archive file. There is no central table of
contents whatsoever. To make things worse, a compressed archive is
compressed in one go from the beginning right up to the end, it is not
possible to access a member in the middle of an archive without having
to decompress all data before it.
Deleting files from an uncompressed archive is rather straightforward
implementation-wise but IO intensive and risky. In contrast, there is no
other way to delete files from a *compressed* tarfile than to make a
copy of it omitting the unwanted files.

[rossmclendon]

In light of Lars's comment on the matter, perhaps this functionality
could be added to zip files only. Surely it can be done, considering
that numerous utilities and even Windows Explorer provide such
functionality. I must confess that I am unfamiliar with the inner
workings of file archives and compression, but seeing as it is
implemented in a number of places already, it seems logical that it
could be implemented in ZipFile as well. I'll spend some time the next
few days educating myself about zip files and how this might be
accomplished.

[loewis]

In light of Lars's comment on the matter, perhaps this functionality
could be added to zip files only. Surely it can be done, considering
that numerous utilities and even Windows Explorer provide such
functionality.

Are you sure they are not creating a new file in order to delete
content? I recall that early zip tools (e.g. pkzip) had a mode
where they would merely delete the entry from the directory, but
leave the actual data in the file. Would you consider that a correct
implementation? If so, *that* can be done, for zipfiles, AFAIU.

I'll spend some time the next
few days educating myself about zip files and how this might be
accomplished.

Please do - you'll find that deletion from zipfiles comes in a can
full of worms.

[victorlee129]

I done it In a very *violent* way.
Is it ok for you thought?
if so, would anybody please fix it into the lib?

[loewis]

I done it In a very *violent* way.
Is it ok for you thought?

In the form in which you have done it, it is clearly
unacceptable for inclusion in the library: we don't
want to add two modules "delete" and "classtools".

In addition, notice that code is for tarfile, whereas
the OP was asking for a similar feature for zipfile.

if so, would anybody please fix it into the lib?

This is not how this works. If you want us to take
action, please submit a complete and correct patch.

[chroipahtz]

I have attempted to implement a ZipFile.remove function. It seems to work fine. I have submitted a patch.

The method of implementation is: find the file's index in the file list, then sum the lengths of the file entries before it to find its location in the archive. Then simply read in all the bytes after it, write them out at that location, and truncate the file x bytes shorter, where x is the length of the record. This works because the directory listing is created when the file is closed, so there's no harm in truncating.

I've also made it truncate the zip file after reading in the existing files upon creation, because the directory information is not used after this point.

This could use some testing on large files.

This is my first patch, so let me know if I've done anything wrong.

[chroipahtz]

My patch had some bugs, I'll need to do some more testing. Sorry about that.