Uh oh!
There was an error while loading. Please reload this page.
Feat/zip features - #139
Conversation
| try { | ||
| // Create a working directory for zip/unzip features | ||
| fs::path wp(m_workingPath); | ||
| if (!fs::exists(wp)) { |
There was a problem hiding this comment.
Use temp_directory_pathhttps://en.cppreference.com/cpp/filesystem/temp_directory_path ?
This way you are guaranteed to get an existing directory, and it might enventually be removed when the machine restarts, if for some reason you were not able to delete it sooner.
Here, if the directory does exists, you might use it, and potentially overwrite user data, since it's not handled as an error.
There was a problem hiding this comment.
Now that I think of it, concerning the two PR comment this seems to address: you might want to split the RAII part you did with the dtor of ZipBuffer, from the utility methods.
- Make ZipBufferUtils (or whatever :)) have these 2 methods of extraction/compression as static helpers
ZipBufferUtils::extract()
ZipBufferUtils::create/compress/...()- create a RAII object responsible to create the tmp dir and deleting it on destruction, something like:
classScopedTmpDir {
ScopedTmpDir () { m_path = temp_directory_path();}
~ScopedTmpDir { fs::remove_all(m_path ); }
fs::path getPath();
};And use it in the static helper function.
This adds the benefit of deleting things ASAP, instead of waiting that the ZipBuffer object goes out of scope, which may be long after the end of the function (which I gues is why you kept deleting things at the end of the methods, instead of relying on the destructor, as I commented elsewhere).
What do you think of this approach ?
There was a problem hiding this comment.
I like this approach :-) Made the modifications.
| try { | ||
| // Check working directory | ||
| fs::path wp(m_workingPath); | ||
| if (!fs::is_directory(wp)) { |
There was a problem hiding this comment.
If temp_directory_path this check is probably not needed (even in the current version). Unless we fear something might happen to this directory between the creation of ZipBuffer and the call to this.
There was a problem hiding this comment.
Removed because temp_directory_path is now used.
| fs::path op(originalPath); | ||
| if (!fs::is_directory(op)) { | ||
| LOG_ERROR("The original path is not a directory: {}", originalPath); | ||
| return FrameworkReturnCode::_NOT_FOUND; |
There was a problem hiding this comment.
Maybe !exists => NOT_FOUND but !directory => ERROR (if we don't have BAD_ARGUMENT) ?
| } | ||
| if (fs::is_empty(op)) { | ||
| LOG_ERROR("The original path is empty: {}", originalPath); | ||
| return FrameworkReturnCode::_NOT_FOUND; |
There was a problem hiding this comment.
SUCCESS? You give an empty dir, you get an empty vector, job done ?
There was a problem hiding this comment.
OK returns SUCCESS but with a warning (and clear the input buffer)
| // Copy data to zip in the working directory | ||
| const auto copyOptions = fs::copy_options::recursive; | ||
| fs::copy(op, wp, copyOptions); |
There was a problem hiding this comment.
Can't we avoid the copy ? For example by invoking zip directly from originalPath ?
std::string command = "cd " + originalPath + ";zip -r " + m_workingPath+ "/data.zip .";
There was a problem hiding this comment.
No, I prefer not to work on the original directory, to ensure I don't modify it.
There was a problem hiding this comment.
Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.
It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.
OK we'll see if it becomes a bottleneck.
There was a problem hiding this comment.
OK, I see the point. I will try to change this behavior.
| fs::remove(zipFile); | ||
| // Copy unzipped data in the destination directory | ||
| const auto copyOptions = fs::copy_options::recursive; |
There was a problem hiding this comment.
Could we avoid the copy by invoking zip from the destination directory?
std::string command = "cd " + destinationPath+ "; unzip " + m_workingPath + " /data.zip";
| // private | ||
| void ZipBuffer::cleanWorkingDirectory() |
There was a problem hiding this comment.
No longer useful at all: removed
| /// * FrameworkReturnCode::_SUCCESS if the process succeeds | ||
| /// * FrameworkReturnCode::_NOT_FOUND if data is not found in original path | ||
| /// * else FrameworkReturnCode::_ERROR_ | ||
| FrameworkReturnCode zipToBuffer(const std::string & originalPath, |
There was a problem hiding this comment.
nit: use directly fs::path i.o. string for path ?
There was a problem hiding this comment.
I thought using std::string avoided forcing the caller to use std::filesystem?
| /// @return | ||
| /// * FrameworkReturnCode::_SUCCESS if the process succeeds | ||
| /// * else FrameworkReturnCode::_ERROR_ | ||
| FrameworkReturnCode bufferToUnzip(const std::vector<unsigned char> & compressedZipBuffer, |
There was a problem hiding this comment.
nit: matter of taste, but with verbs for method ? ZipBuffer::extract()/ZipBuffer::compress()? Or something else.
| * | ||
| */ | ||
| class ZipBuffer { |
There was a problem hiding this comment.
Again naming things is hard... It does not per say represent a ZipBuffer, but more like utils functions... But I'm not sure about ZipBufferUtils... I don't have a better idea :). Something like ZipBufferExtractor (but it does also compress, so ...), or ZipBufferProcessor,...
There was a problem hiding this comment.
What about just ZipUtils ?
There was a problem hiding this comment.
Let's go for ZipBufferUtils class and extractand compressmethods
| #include <vector> | ||
| #include <filesystem> | ||
| namespace fs = std::filesystem; |
There was a problem hiding this comment.
Remove, avoid aliasing in header, especially at this scope.
| * @brief <B>Create a temporary working directory</B> | ||
| * | ||
| */ | ||
| class ScopedWorkingDir { |
There was a problem hiding this comment.
This class is decorrelated to ZipBufferUtils, so 2 suggestions:
- define in its own file for anyboady to reuse
- define it in the cpp file, because it's not referenced here, so it does not have to be defined here.
It's only an implementation detail that does not have to surface here (except if this class is needed elsewhere, but then this would have to be defined in its own file, as suggested in 1.
There was a problem hiding this comment.
Also maybe keep Tmp or Temp in the name (ScopedTmpDir), so that we know it uses the tmp folder API, and this is where the file is created ...?
| class ScopedWorkingDir { | ||
| public: | ||
| ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; } | ||
| ~ScopedWorkingDir() { fs::remove_all(m_workingPath); } |
There was a problem hiding this comment.
Delete copy operations to prevent double deletion
ScopedWorkingDir(const ScopedWorkingDir&) = delete;
ScopedWorkingDir& operator=(const ScopedWorkingDir&) = delete;
ScopedWorkingDir(ScopedWorkingDir&&) = delete;
ScopedWorkingDir& operator=(ScopedWorkingDir&&) = delete;(move operations are already deleted by the fact of defining the destructor, but this way it's explicit)
| */ | ||
| class ScopedWorkingDir { | ||
| public: | ||
| ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; } |
There was a problem hiding this comment.
Make the subdir a (optional?) parameter of the (explicit) ctor?
explicitScopedWorkingDir(const std::string& subDir = "");There was a problem hiding this comment.
This class has moved to the .cpp file, and will only be used by ZipBufferUtils static methods, so I don't think it's usefull to make the sudir configurable
| */ | ||
| class ScopedWorkingDir { | ||
| public: | ||
| ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; } |
There was a problem hiding this comment.
nit: use operator / for path separator, example of syntax:
m_workingPath = m_workingPath / "solar";
m_workingPath /= "solar";There was a problem hiding this comment.
Both operations are available : why this one instead of the other one?
| // Copy data to zip in the working directory | ||
| const auto copyOptions = fs::copy_options::recursive; | ||
| fs::copy(op, wp, copyOptions); |
There was a problem hiding this comment.
Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.
It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.
OK we'll see if it becomes a bottleneck.
| LOG_DEBUG("ZipBufferUtils::extract - Working temporary path: {}", workingDir.getStringPath()); | ||
| // Check/create the working directory | ||
| if (!fs::exists(workingDir.getPath())) { |
There was a problem hiding this comment.
Yes, still needed here: the subdir must be created here before creating the zip file (tested)
| } | ||
| try { | ||
| fs::path dp(destinationPath); |
There was a problem hiding this comment.
In the other function you clear the output buffer, here we could delete the content of the destination path as well ?
There was a problem hiding this comment.
I don't think so: this destination directory can contain other files, we don't know.
| return FrameworkReturnCode::_ERROR_; | ||
| } | ||
| // Delete the zip file |
There was a problem hiding this comment.
Ah yes you do it because the zip is in the same dir as the one you copy later on, OK.
| file.close(); | ||
| // Try to unzip the file content | ||
| std::string command = "cd " + workingDir.getStringPath() + "; unzip data.zip"; |
There was a problem hiding this comment.
Here you could avoid the copy by cd in the dest dir, there's really no risk of messing with existing files, since it's supposed to receive ours, no?
There was a problem hiding this comment.
Yes, I think you're right and I will change it
No description provided.