Feat/zip features - #139

Open
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features
Open

Feat/zip features#139
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features

Conversation

@cutullic

Copy link
Copy Markdown
Contributor

No description provided.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Create a working directory for zip/unzip features
fs::path wp(m_workingPath);
if (!fs::exists(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use temp_directory_pathhttps://en.cppreference.com/cpp/filesystem/temp_directory_path ?

This way you are guaranteed to get an existing directory, and it might enventually be removed when the machine restarts, if for some reason you were not able to delete it sooner.

Here, if the directory does exists, you might use it, and potentially overwrite user data, since it's not handled as an error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I think of it, concerning the two PR comment this seems to address: you might want to split the RAII part you did with the dtor of ZipBuffer, from the utility methods.

  • Make ZipBufferUtils (or whatever :)) have these 2 methods of extraction/compression as static helpers
ZipBufferUtils::extract()
ZipBufferUtils::create/compress/...()
  • create a RAII object responsible to create the tmp dir and deleting it on destruction, something like:
classScopedTmpDir {
ScopedTmpDir () { m_path = temp_directory_path();}
~ScopedTmpDir { fs::remove_all(m_path ); }
fs::path getPath();
};

And use it in the static helper function.

This adds the benefit of deleting things ASAP, instead of waiting that the ZipBuffer object goes out of scope, which may be long after the end of the function (which I gues is why you kept deleting things at the end of the methods, instead of relying on the destructor, as I commented elsewhere).

What do you think of this approach ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach :-) Made the modifications.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Check working directory
fs::path wp(m_workingPath);
if (!fs::is_directory(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If temp_directory_path this check is probably not needed (even in the current version). Unless we fear something might happen to this directory between the creation of ZipBuffer and the call to this.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because temp_directory_path is now used.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::path op(originalPath);
if (!fs::is_directory(op)) {
LOG_ERROR("The original path is not a directory: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe !exists => NOT_FOUND but !directory => ERROR (if we don't have BAD_ARGUMENT) ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment threadsrc/core/ZipBuffer.cpp Outdated
}
if (fs::is_empty(op)) {
LOG_ERROR("The original path is empty: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUCCESS? You give an empty dir, you get an empty vector, job done ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK returns SUCCESS but with a warning (and clear the input buffer)

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we avoid the copy ? For example by invoking zip directly from originalPath ?

std::string command = "cd " + originalPath + ";zip -r " + m_workingPath+ "/data.zip .";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I prefer not to work on the original directory, to ensure I don't modify it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see the point. I will try to change this behavior.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::remove(zipFile);

// Copy unzipped data in the destination directory
const auto copyOptions = fs::copy_options::recursive;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid the copy by invoking zip from the destination directory?

std::string command = "cd " + destinationPath+ "; unzip " + m_workingPath + " /data.zip";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

Comment threadsrc/core/ZipBuffer.cpp Outdated

// private

void ZipBuffer::cleanWorkingDirectory()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer useful at all: removed

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * FrameworkReturnCode::_NOT_FOUND if data is not found in original path
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode zipToBuffer(const std::string & originalPath,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use directly fs::path i.o. string for path ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought using std::string avoided forcing the caller to use std::filesystem?

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// @return
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode bufferToUnzip(const std::vector<unsigned char> & compressedZipBuffer,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: matter of taste, but with verbs for method ? ZipBuffer::extract()/ZipBuffer::compress()? Or something else.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK changed

Comment threadinterfaces/core/ZipBuffer.h Outdated
*
*/

class ZipBuffer {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again naming things is hard... It does not per say represent a ZipBuffer, but more like utils functions... But I'm not sure about ZipBufferUtils... I don't have a better idea :). Something like ZipBufferExtractor (but it does also compress, so ...), or ZipBufferProcessor,...

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just ZipUtils ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go for ZipBufferUtils class and extractand compressmethods

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
#include <vector>
#include <filesystem>

namespace fs = std::filesystem;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove, avoid aliasing in header, especially at this scope.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sorry!

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
* @brief <B>Create a temporary working directory</B>
*
*/
class ScopedWorkingDir {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is decorrelated to ZipBufferUtils, so 2 suggestions:

  1. define in its own file for anyboady to reuse
  2. define it in the cpp file, because it's not referenced here, so it does not have to be defined here.

It's only an implementation detail that does not have to surface here (except if this class is needed elsewhere, but then this would have to be defined in its own file, as suggested in 1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also maybe keep Tmp or Temp in the name (ScopedTmpDir), so that we know it uses the tmp folder API, and this is where the file is created ...?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }
~ScopedWorkingDir() { fs::remove_all(m_workingPath); }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete copy operations to prevent double deletion

ScopedWorkingDir(const ScopedWorkingDir&) = delete;
ScopedWorkingDir& operator=(const ScopedWorkingDir&) = delete;
ScopedWorkingDir(ScopedWorkingDir&&) = delete;
ScopedWorkingDir& operator=(ScopedWorkingDir&&) = delete;

(move operations are already deleted by the fact of defining the destructor, but this way it's explicit)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the subdir a (optional?) parameter of the (explicit) ctor?

explicitScopedWorkingDir(const std::string& subDir = "");

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has moved to the .cpp file, and will only be used by ZipBufferUtils static methods, so I don't think it's usefull to make the sudir configurable

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use operator / for path separator, example of syntax:

m_workingPath = m_workingPath / "solar";
m_workingPath /= "solar";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both operations are available : why this one instead of the other one?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, changed

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

LOG_DEBUG("ZipBufferUtils::extract - Working temporary path: {}", workingDir.getStringPath());

// Check/create the working directory
if (!fs::exists(workingDir.getPath())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, still needed here: the subdir must be created here before creating the zip file (tested)

}

try {
fs::path dp(destinationPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other function you clear the output buffer, here we could delete the content of the destination path as well ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so: this destination directory can contain other files, we don't know.

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
return FrameworkReturnCode::_ERROR_;
}

// Delete the zip file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes you do it because the zip is in the same dir as the one you copy later on, OK.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
file.close();

// Try to unzip the file content
std::string command = "cd " + workingDir.getStringPath() + "; unzip data.zip";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you could avoid the copy by cd in the dest dir, there's really no risk of messing with existing files, since it's supposed to receive ours, no?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think you're right and I will change it

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@cutullic@jim-bcom
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Feat/zip features - #139

Open
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features
Open

Feat/zip features#139
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features

Conversation

@cutullic

Copy link
Copy Markdown
Contributor

No description provided.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Create a working directory for zip/unzip features
fs::path wp(m_workingPath);
if (!fs::exists(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use temp_directory_pathhttps://en.cppreference.com/cpp/filesystem/temp_directory_path ?

This way you are guaranteed to get an existing directory, and it might enventually be removed when the machine restarts, if for some reason you were not able to delete it sooner.

Here, if the directory does exists, you might use it, and potentially overwrite user data, since it's not handled as an error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I think of it, concerning the two PR comment this seems to address: you might want to split the RAII part you did with the dtor of ZipBuffer, from the utility methods.

  • Make ZipBufferUtils (or whatever :)) have these 2 methods of extraction/compression as static helpers
ZipBufferUtils::extract()
ZipBufferUtils::create/compress/...()
  • create a RAII object responsible to create the tmp dir and deleting it on destruction, something like:
classScopedTmpDir {
ScopedTmpDir () { m_path = temp_directory_path();}
~ScopedTmpDir { fs::remove_all(m_path ); }
fs::path getPath();
};

And use it in the static helper function.

This adds the benefit of deleting things ASAP, instead of waiting that the ZipBuffer object goes out of scope, which may be long after the end of the function (which I gues is why you kept deleting things at the end of the methods, instead of relying on the destructor, as I commented elsewhere).

What do you think of this approach ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach :-) Made the modifications.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Check working directory
fs::path wp(m_workingPath);
if (!fs::is_directory(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If temp_directory_path this check is probably not needed (even in the current version). Unless we fear something might happen to this directory between the creation of ZipBuffer and the call to this.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because temp_directory_path is now used.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::path op(originalPath);
if (!fs::is_directory(op)) {
LOG_ERROR("The original path is not a directory: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe !exists => NOT_FOUND but !directory => ERROR (if we don't have BAD_ARGUMENT) ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment threadsrc/core/ZipBuffer.cpp Outdated
}
if (fs::is_empty(op)) {
LOG_ERROR("The original path is empty: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUCCESS? You give an empty dir, you get an empty vector, job done ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK returns SUCCESS but with a warning (and clear the input buffer)

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we avoid the copy ? For example by invoking zip directly from originalPath ?

std::string command = "cd " + originalPath + ";zip -r " + m_workingPath+ "/data.zip .";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I prefer not to work on the original directory, to ensure I don't modify it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see the point. I will try to change this behavior.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::remove(zipFile);

// Copy unzipped data in the destination directory
const auto copyOptions = fs::copy_options::recursive;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid the copy by invoking zip from the destination directory?

std::string command = "cd " + destinationPath+ "; unzip " + m_workingPath + " /data.zip";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

Comment threadsrc/core/ZipBuffer.cpp Outdated

// private

void ZipBuffer::cleanWorkingDirectory()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer useful at all: removed

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * FrameworkReturnCode::_NOT_FOUND if data is not found in original path
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode zipToBuffer(const std::string & originalPath,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use directly fs::path i.o. string for path ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought using std::string avoided forcing the caller to use std::filesystem?

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// @return
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode bufferToUnzip(const std::vector<unsigned char> & compressedZipBuffer,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: matter of taste, but with verbs for method ? ZipBuffer::extract()/ZipBuffer::compress()? Or something else.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK changed

Comment threadinterfaces/core/ZipBuffer.h Outdated
*
*/

class ZipBuffer {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again naming things is hard... It does not per say represent a ZipBuffer, but more like utils functions... But I'm not sure about ZipBufferUtils... I don't have a better idea :). Something like ZipBufferExtractor (but it does also compress, so ...), or ZipBufferProcessor,...

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just ZipUtils ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go for ZipBufferUtils class and extractand compressmethods

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
#include <vector>
#include <filesystem>

namespace fs = std::filesystem;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove, avoid aliasing in header, especially at this scope.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sorry!

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
* @brief <B>Create a temporary working directory</B>
*
*/
class ScopedWorkingDir {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is decorrelated to ZipBufferUtils, so 2 suggestions:

  1. define in its own file for anyboady to reuse
  2. define it in the cpp file, because it's not referenced here, so it does not have to be defined here.

It's only an implementation detail that does not have to surface here (except if this class is needed elsewhere, but then this would have to be defined in its own file, as suggested in 1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also maybe keep Tmp or Temp in the name (ScopedTmpDir), so that we know it uses the tmp folder API, and this is where the file is created ...?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }
~ScopedWorkingDir() { fs::remove_all(m_workingPath); }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete copy operations to prevent double deletion

ScopedWorkingDir(const ScopedWorkingDir&) = delete;
ScopedWorkingDir& operator=(const ScopedWorkingDir&) = delete;
ScopedWorkingDir(ScopedWorkingDir&&) = delete;
ScopedWorkingDir& operator=(ScopedWorkingDir&&) = delete;

(move operations are already deleted by the fact of defining the destructor, but this way it's explicit)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the subdir a (optional?) parameter of the (explicit) ctor?

explicitScopedWorkingDir(const std::string& subDir = "");

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has moved to the .cpp file, and will only be used by ZipBufferUtils static methods, so I don't think it's usefull to make the sudir configurable

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use operator / for path separator, example of syntax:

m_workingPath = m_workingPath / "solar";
m_workingPath /= "solar";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both operations are available : why this one instead of the other one?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, changed

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

LOG_DEBUG("ZipBufferUtils::extract - Working temporary path: {}", workingDir.getStringPath());

// Check/create the working directory
if (!fs::exists(workingDir.getPath())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, still needed here: the subdir must be created here before creating the zip file (tested)

}

try {
fs::path dp(destinationPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other function you clear the output buffer, here we could delete the content of the destination path as well ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so: this destination directory can contain other files, we don't know.

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
return FrameworkReturnCode::_ERROR_;
}

// Delete the zip file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes you do it because the zip is in the same dir as the one you copy later on, OK.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
file.close();

// Try to unzip the file content
std::string command = "cd " + workingDir.getStringPath() + "; unzip data.zip";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you could avoid the copy by cd in the dest dir, there's really no risk of messing with existing files, since it's supposed to receive ours, no?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think you're right and I will change it

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@cutullic@jim-bcom
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feat/zip features - #139

Open
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features
Open

Feat/zip features#139
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features

Conversation

@cutullic

Copy link
Copy Markdown
Contributor

No description provided.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Create a working directory for zip/unzip features
fs::path wp(m_workingPath);
if (!fs::exists(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use temp_directory_pathhttps://en.cppreference.com/cpp/filesystem/temp_directory_path ?

This way you are guaranteed to get an existing directory, and it might enventually be removed when the machine restarts, if for some reason you were not able to delete it sooner.

Here, if the directory does exists, you might use it, and potentially overwrite user data, since it's not handled as an error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I think of it, concerning the two PR comment this seems to address: you might want to split the RAII part you did with the dtor of ZipBuffer, from the utility methods.

  • Make ZipBufferUtils (or whatever :)) have these 2 methods of extraction/compression as static helpers
ZipBufferUtils::extract()
ZipBufferUtils::create/compress/...()
  • create a RAII object responsible to create the tmp dir and deleting it on destruction, something like:
classScopedTmpDir {
ScopedTmpDir () { m_path = temp_directory_path();}
~ScopedTmpDir { fs::remove_all(m_path ); }
fs::path getPath();
};

And use it in the static helper function.

This adds the benefit of deleting things ASAP, instead of waiting that the ZipBuffer object goes out of scope, which may be long after the end of the function (which I gues is why you kept deleting things at the end of the methods, instead of relying on the destructor, as I commented elsewhere).

What do you think of this approach ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach :-) Made the modifications.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Check working directory
fs::path wp(m_workingPath);
if (!fs::is_directory(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If temp_directory_path this check is probably not needed (even in the current version). Unless we fear something might happen to this directory between the creation of ZipBuffer and the call to this.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because temp_directory_path is now used.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::path op(originalPath);
if (!fs::is_directory(op)) {
LOG_ERROR("The original path is not a directory: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe !exists => NOT_FOUND but !directory => ERROR (if we don't have BAD_ARGUMENT) ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment threadsrc/core/ZipBuffer.cpp Outdated
}
if (fs::is_empty(op)) {
LOG_ERROR("The original path is empty: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUCCESS? You give an empty dir, you get an empty vector, job done ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK returns SUCCESS but with a warning (and clear the input buffer)

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we avoid the copy ? For example by invoking zip directly from originalPath ?

std::string command = "cd " + originalPath + ";zip -r " + m_workingPath+ "/data.zip .";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I prefer not to work on the original directory, to ensure I don't modify it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see the point. I will try to change this behavior.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::remove(zipFile);

// Copy unzipped data in the destination directory
const auto copyOptions = fs::copy_options::recursive;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid the copy by invoking zip from the destination directory?

std::string command = "cd " + destinationPath+ "; unzip " + m_workingPath + " /data.zip";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

Comment threadsrc/core/ZipBuffer.cpp Outdated

// private

void ZipBuffer::cleanWorkingDirectory()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer useful at all: removed

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * FrameworkReturnCode::_NOT_FOUND if data is not found in original path
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode zipToBuffer(const std::string & originalPath,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use directly fs::path i.o. string for path ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought using std::string avoided forcing the caller to use std::filesystem?

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// @return
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode bufferToUnzip(const std::vector<unsigned char> & compressedZipBuffer,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: matter of taste, but with verbs for method ? ZipBuffer::extract()/ZipBuffer::compress()? Or something else.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK changed

Comment threadinterfaces/core/ZipBuffer.h Outdated
*
*/

class ZipBuffer {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again naming things is hard... It does not per say represent a ZipBuffer, but more like utils functions... But I'm not sure about ZipBufferUtils... I don't have a better idea :). Something like ZipBufferExtractor (but it does also compress, so ...), or ZipBufferProcessor,...

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just ZipUtils ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go for ZipBufferUtils class and extractand compressmethods

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
#include <vector>
#include <filesystem>

namespace fs = std::filesystem;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove, avoid aliasing in header, especially at this scope.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sorry!

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
* @brief <B>Create a temporary working directory</B>
*
*/
class ScopedWorkingDir {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is decorrelated to ZipBufferUtils, so 2 suggestions:

  1. define in its own file for anyboady to reuse
  2. define it in the cpp file, because it's not referenced here, so it does not have to be defined here.

It's only an implementation detail that does not have to surface here (except if this class is needed elsewhere, but then this would have to be defined in its own file, as suggested in 1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also maybe keep Tmp or Temp in the name (ScopedTmpDir), so that we know it uses the tmp folder API, and this is where the file is created ...?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }
~ScopedWorkingDir() { fs::remove_all(m_workingPath); }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete copy operations to prevent double deletion

ScopedWorkingDir(const ScopedWorkingDir&) = delete;
ScopedWorkingDir& operator=(const ScopedWorkingDir&) = delete;
ScopedWorkingDir(ScopedWorkingDir&&) = delete;
ScopedWorkingDir& operator=(ScopedWorkingDir&&) = delete;

(move operations are already deleted by the fact of defining the destructor, but this way it's explicit)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the subdir a (optional?) parameter of the (explicit) ctor?

explicitScopedWorkingDir(const std::string& subDir = "");

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has moved to the .cpp file, and will only be used by ZipBufferUtils static methods, so I don't think it's usefull to make the sudir configurable

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use operator / for path separator, example of syntax:

m_workingPath = m_workingPath / "solar";
m_workingPath /= "solar";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both operations are available : why this one instead of the other one?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, changed

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

LOG_DEBUG("ZipBufferUtils::extract - Working temporary path: {}", workingDir.getStringPath());

// Check/create the working directory
if (!fs::exists(workingDir.getPath())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, still needed here: the subdir must be created here before creating the zip file (tested)

}

try {
fs::path dp(destinationPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other function you clear the output buffer, here we could delete the content of the destination path as well ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so: this destination directory can contain other files, we don't know.

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
return FrameworkReturnCode::_ERROR_;
}

// Delete the zip file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes you do it because the zip is in the same dir as the one you copy later on, OK.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
file.close();

// Try to unzip the file content
std::string command = "cd " + workingDir.getStringPath() + "; unzip data.zip";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you could avoid the copy by cd in the dest dir, there's really no risk of messing with existing files, since it's supposed to receive ours, no?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think you're right and I will change it

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@cutullic@jim-bcom
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feat/zip features - #139

Open
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features
Open

Feat/zip features#139
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features

Conversation

@cutullic

Copy link
Copy Markdown
Contributor

No description provided.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Create a working directory for zip/unzip features
fs::path wp(m_workingPath);
if (!fs::exists(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use temp_directory_pathhttps://en.cppreference.com/cpp/filesystem/temp_directory_path ?

This way you are guaranteed to get an existing directory, and it might enventually be removed when the machine restarts, if for some reason you were not able to delete it sooner.

Here, if the directory does exists, you might use it, and potentially overwrite user data, since it's not handled as an error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I think of it, concerning the two PR comment this seems to address: you might want to split the RAII part you did with the dtor of ZipBuffer, from the utility methods.

  • Make ZipBufferUtils (or whatever :)) have these 2 methods of extraction/compression as static helpers
ZipBufferUtils::extract()
ZipBufferUtils::create/compress/...()
  • create a RAII object responsible to create the tmp dir and deleting it on destruction, something like:
classScopedTmpDir {
ScopedTmpDir () { m_path = temp_directory_path();}
~ScopedTmpDir { fs::remove_all(m_path ); }
fs::path getPath();
};

And use it in the static helper function.

This adds the benefit of deleting things ASAP, instead of waiting that the ZipBuffer object goes out of scope, which may be long after the end of the function (which I gues is why you kept deleting things at the end of the methods, instead of relying on the destructor, as I commented elsewhere).

What do you think of this approach ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach :-) Made the modifications.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Check working directory
fs::path wp(m_workingPath);
if (!fs::is_directory(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If temp_directory_path this check is probably not needed (even in the current version). Unless we fear something might happen to this directory between the creation of ZipBuffer and the call to this.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because temp_directory_path is now used.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::path op(originalPath);
if (!fs::is_directory(op)) {
LOG_ERROR("The original path is not a directory: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe !exists => NOT_FOUND but !directory => ERROR (if we don't have BAD_ARGUMENT) ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment threadsrc/core/ZipBuffer.cpp Outdated
}
if (fs::is_empty(op)) {
LOG_ERROR("The original path is empty: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUCCESS? You give an empty dir, you get an empty vector, job done ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK returns SUCCESS but with a warning (and clear the input buffer)

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we avoid the copy ? For example by invoking zip directly from originalPath ?

std::string command = "cd " + originalPath + ";zip -r " + m_workingPath+ "/data.zip .";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I prefer not to work on the original directory, to ensure I don't modify it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see the point. I will try to change this behavior.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::remove(zipFile);

// Copy unzipped data in the destination directory
const auto copyOptions = fs::copy_options::recursive;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid the copy by invoking zip from the destination directory?

std::string command = "cd " + destinationPath+ "; unzip " + m_workingPath + " /data.zip";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

Comment threadsrc/core/ZipBuffer.cpp Outdated

// private

void ZipBuffer::cleanWorkingDirectory()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer useful at all: removed

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * FrameworkReturnCode::_NOT_FOUND if data is not found in original path
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode zipToBuffer(const std::string & originalPath,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use directly fs::path i.o. string for path ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought using std::string avoided forcing the caller to use std::filesystem?

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// @return
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode bufferToUnzip(const std::vector<unsigned char> & compressedZipBuffer,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: matter of taste, but with verbs for method ? ZipBuffer::extract()/ZipBuffer::compress()? Or something else.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK changed

Comment threadinterfaces/core/ZipBuffer.h Outdated
*
*/

class ZipBuffer {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again naming things is hard... It does not per say represent a ZipBuffer, but more like utils functions... But I'm not sure about ZipBufferUtils... I don't have a better idea :). Something like ZipBufferExtractor (but it does also compress, so ...), or ZipBufferProcessor,...

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just ZipUtils ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go for ZipBufferUtils class and extractand compressmethods

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
#include <vector>
#include <filesystem>

namespace fs = std::filesystem;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove, avoid aliasing in header, especially at this scope.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sorry!

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
* @brief <B>Create a temporary working directory</B>
*
*/
class ScopedWorkingDir {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is decorrelated to ZipBufferUtils, so 2 suggestions:

  1. define in its own file for anyboady to reuse
  2. define it in the cpp file, because it's not referenced here, so it does not have to be defined here.

It's only an implementation detail that does not have to surface here (except if this class is needed elsewhere, but then this would have to be defined in its own file, as suggested in 1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also maybe keep Tmp or Temp in the name (ScopedTmpDir), so that we know it uses the tmp folder API, and this is where the file is created ...?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }
~ScopedWorkingDir() { fs::remove_all(m_workingPath); }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete copy operations to prevent double deletion

ScopedWorkingDir(const ScopedWorkingDir&) = delete;
ScopedWorkingDir& operator=(const ScopedWorkingDir&) = delete;
ScopedWorkingDir(ScopedWorkingDir&&) = delete;
ScopedWorkingDir& operator=(ScopedWorkingDir&&) = delete;

(move operations are already deleted by the fact of defining the destructor, but this way it's explicit)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the subdir a (optional?) parameter of the (explicit) ctor?

explicitScopedWorkingDir(const std::string& subDir = "");

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has moved to the .cpp file, and will only be used by ZipBufferUtils static methods, so I don't think it's usefull to make the sudir configurable

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use operator / for path separator, example of syntax:

m_workingPath = m_workingPath / "solar";
m_workingPath /= "solar";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both operations are available : why this one instead of the other one?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, changed

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

LOG_DEBUG("ZipBufferUtils::extract - Working temporary path: {}", workingDir.getStringPath());

// Check/create the working directory
if (!fs::exists(workingDir.getPath())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, still needed here: the subdir must be created here before creating the zip file (tested)

}

try {
fs::path dp(destinationPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other function you clear the output buffer, here we could delete the content of the destination path as well ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so: this destination directory can contain other files, we don't know.

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
return FrameworkReturnCode::_ERROR_;
}

// Delete the zip file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes you do it because the zip is in the same dir as the one you copy later on, OK.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
file.close();

// Try to unzip the file content
std::string command = "cd " + workingDir.getStringPath() + "; unzip data.zip";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you could avoid the copy by cd in the dest dir, there's really no risk of messing with existing files, since it's supposed to receive ours, no?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think you're right and I will change it

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@cutullic@jim-bcom
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Feat/zip features - #139

Open
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features
Open

Feat/zip features#139
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features

Conversation

@cutullic

Copy link
Copy Markdown
Contributor

No description provided.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Create a working directory for zip/unzip features
fs::path wp(m_workingPath);
if (!fs::exists(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use temp_directory_pathhttps://en.cppreference.com/cpp/filesystem/temp_directory_path ?

This way you are guaranteed to get an existing directory, and it might enventually be removed when the machine restarts, if for some reason you were not able to delete it sooner.

Here, if the directory does exists, you might use it, and potentially overwrite user data, since it's not handled as an error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I think of it, concerning the two PR comment this seems to address: you might want to split the RAII part you did with the dtor of ZipBuffer, from the utility methods.

  • Make ZipBufferUtils (or whatever :)) have these 2 methods of extraction/compression as static helpers
ZipBufferUtils::extract()
ZipBufferUtils::create/compress/...()
  • create a RAII object responsible to create the tmp dir and deleting it on destruction, something like:
classScopedTmpDir {
ScopedTmpDir () { m_path = temp_directory_path();}
~ScopedTmpDir { fs::remove_all(m_path ); }
fs::path getPath();
};

And use it in the static helper function.

This adds the benefit of deleting things ASAP, instead of waiting that the ZipBuffer object goes out of scope, which may be long after the end of the function (which I gues is why you kept deleting things at the end of the methods, instead of relying on the destructor, as I commented elsewhere).

What do you think of this approach ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach :-) Made the modifications.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Check working directory
fs::path wp(m_workingPath);
if (!fs::is_directory(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If temp_directory_path this check is probably not needed (even in the current version). Unless we fear something might happen to this directory between the creation of ZipBuffer and the call to this.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because temp_directory_path is now used.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::path op(originalPath);
if (!fs::is_directory(op)) {
LOG_ERROR("The original path is not a directory: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe !exists => NOT_FOUND but !directory => ERROR (if we don't have BAD_ARGUMENT) ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment threadsrc/core/ZipBuffer.cpp Outdated
}
if (fs::is_empty(op)) {
LOG_ERROR("The original path is empty: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUCCESS? You give an empty dir, you get an empty vector, job done ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK returns SUCCESS but with a warning (and clear the input buffer)

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we avoid the copy ? For example by invoking zip directly from originalPath ?

std::string command = "cd " + originalPath + ";zip -r " + m_workingPath+ "/data.zip .";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I prefer not to work on the original directory, to ensure I don't modify it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see the point. I will try to change this behavior.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::remove(zipFile);

// Copy unzipped data in the destination directory
const auto copyOptions = fs::copy_options::recursive;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid the copy by invoking zip from the destination directory?

std::string command = "cd " + destinationPath+ "; unzip " + m_workingPath + " /data.zip";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

Comment threadsrc/core/ZipBuffer.cpp Outdated

// private

void ZipBuffer::cleanWorkingDirectory()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer useful at all: removed

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * FrameworkReturnCode::_NOT_FOUND if data is not found in original path
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode zipToBuffer(const std::string & originalPath,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use directly fs::path i.o. string for path ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought using std::string avoided forcing the caller to use std::filesystem?

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// @return
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode bufferToUnzip(const std::vector<unsigned char> & compressedZipBuffer,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: matter of taste, but with verbs for method ? ZipBuffer::extract()/ZipBuffer::compress()? Or something else.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK changed

Comment threadinterfaces/core/ZipBuffer.h Outdated
*
*/

class ZipBuffer {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again naming things is hard... It does not per say represent a ZipBuffer, but more like utils functions... But I'm not sure about ZipBufferUtils... I don't have a better idea :). Something like ZipBufferExtractor (but it does also compress, so ...), or ZipBufferProcessor,...

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just ZipUtils ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go for ZipBufferUtils class and extractand compressmethods

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
#include <vector>
#include <filesystem>

namespace fs = std::filesystem;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove, avoid aliasing in header, especially at this scope.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sorry!

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
* @brief <B>Create a temporary working directory</B>
*
*/
class ScopedWorkingDir {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is decorrelated to ZipBufferUtils, so 2 suggestions:

  1. define in its own file for anyboady to reuse
  2. define it in the cpp file, because it's not referenced here, so it does not have to be defined here.

It's only an implementation detail that does not have to surface here (except if this class is needed elsewhere, but then this would have to be defined in its own file, as suggested in 1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also maybe keep Tmp or Temp in the name (ScopedTmpDir), so that we know it uses the tmp folder API, and this is where the file is created ...?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }
~ScopedWorkingDir() { fs::remove_all(m_workingPath); }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete copy operations to prevent double deletion

ScopedWorkingDir(const ScopedWorkingDir&) = delete;
ScopedWorkingDir& operator=(const ScopedWorkingDir&) = delete;
ScopedWorkingDir(ScopedWorkingDir&&) = delete;
ScopedWorkingDir& operator=(ScopedWorkingDir&&) = delete;

(move operations are already deleted by the fact of defining the destructor, but this way it's explicit)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the subdir a (optional?) parameter of the (explicit) ctor?

explicitScopedWorkingDir(const std::string& subDir = "");

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has moved to the .cpp file, and will only be used by ZipBufferUtils static methods, so I don't think it's usefull to make the sudir configurable

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use operator / for path separator, example of syntax:

m_workingPath = m_workingPath / "solar";
m_workingPath /= "solar";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both operations are available : why this one instead of the other one?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, changed

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

LOG_DEBUG("ZipBufferUtils::extract - Working temporary path: {}", workingDir.getStringPath());

// Check/create the working directory
if (!fs::exists(workingDir.getPath())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, still needed here: the subdir must be created here before creating the zip file (tested)

}

try {
fs::path dp(destinationPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other function you clear the output buffer, here we could delete the content of the destination path as well ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so: this destination directory can contain other files, we don't know.

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
return FrameworkReturnCode::_ERROR_;
}

// Delete the zip file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes you do it because the zip is in the same dir as the one you copy later on, OK.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
file.close();

// Try to unzip the file content
std::string command = "cd " + workingDir.getStringPath() + "; unzip data.zip";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you could avoid the copy by cd in the dest dir, there's really no risk of messing with existing files, since it's supposed to receive ours, no?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think you're right and I will change it

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@cutullic@jim-bcom
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feat/zip features - #139

Open
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features
Open

Feat/zip features#139
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features

Conversation

@cutullic

Copy link
Copy Markdown
Contributor

No description provided.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Create a working directory for zip/unzip features
fs::path wp(m_workingPath);
if (!fs::exists(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use temp_directory_pathhttps://en.cppreference.com/cpp/filesystem/temp_directory_path ?

This way you are guaranteed to get an existing directory, and it might enventually be removed when the machine restarts, if for some reason you were not able to delete it sooner.

Here, if the directory does exists, you might use it, and potentially overwrite user data, since it's not handled as an error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I think of it, concerning the two PR comment this seems to address: you might want to split the RAII part you did with the dtor of ZipBuffer, from the utility methods.

  • Make ZipBufferUtils (or whatever :)) have these 2 methods of extraction/compression as static helpers
ZipBufferUtils::extract()
ZipBufferUtils::create/compress/...()
  • create a RAII object responsible to create the tmp dir and deleting it on destruction, something like:
classScopedTmpDir {
ScopedTmpDir () { m_path = temp_directory_path();}
~ScopedTmpDir { fs::remove_all(m_path ); }
fs::path getPath();
};

And use it in the static helper function.

This adds the benefit of deleting things ASAP, instead of waiting that the ZipBuffer object goes out of scope, which may be long after the end of the function (which I gues is why you kept deleting things at the end of the methods, instead of relying on the destructor, as I commented elsewhere).

What do you think of this approach ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach :-) Made the modifications.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Check working directory
fs::path wp(m_workingPath);
if (!fs::is_directory(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If temp_directory_path this check is probably not needed (even in the current version). Unless we fear something might happen to this directory between the creation of ZipBuffer and the call to this.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because temp_directory_path is now used.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::path op(originalPath);
if (!fs::is_directory(op)) {
LOG_ERROR("The original path is not a directory: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe !exists => NOT_FOUND but !directory => ERROR (if we don't have BAD_ARGUMENT) ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment threadsrc/core/ZipBuffer.cpp Outdated
}
if (fs::is_empty(op)) {
LOG_ERROR("The original path is empty: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUCCESS? You give an empty dir, you get an empty vector, job done ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK returns SUCCESS but with a warning (and clear the input buffer)

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we avoid the copy ? For example by invoking zip directly from originalPath ?

std::string command = "cd " + originalPath + ";zip -r " + m_workingPath+ "/data.zip .";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I prefer not to work on the original directory, to ensure I don't modify it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see the point. I will try to change this behavior.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::remove(zipFile);

// Copy unzipped data in the destination directory
const auto copyOptions = fs::copy_options::recursive;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid the copy by invoking zip from the destination directory?

std::string command = "cd " + destinationPath+ "; unzip " + m_workingPath + " /data.zip";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

Comment threadsrc/core/ZipBuffer.cpp Outdated

// private

void ZipBuffer::cleanWorkingDirectory()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer useful at all: removed

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * FrameworkReturnCode::_NOT_FOUND if data is not found in original path
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode zipToBuffer(const std::string & originalPath,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use directly fs::path i.o. string for path ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought using std::string avoided forcing the caller to use std::filesystem?

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// @return
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode bufferToUnzip(const std::vector<unsigned char> & compressedZipBuffer,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: matter of taste, but with verbs for method ? ZipBuffer::extract()/ZipBuffer::compress()? Or something else.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK changed

Comment threadinterfaces/core/ZipBuffer.h Outdated
*
*/

class ZipBuffer {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again naming things is hard... It does not per say represent a ZipBuffer, but more like utils functions... But I'm not sure about ZipBufferUtils... I don't have a better idea :). Something like ZipBufferExtractor (but it does also compress, so ...), or ZipBufferProcessor,...

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just ZipUtils ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go for ZipBufferUtils class and extractand compressmethods

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
#include <vector>
#include <filesystem>

namespace fs = std::filesystem;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove, avoid aliasing in header, especially at this scope.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sorry!

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
* @brief <B>Create a temporary working directory</B>
*
*/
class ScopedWorkingDir {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is decorrelated to ZipBufferUtils, so 2 suggestions:

  1. define in its own file for anyboady to reuse
  2. define it in the cpp file, because it's not referenced here, so it does not have to be defined here.

It's only an implementation detail that does not have to surface here (except if this class is needed elsewhere, but then this would have to be defined in its own file, as suggested in 1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also maybe keep Tmp or Temp in the name (ScopedTmpDir), so that we know it uses the tmp folder API, and this is where the file is created ...?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }
~ScopedWorkingDir() { fs::remove_all(m_workingPath); }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete copy operations to prevent double deletion

ScopedWorkingDir(const ScopedWorkingDir&) = delete;
ScopedWorkingDir& operator=(const ScopedWorkingDir&) = delete;
ScopedWorkingDir(ScopedWorkingDir&&) = delete;
ScopedWorkingDir& operator=(ScopedWorkingDir&&) = delete;

(move operations are already deleted by the fact of defining the destructor, but this way it's explicit)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the subdir a (optional?) parameter of the (explicit) ctor?

explicitScopedWorkingDir(const std::string& subDir = "");

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has moved to the .cpp file, and will only be used by ZipBufferUtils static methods, so I don't think it's usefull to make the sudir configurable

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use operator / for path separator, example of syntax:

m_workingPath = m_workingPath / "solar";
m_workingPath /= "solar";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both operations are available : why this one instead of the other one?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, changed

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

LOG_DEBUG("ZipBufferUtils::extract - Working temporary path: {}", workingDir.getStringPath());

// Check/create the working directory
if (!fs::exists(workingDir.getPath())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, still needed here: the subdir must be created here before creating the zip file (tested)

}

try {
fs::path dp(destinationPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other function you clear the output buffer, here we could delete the content of the destination path as well ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so: this destination directory can contain other files, we don't know.

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
return FrameworkReturnCode::_ERROR_;
}

// Delete the zip file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes you do it because the zip is in the same dir as the one you copy later on, OK.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
file.close();

// Try to unzip the file content
std::string command = "cd " + workingDir.getStringPath() + "; unzip data.zip";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you could avoid the copy by cd in the dest dir, there's really no risk of messing with existing files, since it's supposed to receive ours, no?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think you're right and I will change it

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@cutullic@jim-bcom
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Feat/zip features - #139

Open
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features
Open

Feat/zip features#139
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features

Conversation

@cutullic

Copy link
Copy Markdown
Contributor

No description provided.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Create a working directory for zip/unzip features
fs::path wp(m_workingPath);
if (!fs::exists(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use temp_directory_pathhttps://en.cppreference.com/cpp/filesystem/temp_directory_path ?

This way you are guaranteed to get an existing directory, and it might enventually be removed when the machine restarts, if for some reason you were not able to delete it sooner.

Here, if the directory does exists, you might use it, and potentially overwrite user data, since it's not handled as an error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I think of it, concerning the two PR comment this seems to address: you might want to split the RAII part you did with the dtor of ZipBuffer, from the utility methods.

  • Make ZipBufferUtils (or whatever :)) have these 2 methods of extraction/compression as static helpers
ZipBufferUtils::extract()
ZipBufferUtils::create/compress/...()
  • create a RAII object responsible to create the tmp dir and deleting it on destruction, something like:
classScopedTmpDir {
ScopedTmpDir () { m_path = temp_directory_path();}
~ScopedTmpDir { fs::remove_all(m_path ); }
fs::path getPath();
};

And use it in the static helper function.

This adds the benefit of deleting things ASAP, instead of waiting that the ZipBuffer object goes out of scope, which may be long after the end of the function (which I gues is why you kept deleting things at the end of the methods, instead of relying on the destructor, as I commented elsewhere).

What do you think of this approach ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach :-) Made the modifications.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Check working directory
fs::path wp(m_workingPath);
if (!fs::is_directory(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If temp_directory_path this check is probably not needed (even in the current version). Unless we fear something might happen to this directory between the creation of ZipBuffer and the call to this.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because temp_directory_path is now used.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::path op(originalPath);
if (!fs::is_directory(op)) {
LOG_ERROR("The original path is not a directory: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe !exists => NOT_FOUND but !directory => ERROR (if we don't have BAD_ARGUMENT) ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment threadsrc/core/ZipBuffer.cpp Outdated
}
if (fs::is_empty(op)) {
LOG_ERROR("The original path is empty: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUCCESS? You give an empty dir, you get an empty vector, job done ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK returns SUCCESS but with a warning (and clear the input buffer)

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we avoid the copy ? For example by invoking zip directly from originalPath ?

std::string command = "cd " + originalPath + ";zip -r " + m_workingPath+ "/data.zip .";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I prefer not to work on the original directory, to ensure I don't modify it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see the point. I will try to change this behavior.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::remove(zipFile);

// Copy unzipped data in the destination directory
const auto copyOptions = fs::copy_options::recursive;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid the copy by invoking zip from the destination directory?

std::string command = "cd " + destinationPath+ "; unzip " + m_workingPath + " /data.zip";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

Comment threadsrc/core/ZipBuffer.cpp Outdated

// private

void ZipBuffer::cleanWorkingDirectory()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer useful at all: removed

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * FrameworkReturnCode::_NOT_FOUND if data is not found in original path
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode zipToBuffer(const std::string & originalPath,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use directly fs::path i.o. string for path ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought using std::string avoided forcing the caller to use std::filesystem?

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// @return
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode bufferToUnzip(const std::vector<unsigned char> & compressedZipBuffer,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: matter of taste, but with verbs for method ? ZipBuffer::extract()/ZipBuffer::compress()? Or something else.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK changed

Comment threadinterfaces/core/ZipBuffer.h Outdated
*
*/

class ZipBuffer {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again naming things is hard... It does not per say represent a ZipBuffer, but more like utils functions... But I'm not sure about ZipBufferUtils... I don't have a better idea :). Something like ZipBufferExtractor (but it does also compress, so ...), or ZipBufferProcessor,...

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just ZipUtils ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go for ZipBufferUtils class and extractand compressmethods

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
#include <vector>
#include <filesystem>

namespace fs = std::filesystem;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove, avoid aliasing in header, especially at this scope.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sorry!

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
* @brief <B>Create a temporary working directory</B>
*
*/
class ScopedWorkingDir {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is decorrelated to ZipBufferUtils, so 2 suggestions:

  1. define in its own file for anyboady to reuse
  2. define it in the cpp file, because it's not referenced here, so it does not have to be defined here.

It's only an implementation detail that does not have to surface here (except if this class is needed elsewhere, but then this would have to be defined in its own file, as suggested in 1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also maybe keep Tmp or Temp in the name (ScopedTmpDir), so that we know it uses the tmp folder API, and this is where the file is created ...?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }
~ScopedWorkingDir() { fs::remove_all(m_workingPath); }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete copy operations to prevent double deletion

ScopedWorkingDir(const ScopedWorkingDir&) = delete;
ScopedWorkingDir& operator=(const ScopedWorkingDir&) = delete;
ScopedWorkingDir(ScopedWorkingDir&&) = delete;
ScopedWorkingDir& operator=(ScopedWorkingDir&&) = delete;

(move operations are already deleted by the fact of defining the destructor, but this way it's explicit)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the subdir a (optional?) parameter of the (explicit) ctor?

explicitScopedWorkingDir(const std::string& subDir = "");

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has moved to the .cpp file, and will only be used by ZipBufferUtils static methods, so I don't think it's usefull to make the sudir configurable

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use operator / for path separator, example of syntax:

m_workingPath = m_workingPath / "solar";
m_workingPath /= "solar";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both operations are available : why this one instead of the other one?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, changed

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

LOG_DEBUG("ZipBufferUtils::extract - Working temporary path: {}", workingDir.getStringPath());

// Check/create the working directory
if (!fs::exists(workingDir.getPath())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, still needed here: the subdir must be created here before creating the zip file (tested)

}

try {
fs::path dp(destinationPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other function you clear the output buffer, here we could delete the content of the destination path as well ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so: this destination directory can contain other files, we don't know.

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
return FrameworkReturnCode::_ERROR_;
}

// Delete the zip file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes you do it because the zip is in the same dir as the one you copy later on, OK.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
file.close();

// Try to unzip the file content
std::string command = "cd " + workingDir.getStringPath() + "; unzip data.zip";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you could avoid the copy by cd in the dest dir, there's really no risk of messing with existing files, since it's supposed to receive ours, no?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think you're right and I will change it

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@cutullic@jim-bcom
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Feat/zip features - #139

Open
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features
Open

Feat/zip features#139
cutullic wants to merge 4 commits into
masterfrom
feat/zip-features

Conversation

@cutullic

Copy link
Copy Markdown
Contributor

No description provided.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Create a working directory for zip/unzip features
fs::path wp(m_workingPath);
if (!fs::exists(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use temp_directory_pathhttps://en.cppreference.com/cpp/filesystem/temp_directory_path ?

This way you are guaranteed to get an existing directory, and it might enventually be removed when the machine restarts, if for some reason you were not able to delete it sooner.

Here, if the directory does exists, you might use it, and potentially overwrite user data, since it's not handled as an error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I think of it, concerning the two PR comment this seems to address: you might want to split the RAII part you did with the dtor of ZipBuffer, from the utility methods.

  • Make ZipBufferUtils (or whatever :)) have these 2 methods of extraction/compression as static helpers
ZipBufferUtils::extract()
ZipBufferUtils::create/compress/...()
  • create a RAII object responsible to create the tmp dir and deleting it on destruction, something like:
classScopedTmpDir {
ScopedTmpDir () { m_path = temp_directory_path();}
~ScopedTmpDir { fs::remove_all(m_path ); }
fs::path getPath();
};

And use it in the static helper function.

This adds the benefit of deleting things ASAP, instead of waiting that the ZipBuffer object goes out of scope, which may be long after the end of the function (which I gues is why you kept deleting things at the end of the methods, instead of relying on the destructor, as I commented elsewhere).

What do you think of this approach ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach :-) Made the modifications.

Comment threadsrc/core/ZipBuffer.cpp Outdated
try {
// Check working directory
fs::path wp(m_workingPath);
if (!fs::is_directory(wp)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If temp_directory_path this check is probably not needed (even in the current version). Unless we fear something might happen to this directory between the creation of ZipBuffer and the call to this.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed because temp_directory_path is now used.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::path op(originalPath);
if (!fs::is_directory(op)) {
LOG_ERROR("The original path is not a directory: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe !exists => NOT_FOUND but !directory => ERROR (if we don't have BAD_ARGUMENT) ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Comment threadsrc/core/ZipBuffer.cpp Outdated
}
if (fs::is_empty(op)) {
LOG_ERROR("The original path is empty: {}", originalPath);
return FrameworkReturnCode::_NOT_FOUND;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUCCESS? You give an empty dir, you get an empty vector, job done ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK returns SUCCESS but with a warning (and clear the input buffer)

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we avoid the copy ? For example by invoking zip directly from originalPath ?

std::string command = "cd " + originalPath + ";zip -r " + m_workingPath+ "/data.zip .";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I prefer not to work on the original directory, to ensure I don't modify it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see the point. I will try to change this behavior.

Comment threadsrc/core/ZipBuffer.cpp Outdated
fs::remove(zipFile);

// Copy unzipped data in the destination directory
const auto copyOptions = fs::copy_options::recursive;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid the copy by invoking zip from the destination directory?

std::string command = "cd " + destinationPath+ "; unzip " + m_workingPath + " /data.zip";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

Comment threadsrc/core/ZipBuffer.cpp Outdated

// private

void ZipBuffer::cleanWorkingDirectory()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer useful at all: removed

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * FrameworkReturnCode::_NOT_FOUND if data is not found in original path
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode zipToBuffer(const std::string & originalPath,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use directly fs::path i.o. string for path ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought using std::string avoided forcing the caller to use std::filesystem?

Comment threadinterfaces/core/ZipBuffer.h Outdated
/// @return
/// * FrameworkReturnCode::_SUCCESS if the process succeeds
/// * else FrameworkReturnCode::_ERROR_
FrameworkReturnCode bufferToUnzip(const std::vector<unsigned char> & compressedZipBuffer,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: matter of taste, but with verbs for method ? ZipBuffer::extract()/ZipBuffer::compress()? Or something else.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK changed

Comment threadinterfaces/core/ZipBuffer.h Outdated
*
*/

class ZipBuffer {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again naming things is hard... It does not per say represent a ZipBuffer, but more like utils functions... But I'm not sure about ZipBufferUtils... I don't have a better idea :). Something like ZipBufferExtractor (but it does also compress, so ...), or ZipBufferProcessor,...

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just ZipUtils ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go for ZipBufferUtils class and extractand compressmethods

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
#include <vector>
#include <filesystem>

namespace fs = std::filesystem;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove, avoid aliasing in header, especially at this scope.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sorry!

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
* @brief <B>Create a temporary working directory</B>
*
*/
class ScopedWorkingDir {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is decorrelated to ZipBufferUtils, so 2 suggestions:

  1. define in its own file for anyboady to reuse
  2. define it in the cpp file, because it's not referenced here, so it does not have to be defined here.

It's only an implementation detail that does not have to surface here (except if this class is needed elsewhere, but then this would have to be defined in its own file, as suggested in 1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also maybe keep Tmp or Temp in the name (ScopedTmpDir), so that we know it uses the tmp folder API, and this is where the file is created ...?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }
~ScopedWorkingDir() { fs::remove_all(m_workingPath); }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete copy operations to prevent double deletion

ScopedWorkingDir(const ScopedWorkingDir&) = delete;
ScopedWorkingDir& operator=(const ScopedWorkingDir&) = delete;
ScopedWorkingDir(ScopedWorkingDir&&) = delete;
ScopedWorkingDir& operator=(ScopedWorkingDir&&) = delete;

(move operations are already deleted by the fact of defining the destructor, but this way it's explicit)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the subdir a (optional?) parameter of the (explicit) ctor?

explicitScopedWorkingDir(const std::string& subDir = "");

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class has moved to the .cpp file, and will only be used by ZipBufferUtils static methods, so I don't think it's usefull to make the sudir configurable

Comment threadinterfaces/core/ZipBufferUtils.h Outdated
*/
class ScopedWorkingDir {
public:
ScopedWorkingDir() { m_workingPath = fs::temp_directory_path(); m_workingPath += "/solar"; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use operator / for path separator, example of syntax:

m_workingPath = m_workingPath / "solar";
m_workingPath /= "solar";

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both operations are available : why this one instead of the other one?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, changed

Comment threadsrc/core/ZipBuffer.cpp Outdated

// Copy data to zip in the working directory
const auto copyOptions = fs::copy_options::recursive;
fs::copy(op, wp, copyOptions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but I'm not sure I see what you fear of invoking zip, if you specify a different destination dit for the zip file, it shouldn't touch the origin one.

It's just that it might be costly to copy a large amount of data. The current implementation is already not ideal performance wise (using a temporary zip file on disk i.o. zipping it in memory, which produces a lot of I/O), and this adds up.

OK we'll see if it becomes a bottleneck.

LOG_DEBUG("ZipBufferUtils::extract - Working temporary path: {}", workingDir.getStringPath());

// Check/create the working directory
if (!fs::exists(workingDir.getPath())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, still needed here: the subdir must be created here before creating the zip file (tested)

}

try {
fs::path dp(destinationPath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the other function you clear the output buffer, here we could delete the content of the destination path as well ?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so: this destination directory can contain other files, we don't know.

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
return FrameworkReturnCode::_ERROR_;
}

// Delete the zip file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes you do it because the zip is in the same dir as the one you copy later on, OK.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

Comment threadsrc/core/ZipBufferUtils.cpp Outdated
file.close();

// Try to unzip the file content
std::string command = "cd " + workingDir.getStringPath() + "; unzip data.zip";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you could avoid the copy by cd in the dest dir, there's really no risk of messing with existing files, since it's supposed to receive ours, no?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think you're right and I will change it

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@cutullic@jim-bcom