ensure atomic writes when saving a file

[basnijholt]

Right now, if a program crashes in the middle of saving, you lose all your data. This ensures that the old file is first moved, then the new file is saved, and only then the old file is removed.

This is not a hypothetical scenario but happens every day for some people that I work with ATM.

[basnijholt]

Maybe adding a "." in front of the fname isn't the best name to choose.

[akhmerov]

This is not a hypothetical scenario but happens every day for some people that I work with ATM.

Maybe that is also worth investigating as a separate issue? Saving usually shouldn't cause a problem if adaptive is stable.

[akhmerov]

The implementation is rather fragile and potentially hitting bad corner cases (overwriting a different file, times being messed up, etc); I think there should be a more systematic way to do an atomic write.

[basnijholt]

Maybe that is also worth investigating as a separate issue? Saving usually shouldn't cause a problem if adaptive is stable.

The reason for the crash is not Adaptive, but merely low priority nodes that can be evicted at any time, that suddenly stop.

[basnijholt]

The implementation is rather fragile and potentially hitting bad corner cases (overwriting a different file, times being messed up, etc); I think there should be a more systematic way to do an atomic write.

Do you have a suggestion?

[akhmerov]

https://github.com/untitaker/python-atomicwrites seems like an option or any other alternatives.

Or we could use all the relevant tricks from it.

[akhmerov]

I also have to bring up another approach: we provide a way to get an in-memory data in adaptive and the user uses atomicwrites on their own. Separation of concerns and all.

[basnijholt]

I also have to bring up another approach: we provide a way to get an in-memory data in adaptive and the user uses atomicwrites on their own. Separation of concerns and all.

There is already _get_data and _set_data that every learner implements. The problem is that you can't use the periodic_saver like this

I do not mind to add the dependency, it's used by 19723 other packages and it is tiny. See the commit I've made. Also, I don't see a reason why people wouldn't want a 'safe' save.