[TW-94] Atomic file writes

[taskwarrior]

Paul Beckingham on 2009-05-25T21:43:36Z says:

Guaranteed atomic file writes via mv

[taskwarrior]

Migrated metadata:

Created: 2009-05-25T21:43:36Z
Modified: 2017-01-16T16:18:47Z

[taskwarrior]

Jakub Wilk on 2013-03-04T23:22:19Z says:

Any news on this?
I've just lost half of my pending.data because I pressed Ctrl+C in a wrong moment. :(

[taskwarrior]

Paul Beckingham on 2013-03-05T01:27:47Z says:

Any news on this?

Nope. No patches received either.

I've just lost half of my pending.data because I pressed Ctrl+C in a wrong moment. :(

Don't do that. If you don't have a backup copy, the pending.data can be recreated from undo.data using "task merge".

[taskwarrior]

Paul Beckingham on 2014-12-27T16:37:07Z says:

The "task merge" advice above no longer applies. Everyone make backups.

[taskwarrior]

Wim Schuermann on 2015-07-22T21:17:57Z says:

The problem with getting this right(tm) is that writes aren't truly atomic unless backlog, completed, pending, and undo.data are all written atomically at the same time. So to get this completely right, one would have to replicate the $TASKDATA directory and atomically move that over once all writes are done.
That this is problematic and bordering on the point of being impossible to get right (What about hooks that use $TASKDATA for data storage, for example?) should be obvious.

Are there better options for this? For example, would writing each individual file atomically be enough if the user were informed about potentially mismatched data files?

[taskwarrior]

Daniel Shahaf on 2015-08-14T01:44:15Z says:

Another workaround: keep ~/.task/ in version control and make automatic commits from the on-exit hook. Then if the data files get corrupted, just revert to the state after the most recent commit.

[taskwarrior]

Paul Beckingham on 2017-01-16T16:18:35Z says:

Files could be individually written, say pending.data could be written to pending.data.temp, completed.data to completed.data.temp, and then finally, rename all the *.temp files. That's easy, would address this reported issue, and although not atomic, is a couple of steps in the right direction.

[vrusinov]

// Looking at the old issues, let me know if it's not helpful.

I took a look at this. There are two related problems here:

Protection against "normal" process termination. This is relatively easy to solve - we just need to catch SIGINT / SIGTERM & co while doing commit. This appears to be already implemented.

Protection against crash - either due power loss, or a bug (related issues #88, #1420). This is much more difficult - there are no simple solutions, APIs are horrible and it's very easy to mess it up. (http://danluu.com/file-consistency/, https://utcc.utoronto.ca/~cks/space/blog/unix/FileSyncProblem, https://lwn.net/Articles/667788/). Basically this involves writing database engine which would carefully arrange journals and fsync/fdatasync calls. I don't think it's worth doing as part of TaskWarriror - it is best to reuse existing library, such as sqlite or some embedded key-value database like LMDB or LevelDB. This was discussed before and once again proposed in recent #2507. This is a big decision so I don't think we should do it here.

Two smaller improvements which may help for some crashes are:

1. add fsync/fdatasync calls to the end of File::close. This will cause small performance penalty so not everyone may be happy.
2. in the non-append codepath of TF2::commit instead of truncating and overwriting file in-place we can be creating a new file (e.g. with .tmp suffix) and then renaming it over the existing one.

I can work on these two things. (2) should also fix corruption during full disk

[tbabej]

// Looking at the old issues, let me know if it's not helpful.

It's definitely helpful!

add fsync/fdatasync calls to the end of File::close. This will cause small performance penalty so not everyone may be happy.

In general, I'm in favour of trading off (small) performance hits for better robustness - I think we should do this. Perhaps some of the cost could be offset by only fdatasyncing if we actually modified the file (though perhaps in that case the cost of the fdatasync is negligible?)

in the non-append codepath of TF2::commit instead of truncating and overwriting file in-place we can be creating a new file (e.g. with .tmp suffix) and then renaming it over the existing one.

💯 Indeed, the full disk corruption has hit some users over the years.