Discussion: Parent file format

[adrybakov]

Topic: Human-readable or binary? Structured or custom?

[adrybakov]

Here is a list of programming languages that shall be considered, based on the surveys

• C
• C++ (in particular C++20)
• Fortran (in particular Fortran90)
• Go
• Matlab
• Python (probably only Python3)
• Rust

Please mention your language if it is not listed

[mducle]

I think that a structured plain text format is preferable because:

• Most of the language listed have libraries which can deserialise / parse such a format (such as json or xml) and this will make implementation much easier than using either a structured binary format like hdf5 or a custom binary format.
• The file can be inspected without any software even if it is not designed to be human readable.

The disadvantage of a text format is that:

• It will be slightly slower / less performant to read in.
• It might not be able to store floating point numbers to the full 64-bit precision (~16 significant digits), depending on string truncation.

However, I don't think that any programs will need to read in the format as part of a tight loop so the format's performance will not be on the critical path; and we can specify that the reference reader/writer write out full-precision numbers.

[adrybakov]

Reflecting on some of your arguments:

Speed

I agree that the speed of reading (or writing) is not the main concern compare to the time of calculations for most of the tools. Perhaps there are use-cases that I am not aware of.

Size

Size of the file might be an issue if one would like to compose a database of spin Hamiltonians. In that case we could optimise in two places: file format itself (.something) and how the parameters are stored by themselves (for instance whether to store i -> j only or both i -> j and j -> j in case of bi-linear interaction). Note that for tri-linear interaction there are 6 equivalent "bonds", 24 for quad-linear and so on (n! for n-linear).

Higher order parameters

The problem of representation of the multidimensional arrays in any txt based-format. I am not aware how well is that handled in the .hdf5, for instance, though.

Floating point

I agree that it is cumbersome, but still any writer can support arbitrary amount of significant digits. Moreover, in my opinion this is out of the scope of the format, as we need to specify how many numbers and in which order to print, but the precision should be decided on the tool's (that produce the file) or user's side. Some might want to default to machine precision, while some might want to base it on the physical relevance, etc.

[adrybakov]

• Most of the language listed have libraries which can deserialise / parse such a format (such as json or xml) and this will make implementation much easier than using either a structured binary format like hdf5 or a custom binary format.

On the other side a structured text format requires a dependency library for reading it in the same way as a structured binary one requires. From that perspective they are identical.

[adrybakov]

• The file can be inspected without any software even if it is not designed to be human readable.

That seems to be the main concern, based on the surveys. In that case if we decide for the format to be human-readable, then the choice is between structured txt or custom txt, the former requires a dependency library in each language. The latter can be read directly by any language, knowing the format specs alone (i.e. without any additional dependency).

Implementation-wise, I think that it is easier to use custom txt format, as then more tools will be ready to use it (does not require the third-party dependency) and the reference implementations of reading/writing will not need to be concerned about the versions and potential bugs in the third-party libraries for the structured txt formats.

Moreover, if the human readability is a secondary goal, then the custom format can be optimised for that as well.

[richard-evans]

Implementation-wise, I think that it is easier to use custom txt format

I agree with this route (keeps things simple), as json etc are a nuisance to read/write. Also once the format is specified we can do a binary equivalent and conversion tool for performance, which works as long as staying within mac/linux/windows.