Stellar
[SEP-54] Ledger Metadata Storage #1678
Replies: 6 comments 19 replies
-
|
@urvisavla pointed out the following differences between the spec and what we have implemented in galexie:
The reason I did not include the Regarding the root directory, I thought it would be useful to have a ledgers directory to separate the ledger keys from the config key. But I am open to feedback on both these points. |
Beta Was this translation helpful? Give feedback.
-
|
Separating the config from the ledgers directory makes sense to me and on some systems may make it easier to list the root and see the config, that otherwise might be lost in a slow need to paginate. Using a .zstd extension to clearly signal that the contents are not xdr, but compressed xdr, seems reasonable to me. The use of extensions to communicate encoding is common place and worth preserving. Files get downloaded, moved around, and put elsewhere. Whilst it may be fine to assume everyone knows the .xdr files are compressed when stored in this place, when they are moved to other places the .zstd will carry the communication about their encoding. Whilst I think this is unlikely, and I wouldn't make a decision on this alone, the .zstd extension also makes it possible to support multiple compression strategies, or to change compression strategy over time with overlap, should that need to occur. |
Beta Was this translation helpful? Give feedback.
-
|
@leighmcculloch do you think we should introduce a new field in the config object to indicate the extension on the batch keys will be |
Beta Was this translation helpful? Give feedback.
-
|
Also, adding the extension makes it clear the files are compressed and reinforces the recommendation to store them that way as storing them as uncompressed xdr would be very inefficient. |
Beta Was this translation helpful? Give feedback.
-
|
I think it'd be sufficient for the SEP to specify a mapping of compression algorithms to extensions. But, if you'd like to avoid needing to update the SEP with other compression algorithms and let folks use whatever they want, then yeah a mapping in the config could be a good idea. But defining it in the SEP is likely to result in more consistency? |
Beta Was this translation helpful? Give feedback.
-
|
I updated the SEP to include the zstandard file name extension but it turns out the filename extension defined for zstandard is |
Beta Was this translation helpful? Give feedback.
-
|
In what use cases can clients expect data providers to batch more than one ledger into a file? As ledgers get bigger and scale increases over time, will it still make sense to batch ledgers? I noticed on the AWS public deployment it is already only one ledger per file and it's tempting to write code that makes the assumption it'll always be one. I'm asking not to change the format, but wondering if for the most part clients can just assume it'll be one ledger per batch in the near future. |
Beta Was this translation helpful? Give feedback.
-
GCS tends to be more efficient when downloading larger files rather than multiple smaller files. If your backend is able to ingest ledgers at a sufficiently high speed then it makes sense to optimize the fetching of ledgers by having larger batch files. Also, keep in mind that the compression will reduce the size of the ledgers by a factor of ~5. So, even if ledgers would become bigger, I don't think it would be significant enough after applying compression. On the other hand having 1 ledger per file works well for the "real-time" use case where you want to ingest the latest ledger as quickly as possible.
The BufferedStorageBackend is able to handle a configurable number of ledgers per file so I don't anticipate clients needing to write any code which is dependent on that property. Clients will be able to scan consecutive ledgers using the BufferedStorageBackend and won't need to worry about the underlying storage in S3 / GCS. |
Beta Was this translation helpful? Give feedback.
-
Hopefully folks will write other clients. I wrote client code today in JavaScript in an app. I don't think we can assume all apps will use Go. That's the benefit of writing a SEP, so that folks can integrate with the file format rather than a library. |
Beta Was this translation helpful? Give feedback.
-
|
Did you implement much of the internal/local buffering aspects in js beyond the file storage schema(ledgers-per-file)? BSB go implementation has some pretty thorough retry and resource buffering beyond just parsing the cloud based file storage schema. It may be interesting to have a docs article on how to package the BSB go implementation as-a-service for an alternative to reimplementing the same in other languages. If you think that makes sense I can write up a request ticket on stellar-docs to track that for a docpocalypse idea. |
Beta Was this translation helpful? Give feedback.
-
The implementation is here: https://github.com/leighmcculloch/mcp-stellar-meta. It does caching. It doesn't do retries. Caching and retries are trivial library drop-ins in most languages, so a microservice that wraps up those two things doesn't occur to me as an obvious need. An article with best practices for integrating, mentioning caching, retries, etc would benefit. |
Beta Was this translation helpful? Give feedback.
-
|
For Partition and Batch Formats, rather than expressing in go fmt, can it be plain text like: |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
I find the code sample more helpful than pseudocode. It's difficult to understand in the pseudocode what is string formatting vs arithmetic. For someone without Go experience, the Go is basic enough that it still does the job as good as pseudocode. |
Beta Was this translation helpful? Give feedback.
-
Yes, maybe that's worth a second Note under Key Format to highlight the hex encoding expectation explicitly just in case the reader may not be as aware of the format mask nuance which provides the same. |
Beta Was this translation helpful? Give feedback.
-
The ledgers path seems somewhat unnecessary. The parameter seems to be a way to store the config file in a different location as the ledgers files. But I would think that has little utility. In the examples it's shown like:
Which implies the config is at the root, and then maybe there are ledgers for pubnet and testnet in different directories. However it wouldn't be possible to have two config files at the root where one references testnet and one references pubnet. In another part of the SEP it examples having the config located closer, which I think is more useful and really reduces the need for the ledger path.
It's a minor detail, I don't feel strongly about this, but it seems like an unnecessary field and the config could be located right next to the partitions. |
Beta Was this translation helpful? Give feedback.
-
|
ok, I think I'll revert back to the earlier proposal where the config path is always located in the directory which contains all the partitions. That will remove the need for a configurable config path. There were concerns that this would result in the config file being interspersed with all the partitions in the lexicographic sorting but I don't think that is such a big deal |
Beta Was this translation helpful? Give feedback.
-
|
That's a good point. I see at one point we were considering a fixed /ledgers/ directory with the config.json sitting just outside it? That seems reasonable too if separated was preferred while keeping the config part of the static structure. It might also be reasonable to not place any requirement on relative location, and allow configuring galexie with the path to the config and path to the ledgers separately. Just sharing ideas. |
Beta Was this translation helpful? Give feedback.
-
|
The stellar history archives contain something similar to the config.json, in the form of the "History Archive State" or "HAS" JSON file, and we can learn a little from what that system found helpful to include: When looking at the HAS file format I see it records the For example: {
"version": 1,
"server": "stellar-core 23.0.0.rc4 (dcf3669574084a50f30d0d5f4dbf35af548486b2)",Ref: https://history.stellar.org/prd/core-live/core_live_001/.well-known/stellar-history.json The version of the file format could be the version of the SEP. The name could be a space separated string of the server name and version of the server. |
Beta Was this translation helpful? Give feedback.
-
|
@urvisavla suggested over at #1764 (comment) that individual files get versioned too:
+1 this would be helpful if the plan was to incrementally update a data store, or only use a new format for new files. Do we have plans or ideas for how migrations will take place? There's a few patterns to do versioning in the XDR. The pattern we've used recently is to add an prefix extension point. A prefix extension point lets you either add extensions or convert the struct into a union in the future while maintaining binary backwards compatibility across the change. For example: struct LedgerCloseMetaBatch
{
+ ExtensionPoint ext;
uint32 startSequence;
uint32 endSequence;
LedgerCloseMeta ledgerCloseMetas<>;
};We could also use a union up front if we wanted to instead of an extension point, it'll just introduce that union into generated code but that may never be used if it isn't extended. Depending on the result of this discussion I started in Slack, we might be able to make a change to the XDR to add an extension point before a final 23.0 xdr release is tagged. I'm just not sure what's happened in that repo with its release process because there's a 23.0 release without a rc version component but it is marked as a prerelease in GitHub. If we can't modify it, we could make a |
Beta Was this translation helpful? Give feedback.
-
|
for every file which is uploaded in the data store we set a bunch of key-value fields as custom metadata. These fields include the version of galexie and core which produced the ledger close meta objects: https://github.com/stellar/go/blob/master/services/galexie/internal/ledger_meta_archive.go#L31-L41 I can update the SEP to include this as part of the spec. custom metadata is also supported by S3 , R2, etc. However, there is no obvious analog when it comes to storing ledger files on disk |
Beta Was this translation helpful? Give feedback.
-
|
The current Metadata section states:
This creates a gap for storage backends that don't support user-defined object metadata, such as:
When implementing a filesystem-based datastore for use in stellar/quickstart, I needed a way to store the per-object metadata (start-ledger, end-ledger, protocol-version, etc.) without relying on GCS/S3-specific features. For storage backends that don't support user-defined object metadata, metadata could be stored in a sidecar JSON file adjacent to the data file with a Example: Sidecar file contents: {
"start-ledger": "2",
"end-ledger": "3",
"start-ledger-close-time": "1622547800",
"end-ledger-close-time": "1622548900",
"protocol-version": "21",
"core-version": "v21.0.0",
"network-passphrase": "Public Global Stellar Network ; September 2015",
"compression-type": "zstd",
"version": "1.0.0"
}A diff to add this to SEP-0054 could be something like: ### Metadata
Every `LedgerCloseMetaBatch` value also has the following metadata associated
with it:
- `start-ledger` - (string) the starting ledger of the range of ledgers
enclosed in the `LedgerCloseMetaBatch` value.
- `end-ledger` - (string) the last ledger of the range of ledgers enclosed in
the `LedgerCloseMetaBatch` value.
- `start-ledger-close-time` - (string) the unix time stamp of the close time
for the starting ledger.
- `end-ledger-close-time` - (string) the unix time stamp of the close time for
the last ledger.
- `protocol-version` - (string) the protocol version of the last ledger.
- `core-version` - (string) the version of stellar-core which produced the
ledgers enclosed in the `LedgerCloseMetaBatch` value.
- `network-passphrase` - (string) the network passphrase for the Stellar
network which produced the ledgers.
- `compression-type` - (string) the compression algorithm used to compress the
`LedgerCloseMetaBatch` value (currently only
[`zstd`]([https://facebook.github.io/zstd/) is supported).
- `version` - (string) the version of
[galexie](https://github.com/stellar/go/tree/master/services/galexie) used to
insert the `LedgerCloseMetaBatch` value in the data store.
-These metadata key-value pairs can be implemented as user-defined object
-metadata in GCS or S3.
+#### Storage Methods
+
+Metadata can be stored using one of the following methods:
+
+1. **Native object metadata** (preferred for GCS/S3): Use user-defined object
+ metadata features provided by the storage backend.
+
+2. **Sidecar JSON files** (for filesystems and other backends): Store metadata
+ in a JSON file adjacent to the data file with a `.metadata.json` suffix
+ appended to the full filename (e.g., `FFFFFFFD--2-3.xdr.zst.metadata.json`).
+ The JSON file contains a single object with the metadata keys and string
+ values as specified above.
+
+When writing data with sidecar files, the metadata file SHOULD be written
+before the data file to ensure atomicity guarantees.
+
+When listing files, implementations SHOULD exclude `.metadata.json` files
+from the results.Another alternative might be to say the metadata is optional for other datastores. |
Beta Was this translation helpful? Give feedback.
-
|
@tamirms suggested this morning that we should treat, at least for now, any file system datastore as a test use case only, so the alternative of simply saying, or assuming, that metadata is optional for datastores that do not explicitly support it is more appropriate. |
Beta Was this translation helpful? Give feedback.
-
|
I agree with that. I think metadata can be an optional feature not supported by all datastores |
Beta Was this translation helpful? Give feedback.
-
|
I think it's worth documenting this, although I could also understand if there was a desire to keep this undefined: |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Discussion for #1677
Simple Summary
A standard for how
LedgerCloseMetaobjects can be stored so that ledgers can be easily and efficiently ingested by downstream systems.
Dependencies
None.
Motivation
galexie is a service which publishes
LedgerCloseMetaXDR objects to a GCS(Google Cloud Storage) bucket. However, the data format and layout of the XDR objects are not formally documented. This
SEP aims to provide a comprehensive specification for storing LedgerCloseMeta objects, enabling third-party developers
to build compatible data stores and clients for retrieving ledger metadata.
Specification
The data store is a key-value store where:
LedgerCloseMetaBatchXDR values.The key-value store must support:
Examples of compatible key-value stores include Google Cloud Storage (GCS) and Amazon S3.
Value Format
Each value in the key-value store is the Zstandard compressed binary encoding of
the following XDR structure:
A LedgerCloseMetaBatch represents a contiguous range of one or more consecutive ledgers.
All batches in a data store instance contain the same number of ledgers.
Currently only Zstandard compression is supported but it is possible to extend
the SEP in the future to allow other compression algorithms.
Key Format
Keys follow a hierarchical directory structure, effectively acting as file paths within the data store. All the ledgers are stored under a configurable
/<ledgers-path>. Within the/<ledgers-path>directory there are subdirectories which represent partitions. Each partition contains a fixed number of batches:If the partition size is 1, the partition is omitted, resulting in:
Partition Format:
Example for
partitionStartLedgerSequence=0andpartitionEndLedgerSequence=15:FFFFFFFF--0-15Batch Format:
Example for
batchStartLedgerSequence=2andbatchEndLedgerSequence=3:FFFFFFFD--2-3.xdr.zstIf the batch size is 1, the format simplifies to:
Example for
batchStartLedgerSequence=2:FFFFFFFD--2.xdr.zstNote the
.zstsuffix is the filename extension defined in the ZstandardRFC. If this SEP is extended to support another compression algorithm
then the standard filename extension for the given compression algorithm will be used as a suffix in the batch name.
Configuration File
The data store includes a configuration JSON object stored under the key
/<config-path>. This file contains thefollowing properties:
networkPassphrase- (string) the passphrase for the Stellar network associated with the ledgers.ledgersPath- (string) the/<ledgers-path>directory which contains all the partitions and batches.compression- (string) the compression algorithm used to compress ledger objects (currently onlyzstdis supported).ledgersPerBatch- (integer) the number of ledgers bundled into eachLedgerCloseMetaBatch.batchesPerPartition- (integer) the number of batches in a partition.Example Configuration:
{ "networkPassphrase": "Public Global Stellar Network ; September 2015", "ledgersPath": "/stellar/pubnet/ledgers", "compression": "zstd", "ledgersPerBatch": 2, "batchesPerPartition": 8 }Example Key Structure
Below is an example list of keys (with
/<config-path>set to/stellar/pubnet/config.json) for ledger batches based on the above example configuration:Note: The genesis ledger starts at sequence number 2, so the oldest batch must have a
batchStartLedgerSequenceof 2.
Design Rationale
Key Encoding (Reversed Ledger Sequence)
encoding the most recent ledgers first, clients can efficiently retrieve the latest data without scanning the entire
dataset.
math.MaxUint32 - startLedgerensures that newer ledgers (with higher sequence numbers)appear before older ones when sorted lexicographically. This avoids the need for additional metadata or indexes to
determine the latest ledger.
Compression Algorithm
zstdwas chosen after evaluatingzstd,lz4, andgzip. It provides the best balance between compression ratioand decompression speed.
Security Concerns
Verifying the validity of the ledgers contained within the data store is outside the scope of this SEP. In otherwords,
this SEP does not provide any mechanism for validating that the ledgers obtained from a data store have not been
altered.
Beta Was this translation helpful? Give feedback.
All reactions