SDKs Binding Generation Events

[leighmcculloch]

Support for events has arrived in SEP-48 Contract Specs in the XDR shipping with protocol 23, and it is currently deployed on Testnet.

The Rust Soroban SDK has support for it via the contractevent macro in version 23.0.0-rc.2.2:

• https://docs.rs/soroban-sdk/23.0.0-rc.2.2/soroban_sdk/attr.contractevent.html

There are a few opportunities that are now available to the tooling ecosystem with regards to working with events in code and data systems:

1. The opportunity for SDKs to offer binding generation for events, so that given a raw event, and given the .wasm that produced the event containing the spec, tooling can generate parsers that unpack the raw event into a code structure that is self-describing, validated, and easy for developers to use.

2. The opportunity to write binding generation to not just code, but also code generate IDL of alternative file formats, such as protobuf, so that any contract can be converted with no custom code.

3. The opportunity to generate processors for the Go SDK's Ingest capability, or to hook these generated types into a generic events processor, such that processes like the Token Transfer Processor could be generated for any contract, or groups of contracts.

There's also an opportunity to present spec informed views of events in tooling UIs, such as Lab (stellar/laboratory#1512), Stellar CLI (stellar/stellar-cli#2086), OpenZeppelin Monitor (OpenZeppelin/openzeppelin-monitor#318), etc, but those are being discussed on their respective issues and this conversation here is intended to focus on coding primitives.

cc @stellar/platform-eng @stellar/data-eng @stellar/devx

To illustrate with an example:

An approve event as defined in SEP-41, can be defined in the upcoming SDK with:

#[contractevent(data_format = "vec")]
pub struct Approve {
    #[topic]
    pub from: Address,
    #[topic]
    pub spender: Address,
    pub amount: i128,
    pub live_until_ledger: u32,
}

The raw event emitted in ledger close meta contains all the values, but none, or little of, the structure:

{
  "ext": "v0",
  "contract_id": "CBUZJXHZ6PBS2YR3SEJZ3CIGMQBYP6367D3KQAR2NB3U2I5AOWLC4DU2",
  "type_": "contract",
  "body": {
    "v0": {
      "topics": [
        { "symbol": "approve" },
        { "address": "GBMBVAHBE6D4AJXJJVTBQTVU4G7SN4FEIJOL5YTOHZ4WCUMKQ52ANL2B" },
        { "address": "GCAN5IE4PWMWSMFZCYQRCJM73MCPG5JD2R7WF5HIFHHHDBWSDJFIWX7X" }
      ],
      "data": {
        "vec": [
          { "i128": "10000" },
          { "u32": 1998742 }
        ]
      }
    }
  }
}

Contract specs include the shape and structure of the event:

{
  "event_v0": {
    "name": "approve",
    "prefix_topics": ["approve"],
    "params": [
      { "name": "from", "type_":"address", "location": "topic_list" },
      { "name": "spender", "type_":"address", "location": "topic_list" },
      { "name": "amount", "type_":"i128", "location": "data" },
      { "name": "live_until_ledger", "type_":"u32", "location": "data" }
    ],
    "data_format": "vec"
  }
}

Which would allow SDKs, such as the JS SDK, Go SDK, Python SDK, etc to offer code generated types that can parse the raw contract event into the code structure:

type Approve struct {
    From            Address
    Spender         Address
    Amount          Int128
    LiveUntilLedger uint32
}

Or, for applications that would like to parse the events into other file formats, generating protobuf, or other IDLs along with the parsing logic.

message Approve {
  string from = 1;
  string spender = 2;
  string amount = 3;
  int live_until_ledger = 4;
}

The intent of this issue is to determine if adding the above types of code generation to SDKs make sense, all or some, which ones, and to brainstorm other ways SDKs or data tooling could make use of them.

There's an example prototype for how to do matching of raw contract events to specs here:

• https://github.com/leighmcculloch/exp-sep48-derived-events-example

[Shaptic]

I think this certainly makes sense for the JS SDK; the looser type system lends itself well to ScVals (e.g., bigint vs. no xdr.Int128 equivalent in Golang, mixed-type maps, etc.) and it would make consuming events super easy. It makes sense for this to go hand-in-hand with stellar/js-stellar-sdk#1163 so that both bindings and events could be generated from within the SDK itself.

We already some plumbing in place for using the spec for the bindings, though there'd be a decent lift to use ScSpecEventDataFormat and friends.

[Shaptic]

Here's a proof of concept that only works for simple types (so far) and the default data_format. There's a lot more work needed to turn this into a full-featured script but it's cool to easy that the spec definitely works as advertised 👍

[overcat]

I think I can extend stellar-contract-bindings to add corresponding support.

[sreuland]

The opportunity to write binding generation to not just code, but also code generate IDL of alternative file formats, such as protobuf, so that any contract can be converted with no custom code.

I'm trying to understand how IDL codegen of events or other XDR spec models will result in net gain for apps.

Since the applications will start with ledger metadata in XDR or optionally obtainable as JSON, neither of those formats can be directly deserialized by other IDL serialization libraries as each requires their own proprietary wire(serialization) format. It seems like applications will always have to have the XDR (de)serialization bindings first to use on metadata such as SDK binding approaches have illustrated here.

The application could establish a second IDL which it generates from the meta specs. But at that point the application would need to run an adapter pattern with both de(ser) code libs and generate new instances of second IDL objects from deserialized XDR objects. One benefit at that point is the removal of XDR dependency and the data is now portable purely in second IDL.

[leighmcculloch]

The idea is for example is that a code gen tool could generate .proto files, along with code for converting Go XDR to Go Protos that are code generated by the protoc tool. I don't see a reason that the codegen tool couldn't generate just Go XDR <> Go Types bindings, but could do so for types that implement other file formats like protobuf. What do you see as the blocker for that?

[sreuland]

That sounds like chicken before egg or at least this is how I was understanding the flow - When the code gen tool generates the .proto, then next step would need to generate bindings from the .proto and then how will the code gen automate that last step of '...along with code for converting Go XDR to Go Protos'? That seems like will be an intricate step to automate as there's no guarantee on the generated stubs having similar signatures between <Language> XDR and <Language> Proto. Am assuming this last step for conversion from XDR binding to Proto binding would ultimately need to be manually coded by a person.