Unified Golang SDK for Stellar #1716
Replies: 11 comments 21 replies
-
|
On initial read I really liked this idea and supported it, but I have a couple comments now from a logistical standpoint: +1, I do think both ingest and processors suffer from a lack of discoverability within the go monorepo. Processors more obviously so because the code is directly coupled to Horizon. I am in favor of moving ingest and processors out, plus all these helper functions you describe, but am hesitant on whether moving into an SDK with RPC + Horizon Clients is the right move.
These concerns are largely implementation details and should not inhibit moving forward with this decision. I do not think processors should live within the ingest SDK, and I am supportive of them being a distinct entity, whether that is a top-level concept within the SDK or as a completely separate repo. Imo, we should try and keep these top-level concepts simple and action oriented. A developer should be able to easily determine which part of the SDK to use: I want to ingest raw data, versus, I want to parse [transform] [process] raw data. One of the up sides of putting all this code in one SDK is the horizon and RPC clients could potentially benefit from easy discoverability of processors as well. |
Beta Was this translation helpful? Give feedback.
-
Upside of doing this is it would force us to archive or specify true owners of services that continue to live on in the monorepo largely by happenstance when in reality they are not actively maintained. |
Beta Was this translation helpful? Give feedback.
-
|
I thought part of the win of putting the SDK in its own repo would be a small narrower dependency graph so that imports of the lib don't end up with a massive dep graph too. I just did a quick look at what the dep graph would look like removing just services, and ~33 deps, meaning the repo would still have 135 deps.
Because of the large dep graph, there might be a lot of "app code" hiding in utilities/libraries that while useful and maybe could be shared, really need to be removed from the "go sdk" to make the sdk hyper-focused on the things folks need when building on Stellar. |
Beta Was this translation helpful? Give feedback.
-
+1 I think this plan will benefit folks overall. There's tradeoffs every way we go. But Go's tooling is designed well to handle multiple packages in a shared repository. And while someone might develop with only a part of the SDK, they're likely to develop with multiple. A dev using RPC, Ingest, or Horizon will likely use the XDR and strkey utilities, and maybe the hash and even txnbuild utilities if they parse transactions into objects. There's a lot of overlap. As far as I can tell there's no optimisation or compiler reason to separate these packages into separate modules like exist in some other ecosystems like the Rust ecosystem, where smaller crates are preferred because of the impact on compile times of mega crates. It's worth mentioning we went a different way with the Rust libraries and it seems to be working out just fine:
However, the Rust tooling lets us do this hybrid/mixed approach, while still shipping separate micro crates, so the hybrid repos which is a good balance for maintainers doesn't bleed into a developers experience of using the crates. Someone can just use the The same hybrid approach doesn't really work in Go because the import paths make the structure fully visible in every go file someone imports it. So unless we plan to do some sort of micro-repo split up, I think a single Go SDK presents the best balance for both maintainers, and developers discovering functionality. As mentioned in https://github.com/stellar/go/discussions/5651#discussioncomment-12759163, we have to work hard to present all the meaningful components of the Go SDK, while hiding / * The xdr and strkey Rust libs live in their own repos because we wanted the xdr lib to be available instantly for any updated xdr without needing to update any applications or anything else. The strkey lib ended up in its own repo mostly because it was one of the early ones and we had no rust repos. Both are also included into core and have a different level of risk to changes. Many libs live in the sdk and env repos because they largely change at the same time as those products. This mix of approaches for the rust libs works just fine because the import name/path of the libs doesn't use the repos, so it doesn't matter where we store it. And so far we haven't seen any negative impact wrt contributions from others. The main place folks interact with docs is also on crates.io and docs.rs rather than via GitHub, so GitHub isn't the entry point. |
Beta Was this translation helpful? Give feedback.
-
|
I'm fully supportive of creating a unified Golang SDK, but I think it is worth reframing our justification for doing so. Yes, we want the ecosystem to transition from using Horizon's higher-level abstractions to using Stellar's underlying data model which is exposed through other means such as RPC, data lakes, and captive core. However, I'm skeptical that our ingest SDK as it exists today is the best solution for doing so, and I don't think we should be creating a Golang SDK primarily for the sake of promoting the ingest SDK. We should refactor our monorepo to create a Golang SDK for its own sake, so that our golang tooling is consistent with the rest of our ecosystem. The scope should be the same as the scope of the rest of the SDKs:
Once we have a proper Golang SDK, we can start thinking about how to increase the scope of all of our SDKs to support consuming raw network data and design the interface in a way that is consistent with how the SDKs are designed. I'd argue that the Ingest SDK consists of a few different features we should consider decomposing and integrating into the SDKs in different ways:
IMO, we could start introducing pieces of the features within the Ingest SDK into the normal SDKs iteratively rather than adding the whole thing as its own more-or-less standalone package. I would think the first step is to introduce a captive core client and data lake client in the same way we have Horizon and RPC clients. |
Beta Was this translation helpful? Give feedback.
-
+1 to relocation of processor implementations(
The relocation of processor implementations out of current ingest sdk to another repo could be an opportunity in it's wake to define generic interfaces for processor and event in the proposed Golang SDK to promote consistent approach for consuming network data with stream oriented, event driven processor implementations which follow typical roles(source, sink, transformer) with inbound/outbound events. |
Beta Was this translation helpful? Give feedback.
-
|
in the unified sdk proposal, there was mention in team discussion of it providing an abstraction layer first which provides interfaces/functions/classes/data structs which constitute a client facing facade. The facade would have intelligence internally to know when to route client requests to existing sdk or api's such as rpc client, ingest sdk(ledger backends for datastore or captive core), data warehouse. So clients don't have to maintain the know-how to choose, they just request the data and get back the appropriate derived model related to the request, i.e. This sounds like an interesting angle to explore if it can hold up against the various client use cases which span from tx-sub, to range of queries from current state to very aged historical timeframes. We may be able to mock up the facade as a proto v3 IDL file which allows defining rpc methods and data models and leverage the built-in streaming transports on request/responses also. |
Beta Was this translation helpful? Give feedback.
-
|
❗Feedback needed ❗ We have moved forward with this initiative and signification changes to the Go monorepo have already taken place. The full list of changes is being tracked here, but the TLDR is:
This all effectively means that all major structural changes that we wanted to make, have been done. So far, none of these changes should be "breaking" for anyone depending on the Go monorepo, aside from anyone that was building one of these services from source, in which case they'll just need to build from the new repo. However, we'd like more input on a potentially significantly more breaking change that we're considering. That change is, renaming the repo itself from go -> go-stellar-sdk. If we were to do such a change, this would mean:
Pros
Cons
|
Beta Was this translation helpful? Give feedback.
-
|
Starting a thread to discuss versioning for the new go / go-stellar-sdk, whatever it is called.
I don't see discussion here about versioning other than the comment above (#1716 (comment)) that assumes the SDK will continue to use protocol versioning. I think we should consider the new go / go-stellar-sdk starting with a new v1 version, and like most libraries in the Go ecosystem consider if we could endeavour to never change the major version. Many of the tools and services are versioned with their protocol version because most of the time they need updating for every protocol in breaking ways — either XDR exposed changes, or big migrations are needed to deal with protocol changes — and so it gives us a coordinated point in time to limit those breaking experiences to and makes it easy to see if developers and operators are using compatible software. However, when we introduced protocol versioning @mollykarcher and @Shaptic pointed out that those benefits would be better to trade away for SDKs. The js-stellar-sdk as a result still uses its own semver versioning scheme, and releases major versions on its own timeline. The only reason the go sdk uses protocol versioning today was apparently due to its proximity to Horizon. SDKs benefit from releasing minimal major versions over time, especially Go libraries because updating a Go library major version requires every importer changing every file the import is used. The only thing that breaks for SDKs at new protocols is the XDR types they contain, but for each ecosystem that may have little or a lot of impact. For the Rust ecosystem the impact is high because of the rigidity of its type system and the exhaustive checks the compiler performs on enums, which is why we protocol version the xdr lib and everything downstream of it. For the JS, Go, and many other ecosystems the fact that an XDR type gets an additional union or enum can typically go unnoticed if an application doesn't interact with it, and so issuing major versions for these SDKs is not required. Regardless of what name the go sdk uses, either |
Beta Was this translation helpful? Give feedback.
-
|
Thanks all for your engagement here! We'll be closing this discussion as all of the planned changes related to this have been rolled out. If you're a go repo user, make sure to update: https://github.com/stellar/go/blob/master/MIGRATION.md |
Beta Was this translation helpful? Give feedback.
-
|
I'm a user of the Go SDK, and for the record, this change is very much not welcome for me. Please don't take the habit of introducing breaking changes just to make things more pretty. Code built on Stellar's Go SDK has built for years without requiring maintenance, this was making it exceptional. Anyway, thanks for your work, I just thought you had to know not all users liked this. |
Beta Was this translation helpful? Give feedback.


Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
What problem does your idea address?
While the Ingest SDK has been it's own package/library for years, it has little->no usage outside of the Platform and Data teams at SDF. There is a rust port of it that xycloo labs uses as well. But aside from this, even internally to SDF, the ingest SDK has seen little usage and poor discoverability/branding. It was not until very recently that it had first-class treatment in the developer docs, and it has become clear that it's not obvious to partners/ecosystem members that these tools are available to them.
While seemingly minor, this has implications that ripple back to the Horizon -> RPC (+CDP) shift we'd like to see the ecosystem make. The API that is XDR/LCM is far more raw/minimalistic than the indexed APIs Horizon provides, which means application developers will need to be parsing and indexing their own data out of LCM. If there aren't tools available to them that make this easier, then it's likely they'd choose to remain on Horizon. We also know that this has hampered Galexie adoption for several data/analytics providers that have onboarded within the last year.
We'd also like to expand the scope of responsibility for our external SDK developers that receive infrastructure grants, to include the ability to interact with a Galexie-exported data lake. Previously, we had imagined this being an ask to do a direct port of the Ingest SDK. However, realistically all we likely need them to do is an implementation of this eventual SEP. So to that end, all we realistically need is an understandable structure within the Ingest SDK so that we can reference back to it as an example of that SEP implementation.
What would you like to see?
Build out a new, independent repository that houses any and all utilities that someone integrating with Stellar and developing in golang could benefit from. It would not be opinionated about where they are getting their data (Captive Core, data lake, RPC, Horizon, etc), and the general scope would just be anything that could be useful for a golang developer developing on Stellar. This would include:
Having all of these things in a single place would help with branding and discoverability. This structure would also align with how most other Stellar SDKs are structured. At present, the Go SDK is the only one that does not co-locate the Horizon and RPC clients. We have discussed in the past asking SDK providers to support pieces of functionality from the ingest SDK, and if they were to do so, then in all likelihood that would end up in the same repositories as all the existing code.
Separately, I'd like to see us reassess if the processors should live within the ingest SDK, or if they could live as their own top-level concept within this now larger SDK. The motivation for this would be that they are somewhat of a higher-level or application-level concept. In the original plan for CDP we had planned on separate concepts for Ledger Backend (equivalent to "extract" in ETL architectures) and Transforms (aka processors, equivalent to "transform" in ETL architectures). We then pushed both of these into the existing "Ingest SDK", but I think it could be valuable for them to have independent identities.
What alternatives are there?
In the past, we've debated and explored many other alternative ideas around monorepo/ingest SDK structure that get at the same end goal. For example:
Beta Was this translation helpful? Give feedback.
All reactions