Stellar
Unified Golang SDK for Stellar #1716
Replies: 11 comments 21 replies
-
|
On initial read I really liked this idea and supported it, but I have a couple comments now from a logistical standpoint: +1, I do think both ingest and processors suffer from a lack of discoverability within the go monorepo. Processors more obviously so because the code is directly coupled to Horizon. I am in favor of moving ingest and processors out, plus all these helper functions you describe, but am hesitant on whether moving into an SDK with RPC + Horizon Clients is the right move.
These concerns are largely implementation details and should not inhibit moving forward with this decision. I do not think processors should live within the ingest SDK, and I am supportive of them being a distinct entity, whether that is a top-level concept within the SDK or as a completely separate repo. Imo, we should try and keep these top-level concepts simple and action oriented. A developer should be able to easily determine which part of the SDK to use: I want to ingest raw data, versus, I want to parse [transform] [process] raw data. One of the up sides of putting all this code in one SDK is the horizon and RPC clients could potentially benefit from easy discoverability of processors as well. |
Beta Was this translation helpful? Give feedback.
-
FWIW, there is no well-maintained Rust SDK for offchain clients. |
Beta Was this translation helpful? Give feedback.
-
|
The risk with both approaches is that ingest doesn't get sunlight to be discoverable. I think with either approach this is possible, and so we have to work hard to document and present the SDK and its components, or the separate repos, well. |
Beta Was this translation helpful? Give feedback.
-
Yeah what Jake said. I don't think there was any reason other than that the product that was writing the rust port (xycloo labs) was not currently maintained a rust API client SDK for horizon/RPC, nor did they need those things to support their products.
Both of these points are fair and regardless of the path we choose, we'd definitely have to combat with solid documentation and package structuring. Ingest definitely serves a slightly different purpose, but one could argue that it's also an API client, where the API is LedgerCloseMeta. |
Beta Was this translation helpful? Give feedback.
-
Upside of doing this is it would force us to archive or specify true owners of services that continue to live on in the monorepo largely by happenstance when in reality they are not actively maintained. |
Beta Was this translation helpful? Give feedback.
-
|
I thought part of the win of putting the SDK in its own repo would be a small narrower dependency graph so that imports of the lib don't end up with a massive dep graph too. I just did a quick look at what the dep graph would look like removing just services, and ~33 deps, meaning the repo would still have 135 deps.
Because of the large dep graph, there might be a lot of "app code" hiding in utilities/libraries that while useful and maybe could be shared, really need to be removed from the "go sdk" to make the sdk hyper-focused on the things folks need when building on Stellar. |
Beta Was this translation helpful? Give feedback.
-
+1 I think this plan will benefit folks overall. There's tradeoffs every way we go. But Go's tooling is designed well to handle multiple packages in a shared repository. And while someone might develop with only a part of the SDK, they're likely to develop with multiple. A dev using RPC, Ingest, or Horizon will likely use the XDR and strkey utilities, and maybe the hash and even txnbuild utilities if they parse transactions into objects. There's a lot of overlap. As far as I can tell there's no optimisation or compiler reason to separate these packages into separate modules like exist in some other ecosystems like the Rust ecosystem, where smaller crates are preferred because of the impact on compile times of mega crates. It's worth mentioning we went a different way with the Rust libraries and it seems to be working out just fine:
However, the Rust tooling lets us do this hybrid/mixed approach, while still shipping separate micro crates, so the hybrid repos which is a good balance for maintainers doesn't bleed into a developers experience of using the crates. Someone can just use the The same hybrid approach doesn't really work in Go because the import paths make the structure fully visible in every go file someone imports it. So unless we plan to do some sort of micro-repo split up, I think a single Go SDK presents the best balance for both maintainers, and developers discovering functionality. As mentioned in https://github.com/stellar/go/discussions/5651#discussioncomment-12759163, we have to work hard to present all the meaningful components of the Go SDK, while hiding / * The xdr and strkey Rust libs live in their own repos because we wanted the xdr lib to be available instantly for any updated xdr without needing to update any applications or anything else. The strkey lib ended up in its own repo mostly because it was one of the early ones and we had no rust repos. Both are also included into core and have a different level of risk to changes. Many libs live in the sdk and env repos because they largely change at the same time as those products. This mix of approaches for the rust libs works just fine because the import name/path of the libs doesn't use the repos, so it doesn't matter where we store it. And so far we haven't seen any negative impact wrt contributions from others. The main place folks interact with docs is also on crates.io and docs.rs rather than via GitHub, so GitHub isn't the entry point. |
Beta Was this translation helpful? Give feedback.
-
|
I'm fully supportive of creating a unified Golang SDK, but I think it is worth reframing our justification for doing so. Yes, we want the ecosystem to transition from using Horizon's higher-level abstractions to using Stellar's underlying data model which is exposed through other means such as RPC, data lakes, and captive core. However, I'm skeptical that our ingest SDK as it exists today is the best solution for doing so, and I don't think we should be creating a Golang SDK primarily for the sake of promoting the ingest SDK. We should refactor our monorepo to create a Golang SDK for its own sake, so that our golang tooling is consistent with the rest of our ecosystem. The scope should be the same as the scope of the rest of the SDKs:
Once we have a proper Golang SDK, we can start thinking about how to increase the scope of all of our SDKs to support consuming raw network data and design the interface in a way that is consistent with how the SDKs are designed. I'd argue that the Ingest SDK consists of a few different features we should consider decomposing and integrating into the SDKs in different ways:
IMO, we could start introducing pieces of the features within the Ingest SDK into the normal SDKs iteratively rather than adding the whole thing as its own more-or-less standalone package. I would think the first step is to introduce a captive core client and data lake client in the same way we have Horizon and RPC clients. |
Beta Was this translation helpful? Give feedback.
-
+1 to relocation of processor implementations(
The relocation of processor implementations out of current ingest sdk to another repo could be an opportunity in it's wake to define generic interfaces for processor and event in the proposed Golang SDK to promote consistent approach for consuming network data with stream oriented, event driven processor implementations which follow typical roles(source, sink, transformer) with inbound/outbound events. |
Beta Was this translation helpful? Give feedback.
-
+1 here - when i think ingestion, i think of simply the means to consume data - not transforming it. Higher level processors like TTP provide users with a way to derive value from that extracted data, which we should separate and market as such. |
Beta Was this translation helpful? Give feedback.
-
|
in the unified sdk proposal, there was mention in team discussion of it providing an abstraction layer first which provides interfaces/functions/classes/data structs which constitute a client facing facade. The facade would have intelligence internally to know when to route client requests to existing sdk or api's such as rpc client, ingest sdk(ledger backends for datastore or captive core), data warehouse. So clients don't have to maintain the know-how to choose, they just request the data and get back the appropriate derived model related to the request, i.e. This sounds like an interesting angle to explore if it can hold up against the various client use cases which span from tx-sub, to range of queries from current state to very aged historical timeframes. We may be able to mock up the facade as a proto v3 IDL file which allows defining rpc methods and data models and leverage the built-in streaming transports on request/responses also. |
Beta Was this translation helpful? Give feedback.
-
|
Indeed there was not. I only just started a new discussion on that over here: https://github.com/stellar/stellar-rpc/discussions/401 I was actually internally debating what you're getting at as I was writing that up, but I was thinking about it in a much less broad/sweeping way than you are proposing. We were discussing a "facade", and I was actually imagining that to be a facade service, but the facades could also be the clients/SDKs, which would intelligently route to the right service/API. So like in that linked discussion, when debating between options 1 + 2, there's little difference IMO aside from the fact that 2 is moving the problem of creating a "unified API" to the client level, rather than unifying around RPC's existing API. As it relates to this discussion, I was originally not viewing "unified golang SDK" as being anything more than shoving all the things together in another repo somewhere. So I wasn't proposing adding any additional facades/APIs to effectively hide away from the client/SDK user what API they are hitting. I have to think more about it, but I don't think we'd want to do this in a big sweeping way, but it could definitely be a useful tool if used in some surgical/targeted ways. The main one I'm thinking of, is if we were to force some bits of the Horizon -> RPC migration by just merely changing the implementations of the existing clients (ignoring backwards compatibility concerns just for the thought experiment). |
Beta Was this translation helpful? Give feedback.
-
|
Yes, picking the layer where the intelligence of choosing the appropriate backend/sources resides is key, it could conceptually be in the client side as SDK code or on server side with existing and new endpoints. On the SDK side, the facade could be a helper style SDK package on the side, which some clients opt into, or they can go around it and use the existing SDK's which would be present publicly in the SDK also. This has some similarity with client-side load balancing pattern, not sure if that is good! Or this intelligence to determine which back-ends to use for resolving data requests resides as a service layer in RPC endpoints or as a new smart-proxy service, which can see similar in architecture to GraphQL. |
Beta Was this translation helpful? Give feedback.
-
|
❗Feedback needed ❗ We have moved forward with this initiative and signification changes to the Go monorepo have already taken place. The full list of changes is being tracked here, but the TLDR is:
This all effectively means that all major structural changes that we wanted to make, have been done. So far, none of these changes should be "breaking" for anyone depending on the Go monorepo, aside from anyone that was building one of these services from source, in which case they'll just need to build from the new repo. However, we'd like more input on a potentially significantly more breaking change that we're considering. That change is, renaming the repo itself from go -> go-stellar-sdk. If we were to do such a change, this would mean:
Pros
Cons
|
Beta Was this translation helpful? Give feedback.
-
|
One additional step that should happen is the |
Beta Was this translation helpful? Give feedback.
-
If we do a straight up rename, and then create a new repo named |
Beta Was this translation helpful? Give feedback.
-
|
I have a slight preference towards not renaming the repo. I think if we add "SDK" to the description of the github repo it will make stellar/go appear in the github search results. This is how the anchor-platform github repo shows up in the "SDK" search results: 
I think if we update the github repo description and our public docs that should make the go sdk sufficiently discoverable. |
Beta Was this translation helpful? Give feedback.
-
|
Starting a thread to discuss versioning for the new go / go-stellar-sdk, whatever it is called.
I don't see discussion here about versioning other than the comment above (#1716 (comment)) that assumes the SDK will continue to use protocol versioning. I think we should consider the new go / go-stellar-sdk starting with a new v1 version, and like most libraries in the Go ecosystem consider if we could endeavour to never change the major version. Many of the tools and services are versioned with their protocol version because most of the time they need updating for every protocol in breaking ways — either XDR exposed changes, or big migrations are needed to deal with protocol changes — and so it gives us a coordinated point in time to limit those breaking experiences to and makes it easy to see if developers and operators are using compatible software. However, when we introduced protocol versioning @mollykarcher and @Shaptic pointed out that those benefits would be better to trade away for SDKs. The js-stellar-sdk as a result still uses its own semver versioning scheme, and releases major versions on its own timeline. The only reason the go sdk uses protocol versioning today was apparently due to its proximity to Horizon. SDKs benefit from releasing minimal major versions over time, especially Go libraries because updating a Go library major version requires every importer changing every file the import is used. The only thing that breaks for SDKs at new protocols is the XDR types they contain, but for each ecosystem that may have little or a lot of impact. For the Rust ecosystem the impact is high because of the rigidity of its type system and the exhaustive checks the compiler performs on enums, which is why we protocol version the xdr lib and everything downstream of it. For the JS, Go, and many other ecosystems the fact that an XDR type gets an additional union or enum can typically go unnoticed if an application doesn't interact with it, and so issuing major versions for these SDKs is not required. Regardless of what name the go sdk uses, either |
Beta Was this translation helpful? Give feedback.
-
|
@leighmcculloch do we have to bump the version in the module name? I see that stripe does that: https://github.com/stripe/stripe-go/blob/master/go.mod#L1 But, I'm wondering why we can't do the versioning using git tags alone, e.g. |
Beta Was this translation helpful? Give feedback.
-
Yup. Every major version changes the import path. The major version in a semver version has to be the suffix of the import path, it has to match in the Afaik the only way around that is to do what we were doing before and a non-standard tag name like |
Beta Was this translation helpful? Give feedback.
-
could you describe these rules or link to some documentation about it? The downsides you've pointed out about having to update every import statement with the new version is clear. But, I'm not familiar with what the benefits are of using this approach.
Could you elaborate on why that is a poor developer experience? |
Beta Was this translation helpful? Give feedback.
-
|
Documentation about go modules and managing major versions:
If go-stellar-sdk tags
The go sdk has been a poor experience in these ways for a long time, and it would be great to clean this up with the new sdk. |
Beta Was this translation helpful? Give feedback.
-
|
@leighmcculloch now that I understand all the pain points, I think your original proposal makes sense. If consumers of the sdk will have to update their import paths then we might as well rename the repo while we're at it |
Beta Was this translation helpful? Give feedback.
-
|
Thanks all for your engagement here! We'll be closing this discussion as all of the planned changes related to this have been rolled out. If you're a go repo user, make sure to update: https://github.com/stellar/go/blob/master/MIGRATION.md |
Beta Was this translation helpful? Give feedback.
-
|
I'm a user of the Go SDK, and for the record, this change is very much not welcome for me. Please don't take the habit of introducing breaking changes just to make things more pretty. Code built on Stellar's Go SDK has built for years without requiring maintenance, this was making it exceptional. Anyway, thanks for your work, I just thought you had to know not all users liked this. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for taking the time to share this, and very sorry to hear it wasn't welcome. Obviously you've found and read this discussion so you're probably aware, but a big part of the motivation behind the repo restructure was to fix long-standing issues with discoverability, documentation, and versioning that have created their own kind of maintenance burden. We don't often get much feedback from developers outside of SDF (as is evidenced by this discussion itself), or partners that we work directly with on integrations (which is skewed towards new/recent projects). So I'm genuinely grateful for feedback like yours, it's what keeps us grounded in the real experience of SDK users. We do hope to maintain the tradition of keeping breaking changes to an absolute minimum, so if you do choose to move over to |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, great to hear backward compatibility stays a concern for you guys! And yeah, sorry, I guess this is an other example of "you only hear from users when they have problems". I think it's worth saying that the Go SDK have been working wonders for me. I've implemented my own wallet with it, then trading features for the SDEX, going more and more advanced, and it never got in the way. It is a perfect case of "boring in the right way", software that just works. It's not just the Go SDK either. I've been using the Horizon HTTP API directly to fetch and store my data locally, and the stellar-xdr repos to find the definition of the XDR types (I made a SQLite extension to store my data as XDR and decode it as JSON on the fly so I can store in XDR, query in JSON), it's all been a breeze, with everything being properly documented and legible. Good job to everyone involved. :) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
What problem does your idea address?
While the Ingest SDK has been it's own package/library for years, it has little->no usage outside of the Platform and Data teams at SDF. There is a rust port of it that xycloo labs uses as well. But aside from this, even internally to SDF, the ingest SDK has seen little usage and poor discoverability/branding. It was not until very recently that it had first-class treatment in the developer docs, and it has become clear that it's not obvious to partners/ecosystem members that these tools are available to them.
While seemingly minor, this has implications that ripple back to the Horizon -> RPC (+CDP) shift we'd like to see the ecosystem make. The API that is XDR/LCM is far more raw/minimalistic than the indexed APIs Horizon provides, which means application developers will need to be parsing and indexing their own data out of LCM. If there aren't tools available to them that make this easier, then it's likely they'd choose to remain on Horizon. We also know that this has hampered Galexie adoption for several data/analytics providers that have onboarded within the last year.
We'd also like to expand the scope of responsibility for our external SDK developers that receive infrastructure grants, to include the ability to interact with a Galexie-exported data lake. Previously, we had imagined this being an ask to do a direct port of the Ingest SDK. However, realistically all we likely need them to do is an implementation of this eventual SEP. So to that end, all we realistically need is an understandable structure within the Ingest SDK so that we can reference back to it as an example of that SEP implementation.
What would you like to see?
Build out a new, independent repository that houses any and all utilities that someone integrating with Stellar and developing in golang could benefit from. It would not be opinionated about where they are getting their data (Captive Core, data lake, RPC, Horizon, etc), and the general scope would just be anything that could be useful for a golang developer developing on Stellar. This would include:
Having all of these things in a single place would help with branding and discoverability. This structure would also align with how most other Stellar SDKs are structured. At present, the Go SDK is the only one that does not co-locate the Horizon and RPC clients. We have discussed in the past asking SDK providers to support pieces of functionality from the ingest SDK, and if they were to do so, then in all likelihood that would end up in the same repositories as all the existing code.
Separately, I'd like to see us reassess if the processors should live within the ingest SDK, or if they could live as their own top-level concept within this now larger SDK. The motivation for this would be that they are somewhat of a higher-level or application-level concept. In the original plan for CDP we had planned on separate concepts for Ledger Backend (equivalent to "extract" in ETL architectures) and Transforms (aka processors, equivalent to "transform" in ETL architectures). We then pushed both of these into the existing "Ingest SDK", but I think it could be valuable for them to have independent identities.
What alternatives are there?
In the past, we've debated and explored many other alternative ideas around monorepo/ingest SDK structure that get at the same end goal. For example:
Beta Was this translation helpful? Give feedback.
All reactions