docs: clarify advanced extensions #833

wackywendell · 2025-07-25T20:43:46Z

The Advanced Extensions section of the docs seemed somewhat mismatched from what was in the protobufs, so I tried to clarify it, based on what was in the protobuf definitions.

Thoughts welcome! Also could use some sanity-checking on the details!

Naming

I also included custom relations (ExtensionLeafRel, …) and custom reads and writes (ExtensionTable, …) all under the heading "Advanced Extensions", even though there is an AdvancedExtension message that doesn't cover those. The name "Advanced Extension" seems a bit ambiguous here - does it cover all of the above, or only the enhancements and optimizations in the AdvancedExtension message? - but it seems to be what we have, so I went with it. Thoughts?

Guidance

This is a bit low on guidance on when to use which - e.g. ExtensionLeafRel, ExtensionTable, or ReadRel.advanced_extension.enhancement could potentially all be used for an unusual kind of read. I'm not sure what the guidance here should be, or if there should be any, so I left it out; I'm not sure any guidance here would be that helpful.

yongchul · 2025-07-29T23:03:31Z

site/docs/extensions/index.md

+
+Advanced extensions come in several main forms, discussed below:
+
+1. Embedded extensions: These use the `AdvancedExtension` message for adding custom data to existing Substrait elements


I'm not sure or don't remember whether element is commonly used across the doc. Now that I check again, advanced extensions are generously sprinkled over the proto file... :) That said, it would be clearer to mention the scope of the embedded extension like adding custom data to enclosing Substrait message (or element).

The above mentioned protobuf and call out Any. So I'm wondering whether we can stick to Substraite message rather than using element.

Makes sense, message makes more sense than element for consistency!

yongchul · 2025-07-29T23:06:13Z

site/docs/extensions/index.md

+| **`RelCommon`**                   | Extensions for any relational operator      |
+| **Relations** (e.g. `ProjectRel`) | Extensions for a specific relation type     |
+| **Hints**                         | Extensions within optimization hints        |
+| **`ReadRel.NamedTable`**          | Add custom metadata to named table references |


nit: remove Add to be consistent with the rest of Usages? (i.e., either all starts with verbs or nouns)

yongchul · 2025-07-29T23:07:54Z

site/docs/extensions/index.md

+| **`RelCommon`**                   | Extensions for any relational operator      |
+| **Relations** (e.g. `ProjectRel`) | Extensions for a specific relation type     |
+| **Hints**                         | Extensions within optimization hints        |
+| **`ReadRel.NamedTable`**          | Add custom metadata to named table references |


looks like LocalFiles also has advanced extension.

yongchul · 2025-07-29T23:11:36Z

site/docs/extensions/index.md

+| **Hints**                         | Extensions within optimization hints        |
+| **`ReadRel.NamedTable`**          | Add custom metadata to named table references |
+| **`WriteRel.NamedObjectWrite`**   | Add custom metadata to write targets        |
+| **`DdlRel.NamedObjectWrite`**     | Add custom metadata to DDL targets          |


noticed some of the advanced extension tags are not consistent in these Ddl related messages. Just note to self.

site/docs/extensions/index.md

wackywendell · 2025-07-30T20:47:55Z

@yongchul - thanks for the review! Good comments - I've updated to address them. Can you give them a look?

Thanks!

yongchul

Thank you for taking suggestions! Looks good to me :)

site/docs/extensions/index.md

wackywendell · 2025-07-31T19:38:52Z

Thanks again! Incorporated. I also realized I had the !!! note sections incorrect (they needed indenting), so I fixed that, and ran mkdocs locally to verify it worked - and it did!

wackywendell · 2025-11-18T19:27:51Z

^ Thanks, @yongchul for the merge! I'm wondering - what do we need to do next to advance this? Do we need additional reviewers so we can get it approved and merged?

yongchul · 2025-11-19T00:42:15Z

^ Thanks, @yongchul for the merge! I'm wondering - what do we need to do next to advance this? Do we need additional reviewers so we can get it approved and merged?

Thank you for your patience! We will need another SMC review I believe. I'll bring this up in the sync up.

site/docs/extensions/index.md

- Update proto syntax to use new %%% format - Reorder sections to prioritize custom read/write types over custom relations

benbellick

Left some minor comments but this is a welcome change!

site/docs/extensions/index.md

benbellick · 2025-11-20T16:30:23Z

site/docs/extensions/index.md

+!!! note "Enhancements vs Optimizations"
+
+    * Use **optimizations** for performance hints that don't change semantics and can be safely ignored.
+    * Use **enhancements** for semantic changes that must be understood by consumers or the plan cannot be executed correctly.


nit: perhaps we could move this section to after the two sections below? It feels like it makes sense to compare them only after explaining what each thing is.

Hm - I see your point, although there's already a note there about Enhancement Constraints. Should we put the two notes one after the other, or leave this where it is?

Yeah that's a good point. 🤷 lets just leave it as it is for now then.

benbellick · 2025-11-20T16:32:31Z

site/docs/relations/physical_relations.md

+| Inputs               | 2                                                                 |
+| Outputs              | 1                                                                 |
+| Property Maintenance | Distribution is maintained. Orderedness is eliminated.            |
+| Input Order          | Same as the [Join](logical_relations.md#join-operation) operator. |


Thank you for doing all of these changes here 🚀

site/docs/extensions/index.md

benbellick · 2025-11-20T16:47:12Z

@yongchul @wackywendell As I understand it, the governance rules dictate that only one SMC approval is necessary to get documentation changes in.

Since @yongchul is an SMC, do we have to wait for any other approvals? Or can we just merge it?

yongchul · 2025-11-20T20:19:49Z

@yongchul @wackywendell As I understand it, the governance rules dictate that only one SMC approval is necessary to get documentation changes in.

Since @yongchul is an SMC, do we have to wait for any other approvals? Or can we just merge it?

I'm going to merge after re-reading the changes one last time. :smile

yongchul · 2025-11-20T20:22:57Z

site/docs/extensions/index.md

+
+Advanced extensions come in several main forms, discussed below:
+
+1. Embedded extensions: These use the `AdvancedExtension` message for adding custom data to existing Substrait messages


not in this PR but it would be great if we can just easily reference this to the actual proto message definition. perhaps, onetime clean up is necessary.

yongchul · 2025-11-21T04:26:02Z

site/docs/extensions/index.md

+- Modify the semantic behavior of operations
+- Must be understood by consumers or the plan cannot be executed correctly
+- Only one enhancement per message
+- Examples: specialized join conditions (e.g. fuzzy matching, geospatial) or sorting (e.g. clustering)


clustering is unclear. can we have more detailed example or expand clustering a bit? Fetch with tie is a good example but I have a pending PR #797 for this 😆

Good point. I was thinking something like similarity matching, but that's a bit of a complicated example to explain in a few words, so I switched for locale-aware string ordering; does that make sense to you?

locale aware sort is possible already as you can specify a custom comparison operation per field. Shall we just drop it for now and add an example later once we have a good one?

Oh, you're right - I hadn't seen the custom comparison operator before.

I've dropped the sorting example.

site/docs/extensions/index.md

yongchul · 2025-11-21T04:35:00Z

@wackywendell thank you for the changes and sorry about taking it so long! Please resolve the two minor changes and it looks good to me to go!

wackywendell · 2025-11-21T15:18:48Z

Thanks, @yongchul! I updated, let me know how it looks to you - especially the sorting example, clustering was unclear as you noted!

jacques-n · 2025-12-03T18:00:49Z

The one issue I see with the extensions updates here is that they act like the only way to express something is using protobuf. Our goal was to spec first, with proto being an implementation. That being said, the reality is a bit different than that.

Clarify in the advanced extensions description that Protobuf is *an* implementation of Substrait. It's the only one for now, but not the only possible one.

wackywendell · 2025-12-08T18:50:16Z

@jacques-n - good point. I was looking at it, and while I think most of the protobuf references are hard to get away from - e.g. field names, hierarchy - the top-level description did clearly treat the Protobuf representation as the only one, so I changed that; see 5fce283.

For the other places where it references Protobuf messages directly, that seems to be in line with the rest of the docs, and also seems like a good place to show implementations; and in theory, they could also be updated to multiple 'dropdowns' showing the representations in various forms if other implementations existed.

Does that make sense to you, or were there more specific ways we could address this?

docs: clarify advanced extensions

e0cd598

wackywendell force-pushed the extensions-doc branch from 624b170 to e0cd598 Compare July 25, 2025 20:45

wackywendell marked this pull request as ready for review July 25, 2025 20:46

wackywendell requested review from EpsilonPrime, cpcloud, jacques-n, vbarua and westonpace as code owners July 25, 2025 20:46

yongchul reviewed Jul 29, 2025

View reviewed changes

docs: add guidelines back, some clarifications

b537d3d

yongchul reviewed Jul 30, 2025

View reviewed changes

site/docs/extensions/index.md Outdated Show resolved Hide resolved

site/docs/extensions/index.md Outdated Show resolved Hide resolved

site/docs/extensions/index.md Outdated Show resolved Hide resolved

yongchul previously approved these changes Jul 30, 2025

View reviewed changes

docs: reformat notes in extensions

e7380b3

wackywendell force-pushed the extensions-doc branch from 7c70ca1 to e7380b3 Compare July 31, 2025 19:37

wackywendell dismissed yongchul’s stale review via e7380b3 August 5, 2025 17:41

Merge branch 'main' into extensions-doc

d3a6f5f

yongchul approved these changes Nov 19, 2025

View reviewed changes

site/docs/extensions/index.md Outdated Show resolved Hide resolved

benbellick reviewed Nov 19, 2025

View reviewed changes

site/docs/extensions/index.md Outdated Show resolved Hide resolved

wackywendell added 2 commits November 20, 2025 11:12

docs: apply PR feedback for extensions documentation

85c96e6

- Update proto syntax to use new %%% format - Reorder sections to prioritize custom read/write types over custom relations

docs: broken link to join-operator pointed out by mkdocs

349070a

benbellick approved these changes Nov 20, 2025

View reviewed changes

docs: small restructuring for pr comments

be12b27

yongchul requested changes Nov 21, 2025

View reviewed changes

docs: add get-in-touch link, clarify example

5f002e8

docs: remove poor example

1b922a7

benbellick mentioned this pull request Dec 3, 2025

docs: bad mkdoc link to logical_relations.md#join-operator #904

Merged

wackywendell added 2 commits December 8, 2025 13:39

Merge remote-tracking branch 'upstream/main' into extensions-doc

b865b3a

docs: clarify advanced extensions description

5fce283

Clarify in the advanced extensions description that Protobuf is *an* implementation of Substrait. It's the only one for now, but not the only possible one.


		Advanced extensions come in several main forms, discussed below:

		1. Embedded extensions: These use the `AdvancedExtension` message for adding custom data to existing Substrait elements

docs: clarify advanced extensions #833

Are you sure you want to change the base?

docs: clarify advanced extensions #833

Uh oh!

Conversation

wackywendell commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Naming

Guidance

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wackywendell commented Jul 30, 2025

Uh oh!

yongchul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wackywendell commented Jul 31, 2025

Uh oh!

wackywendell commented Nov 18, 2025

Uh oh!

yongchul commented Nov 19, 2025

Uh oh!

Uh oh!

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

benbellick commented Nov 20, 2025

Uh oh!

yongchul commented Nov 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yongchul commented Nov 21, 2025

Uh oh!

wackywendell commented Nov 21, 2025

Uh oh!

jacques-n commented Dec 3, 2025

Uh oh!

wackywendell commented Dec 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

wackywendell commented Jul 25, 2025 •

edited

Loading