Overhaul button drawing/styling

[Julusian]

For anyone stumbling upon this curious about what this is;

We want your examples for complex buttons or weird scenarios that are hard to do in the current system.

---

There are a few threads around different aspects of this already, but things have changed in companion since they were opened and there are additional requirements now.
The ones I found: #262, a snippet of #2387, #2237, #3067, #2367

I want to focus on the aspects of:

1. Supporting multiple aspect ratios of drawing
2. Supporting more drawing options without cluttering the ui

A lot of this builds heavily on what @dnmeid has already written, primarily around adding layers.

I think that doing anything around this is going to 'break' advanced feedbacks, because they are very tied to the current structure of properties we use for drawing. This structure has become a bit of a mess as new bits have been added over time, but has been unable to be tidied up the layout due to it being used in feedbacks and the module-api.

I would propose that this would probably be best achieved as a new button type. We wouldn't touch existing buttons (for now), once this is done we could look at auto converting some/all existing buttons across, but I expect that not everything will translate perfectly.
This way, we don't break things for existing users, and don't limit ourselves by trying to minimise this breakage.
We can then drop the old rendering in a future major version, or sooner if we can figure out acceptable automatic translation.

I feel like this should be implemented largely in the open in main, rather than hidden on a branch until it is complete. We can label the new type as alpha/beta, and if we want to do a stable release we could leave it in or hide the ui option

Layers

As has already been proposed, I agree that buttons should have layers. I would suggest that each layer should do a single thing, ie, a layer can be a png layer or a text layer. This will allow us to add more options within each layer without making the ui too cluttered. And at the same time, we can add new layer types without cluttering up the ui,
The user should be free to add and remove as many layers as they like into this stack, the one they will not be able to remove is the fixed background layer which will be where the background colour is set.
UI clutter is a big concern I have around improving the drawing, the existing style properties don't feel very organised and any changes to the current system feel like they will make it worse unless we can categorise things better..

Once we have this structure, adding many of the things from https://github.com/thedist/companion-module-utils would be much simpler, as they would be their own layer types.

A thought for myself, perhaps we could utilise the layering for some extra performance enhancements? We could draw each layer as its own bitmap and then flatten afterwards. Doing this is excessive, but we could utilise it for any costly layers, or to avoid repeating ourselves too much when it comes to gifs or frequent redraws. Not in scope of this, but something to think about in the future.

What about feedbacks

Now for one of the more radical ideas I have here; we should move the feedbacks to be defined as part of this layering.
For boolean feedbacks, this would be a pretty trivial way of adding an enable rule on each layer. This will likely need a way of linking layers to accommodate this fully, or a way of defining some 'button local' variables based on feedbacks. I am not sure on the best approach here.
For 'advanced' feedbacks, as written above I dont think these will translate well to this new model. I would be tempted to call these 'legacy feedbacks', and be able to mark layers as 'override for legacy feedbacks'. And at the same time, define a new 'advanced' feedback type which would fit better into this model.

I forgot to consider here that feedbacks override the defaults currently..
Perhaps then each layer type should have its own feedback stack to allow applying overrides to itself? That feels like it will be a bit clunky, but if we encourage doing it via local variables (or only allow via local variables?), to be able to share feedbacks across layers then it wouldnt be terrible? Definitely better than forcing enabling/disabling of layers
I am very open to ideas here

Aspect ratios

Finally, we need to be able to handle different aspect ratios of buttons. The Stream Deck Studio has slightly widescreen buttons that we 'letterbox', the Neo has the lcd strip that we ignore, the Plus has the LCD strip that we leave space around each button, the loupedecks have vertical strips that we ignore, and more quirks will come in the future.

I am not 100% sure how this should work, but I think long term we need to allow the user control over how different aspect ratios render.
The MVP could be to just perform the same operations on the different aspect ratio, and hope it works out. This will likely do for many users, as they will likely only look at buttons on one type of surface. We would need to figure out how to show this to them in the UI, but I dont think that is in scope of this task.

For a proper solution, I feel like we would need to have a separate stack of layers to draw, but that will get tedious if you just need to tweak one thing. Perhaps they share a stack and layers can be marked as 'only for X', and 'except for X'. I'm not sure, and I think it would be best to leave this to a follow up task.

The UI

I am not sold on these ideas, just my immediate thoughts for future me.

For showing this in the UI, I think this would be a case of adding this as a new tab in the action step tab list. Probably as the first and default tab.
The 'progress' and 'rotary actions' toggles should probably be moved to an 'options'/'settings' tab there too.

This tab could be split with the top portion showing a larger preview of the button, with a layer list beside it.
With the editor for the currently selected layer below.

[Julusian]

define a new 'advanced' feedback type which would fit better into this model.

From memory, there times where advanced feedbacks are used are:

1. module hasn't been updated to boolean feedbacks
2. feedback produces a highly custom imageBuffer
3. feedback produces text/colour values based on some device state.

For 1, we can ignore these, they should be migrated to boolean.
For 2, we should continue to support this, perhaps a new 'imageBuffer' feedback type, which does just this? There is the question here on how to handle different aspect ratios, do we want to tell the feedback about all the aspect ratios it should do, or tell the module about 'fake' feedbacks for the non-standard ones.
For 3, perhaps we need a way for a feedback to produce variables? If the feedback produced some 'button local' variables, they could then be used by the user in one of their layers.

I don't know if modules do 2&3 in the same feedback, hopefully not. Or if they do, then maybe any feedback should be able to produce these variables (including boolean feedbacks). We would then need somewhere to park feedbacks which are used just for their variables, but that should be solvable.

[Julusian]

Some other things will 'break' as part of this

The feedback 'use another buttons style' won't work fully, as one of those 'advanced' feedbacks. I expect we will leave this 'broken' when going across types, and add a new one for this new button type.

The 'set button text' and similar api endpoints and actions wouldn't nicely translate across to this. We could take the opportunity to remove them, or we could expand them to allow choosing a layer index (which will be brittle) defaulting to the first text layer. I would lean towards removing them, everything they support will be possible with custom variables instead in this.

Satellite API can provide text and text colour to clients. We need to define a rule on how those should be determined from a multi-layer button.

[dnmeid]

I appreciate this summary and it is mostly also what I'd suggest.
There is maybe this one to link where I maintained the work of this project so far Graphics generation overhaul but this can also transfer to Companion plan.

• Layers: d'accord with the general specs. I'd maybe add the possibility of (nested) layer groups. So one could also switch on/off a complete group or have it switched by a feedback/variable
• Feedbacks: I'd propose to pretty much rely on the local variables here. Let's assume every feedback just writes its original data to a local variable. Now there is this expressions everywhere idea (Extend expressions to action options #2345) and that means you can simply switch the enable of a layer to be fed from an expression and use the boolean of a boolean feedback. Or you can switch the color to an expression and can use the color part of an advanced feedback by accessing it from the json. Or using the text of a feedback in a text layer and combine it with some other text. That way we could even fit all todays functionality in the new layers and wouldn't necessarily need a new button style. The styling options in the feedback itself could be dropped and would be replaced by the style of the layer or what you write in the expression. The downside I see, is much more work for the user to style the button. A lot users want to have more flexibility but not more work. I can only think of some nice workflow enhancements to streamline the setup process, e.g. drag and drop a feedback to an option and it will switch to an expression with the according local variable filled. Or having the expression prefilled with the most likely variable when switching to it.
  Another issue I see, is to transfer the current way of feedback stacking, i.e. higher feedbacks overwrite lower ones. One can easily stack the boolean sources of boolean feedbacks. Also it would be possible to just have e.g. one solid color layer per boolean feedback and switch it on/off from the feedback. This leads to the same result, but I think this may be too much effort for many users. Even worse with stacked advanced feedbacks. What so far has been an easy to use feedback will then be some programming like $(this:fb3.text) || $(this:fb2.text) || $(this:fb1.text) . We could maybe offer a pre-stacked preudo-feedback like$(this:advanced_feedbacks_stacked).
  We could also offer this as layer types, eg. a layer for all stacked advanced feedback backgrounds, a layer for all stacked advanced feedback images, a layer for all stacked advanced feedback texts.
  I suspect that going that route the advanced feedbacks will be completely replaced in the long run by "value" feedbacks, like a string feedback that returns a string or a number feedback that returns a number and this will have the same workflow like the boolean feedbacks, just with different datatypes.
• Aspect ratios: My thoughts here are two possibilities: First possibility is to have sizes either in pixels/pt or in percent, second is to add something like CSS breakpoints, i.e. some conditional logic what to use for a given size or a/r.
  There are two questions. How can we make a UI for the conditional logic and should we switch complete layers or only some properties. When we have expression possibilities in size properties we automatically get the ternary operator with it. We would just need to add the button size and/or a/r to local variables and then you could use it in the expression. This would also mean that a button image can only be rendered at surface level but I think that is anyway what we need to do. I'm not sure if we should do it per surface or per surface type or per output type and where in the code (thinking of the deamon idea here).
  For the fixed/proportional value entry I see two possible solutions. One would be to simply have a dropdown per option where you can switch if the entered value or the value from the expression is pixels, percent (0-100) or a float (0.0 - 1.0) and the other one would be some automatic switching for regular options like css does where you can enter 17 or 17px for pixels and 0.5 or 50% . For expressions we would need to either force the users to generate a size string then with the named suffixes and/or offer functions to convert to special data types like asPixels() or asPercent().
  The first solution is easier to understand, the second is more flexible as you could also switch between fixed and proportional sizing by logic. I wouldn't necessarily offer expression possibility for the dropdown of the first solution because this would be overkill to have possibly two independent expressions to generate one size.
• breaking stuff: This stuff would not necessarily break the existing button/feedback system and can in theory be upgraded automatically. It would not nessarily break the advanced feedbacks but automatic upgrades may not look very nice.
  Im not sure if two different button types would be a good solution. If we keep the current button as it is, we would also need to keep the feedbacks as they are, that means we would e.g. still need the style options for a boolean feedback. If we keep the button mostly as ist is but make the feedbacks part of that legacy button, we would have two different feedback presentations. In the legacy button there would be style options and in the new one we wouldn't show them. Additionaly I'd consider the feedbacks more to be part of the control itself if they feed local variables and not part of the graphics, because local variables can be used everywhere in the control.
  Satellite text and color transmission I think is a easy solution, I'd provide a as good as possible backward compatible solution and add the new features to it. I expect a good adoption rate of users that make use of these special features and others will just not update.
  For the set text and similar endpoints I'd suggest to just always direct them to the most probable layer, e.g. we could say one has to call a solid color layer Background and then it will receive the background color and if that name is not found it will be the lowest solid color layer. This I'd just do to not break existing configurations, but I'd also discourage using such workflows in the future and prefer user variables instead.
  "use another button's style" could use the same logic of mapping existing layers to legacy style options. That would at least make sure that automatically updated configs are still usable. But also in the long term I would deprecate it. I think the whole thing of borrowing a style from a different button is kind of a workaround and we should think of what the real goal behind it is and if there maybe is a better solution.

There's now not an answer for all of the open questions, but I tend to write too much and I think we'll find more solutions along the way if we agree first on the concept.

[Julusian]

Layers: d'accord with the general specs. I'd maybe add the possibility of (nested) layer groups.

Yeah I was thinking about that too, but didn't write it.

Sounds like this could be started on then, either as unit tests, or a standalone browser based tool so that we can play around with it and see how aspect-ratios behave.

Feedbacks: I'd propose to pretty much rely on the local variables here.

No objection from me on this. It will be harder for users, but that might just be the cost of necessity. This whole change will need some UX focus, hopefully we can mitigate confusion with some tooltips or better inline help/or links to help.

I can only think of some nice workflow enhancements to streamline the setup process

An idea: make it one click to create a new local variable and jump to where to setup the value

What so far has been an easy to use feedback will then be some programming like

For advanced feedbacks, I don't mind making them awkward/hard to use. In a way, I want to be 'hostile' against using them, to motivate pushing modules to newer feedback types that fit our new systems.
As I see it, we either make it painful to use and hope it pushes modules away from it, or it could be annoying to use and modules wont bother changing as quickly as its fine and works.

The other challenge with advanced feedbacks, is that we don't know what properties they will give us. And the properties they give us can change each time they give us a value. So we either need to do something to semi automatically link all the properties, or leave that for the user to do and hope they know what the feedback will give them.

But you're right, this will mean that replacing the text will become more programming like; which I think is unavoidable its just a question of how we expose/encourage that programming to be done (expressions vs logic feedbacks vs override layers). I don't think we need to decide on this immediately, this feels like something that can be iterated on once we have something to look at and play with.

I suspect that going that route the advanced feedbacks will be completely replaced in the long run by "value" feedbacks

Yep, I agree. It should be able to produce multiple values, with a predefined list of values to expect in the feedback definition. (to allow for cases where today they return a bgcolor and fgcolor or whatever)

Aspect ratios: My thoughts here are two possibilities: First possibility is to have sizes either in pixels/pt or in percent,

Thats a good point, maybe using things like this would get us 99% of the way there. Throw in using expressions, and some new variables as input to the expression of the width and height of the current drawing, then complex formula become possible.
We probably don't need to make this 'simple' to use, I suspect it will be a minority case where fine control is needed over the drawing. Maybe something about em/rem too for sizing? I don't fully know what they do, but know theyre relative to something..
Making expression output be consistent for this will need some work, if things like randomInt are used but all doable.

I'm going to skip over the rest of this for now, as its not critical for a MVP, and I think will be easier to reason about once we have something we can play/test ideas with.

breaking stuff: This stuff would not necessarily break the existing button/feedback system and can in theory be upgraded automatically.
If we keep the current button as it is, we would also need to keep the feedbacks as they are, that means we would e.g. still need the style options for a boolean feedback

Well, the style properties of boolean feedbacks belong to companion, and they are already ignored when used as conditions, so I don't see this as a problem. Sure, it would be nice to clean them from the code, but thats not an urgent thing.

Im not sure if two different button types would be a good solution.

Long term no, but for a couple of releases I think it would be a good idea to. And maybe depending on user feedback and our feeling on how complex the new system is to use, I would be open to keeping a slightly simplified form of it if there is enough demand for it.

Satellite text and color transmission I think is a easy solution, I'd provide a as good as possible backward compatible solution and add the new features to it

My main uncertainty with it was what text do we send. But I suppose it would be taking the text from the top or bottom most enabled text layer.
I am very hesitant to add anything resembling the full layer blob to the satellite protocol, as then suddenly the semver of the satellite api becomes tied to how we represent this layer drawing. We will always need to support forward converting the structure for imports and presets, but for satellite that would mean converting it to older versions too.
I much prefer making companion do the compositing when wanting an actual pixel buffer; for me the text/color options are there to support simple surface types that don't have full rgba screens and can only do simple things like showing a line of text possibly with colours.

---

So to summarise, I think I have enough answers to be able to start playing around with this.

I think the next steps are:

• Define the layers data-structure, and implement the rendering, possibly as its own tool.
• Do a minimal version of this button type, using purely static data.
  • During this stage, the editor should probably show a few different aspect ratios at all times, square, SDS, really tall and really wide, just so we can get a feel for what does/doesnt work.
• Add local variables to buttons. Some thought might be needed on how to tell users that these won't work in most actions
• Expand this to be minimally 'dynamic', being able to enable/disable layers and put expressions in every field.
• Then it will be time to pickup discussions on what to add to improve the UX, and the other interactions that have been neglected (eg advanced feedbacks)

And I am going to insist that for now this should be its own type, as it means we can push it to the betas/experimental before it is feature complete, and allow testing/experimenting in parallel with other usage. Based on the steps above, I could probably convert 90% of my office/test setup across (I follow the latest betas there, it has found a bunch of bugs I missed during development, so I want to keep it to betas).
I am open to discussing whether it will be released like this when we are nearer to it being ready for general use. I don't think that having both will be a massive cost (short-term), we should try and write the code to be as separated as possible anyway.

[dnmeid]

I can go with the new button being a different type during development, but I'd really prefer it to replace the current button before it is released and not have both in a couple of releases. I don't see a big stopper where we wouldn't have an upgrade path. My main concern is not having two button types in parallel, but having two feedback workflows in parallel. If we make a clean transition, some users will praise it, most users will just take it and some will dislike it. If we offer both workflows in parallel, it will feel much more like we're taking somthing away later instead we're improving it. The frustration will be higher. We can see this now a little with the delay to wait change. Just because the space where delays had been is empty now, many users have the feeling that we have taken something away.
I can see your point of having it in the main branch for better access and testing but if we feel that this will not make it in the next release we should keep it in develop branch. We can merge in main as soon as we are sure that it will be ready with the next release and that means at least some months of testing time in a beta.

I suspect that going that route the advanced feedbacks will be completely replaced in the long run by "value" feedbacks

Yep, I agree. It should be able to produce multiple values, with a predefined list of values to expect in the feedback definition. (to allow for cases where today they return a bgcolor and fgcolor or whatever)

I'd rather vote for one value per feedback like a function also has only one return value. Just like the boolean feedback today returns a boolean, but with some companionabel types. E.g. text, color, number, image. Internally we can derive the type either by the type of the value or we pack it in a JSON with additional descripor for ambigous types. I wonder if we should introduce something like ENUM, or how do we want to deal with switching the alignment from a feedback (do we want to have this at all or drop it).
For the UX I'd go more with the atomic approach, when you want to e.g. change a foreground color and a background color, use the same feedback two times. For the real advanced stuff we can allow objects like arrays or JSON as the return type. This would need some skills of the users but it would be the replacement of today's advanced feedback. I'm also fine with this being so complex and actually motivates the module devs to break down the still existing advanced feedbacks into smaller feedbacks.
As a side effect this would lead to the opportunity of a better unification of module variables and feedbacks. A feedback usually is associated with a control and a variable with a connection, but besides that they do the same thing: returning values. A feedback gives a local variable and a variable gives a global variable with the connection namespace. We could use the existing definitions for module variables as well. A current variable would translate into a feedback without options but modules then could also offer to publish user defined variables. Today this already often done with feedbacks. Some feedbacks only exist to generate a variable, but you have to put it on a unused control to work.

[Julusian]

Something else to do as part of this:

Stop using \n in the ui for a line break. We could just as easily turn the field into a multiline field (it already does this when in expression mode), and use real line breaks.

Doing this is messy for presets, but if we do it at the same time as introducing a new preset type, we can leave the old behaviour for old presets, with new behaviour for new presets

[Julusian]

Decide on topbar being an element or property of the canvas

So far the topbar is a property of the canvas instead of an element.

@dnmeid said in #3299:

I think the biggest advantage of a fixed topbar is that it remains possible to switch the topbar on and off and the remaining button just rescales automatically (if the elements used percent).

On the other hand the biggest advantage of a separate topbar is the styling flexibility. You can position the bar at the top or at the bottom or somewhere else. You can have some containers with a topbar and some without and switch between them programatically (e.g. per step). You can resize the topbar to your liking. Currently it is 18% of the height, when I think of e.g. a 640x400px display, I don't think that a 18% topbar will be what you want. Maybe you want it to be only 5% high and only 20% wide.

We could of course offer both possibilities, the fixed topbar and a statusbar element at the same time. I just think then we are carrying unnecessary code and UI stuff over, because actually only one system is needed. Maybe it's an idea to have both possibilities for a testing period and then we see what we like more or how some test users react before releasing it.

which I think is a good summary of the options.

I have another spanner to add to this; which is the question of how to handle existing configs (assuming we do manage to convert existing buttons across).
When importing a config, we won't know for certain whether the user wants the topbar enabled or not. (full exports dont currently store the userconfig value).

If this is a property of the canvas, it is not that bad (like today), where the current global config will remain that they must correct manually and everything will be ok.

But if this is an element, we would either need to start prompting for this, or we will import with the top bar shown (matching existing behaviour).
But if the user had changed this config and disagrees with what we assumed, they will now be forced to go through each button and remove the extra elements we added.
Alternatively we could default to it being hidden, and force them to add it if they want it.

So this has the potential to upset a bunch of users, and force a load of work on them to tidy up our assumptions.

Because of this, and my other uncertainty about changing this to be an element, I am leaning towards keeping it as a property of the canvas/button for now. Changing it to be an element later on will be just as easy as making the change now, going the other way is much harder.

Maybe the canvas should also get a property to control the height of the topbar, so that the user can still choose the size of it. Alternatively, having it as an element too I don't expect will be much work, most of the effort there will be making sure it behaves sensibly at different sizes.

On a side note, I wonder if making it default hidden would have the side effect of it rarely being enabled. Do people want the statusbar there, or tolerate it because they don't realise they can change it?
Personally I mostly have it enabled because its always been there and buttons look weird to me without it.

[Julusian]

Maybe the canvas should also get a property to control the height of the topbar, so that the user can still choose the size of it. Alternatively, having it as an element too I don't expect will be much work, most of the effort there will be making sure it behaves sensibly at different sizes.

Perhaps we could have the surfaces provide a DPI like value, so that we can auto-size the topbar to be roughly the same physical size. That will probably get pretty complicated, and only work maybe 90% of the time, so that sounds like a follow up to have an auto-size mode

[Julusian]

Solving 'advanced' feedbacks

I don't think we have reached an agreement on how these should be handled in the new drawing. I think we do both agree that they should be replaced with a new feedback type, but haven't agreed on exactly that will be.
For this thread lets shelve what the new thing is, and instead think about how to support the old feedbacks.

For this, I think we should 'rush' to drop advanced feedbacks from the proposed 2.0 of the module api. I think it better to do it slightly prematurely (modules dont have to update immediately) than to have to keep support for the old advanced feedbacks for years.
Perhaps this needs a discussion too, but I am assuming that we don't want to have the module-api claim to support these 'advanced' feedbacks when companion does not. The best way to discourage their use is to stop them being supported by newer module-api versions.
And as existing modules will be producing these feedbacks, we will want to keep some support in companion until we raise the minimum module-api to one that does not support them. Whether this happens in companion 5.0 or earlier is a discussion to take later.

But short term, if we want to be able to push user configs across to the new buttons automatically, we will need to support these 'advanced' feedbacks in a reasonable way.

My Pitch

This feels messy, but not too horrific. So I am very open to other ideas.

Somewhere would be a panel which allows for building a stack of these 'legacy' feedbacks. This panel should be clearly labelled as legacy/deprecated, with a warning that it should not be used if possible and will be removed in a future release.
These feedbacks would have their output flattened (like today), and would override the appropriate elements based on the usage rules introduced by #3444.

With this, I think that will preserve all behaviour in a backwards compatible way, and not be too invasive into the UX of the new system.
And for the short-medium term, this wont be too painful for users who need to use these feedbacks on new buttons.

I don't know where would be best to put this panel; Perhaps it should be an element type (limited to being added once), or this could go on the canvas element, or be its own tab (which could be used to nudge the users looking for feedbacks into the new flows, but could also have the opposite effect due to muscle memory), or something else.
I do worry that the automatic overriding of values could be confusing, but I think this is a necessary evil to avoid this being incredibly clunky and error prone. Alternatively these computed values could be done via some local variables, but that would require them to be manually hooked up to the right places, which will be tedious and error prone. Maybe this confusion can be mitigated by a banner at the top of the new layer editor when any legacy feedbacks have been added.

When it comes to migrating existing configs, I suspect we will need to handle two cases separately:

• using just boolean feedbacks - I suspect this should be possible to rebuild properly using just local variables
• using advanced feedbacks - I suspect this will need to keep the feedbacks untouched, to ensure we dont break the order of the layering.
  Maybe we can provide a one click tool in the ui to convert from this to the new flow when we detect it will work; I am thinking here of the case of when a module 'upgrades' an advanced feedback to be boolean, which could mean that we no longer need to be in this old flow.

[Julusian]

New feedback type

We agree that the 'advanced' feedback should be replaced with something new.

#3314 introduced internally the concept of 'value' feedbacks. Those were made to be able to populate local-variables with user defined values/expressions or the result of a boolean feedback.

So I suppose the question is whether we should simply allow modules to define these too. Modules could populate these values with either a simple primitive, or an object. Essentially, any type that we already allow for custom-varibles.

With just this, it should be possible to replace 90% of the values that advanced feedbacks can currently produce, of course it will be up to the user to hook up the values it produces. We should define some rules, but It is up to them to decide how to format the output. We should also give some guidance on where to document the output structure.

The one bit which is unhandled by this is feedbacks producing imageBuffers. The values from those would fit into this structure, but they also need to know the dimensions to draw at, so will need some extra thought. (See the next thread for my thoughts on these)

[Julusian]

While we could interpret every advanced feedback as this value type, I feel like we shouldnt simply to encourage modules to do it properly.
The existing ones are likely not very thought out, often they return objects of entirely different shapes depending on an internal condition. This will make using them very hard without knowing the implementation and handling the undefined values sanely.

[Julusian]

Handling imageBuffers

In the old flow, buttons were drawn at one size. So feedbacks were able to produce an imageBuffer and it would match the button size for drawing.
In the new flow, buttons can be drawn at multiple sizes/aspect ratios simultaneously.

For any 'advanced' feedbacks in the backwards compatibility flow, it would be good enough to simply do a 'fit' draw of the buffer resized to match the space. It won't look the best in all cases, but its good enough.

But what should the new approach be?

At the beginning of this overhaul, I was thinking that we could figure out what sizes of the buffer we need and tell the module about 'virtual instances' of a feedback to collect renders at all the needed sizes. But that only works if we know that a feedback needs to know about the dimensions and will produce an image buffer. It won't work once we stuff that value into a variable, as by doing that we lose the ability to alias the value to the correct bitmap.
So that won't work if this is a value feedback.

The other problem is that the bitmaps may not be drawn at full size, with the positioning available for each element, we don't know what resolution we need until it gets drawn. And it could change for each draw, we can't really preempt what we will need.

So I don't have any good ideas right now on how to support these.

It feels like part of the solution is to have a dedicated feedback type for this, and a 'connection imagebuffer' element type, which is where these feedbacks will be used. In that way we avoid issues with aliasing of values, and it allows us to do any 'virtual instance' stuff we want to get multiple renders.

As long as we preserve aspect ratio, is drawing 1:1 pixel size important?

Perhaps we can preempt the resolution by saying that if the size is an expression we always ask for canvas size (possibly with user a aspect ratio override)?

I have a bunch of thoughts here, but no clear plan on how to handle these. Input is very much appreciated

[Julusian]

I am wondering if a good enough solution is to also deprecated any form of pixelbuffer output from feedbacks.
It will be possible to produce a png in a value feedback, which can be bound into an element, so if a module really needs it they can still do something, but this encourages that we don't support informing about dimensions of these, and they wont be drawn 1:1 like today.

[WNTLtd]

While the Layer idea is thought about - any chance we could also have the ability of adding button Boarders - a bit like you can do in Excel with cells. That way, when working with multiple devices on one streamdeck page, you can visibly separate the button areas per device/connection - leaving colours for feedback.

[Julusian]

I've just copied in a proposed list of element types: #3293 (comment)

That proposal includes a border for the box element, which will give you what you want.

[Julusian]

Proposed element types (originally posted #3299 (comment))

Let's collect some ideas for elements to see how tey can build a useful set with distinct functionality.
Top priority

• Container: Can hold other elements. Has X,Y,W,H. Maybe rotation? Child elements use the extends as their 0%-100% range, but can also draw outside of the container area. E.g. container x is at 5px and child x is at -5px, actual x will be at 0.
• Matte: draws a color or a gradient (maybe gradient as a separate element?). Has color. Does always draw to the extends of the parent element
• Box: Has XYWH, rotation, fill color/gradient, border width, border color, border alignment, corner radius?. Maybe we make the box the container? Easier sometimes but more unused options most often
• Text: Has text, font size, font, horizontal alignment, vertical alignment, color, style, line height. Maybe later animation like scroll? Maybe enable interpretion dropdown for plain|md|html, maybe formatting option like in excel.
• Image: Has XYWH, rotation, image, alignment horizontal, alignment vertical, crop/fit. Can use different bitmap and vector image formats. Maybe gif support? (usually super ugly but highly requested by users). Maybe Images are taken as a reference from a image storage. Drag and drop from the OS should be possible either to the image option of an existing element or directly in the layer stack.
• Statusfield: Has XYWH, rotation, color, border color. Inner elements will adopt to size/aspect of element.

Medium priority

• Line: Has X1, Y1, length, angle, width, color, style, cap start, cap end
• Gauge: Has XYWH, rotation, value, style (horizontal, vertical, circular...), inactive color, active color, indicator width, indicator color, border width, border color
• Circle: Has XY, diameter, angle start, angle end, fill color/gradient, border width, border color, border alignment
• Icon: Has XYWH, rotation, icon from a super exhaustive set of vector icons, overlay color

Low priority

• Gauge: add possibility to use custom style for power users or to export from presets
• Videostream: like image, but can show a videostream (e.g. CITP, NDI, RTMP)
• Web: Has XYWH, rotation, scale, refresh. renders html from URL (probably with CSS but without JS). This may add limited value to a tiny button but will be very useful at larger areas
• Reference: Has XYWH, rotation, source, element, alignment horizontal, alignment vertical, crop/fit. Can reuse other containers or elements of the same graphics or from other buttons or complete other buttons

Any more?

All elements should have a Name/ID, Show/hide toggle, opacity and draw mode.
When you select an element, an outline should be shown in preview.

[Julusian]

For anyone curious about how to add a new element type, the complete change for adding the line type is: a61cc21

[lukescott]

I just stumbled upon this, so I thought I would share my complex button example:

I have a "hot bar" page where every button can be 1 out of 80-90 different button types. When a button is "not assigned" it shows an "x". When you press it, it takes you to another page with a bunch of different actions (multi-page). When you choose one, it takes you back to the original page and "assigns" that action to that button. When you press it again, it does that operation.

Where this gets complex is every button on the "hot bar" page has ~80 feedbacks. Each operation has a unique png icon to represent the operation. So I basically have 80 * 20 conditions and pngs. So the 80 images are repeated on every button as a feedback. That's a lot of duplication.

If that sounds complicated, it is. I have a script that generates a companion config file just to import all of this. I say 80-90 because new operations get added over time. When a new operation is added, I have to rebuild my import file and import it.

What would have made this far easier is if png64 could take a variable, and I could have png's stored in a global variable with an id for each one. When the button operation gets assigned, I could just reference the correct icon. (I actually forked companion and added png64_text, just to make it work; and I stumbled upon local variables; really exciting!)

What makes my use-case even more complex is for every button I have to create a global variable to store the operation's id, which is fed into a module that does the prescribed operation.

Does this new feedback system make this use-case easier?

I also think your work with local variables might also be very useful for me because I could store the operation id within the button itself instead of creating a global variable. Which means I could potentially duplicate buttons and move them around. The only thing that I would need is a way to address the button and set its local variable externally because I need to set the value from the "chooser" pages.

Which brings up another thing for me, I wish a module could ether have full control over a button (define a new button type) or even have full control over a page and the buttons on it to create some sort of ui. I've clumsily done this with import file generator, but it would be nice if I could build a module to handle all that.

You may ask 'why not use the companion ui to configure the buttons?'. The reason is you have to be able to configure button operations on the fly, quickly, without clicking thru a ui on the screen. Being able to tap two buttons to configure is convenient, and you can do it with muscle memory. On the same page, I have buttons that will clear out an entire row (or the page itself) so they can be re-assigned.

[Julusian]

It sounds like this will become much easier in the new system, largely due to local variables. The action to change a local variable can change a local variable on any button already.

What would have made this far easier is if png64 could take a variable, and I could have png's stored in a global variable with an id for each one. When the button operation gets assigned, I could just reference the correct icon.

This is basically exactly what the image library will support you in doing. And this drawing rework is going to allow a variable to be used for every styling field.

[thedist]

When the button operation gets assigned, I could just reference the correct icon.

Have you considered using a single internal: Button: Use another buttons style feedback on each of your Hot Bar buttons, and just using an expression that when assigned will copy the style of another button with the icon ready to go. Then it's just 1 feedback on each Hot Bar button, and 1 button for each image stored on a page somewhere to be referenced as needed.

Which brings up another thing for me, I wish a module could ether have full control over a button (define a new button type) or even have full control over a page and the buttons on it to create some sort of ui. I've clumsily done this with import file generator, but it would be nice if I could build a module to handle all that.

Personally I'm against this, it's beyond the scope of what a module should be doing, and if a module needs to manipulate buttons/pages perhaps it shouldn't be a module but instead integrated into Companion either directly or as an external app.

[lukescott]

The challenge is that with the buttons being able to have an infinite number of elements, with unpredictable ids and order, of varying types, allowing modules to interact with that is not going to be simple or easy to understand.

If I was able to do something closer to my second example, I wouldn't need advanced. All I need is:

• A way to specify the image and text (not just a default)
• Nice to have: Support for svg vectors (w/ a size option) to keep the image sharp - with my first example, some reason drawScale 0.5 is sharper. If I could just pass the svg base64 and let companion figure out the best way to scale/render it, that would simplify things quite a bit.

Which mostly answers this too. But also worth noting that a value variable can produce anything, so it could produce an object like:

{
  image: 'data:image/png;base64,aaaaa',
  text: 'something'
}

and allow that to be referenced elsewhere in the button where expressions are supported with something like $(local:my-thing)['image']

Sorry just trying to wrap my head around this: How would this work on the module side? Is this mostly a user-facing thing? Can a module set a local button variable somewhere?

My goal is to have my module have some icon and text combinations baked in. The user should be able to specify just the id, either statically or as variable. Currently my custom feedback takes an id parameter.

I would be happy to beta/alpha test this and see how it fits with my use-case. My module is still in the very early stages of development.

[Julusian]

Sorry just trying to wrap my head around this: How would this work on the module side? Is this mostly a user-facing thing? Can a module set a local button variable somewhere?

Its not implemented yet, but as part of local variables a new 'value' feedback type will be added so that modules can produce values to put into these local variables

[lukescott]

ts not implemented yet, but as part of local variables a new 'value' feedback type will be added so that modules can produce values to put into these local variables

Sweet, I'll keep an eye on it. I can test it out, even if it's just a dev build.

With that initial example, I have a variable const scale, which if I set to 2, drawScale becomes 0.5. Do you know why this would make the icon look sharper and less blurry? Everything I tried with the 'sharp' library didn't change much, changing the scaling method, etc. Only thing that made a difference was drawScale. Uploading pre-scaled SVG's are the same story as drawScale 1.

(It would be awesome of companion supported SVG's with width and/or height (max height)).

[Julusian]

With that initial example, I have a variable const scale, which if I set to 2, drawScale becomes 0.5. Do you know why this would make the icon look sharper and less blurry?

We claim to modules that we are drawing at 72x72, but we are actually drawing these days at 4x that. But we lie to modules for backwards compatibility reasons. By setting drawscale, you are opting into providing that higher resolution bitmap that we can handle

(It would be awesome of companion supported SVG's with width and/or height (max height)).

As I said above, that is already suported in this overhaul.
It isnt decided what release all of this will be in though

[lukescott]

We claim to modules that we are drawing at 72x72, but we are actually drawing these days at 4x that. But we lie to modules for backwards compatibility reasons. By setting drawscale, you are opting into providing that higher resolution bitmap that we can handle

Ah, that clears that up, thank you!

As I said above, that is already suported in this overhaul.
It isnt decided what release all of this will be in though

Oh, sorry I missed that comment. I'm excited to try it out. I've already got a fork running, so I'm willing to pull a branch down and try things out. I've already somewhat played with local variables, at least the parts on the main branch.