Separator not emitted for nulls until a value is hit

[rrs]

We use string templates for a variety of data exports. One of which is simple CSV.

If we have an input from a table of data where as long as the first value in each row is non null then the CSV is correctly formed. When the first value is a null a separator is not emitted until there is a value.

Imagine the following table of data

Name, Age, Color
Ron, 10, Pink
Paul,15,Brown

and the template

  printRow(row) ::= <%
    <items.Columns:{ k | <row.(k)>}; separator=","><\n>
  %>
  
  outputTemplate(items) ::= <%
    <items.Columns; separator=","><\n>
    <items.Rows:printRow()>
  %>

this will output
Name, Age, Color
Ron, 10, Pink
Paul,15,Brown

if however the input data was
Name, Age, Color
Ron, 10, Pink
null,15,Brown

Then the output would be

Name, Age, Color
Ron, 10, Pink
15,Brown

and not

Name, Age, Color
Ron, 10, Pink
,15,Brown

I can see it is because of this section of code

because the seenAValue is false until the first value in the enumerable is non null.

In this case there is a work around, which is to substitute nulls with "" which will generate for the example above

Name, Age, Color
Ron, 10, Pink
"",15,Brown

However I was looking at the possibility of adding another option like emitAll="true" or something like that which would be used along with the separator option to still output, something like

boolean needSeparator = (seenAValue || (emitAll && separator!=null)) &&
               separator!=null &&            // we have a separator and
               (iterValue!=null ||           // either we have a value
                   options[Option.NULL.ordinal()]!=null); // or no value but null option

However I do not have a clear idea of the impact this could have and was hoping someone for familiar with the code could advise me

[parrt]

I can't remember but I thought we had a template option that would test whether value is null or not and use a replacement. Oh, yeah, this thing: options[Option.NULL.ordinal()]!=null.

[rrs]

At this point its worth mentioning I am debugging the c# version so its possible there are differences. The codebase is largely neglected. However as those parts of the code are identical I thought I would ask here. Seems like the need to gobble up nulls at the start of a series is necessary to support some of the formatting allowed in the templates. Its the fact that the same code is also used with the separator option that gives me some hope it can be tweaked.

I will get a simple Java version going if I can to test it, but it seems in this context that null never gets triggered

My 'List' that is being iterated is actually an OrderedDictionary so it's possible that in a simpler context of just a plain list the null gets realised but here it doesn't. I'll have another play later now I have some more leads to follow.

Something else is that the value ultimately gets written here

and not in the with options variant, I don't quite understand how those opcodes are generated at this point

[rrs]

Here is a very simple example project in Java exhibiting the same behaviour

https://github.com/rrs/StringTemplateEmitSeparator

[parrt]

Dang. Does seem broken (haven't run code yet though).