Improvements to section 4.4 about time coordinates

[JonathanGregory]

Moderator

None yet

Moderator Status Review

Summary

This proposal aims to

• clarify the concepts involved in time coordinates in CF, especially calendars;

• introduce some restrictions on the use and format of time units and of time zone offsets in the reference datetime in the units attribute of a time coordinate variable, in order to avoid confusion, mistakes and bugs in UDUNITS software;

• exclude dates before 1972 in the utc calendar, when it came into force with its present definition;

• replace the units_metadata leap_seconds convention introduced in CF 1.12 with simpler recommendations for how to deal with leap seconds and which calendar to choose when writing new data.

Status Quo

Some of the changes introduced 4.4 in CF 1.12 have been disputed regarding clarity and accuracy. In discussion 400 a small group was convened to consider these concerns and propose changes to remedy the problems raised. In three intense meetings in the summer of 2025, during which everyone compromised on their positions, we made the decisions listed below as our detailed proposal. The intention of the present PR is to implement this proposal. Its precise wording has been worked on in previous PRs during the last two months by members of the group.

Associated pull request

#612

Detailed Proposal

• We need some explanation of the concepts in the conventions document, sufficient to understand the convention, but long discussion belongs better outside the conventions document.

• We will stick with the CF 1.12 convention of using the single calendar attribute to identify the treatment of the combination of timescales (for the annual cycle, the diurnal cycle, and any others which might be relevant) as a bundle.

• We propose that CF 1.13 disallow the use of time-zone offsets in the reference date-time of the units of a time coordinate variable in the utc and tai calendars. This is a backward-incompatible change (in the generous interpretation of principle 9 of Sect 1.2), but we don't think it matters because these calendars were newly introduced in CF 1.12 and they probably have not yet been used in archived data. In addition, we propose to disallow some of the formats supported by UDUNITS for time zone offsets because UDUNITS does not implement them correctly.

• In the other calendars (which all existed before CF 1.12) numeric time-zone offsets are allowed (as they always have been), but we propose recommending in CF 1.13 that time-zones should not be used. This is because it's easy to make mistakes with them, especially with the sign and in allowing for seasonal changes in clocks. (Principle 7 in Sect 1.2 is "The conventions should minimise the possibility for mistakes by data-writers and data-readers.") A time-zone offset is simply a number of hours to be added or subtracted when converting between time coordinates and date-times. Exactly the same can be achieved by modifying the time coordinates i.e. converting them to the zulu time-zone. Note that named time-zones e.g. GMT and MDT are never allowed. CF 1.12 says so in 4.4.1, but we could say it more strongly.

• In CF 1.12, the utc calendar can be used for date-times from 1958-1-1. We propose that it should be changed in CF 1.13 to start at 1972-1-1, when the present version of UTC began, including leap seconds. Again, this is a backward-incompatible change which we don't think matters.

• In CF 1.12, we already note that the units of day, minute and hour are defined as a fixed number of seconds, and that this means they don't correspond to the expected date-time differences when there is a leap second. We propose recommending in CF1.13 not to use the units of day, hour and minute in the utc calendar, because it could cause mistakes and confusion.

• In CF 1.12, we introduced the leap_seconds keyword to units_metadata, to clarify the standard calendar. We propose that CF 1.13 should remove this feature. In CF 1.13 we would recommend that model data in the Gregorian calendar without leap seconds should set the calendar attribute to proleptic_gregorian, and we will clarify the definition of that calendar to say that it does not include leap seconds. This recommendation would replace leap_seconds: none with the standard calendar in CF 1.12. It would be good if CMIP7 and CMOR could adopt this new recommendation straight away, for any models that are using the standard calendar at present. For real-world data written in the standard calendar, we would say that when time coordinates are converted to date-times, or time intervals calculated as differences between date-times, the result might be inaccurate by a number of seconds, because it is not known how the data-writer dealt with leap seconds; this is the meaning of leap_seconds: unknown in CF 1.12.

[JonathanGregory]

Please see HTML versions of conventions and conformance documents as modified by PR #612.

[JonathanGregory]

Dear all

In PR #612 I've completed changes arising from Lars's comments on 4.4.3 and emails with Chris about proleptic_gregorian and utc calendars. Please see the changes (between yesterday and now) and the current conventions and conformance documents. The largest changes are:

• Simplify the table and its caption, and remove some duplication with the following text.

• Insert a WARNING (using the asciidoc "admonitions" syntax) that writing time coordinates with utc requires the user to have leap-second-aware software. Chris suggested something along these lines.

• Insert extra statements, arising from emails with Chris, for

  • proleptic_gregorian: "If the time coordinates identify real-world instants from 1972 onwards, they represent UTC datetimes. In this case, the difference between two time coordinates will not exactly equal the length of the interval between the instants concerned if there are any intervening leap seconds."
  • utc: "If time coordinates in the utc calendar are converted to datetimes by software that is unaware of leap seconds, they will be later than the correct UTC datetimes by a number of seconds."

• Move some of the requirements and recommendations in the conformance document, because they were in the wrong subsections.

Next I will study Lars's comments on the appendix.

Jonathan

[ChrisBarker-NOAA]

Thanks Jonathan:

Insert a WARNING (using the asciidoc "admonitions" syntax) that writing time coordinates with utc requires the user to have leap-second-aware software.

This looks great, thanks!

Simplify the table and its caption, and remove some duplication with the following text.

This looks good, too. The one gap is that we don't define what "leap seconds" means is that table.

I think "unknown" for standard is unavoidable, so sure.

And "yes" for UTC makes perfect sense

And "no" for TAI also makes perfect sense.

And "no" for the not-the-actual-earth ones makes sense.

But "no" for proleptic_gegorian is a touch confusing -- the calendar itself does not support leap seconds, but it may (always?) have been used to encode UTC datetimes that DO include leap seconds (at least for some time periods)

So what "no" means here is that:

• no leap-second datetime (i.e. 61 second minute) can be encoded with this calendar.
• no leap-seconds were used to encode the data, and should not be used to decode the data.

I'm not sure that's clearly spelled out elsewhere.

And here's where the confusion comes up:

if you are writing a model dataset following the real-world Gregorian calendar for dates since 1972. It is recommended to use the proleptic_gregorian calendar in preference to the standard calendar in this case, in order to be completely clear to the dataset user that there are no leap seconds. (In principle a model could be programmed to include leap seconds, but we assume that this is not the case.)

But as far as an encoded time variable is concerned, the standard calendar never has leap seconds anyway, just like the proleptic_gregorian. The "unknown" means that the values could have been encoded from a UTC time with leap seconds, or it could have been encoded with no concern for leap seconds at all (e.g. generated by a model from a given startime).

For proleptic_gregorian, we are recommending that it be used for encoding UTC datetimes (e.g. from an observation) if proper leap-second aware software is not available. Which means we are saying that after decoding a proleptic_gregorian time coordinate, you will get proper UTC with leap seconds time. But that's not what a model is producing -- it's producing correct time-deltas, and that will break if leap seconds are assumed to be included.

So I think we should recommend "standard" for model results -- and maybe state that "unknown" means "don't count on that level of precision".

Unfortunately, we are still struggling with how to address two distinct use-cases with inadequate software [*]:

1. Time stamps from observations in UTC.
2. Results from models that (hopefully) started with a UTC timestamp, but then are "time-delta correct", e.g. no leap seconds.

[*] I say inadequate software because the correct way to do (1) is to use the utc calendar.

It seems in this draft that we are recommending proleptic_gregorian for both (1) and (2) -- but the user has no way to know which of those it is. In practice, it probably doesn't matter (that's why we've gotten away with this for so long) because:

A) The user generally does know whether the data came from observations or a model, and can make different assumptions for each

and/or

B) second-level precision isn't important

and/or

C) the epoch used is close to the data, so very few leap seconds play a role.

But now that we are doing all this work, I think we should make things as well defined as possible.

NOTE: this probably is the biggest issue if someone is comparing model results with observations -- in that case, (B) probably applies, but CF is supposed to be pedantic.

Anyway, my suggestions:

We need a way to spell: encoded-from-UTC-with non-leap-second-aware-software

If that's what proleptic_gregorian means, which is what it seems to mean (" If the time coordinates identify real-world instants from 1972 onwards, they represent UTC datetimes") -- but then we should recommend the standard calendar for models. (or TAI, but then they might well have used a UTC time for the initial start time, so that's wrong).

[JonathanGregory]

Dear all

I'm sure you're aware that today is the last chance for any substantial changes that are to be included in CF 1.13. After today, we can make only minor and uncontroversial edits in the PR.

Thanks for your recent comments on PR #612. I have answered these and made revisions to the PR except for those discussed below. I have not yet renamed Appendix L to M, because that would make it harder to compare commits up to now. I'll do that once we are all happy. Here is the updated conventions document according to the present state of PR #612.

---

In 4.4.2, this version says

In CF, following UDUNITS, any unit may optionally have one of the decimal prefixes for multiples and submultiples (Table 3.1) e.g. millisecond or ms; however, we note that [SI] (chapter 4) does not allow these prefixes for any unit of time other than second.

Chris asked, "Is it too late for us to say that CF does not recommend using pre/suffixes for any time unit other than second?" To do that, I would suggest changing the text to read

In CF, following UDUNITS, any unit may optionally have one of the decimal prefixes for multiples and submultiples (Table 3.1) e.g. millisecond or ms. CF recommends not to use these prefixes with any unit of time other than second, because [SI] (chapter 4) does not allow them except with second.

Do you (all) think we should do that (and a corresponding addition to the conformance document)?

---

In his comment above, Chris writes concerning the table of calendars,

"no [leap seconds]" for proleptic_gegorian is a touch confusing -- the calendar itself does not support leap seconds, but it may (always?) have been used to encode UTC datetimes that DO include leap seconds (at least for some time periods)

He suggests ways to define exactly what "no" and "unknown" mean for leap seconds in the table. He also raises concerns about recommending proleptic_gregorian for real-world data, suggests we could recommend standard for models, and remarks,

We need a way to spell: encoded-from-UTC-with non-leap-second-aware-software.

In his comments, Lars writes,

Do we really want/need to make this blanket recommendation against using the standard calendar? As we know, almost all use cases do not need (or want) this second-precision accuracy. We must not be unaware of the fact that making the recommendation as stated may initiate a lot of work on changing workflow and software systems. Or, alternatively, that people and organisations decide not to follow the stated recommendation, which only undermines the status of the CF Conventions.

and in his email

we are talking about "leap second aware (and the opposite) without giving any indication what this means.

These points aren't currently dealt with in PR #612.

In the summer, we agreed to define proleptic_gregorian as having no leap seconds (meaning that they are not valid datetimes and intervals of time do not include them), and to recommend it for model data using the Gregorian calendar. We hoped (and requested, I think) that this would be adopted in CMIP7. I believe we should stick to this decision.

It was my more recent suggestion that we should recommend proleptic_gregorian for writing real-world obs data if the data-writer chooses not to require leap-second-aware software for UTC. That's what we currently have in PR #612. In view of Lars's and Chris's concerns, I think that was a mistake and I retract it.

In order to address these outstanding concerns, I've made another new version here. In this document, I've marked insertions wrt PR #612 with yellow highlight, and deletions with strikethrough and red highlight. Where I've moved a sentence without changing it, I have not marked it up. These changes are in 4.4.3 and Appendix L.

In this version,

• A definition of "no leap seconds" is added to the table caption, consistent with what Chris said, viz. leap seconds are not valid datetimes, conversion betwee datetimes and time coordinates should assume 60 seconds in every minute, as if there were no leap seconds (even if there might have been).

• In its description, proleptic_gregorian is specified to be a model calendar, with no leap seconds. Actually, it must be a model calendar, because it isn't Julian before 1582. It is now not recommended for real-world data (reversing my recent suggestion).

• I have inserted a column into the table for "exact intervals". This states whether or not the difference between two time coordinates exactly equals the intervening time interval. It is true in all cases except for the coordinates identifying UTC datetimes in the standard calendar. I think this distinction is useful because it isn't the same distinction as the "leap seconds" column makes. This kind of statement has appeared several times in our text, and hasn't been criticised as unclear or misleading, so I hope it'll be acceptable.

• To avoid the uncertainty that afflicts existing data in the standard calendar, I have made a distinction in the table and text between standard before CF 1.13, and standard since CF 1.13. The latter defines standard to have no leap seconds in CF 1.13 (but doesn't affect earlier data, of course). The CF 1.13 definition is one of the possibilities previously allowed, so I don't think it breaks backward compatibility in any problematic way. Do you? We can maintain this new clean definition of standard through our recommendations in CF 1.13 that model data is written with the proleptic_gregorian calendar, and that leap seconds must be ignored when converting UTC datetimes to time coordinates in the standard calendar.

• The term "leap-second-[un]aware" is not used anywhere.

• In the Appendix, I have inserted a link to the IERS leap second list, to explain what the "leap-second-aware" idea means.

• if the writer chooses not to use utc, standard is recommended for real-world data (not proleptic_gregorian, reversing my previous suggestion).

• standard datetimes are stated to be UTC after 1972.

• I have stated that the Appendix is focussed on guidance for the data-writer. This means we can repeat less stuff for the sake of the data-user. What they need to know is in Sect 4.4.3.

• I have deleted the sentence we have just been working on: For example, equipment that automatically records data with UTC datetimes can insert a new leap second only if it has prior knowledge of when this is to be done, or is appropriately synchronized to UTC time services that deliver leap second support. Although this is true, I suggest that we don't need to say it. If the data-writer is going to use the standard calendar for real-world data since 1972, it must be their responsibility to correct their UTC datetimes for problems of this kind before converting them into time coordinates (when leap seconds are excluded). Moreover, there haven't been any new leap seconds since 2016 and there probably won't be, so I guess there are not likely to be any new datasets produced from obs data with raw uncorrected elapsed times that might be affected by leap seconds. This could be a legacy issue.

I realise of course that all the above changes come at the last-minute and aren't entirely trivial. I am hoping that they are consistent with the views we've been converging on, though, and that therefore you will all find them acceptable. This would clarify things for users of the standard, I believe.

I'm curious to hear your reactions!

Happy Monday.

Jonathan

[davidhassell]

Question (time's short, no preamble :)) - at CF-1.13, what if a model with no leap seconds does (which is allowed) use the standard calendar for datetimes >= 1972 (rather than proleptic_gregorian)? We'd be back in some sort of "unknown" territory, I think. Shouldn't we say that the standard calendar must not be used for model data after 1972 that does not use leap seconds?

[JonathanGregory]

In the summer we agreed to recommend proleptic_gregorian for model data. My newer version of this weekend (the one with highlighting) is relying on that recommendation to keep standard "clean", meaning its datetimes are UTC, but its time coordinates are encoded as if no leap seconds. As you say, we could require that model data must not be written with the standard calendar for datetimes after 1972. I think that would be fine. What do others think?

In this newer version, both proleptic_gregorian and standard require non-leap-second-aware conversion between datetimes and time coordinates, and leap second are invalid datetimes in both of them. The difference is that proleptic_gregorian truly has no leap seconds, whereas standard has UTC datetimes but leap seconds are excluded and not counted. This is the same as the distinction between units_metadata leap_seconds of none and utc in CF 1.12 (I think!) but I have not said that because it would confuse the present issue.

[larsbarring]

Regarding @JonathanGregory's second point

In its description, proleptic_gregorian is specified to be a model calendar, with no leap seconds. Actually, it must be a model calendar, because it isn't Julian before 1582. It is now not recommended for real-world data (reversing my recent suggestion).

In the world outside the usual CF domains a proleptic calendar is widely used to extend further back in time what the same calendar otherwise can not define.

Wikipedia has a brief general entry on Proleptic calendar:

---

A proleptic calendar is a calendar that is applied to dates before its introduction. Examples include:

• Proleptic Gregorian calendar
• Proleptic Julian calendar
• Symmetry454

---

There are several ISO standards that specifically mention the proleptic calendar, and refers to ISO 8601 for its definition, for example:

• ISO 19111:2019(en): Geographic information — Referencing by coordinates (also open OGC standard)
• ISO 19162:2019(en): Geographic information — Well-known text representation of coordinate reference systems (also open OGC standard)
• ISO 34000:2023(en): Date and time — Vocabulary

I do think that we are doing CF a disservice if we are ignoring these standards!

Currently we have no use case for models having (leap) second precision, which mean that we can recommend that model data use the proleptic_gregorian even if the leap second is not explicitly represented. I think that we have a long way to go until there is a problem if model results are out of phase with observed conditions by about half a minute over 50 years. There are bigger fish to fry both for CF to handle and model developers, as well as for observational data collectors.

---

Finally, the new version of Appendix L contains many changes. I t will take some time to digest and contemplate, I will do my best...

[larsbarring]

And to answer the specific question: Yes, I think we should do that.

(As always, almost, CF only recommend -- or not -- things, which means that if there ever is a reasonable use case for for having Yhror fmin, or ...., it is indeed CF compliant!)

In CF, following UDUNITS, any unit may optionally have one of the decimal prefixes for multiples and submultiples (Table 3.1) e.g. millisecond or ms. CF recommends not to use these prefixes with any unit of time other than second, because [SI] (chapter 4) does not allow them except with second.

Do you (all) think we should do that (and a corresponding addition to the conformance document)?

[JonathanGregory]

Dear Lars

You write:

Currently we have no use case for models having (leap) second precision, which mean that we can recommend that model data use the proleptic_gregorian even if the leap second is not explicitly represented.

Yes, I agree. In one or two places the text says we're assuming that models don't have leap seconds, although in principle they could. If in future there are models with leap seconds, they can use the utc calendar. We wouldn't need to change anything about the convention except for clarifying this point.

Thanks for looking at the new Appendix. I agree that there is quite a lot of yellow, but it's not supposed to have any new ideas. The only change in substance is to revert to recommending standard for real-world data, if it's not utc. That's what we agreed in the summer, and what we had until I suggested something else a couple of weeks ago. Reverting was motivated by your and Chris's concerns that we shouldn't deprecate the standard calendar, and that it could be confusing to use the same calendar for a situation that is truly without leap seconds and a situation which has leap seconds but they're not being counted. Other changes are also intended to address comments that were made in the last few days.

I wonder what you think of it, @ChrisBarker-NOAA, Patrick @pvanlaake and @sethmcg?

Best wishes

Jonathan

[pvanlaake]

This being a very late hour to make any more changes, I am not going to do so.

But having considered this for a full year now, most of the concepts are much clearer to me now, as will be the case for all of us. One thing that stands out is that the standard calendar is not standard at all by any standard, the proleptic_gregorian calendar is as Lars made clear with his references to various ISO and OGC standards. It appears that CF calls the standard calendar like so because it is the default calendar in UDUNITS. Perhaps something to consider for CF-1.14?

I'm happy with the text as it currently is.

[ChrisBarker-NOAA]

One thing that stands out is that the standard calendar is not standard at all by any standard ...

indeed -- I don't know what it got that name back in the day, but oh well, we're kinda stuck with it now. And it's not that bad -- "standard name" means: "standard in CF" and that's also the case for the standard calendar.

In the summer we agreed to recommend proleptic_gregorian for model data. My newer version of this weekend (the one with highlighting) is relying on that recommendation to keep standard "clean", meaning its datetimes are UTC, but its time coordinates are encoded as if no leap seconds. As you say, we could require that model data must not be written with the standard calendar for datetimes after 1972. I think that would be fine. What do others think?

I'm still twisted around with this in my head. There are (at least) two use-cases here that I'm concerned about.

(1) Folks started with UTC datetimes and encoded them -- "real world data"

(2) Folks started with model generated data -- so presumably the deltas are correct, but the epoch may have who knows what leap seconds included. I say that because folks usually start with datetimes, and then encode to an arbitrary epoch [sometimes poorly chosen] on writing the file ...

• key to this is that in this case, the precision is such that no one knows or cares about leap seconds.

I think the challenge is that we've been focused on what the calendar definition itself is - that is, what days, months, years, seconds, it includes, etc. And I think we have that pretty well nailed down.

But we don't quite have the use cases (that is, the wonky use-cases) nailed down.

I don't know that we need another name or calendar, but we need to nail down the meanings.

I think we should have one that means:

"we're not really sure what the story is with leap seconds, but we also don't care"

• I thought that was the standard calendar

and one that means:

"there are no leap-second datetimes in here, but when you decode, you'll get UTC datetimes (with leap seconds) back"

• I thought, until today, that that was what proleptic_gregorian was.

But now it seems we want to swap those meanings?

This is complicated by the fact that the proleptic_gregorian and standard calendars differ in how they handle pre-gregorian times. -- which means we kinda need four calendars to be complete, but I don't think there's a use case -- if you are dealing with pre-gregogian times, you can't possibly care about leap second precision.

Clear as mud?