lines property in Zope 5.3: new one has text, old one has bytes #987
Description
Zope 5.3 changed the lines property from storing bytes to storing text. See issue #962 and PR #965.
What now happens:
- Create a Zope 5.2.1 site, add a Folder A, and in the ZMI add a lines property on it. It contains a tuple of bytes.
- Update to Zope 5.3. (I simply switch between git tags.)
- Create folder B and add a lines property. It contains a tuple of texts (strings).
- The lines property in folder A still contains a tuple of bytes.
- Save the properties form in folder A, and it is converted to a tuple of strings.
So depending on when you created or edited the lines property, getProperty in Zope 5.3 may return bytes or string.
I don't really know if this is a problem or not.
But there are already some test failures in GenericSetup, which is why I started looking. Apart from some expectations in tests which need to be updated, GenericSetup needs a few fixes in the part of its code where it is adding or updating properties, so the affected code is limited. See PR. The most difficult about that PR, is that there is code in GenericSetup which supports non-utf8 encodings in profile files, so the changes are more convoluted than they should have been. Other packages which do similar things, would need to update their code as well.
If we think this should be fixed in Zope, I can see two solutions:
- In
getPropertycall the converter of the field and return the result. But this means extra code gets run whenever you callgetProperty. Not nice for performance. - Add a function that goes over all properties (or all lines properties) of an object, calls the converters, and saves changes. Integrators could apply this with
zopeFindAndApply. Plone could choose to do this in a GenericSetup upgrade step, although this may take long when your Data.fs is 100 GB.
Additionally, a function could be useful to go over all deprecated properties, for example turning ulines into lines. The two functions could be combined.
Does someone already have code like this? Even when Zope itself does not call the function anywhere, having it available in Zope would be useful.
cc @jugmac00
O, let me share some output from debugging. I started with Zope 5.2.1 and several properties containing non-ascii: äh. In the ZMI this looks fine in all fields, except for the lines field, where it looks wrong: äh. With bin/instance debug:
>>> app.testfolder.getProperty("lines")
(b'foo', b'\xc3\xa4h')
>>> app.testfolder.getProperty("lines")[1].decode("utf-8")
'äh'
Save it a couple of times, and the ZMI show äh:
>>> app.testfolder.getProperty("lines")
(b'foo', b'\xc3\x83\xc6\x92\xc3\x86\xe2\x80\x99\xc3\x83\xe2\x80\x9a\xc3\x82\xc2\xa4h')
So a lines field with non-ascii in older Zopes was already wrong: you should have used a ulines field. So probably the real problems are only there when you already had a problem in the earlier Zopes.
Start in Zope 5.3 and it it still looks wrong, because the original data is bad. Saving it once (or multiple times), changes the bytes to string, but it is still looking wrong:
>>> app.testfolder.getProperty("lines")
('foo', 'äh')
Manually insert the proper äh in the ZMI and save, and then it is fine:
>>> app.testfolder.getProperty("lines")
('foo', 'äh')