Various enhancements for zopen tools #58

IgorTodorovskiIBM · 2024-04-22T18:21:24Z

pthread_create-created threads do not set autocvt by default. This can be problematic if _BPXK_AUTOCVT is unset in the user environment. This PR adds a change to override pthread_create which sets AUTOCVT=ON in a similar way to the startup routine in zos.cc.
adds netinet/in_systm.h is missing, affects honggfuzz and nmap
adds getline and getdelim - affects honggfuzz, libssh2 and moreutils
Adds _SC_NPROCESSORS_ONLN to sysconf - encountered too many times and rather than editing the application code, I extended sysconf to support _SC_NPROCESSORS_ONLN
It also modifies the untagged read heuristic to remove the restriction on the min required bytes (currently set to 8 bytes). The reason for this is that there are <8 byte EBCDIC files such as /etc/syslog.pid and /etc/sshd.pid and based on user experience, untagged files which are small in size typically contain EBCDIC data and not binary data or ascii data so we should lean towards marking it as EBCDIC. Additionally, all z/OS Open Tools tag their data. Note, the __UNTAGGED_READ_MODE environment variable still takes precedence when set but most users do not set this environment variable.
- This will help resolve:
Also changes chgfdccsid to not set the txtflag when ccsid = 0
Adds more tests

MikeFultonDev · 2024-06-26T18:40:51Z

src/zos-char-util.cc

+    if (ccsid == 1047 && len == cnt) {
+      if (no_tag_read_behaviour == __NO_TAG_READ_DEFAULT_WITHWARNING) {
+        if (name) {
+          len = strlen(name) + 1;


minor thing but i would typically have len be the strlen and then allocate the buffer to len+1 since you don't need to convert a 0 to a 0.
But perhaps just me.

MikeFultonDev · 2024-06-26T18:41:14Z

src/zos-char-util.cc

-        return 1;
      }
-    }       // cnt > 8
+      fdcache.set_attribute(fd, 0x0000000000020000UL);


use a name for the magic number?

MikeFultonDev · 2024-06-26T18:42:50Z

src/zos-io.cc

+    size_t pos = 0;
+
+    if (*lineptr == NULL || *n == 0) {
+        *n = 128;


why 128? and why 2 farther down? would those be good to have as buffer length and growth size macros (assuming that's what they are?)

MikeFultonDev · 2024-06-26T18:43:47Z

src/zos-io.cc

+  int c;
+
+  if (buf == NULL || bufsize == 0) {
+    bufsize = 128;


same comment as above.

test/test-clib-override.cc

MikeFultonDev

reviewed - minor comments.

include/zos-v2r5-symbolfixes.h

IgorTodorovskiIBM · 2024-07-11T19:37:10Z

Added a readlink override to address zopencommunity/coreutilsport#74 and zopencommunity/gitport#122

AnthonyGiorgio · 2024-07-16T11:27:54Z

src/zos-io.cc

-        *lineptr = (char *)malloc(*n);
+    if (*lineptr == NULL || *n == 0) {
+        *n = 120;
+        *lineptr = (char *)realloc(*lineptr, *n);


Calling realloc() this way is incorrect. If the memory allocation fails, then the original value of lineptr is lost, and the memory it pointed to is leaked. cppcheck complains about this, and suggests you use a temporary pointer to hold the return value, and then assign it to lineptr after validation.

https://stackoverflow.com/questions/21006707/proper-usage-of-realloc

- Override pthread_create to always set autocvt=on - Modify untagged heuristic to remove restriction on min required bytes (8) - Add tests

gabylb · 2025-06-03T12:08:18Z

include/sys/file.h

+#if (__TARGET_LIB__ < 0x42050000) || \
+    (__TARGET_LIB__ >= 0x42050000 && \
+    (((_POSIX_C_SOURCE + 0) < 200809L) && !defined(_XPLATFORM_SOURCE)))


Suggested change

#if (__TARGET_LIB__ < 0x42050000) || \

(__TARGET_LIB__ >= 0x42050000 && \

(((_POSIX_C_SOURCE + 0) < 200809L) && !defined(_XPLATFORM_SOURCE)))

#if (__TARGET_LIB__ < 0x42050000) || \

(((_POSIX_C_SOURCE + 0) < 200809L) && !defined(_XPLATFORM_SOURCE))

If a is not < b then a >= b. Same applies to the change in zos-getentropy.h (and a few other places).

… works in fopen in addition to open

…n() existing behaviour

* fix: fdopendir and statfs conflict override * feat: strerror without EDC prefix * fix: get rid of fdopendir * fix: get rid of _Packed (maintenance) * refactor: igor - make changes consistent with zoslib * fix: strerror not working due to linker * Update include/zos-v2r5-symbolfixes.h Co-authored-by: Igor Todorovski <[email protected]> * Update include/dirent.h Co-authored-by: Igor Todorovski <[email protected]> * document statfs override --------- Co-authored-by: Igor Todorovski <[email protected]>

* feat: aligned operator new and delete * add todo

* Add initial implementation of ptrace and tests * Guard clock_gettime and nanosleep in time.h to fix build error in zstd * Use _Z_EXPORT in ptrace

* Add error function * Fix various warnings * revert issue * Further updates

* Add mount readonly flag, spawn setsigdefault * Add duplocale. Introduce macro to use zoslib locale

* Add scandir, timegm implementation * Add portable remove_all function * Define FNM_EXTMATCH * Add cpp guard for filesystem header

This commit introduces the `__UNTAGGED_FILE_ENCODING` environment variable, which deprecates `__UNTAGGED_READ_MODE`, to control the handling of untagged files. The implementation prioritizes the new variable, falling back to the old one for compatibility. Documentation and tests have been updated accordingly.

… set

This reverts commit 2b9235e.

IgorTodorovskiIBM requested review from MikeFultonDev, gabylb and mfsysprog April 22, 2024 18:21

IgorTodorovskiIBM marked this pull request as draft April 22, 2024 23:47

HarithaIBM approved these changes May 6, 2024

View reviewed changes

IgorTodorovskiIBM mentioned this pull request Jun 24, 2024

cat issue with untagged text zopencommunity/coreutilsport#68

Open

MikeFultonDev reviewed Jun 26, 2024

View reviewed changes

src/zos-io.cc Outdated

int c;

if (buf == NULL || bufsize == 0) {

bufsize = 128;

Copy link

Collaborator

MikeFultonDev Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as above.

MikeFultonDev reviewed Jun 26, 2024

View reviewed changes

test/test-clib-override.cc Show resolved Hide resolved

MikeFultonDev reviewed Jun 26, 2024

View reviewed changes

test/test-clib-override.cc Show resolved Hide resolved

MikeFultonDev approved these changes Jun 26, 2024

View reviewed changes

gabylb reviewed Jun 27, 2024

View reviewed changes

include/zos-v2r5-symbolfixes.h Show resolved Hide resolved

AnthonyGiorgio reviewed Jul 16, 2024

View reviewed changes

IgorTodorovskiIBM added 9 commits July 16, 2024 09:05

zopen enhancements:

d95312c

- Override pthread_create to always set autocvt=on - Modify untagged heuristic to remove restriction on min required bytes (8) - Add tests

Add missing header netiniet

11eeb20

Add support for getline, getdelim, sysconf, memset

ebe82e0

Add utimesat and PTHREAD_RWLOCK_INITIALIZER

b32dcd4

Add readlink override to resolve PARMLIB variables like

7d497c6

Fix getdelim and readlink override

8253dfe

Fix typo

153ef5f

Fix getdelim again

b7074a5

Fix missing }

b0cd547

IgorTodorovskiIBM force-pushed the zopen2 branch from 5bbcd18 to b0cd547 Compare July 16, 2024 13:11

IgorTodorovskiIBM added 2 commits July 16, 2024 13:40

Address comments

276feb4

Address comments

827486b

IgorTodorovskiIBM marked this pull request as ready for review July 16, 2024 17:43

Add __threading_support - needed for boost and clang 1.1 builds

4145cf9

IgorTodorovskiIBM and others added 5 commits May 14, 2025 22:06

V2R5 considerations, malloc and posix_memalign changes

c3ff9d7

Fix condition in v2r5 symbosl check

2826707

Avoid defining XPLAT and cleanup override headers (#82)

8c3872e

Add sys/poll.h

5d82be6

Update to v2r5 (#83)

0821373

gabylb reviewed Jun 3, 2025

View reviewed changes

IgorTodorovskiIBM and others added 18 commits June 5, 2025 17:49

Use fcntl F_SETTAG instead of __fchattr for __tag_new_file so that it…

404cf6f

… works in fopen in addition to open

Introduce __tag_new_file_fp for FILE services so as not to affect ope…

6bdd87e

…n() existing behaviour

Update poll.h - fix extra _ in #define for sys/poll.h

04aabf5

invoke ascii strerror_r (#87)

338e49a

Aligned operator new and delete (#88)

232a618

* feat: aligned operator new and delete * add todo

Add initial implementation of ptrace + test cases (#85)

e7295d4

* Add initial implementation of ptrace and tests * Guard clock_gettime and nanosleep in time.h to fix build error in zstd * Use _Z_EXPORT in ptrace

Add error() routine and resolve zoslib compiler warnings (#89)

7584f6a

* Add error function * Fix various warnings * revert issue * Further updates

Add fnmatch code (#90)

fb6f6e3

Use __asm instead of asm which is a gnu extension

caa88f6

Add mount readonly flag, spawn setsigdefault (#91)

7d0fe23

* Add mount readonly flag, spawn setsigdefault * Add duplocale. Introduce macro to use zoslib locale

Correct locale macro check (#92)

8fb09ef

Fix locale issue when stdlib included (#93)

fdccdad

Add scandir, timegm, portable_remove_all implementation (#94)

9e9de33

* Add scandir, timegm implementation * Add portable remove_all function * Define FNM_EXTMATCH * Add cpp guard for filesystem header

Update zos-io.cc - update __chgpathccsid to set txtflag if ccsid != 0

a39f737

undef strnlen, strpcpy, and strsignal from string.h

0e6da67

Update string.h - remove faulty functions in LE

19364de

Update string.h - fix typos

aba8e73

v1gnesh mentioned this pull request Oct 8, 2025

Unable to build due to multiple errors zopencommunity/duckdbport#29

Open

sachintu47 and others added 5 commits November 17, 2025 20:10

Add pkgconfig

bd296d5

Add FTS (File Tree Stream) implementation from musl-fts

f96d9f9

Fix error() function and expose only if ZOSLIB_OVERRIDE_CLIB_ERROR is…

45267d8

… set

Revert "feat: Add __UNTAGGED_FILE_ENCODING envar"

e5599f3

This reverts commit 2b9235e.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Various enhancements for zopen tools #58

Various enhancements for zopen tools #58

Uh oh!

IgorTodorovskiIBM commented Apr 22, 2024 •

edited

Loading

Uh oh!

MikeFultonDev Jun 26, 2024

Uh oh!

MikeFultonDev Jun 26, 2024

Uh oh!

MikeFultonDev Jun 26, 2024

Uh oh!

MikeFultonDev Jun 26, 2024

Uh oh!

Uh oh!

Uh oh!

MikeFultonDev left a comment

Uh oh!

Uh oh!

IgorTodorovskiIBM commented Jul 11, 2024

Uh oh!

AnthonyGiorgio Jul 16, 2024 •

edited

Loading

Uh oh!

gabylb Jun 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Various enhancements for zopen tools #58

Are you sure you want to change the base?

Various enhancements for zopen tools #58

Uh oh!

Conversation

IgorTodorovskiIBM commented Apr 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MikeFultonDev Jun 26, 2024

Choose a reason for hiding this comment

Uh oh!

MikeFultonDev Jun 26, 2024

Choose a reason for hiding this comment

Uh oh!

MikeFultonDev Jun 26, 2024

Choose a reason for hiding this comment

Uh oh!

MikeFultonDev Jun 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MikeFultonDev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

IgorTodorovskiIBM commented Jul 11, 2024

Uh oh!

AnthonyGiorgio Jul 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabylb Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

IgorTodorovskiIBM commented Apr 22, 2024 •

edited

Loading

AnthonyGiorgio Jul 16, 2024 •

edited

Loading