Fails to build on OpenSSL-4.x-dev

[mouse07410]

Problem

OpenSSL-4 officially removed support for Engine API from the code. As a result, attempt to build this repo with OpenSSL-4-dev fails:

/opt/local/bin/cmake -S/Users/ur20980/src/grasshopper-engine -B/Users/ur20980/src/grasshopper-engine/build --check-build-system CMakeFiles/Makefile.cmake 0
/opt/local/bin/cmake -E cmake_progress_start /Users/ur20980/src/grasshopper-engine/build/CMakeFiles /Users/ur20980/src/grasshopper-engine/build//CMakeFiles/progress.marks
/Applications/Xcode.app/Contents/Developer/usr/bin/make  -f CMakeFiles/Makefile2 all
/Applications/Xcode.app/Contents/Developer/usr/bin/make  -f CMakeFiles/test_digest.dir/build.make CMakeFiles/test_digest.dir/depend
cd /Users/ur20980/src/grasshopper-engine/build && /opt/local/bin/cmake -E cmake_depends "Unix Makefiles" /Users/ur20980/src/grasshopper-engine /Users/ur20980/src/grasshopper-engine /Users/ur20980/src/grasshopper-engine/build /Users/ur20980/src/grasshopper-engine/build /Users/ur20980/src/grasshopper-engine/build/CMakeFiles/test_digest.dir/DependInfo.cmake "--color="
/Applications/Xcode.app/Contents/Developer/usr/bin/make  -f CMakeFiles/test_digest.dir/build.make CMakeFiles/test_digest.dir/build
[  1%] Building C object CMakeFiles/test_digest.dir/test_digest.c.o
/opt/local/bin/clang -DENGINE_DIR=\"/Users/ur20980/src/grasshopper-engine/build/bin\" -DL_ENDIAN -isystem /Users/ur20980/openssl-4/include -O3 -std=gnu18 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -O2 -std=gnu90 -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX26.2.sdk -Werror -Wall -Wno-unused-parameter -Wno-unused-function -Wno-missing-braces -Qunused-arguments -Wno-deprecated-declarations "-Wno-error=#warnings" -MD -MT CMakeFiles/test_digest.dir/test_digest.c.o -MF CMakeFiles/test_digest.dir/test_digest.c.o.d -o CMakeFiles/test_digest.dir/test_digest.c.o -c /Users/ur20980/src/grasshopper-engine/test_digest.c
In file included from /Users/ur20980/src/grasshopper-engine/test_digest.c:33:
/Users/ur20980/src/grasshopper-engine/gost_lcl.h:34:5: warning: "Gost-engine is built against not fully supported version of OpenSSL. EVP_CTRL_SET_TLSTREE_PARAMS definition in OpenSSL is expected." [-W#warnings]
   34 | #   warning "Gost-engine is built against not fully supported version of OpenSSL. \
      |     ^
/Users/ur20980/src/grasshopper-engine/gost_lcl.h:46:5: warning: "Gost-engine is built against not fully supported version of OpenSSL. NID_magma_mgm definition in OpenSSL is expected. No magma mgm functionality is guaranteed." [-W#warnings]
   46 | #   warning "Gost-engine is built against not fully supported version of OpenSSL. \
      |     ^
/Users/ur20980/src/grasshopper-engine/gost_lcl.h:59:5: warning: "Gost-engine is built against not fully supported version of OpenSSL. NID_kuznyechik_mgm definition in OpenSSL is expected. No kuznyechik mgm functionality is guaranteed." [-W#warnings]
   59 | #   warning "Gost-engine is built against not fully supported version of OpenSSL. \
      |     ^
3 warnings generated.
[  2%] Linking C executable bin/test_digest
/opt/local/bin/cmake -E cmake_link_script CMakeFiles/test_digest.dir/link.txt --verbose=1
Undefined symbols for architecture arm64:
  "_ENGINE_by_id", referenced from:
      _warn_all_untested in test_digest.c.o
  "_ENGINE_finish", referenced from:
      _warn_all_untested in test_digest.c.o
  "_ENGINE_free", referenced from:
      _engine_is_available in test_digest.c.o
      _warn_all_untested in test_digest.c.o
      _warn_all_untested in test_digest.c.o
  "_ENGINE_get_digest", referenced from:
      _warn_all_untested in test_digest.c.o
  "_ENGINE_get_digests", referenced from:
      _warn_all_untested in test_digest.c.o
  "_ENGINE_get_first", referenced from:
      _engine_is_available in test_digest.c.o
      _warn_all_untested in test_digest.c.o
  "_ENGINE_get_id", referenced from:
      _engine_is_available in test_digest.c.o
      _warn_all_untested in test_digest.c.o
  "_ENGINE_get_next", referenced from:
      _engine_is_available in test_digest.c.o
      _warn_all_untested in test_digest.c.o
  "_ENGINE_init", referenced from:
      _warn_all_untested in test_digest.c.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
/opt/local/bin/clang -O3 -std=gnu18 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -O2 -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX26.2.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/test_digest.dir/test_digest.c.o -o bin/test_digest  /Users/ur20980/openssl-4/lib/libcrypto.dylib
make[2]: *** [bin/test_digest] Error 1
make[1]: *** [CMakeFiles/test_digest.dir/all] Error 2
make: *** [all] Error 2

In case configuration matters:

-- The C compiler identification is AppleClang 17.0.0.17000603
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/local/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found OpenSSL: /Users/ur20980/openssl-4/lib/libcrypto.dylib (found suitable version "4.0.0", minimum required is "3.4")
-- Found OpenSSL 4.0..
CMAKE_BUILD_TYPE = Release
-- Found OpenSSL includes:   /Users/ur20980/openssl-4/include
-- Found OpenSSL crypto lib: /Users/ur20980/openssl-4/lib/libcrypto.dylib
-- Found OpenSSL application: 
-- Found OpenSSL application: /Users/ur20980/openssl-4/bin/openssl
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - not found
-- LITTLE_ENDIAN
-- Performing Test ADDCARRY_U64
-- Performing Test ADDCARRY_U64 - Failed
-- Performing Test RELAXED_ALIGNMENT
-- Performing Test RELAXED_ALIGNMENT - Success
-- No Test2::V0 perl module (engine and provider tests skipped)
CMake Deprecation Warning at libprov/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.


CMake Warning (dev) at CMakeLists.txt:493 (install):
  Target lib_gost_engine has PUBLIC_HEADER files but no PUBLIC_HEADER
  DESTINATION.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done (1.4s)
-- Generating done (0.2s)
-- Build files have been written to: /Users/ur20980/src/grasshopper-engine/build

Proposed solution

Split cleanly between Engine and Provider build - make it possible to build one or the other, via make target, and describe this process in the README.

[Mironenko]

Thank you for the report. Unfortunately, the problem is deeper: provider depends on removed ENGINE interfaces, so a huge refactoring is anticipated.

[mouse07410]

provider depends on removed ENGINE interfaces, so a huge refactoring is anticipated

I suspected as much. ;-)

[chipitsine]

let's try the power of copilot ))

[chipitsine]

@mouse07410 can you have a look please #492

[mouse07410]

@chipitsine looks promising! Unfortunately, Copilot does not seem to accept commands from me - please feel free to take a look at #492 (comment) and tell Copilot to fix it.

[mouse07410]

Problem persists - summary

#492 fails to compile with OpenSSL-4.0-dev, and gostprov fails some tests on OpenSSL-3.5.4.

Details

OpenSSL-4.0-dev

Configuration:
cmake-4-out.txt

Failing build:
make-4-out.txt

Obviously, tests fail:
test-4-out.txt

Gist:

[  9%] Building C object CMakeFiles/gost_core.dir/gost_pmeth.c.o
/opt/local/bin/clang -DHAVE_ADDCARRY_U64 -DL_ENDIAN -isystem /Users/ur20980/openssl-4/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -g -O2 -std=gnu90 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX26.2.sdk -fPIC -Werror -Wall -Wno-unused-parameter -Wno-unused-function -Wno-missing-braces -Qunused-arguments -Wno-deprecated-declarations "-Wno-error=#warnings" -MD -MT CMakeFiles/gost_core.dir/gost_pmeth.c.o -MF CMakeFiles/gost_core.dir/gost_pmeth.c.o.d -o CMakeFiles/gost_core.dir/gost_pmeth.c.o -c /Users/ur20980/src/grasshopper-engine/gost_pmeth.c
In file included from /Users/ur20980/src/grasshopper-engine/gost_pmeth.c:19:
/Users/ur20980/src/grasshopper-engine/gost_lcl.h:36:5: warning: "Gost-engine is built against not fully supported version of OpenSSL. EVP_CTRL_SET_TLSTREE_PARAMS definition in OpenSSL is expected." [-W#warnings]
   36 | #   warning "Gost-engine is built against not fully supported version of OpenSSL. \
      |     ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:813:24: error: implicit declaration of function 'EVP_MD_meth_get_ctrl' [-Werror,-Wimplicit-function-declaration]
  813 |                 return EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |                        ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:813:24: note: did you mean 'BIO_meth_get_ctrl'?
/Users/ur20980/openssl-4/include/openssl/bio.h:1018:30: note: 'BIO_meth_get_ctrl' declared here
 1018 | OSSL_DEPRECATEDIN_3_5 long (*BIO_meth_get_ctrl(const BIO_METHOD *biom))(BIO *, int,
      |                              ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:814:21: error: called object type 'int' is not a function or function pointer
  813 |                 return EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  814 |                     (mctx, EVP_MD_CTRL_SET_KEY, 0, key);
      |                     ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:816:24: error: implicit declaration of function 'EVP_MD_meth_get_ctrl' [-Werror,-Wimplicit-function-declaration]
  816 |                 return EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |                        ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:817:21: error: called object type 'int' is not a function or function pointer
  816 |                 return EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  817 |                     (mctx, EVP_MD_CTRL_SET_KEY, 32, &(data->key));
      |                     ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:946:24: error: implicit declaration of function 'EVP_MD_meth_get_ctrl' [-Werror,-Wimplicit-function-declaration]
  946 |                 return EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |                        ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:947:21: error: called object type 'int' is not a function or function pointer
  946 |                 return EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  947 |                     (mctx, EVP_MD_CTRL_SET_KEY, 0, key);
      |                     ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:949:24: error: implicit declaration of function 'EVP_MD_meth_get_ctrl' [-Werror,-Wimplicit-function-declaration]
  949 |                 return EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |                        ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:950:21: error: called object type 'int' is not a function or function pointer
  949 |                 return EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  950 |                     (mctx, EVP_MD_CTRL_SET_KEY, 32, &(data->key));
      |                     ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1135:5: error: implicit declaration of function 'EVP_MD_meth_get_ctrl' [-Werror,-Wimplicit-function-declaration]
 1135 |     EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |     ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1136:9: error: called object type 'int' is not a function or function pointer
 1135 |     EVP_MD_meth_get_ctrl(EVP_MD_CTX_md(mctx))
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 1136 |         (mctx, EVP_MD_CTRL_MAC_LEN, data->mac_size, NULL);
      |         ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1153:14: error: implicit declaration of function 'EVP_PKEY_meth_new' [-Werror,-Wimplicit-function-declaration]
 1153 |     *pmeth = EVP_PKEY_meth_new(id, flags);
      |              ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1153:12: error: incompatible integer to pointer conversion assigning to 'EVP_PKEY_METHOD *' (aka 'struct evp_pkey_method_st *') from 'int' [-Wint-conversion]
 1153 |     *pmeth = EVP_PKEY_meth_new(id, flags);
      |            ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1160:9: error: implicit declaration of function 'EVP_PKEY_meth_set_ctrl' [-Werror,-Wimplicit-function-declaration]
 1160 |         EVP_PKEY_meth_set_ctrl(*pmeth,
      |         ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1160:9: note: did you mean 'EVP_PKEY_asn1_set_ctrl'?
/Users/ur20980/openssl-4/include/openssl/evp.h:1586:6: note: 'EVP_PKEY_asn1_set_ctrl' declared here
 1586 | void EVP_PKEY_asn1_set_ctrl(EVP_PKEY_ASN1_METHOD *ameth,
      |      ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1162:9: error: implicit declaration of function 'EVP_PKEY_meth_set_sign' [-Werror,-Wimplicit-function-declaration]
 1162 |         EVP_PKEY_meth_set_sign(*pmeth, NULL, pkey_gost_ec_cp_sign);
      |         ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1163:9: error: implicit declaration of function 'EVP_PKEY_meth_set_verify' [-Werror,-Wimplicit-function-declaration]
 1163 |         EVP_PKEY_meth_set_verify(*pmeth, NULL, pkey_gost_ec_cp_verify);
      |         ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1165:9: error: implicit declaration of function 'EVP_PKEY_meth_set_keygen' [-Werror,-Wimplicit-function-declaration]
 1165 |         EVP_PKEY_meth_set_keygen(*pmeth, NULL, pkey_gost2001cp_keygen);
      |         ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1167:9: error: implicit declaration of function 'EVP_PKEY_meth_set_encrypt' [-Werror,-Wimplicit-function-declaration]
 1167 |         EVP_PKEY_meth_set_encrypt(*pmeth,
      |         ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1170:9: error: implicit declaration of function 'EVP_PKEY_meth_set_decrypt' [-Werror,-Wimplicit-function-declaration]
 1170 |         EVP_PKEY_meth_set_decrypt(*pmeth, NULL, pkey_gost_decrypt);
      |         ^
/Users/ur20980/src/grasshopper-engine/gost_pmeth.c:1171:9: error: implicit declaration of function 'EVP_PKEY_meth_set_derive' [-Werror,-Wimplicit-function-declaration]
 1171 |         EVP_PKEY_meth_set_derive(*pmeth,
      |         ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
1 warning and 20 errors generated.
make[2]: *** [CMakeFiles/gost_core.dir/gost_pmeth.c.o] Error 1

OpenSSL-3.5.4

Configuration:
cmake-3m-out.txt

Build:
make-3m-out.txt

Tests (provider tests fail, engine tests pass):
test-3m-long-out.txt

Gist:

Running tests...
/opt/local/bin/ctest --force-new-ctest-process 
Test project /Users/ur20980/src/grasshopper-engine/build
    Start 1: digest-with-provider
1/6 Test #1: digest-with-provider .............Subprocess aborted***Exception:   0.02 sec
    Start 2: ciphers-with-provider
2/6 Test #2: ciphers-with-provider ............Subprocess aborted***Exception:   0.02 sec
    Start 3: curves
3/6 Test #3: curves ...........................   Passed    0.07 sec
    Start 4: context-with-provider
4/6 Test #4: context-with-provider ............Subprocess aborted***Exception:   0.02 sec
    Start 5: keyexpimp
5/6 Test #5: keyexpimp ........................   Passed    0.04 sec
    Start 6: gost89
6/6 Test #6: gost89 ...........................   Passed    0.02 sec

50% tests passed, 3 tests failed out of 6

Total Test time (real) =   0.19 sec

The following tests FAILED:
          1 - digest-with-provider (Subprocess aborted)
          2 - ciphers-with-provider (Subprocess aborted)
          4 - context-with-provider (Subprocess aborted)

[chipitsine]

I think we must confess that copilot did its best and we won't get more from it

[mouse07410]

I think we must confess that copilot did its best and we won't get more from it

Respectfully disagree - in my experience, Copilot can do a lot more - with a lot of continuous prodding. If you expect to just point it at a problem and get a complete solution by its next iteration - you're likely to be very disappointed.

(As for me - Copilot helped very much in clearing out all the issues in one pretty complex repo I maintain.)

Update

Just pass my reported updates to it. I would've done it myself, but in this repo it appears to react only to you.

[mouse07410]

OK, I'm done here too.