gatsby 4 failing in gitlab-runner

[kmajzlik]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

While upgrade Gatsby from 3 to 4 i have failing pipelines with gitlab-runner.
I have shell runner, tried to use both Node 14 and 16.
If i run gatsby build --verbose under gitlab-runner user in same directory it succeeds.
I also tried FORCE_COLOR=0 ~/.yarn/bin/gatsby build --verbose &> build-log.txt and finish in build-log with same result.

On multiple projects i have same result with Gatsby 4 (tried 4.0.0, 4.2.0 and also 4.2-next):

Reproduction Link

none

Steps to Reproduce

1. use Gatsby4 in gitlab-runner

Expected Result

Successful build.

Actual Result

Failing build.

error UNHANDLED EXCEPTION write EPIPE
  Error: write EPIPE
  - child_process:846 ChildProcess.target._send
    node:internal/child_process:846:20

  - child_process:719 ChildProcess.target.send
    node:internal/child_process:719:19

  - index.js:290 WorkerPool.sendMessage
    [gatsby]/[gatsby-worker]/dist/index.js:290:19

  - worker-messaging.ts:22
    [gatsby]/[gatsby]/src/utils/jobs/worker-messaging.ts:22:22
not finished Merge worker state - 0.667s

Environment

System:
    OS: Linux 5.4 Ubuntu 20.04.3 LTS (Focal Fossa)
    CPU: (4) x64 Intel(R) Pentium(R) CPU J4205 @ 1.50GHz
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 16.13.0 - /usr/bin/node
    Yarn: 1.22.15 - /usr/bin/yarn
    npm: 8.1.0 - /usr/bin/npm
  Browsers:
    Firefox: 94.0
  npmPackages:
    gatsby: ^4.2.0 => 4.2.0
    gatsby-plugin-config: ^1.0.7 => 1.0.7
    gatsby-plugin-global-context: ^1.0.0 => 1.0.0
    gatsby-plugin-image: ^2.2.0 => 2.2.0
    gatsby-plugin-manifest: ^4.2.0 => 4.2.0
    gatsby-plugin-react-helmet: ^5.2.0 => 5.2.0
    gatsby-plugin-sass: ^5.2.0 => 5.2.0
    gatsby-plugin-sharp: ^4.2.0 => 4.2.0
    gatsby-plugin-styled-components: ^5.2.0 => 5.2.0
    gatsby-plugin-typography: ^4.2.0 => 4.2.0
    gatsby-plugin-use-query-params: ^1.0.1 => 1.0.1
    gatsby-source-drupal: ^5.2.0 => 5.2.0
    gatsby-source-filesystem: ^4.2.0 => 4.2.0
    gatsby-transformer-sharp: ^4.2.0 => 4.2.0

Config Flags

None

[LekoArts]

Hi!

This most probably means that your runner is running out of memory. Try to monitor the memory or give the runner more memory and it should work. The new abstraction with running queries in workers needs a bit more memory at the moment.

[kmajzlik]

Hey, any advice how to fix it? How to make build running longer but do not fail.
I also tried to run in Docker. 6GB of RAM not enough?

My docker-compose.yml

version: '3'
services:
  node:
    container_name: "gatsby-builder"
    build:
      dockerfile: Dockerfile
      context: ./
    environment:
      GATSBY_CPU_COUNT: 1
      NODE_OPTIONS: '--max_old_space_size=8192'
    volumes:
      - "../gatsby/:/opt/app"
    deploy:
      resources:
        reservations:
          cpus: '2.00'
          memory: 6G
        limits:
          cpus: '3.00'
          memory: 8G

Error:

⠋ Merge worker state
/usr/local/share/.config/yarn/global/node_modules/yoga-layout-prebuilt/yoga-layout/build/Release/nbind.js:53
        throw ex;
        ^

Error: write EPIPE
    at ChildProcess.target._send (node:internal/child_process:846:20)
    at ChildProcess.target.send (node:internal/child_process:719:19)
    at WorkerPool.sendMessage (/opt/app/node_modules/gatsby-worker/dist/index.js:290:19)
    at /opt/app/node_modules/gatsby/src/utils/jobs/worker-messaging.ts:22:22
Emitted 'error' event on ChildProcess instance at:
    at node:internal/child_process:850:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

[kmajzlik]

docker-compose.yml

version: '2'
services:
  node:
    container_name: "gatsby-build"
    build:
      dockerfile: Dockerfile
      context: ./
    environment:
      GATSBY_CPU_COUNT: 1
      NODE_OPTIONS: '--max_old_space_size=4096'
    cpus: 2
    mem_limit: 8g
    mem_reservation: 6g
    privileged: true

• Running on Atom J4205 server with 16GB, Samsung PRO SSD, memory usage before build is ~500M (basically samba server for home sharing - no traffic, on other workload).
• Ubuntu 20.04 server (86_64 of course).
• Docker image based on node:16.
• While building memory usage is higher and higher, it goes to between 3 and 4GB and fails.

Should Gatsby be really enterprise-only product only for those with big servers?
My site had no problem on Gatsby V3. It seems in issues that more people is facing some problems with G4 failing for unknown reasons.

and while later

info bootstrap finished - 1385.458s
success write out requires - 0.053s
success Building production JavaScript and CSS bundles - 118.034s
success Building HTML renderer - 64.891s
success Execute page configs - 0.059s
success Caching Webpack compilations - 0.002s
success run queries in workers - 534.753s - 7040/7040 13.16/s
error UNHANDLED EXCEPTION write EPIPE
  Error: write EPIPE
  
  - child_process:846 ChildProcess.target._send
    node:internal/child_process:846:20
  
  - child_process:719 ChildProcess.target.send
    node:internal/child_process:719:19
  
  - index.js:290 WorkerPool.sendMessage
    [gatsby]/[gatsby-worker]/dist/index.js:290:19
  
  - worker-messaging.ts:22 
    [gatsby]/[gatsby]/src/utils/jobs/worker-messaging.ts:22:22
  
not finished Merge worker state - 0.806s

[kmajzlik]

Crosslink: #34106
So maybe we are related sharp.

[graysonhicks]

@kmajzlik You can lower peak memory usage following this suggestion: #34215 (comment)

[kmajzlik]

@graysonhicks - thanks for advice. Originally i had 4 and worked well with Gatsby 3. After upgrade i tried to decrease to 2 and also 1.
With cpu count = 1 i see in log: "1 worker started" (number was different with different setting so setting is applied).
Everytime this fail refers to "messaging" and "childProcess".

[kmajzlik]

Crosslink: #34215

[feedm3]

Not exactly the same, but within our CI runners we also see this error around every second build:

error An error occurred during parallel query running.
Go here for troubleshooting tips: https://gatsby.dev/pqr-feedback


      Error: Worker exited before finishing task
      
      - index.js:193 
        [SERVER_NAME]/[gatsby-worker]/dist/index.js:193:31
      
    
error Command failed with exit code 1.

Super annoying. There is no indication of what the error is. All we do is to start the build again, usually, it works then. Those CI servers don't have a lot of memory (don't know how much to be honest), but we never had any issues with Gatsby 3.

I also added a comment about the memory usage - that increased dramatically - on our production server here: #34215 (comment)