[BUG]? With Lambda layer in version >= 137 `page.pdf()` consistently fails on second request

[jandppw]

We have an issue with Chromium on Lambda since version 137: browser.newPage() page.pdf() consistently fails on the second request. We are at wit’s end. Anybody any ideas?

(Edited: see follow-up post)

Environment

• chromium Version: 137 – 138
• puppeteer / puppeteer-core Version: 24.10.2 – 24.19.0
• Node.js Version: v22.15.1 on Lambda
• Lambda / GCF Runtime: nodejs22.x (which currently reports as v22.15.1)
• Runtime Architecture: x86_64

Expected Behavior

A new page should become available when requested.

Current Behavior

Edited: this is wrong

ConnectionClosedError: Connection closed.
at Connection.\_rawSend
(/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Connection.js:97:35) at Connection.send
(/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Connection.js:90:21) at CdpBrowser.\_createPageInContext
(/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Browser.js:181:53) at CdpBrowserContext.newPage
(/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/BrowserContext.js:123:40) at async CdpBrowser.newPage
(/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Browser.js:178:16) at async createAndStorePDF
(/var/task/lib/document/createAndStorePDF.js:49:16) at async contractFunction.getPDFDownloadURL
(/var/task/lib/document/DocumentGenerator.js:641:7) at async getDocument (/var/task/lib/getDocument.js:56:32) at async
Runtime.handlerImpl (/var/task/lib/service.js:171:22)

Corrected:

TargetCloseError: Protocol error (Page.printToPDF): Target closed
    at CallbackRegistry.clear (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/common/CallbackRegistry.js:81:36)
    at CdpCDPSession.onClosed (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/CdpSession.js:114:25)
    at #onClose (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Connection.js:181:21)
    at WebSocket.<anonymous> (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/node/NodeWebSocketTransport.js:48:30)
    at callListener (/var/task/node_modules/ws/lib/event-target.js:290:14)
    at WebSocket.onClose (/var/task/node_modules/ws/lib/event-target.js:220:9)
    at WebSocket.emit (node:events:518:28)
    at WebSocket.emitClose (/var/task/node_modules/ws/lib/websocket.js:272:10)
    at Socket.socketOnClose (/var/task/node_modules/ws/lib/websocket.js:1341:15)
    at Socket.emit (node:events:518:28)
    at TCP.<anonymous> (node:net:351:12)

and the browser disconnects.

Project history

This project has been running for years without much trouble. In the past we fought closing the browser in test, but
apart from that upgrades have been seamless. Thank you, @Sparticuz and contributors, for the excellent work.

This is still a CommonJS project.

Last working situation

The latest version combination that has been running smoothly for months was

• @sparticuz/chromium 133.0.0 as dependency
• puppeteer and puppeteer-core 24.10.2

puppeteer-core is used in production, on Lambda. puppeteer is used for local tests (macOS) and in the node:22 and
node:24 Docker image on CI.

On Lambda, we used a self-build layer, build from @sparticuz/chromium 133.0.0.

Lambda is --compatible-architectures x86_64 nodejs22.x (which currently reports as v22.15.1).

Upgrading

At this time the latest versions are

• @sparticuz/chromium 138.0.2
• puppeteer and puppeteer-core 24.19.0, targeted at Chromium 140.

To upgrade from @sparticuz/chromium 133.0.0 to @sparticuz/chromium 137.0.0, we needed to
take into account that we now have to supply arguments
and other options ourselves, and that the package now supports ESM. We cannot use @sparticuz/chromium 137.0.0,
137.0.1, 138.0.0, or 138.0.1 cannot be used in a CommonJS project (without jumping through some hoops). So, we
intend to upgrade to 138.0.2, which again supports CommonJS. The addition of AMD support should have no influence,
since at this time we are continuing to use x64.

Research and attempts

We literally tried every possible combination of versions over the last days. The error is consistent when we use either
a self-build layer or the layer downloaded from GitHub Assets > 133.0.0.

• Testing: locally (macOS) and in the node:22 and node:24 Docker image on CI
  • All puppeteer and puppeteer-core versions between 24.10.2 and 24.19.0 work with both
    @sparticuz/chromium 133.0.0 and 138.0.2 as devDependency
• Lambda: --compatible-architectures x86_64 nodejs22.x (which currently reports as v22.15.1)
  • All puppeteer and puppeteer-core versions between 24.10.2 and 24.19.0 work with both
    @sparticuz/chromium 133.0.0 and 138.0.2 as devDependency with the self-build and with the downloaded layer
    v133.0.0
  • All puppeteer and puppeteer-core versions between 24.10.2 and 24.19.0 fail with both
    @sparticuz/chromium 133.0.0 and 138.0.2 as devDependency on Lambda with the self-build and downloaded layer
    v138.0.2 and
    with the downloaded layer
    v137.0.1

The combination

• @sparticuz/chromium 138.0.2 as devDependency
• puppeteer 24.19.0 as devDependency
• puppeteer-core 24.19.0 as dependency
• with the downloaded layer
  v133.0.0

is the most up-to-date combination that works locally (macOS), in the node:22 and node:24 Docker image on CI, and in
Lambda x86_64 (which currently reports as v22.15.1).

Code

On first need, a browser is started, and the instance is cached. On later use, this instance is returned from the
cache. The instance is never closed on Lambda (there is no need).

// browser.js

/* NOTE: '@sparticuz/chromium' is a weird code structure. It exports a class Chromium, that only has static methods,
         _but uses `this` and state in its code_. So the static methods _must_ be called OO-style. */
const chromium = require('@sparticuz/chromium')
const log = require('../ppwcode/simpleLog')
const assert = require('assert')
const { join } = require('path')
const { kill } = require('../_util/kill')

const fontVariants = ['Bold', 'BoldItalic', 'Italic', 'Regular']
const fontBasePath = join(__dirname, 'fonts')
const fontFilePaths = fontVariants.map(fv => join(fontBasePath, `Roboto/Roboto-${fv}.ttf`))

/**
 * As advised by https://github.com/Sparticuz/chromium/releases/tag/v137.0.0
 */
const defaultViewport = {
  deviceScaleFactor: 1,
  hasTouch: false,
  height: 1080,
  isLandscape: true,
  isMobile: false,
  width: 1920
}

/**
 * As advised by https://github.com/Sparticuz/chromium/releases/tag/v137.0.0
 */
const headless = 'shell'

/**
 * As advised by https://github.com/Sparticuz/chromium/releases/tag/v137.0.0, extended as documented in
 * notes/ChromiumLaunchOptions/Puppeteer Chromium Options.xlsx
 */
const args = [
  /* NOTE: On Bitbucket Error: Failed to launch the browser process!
           xxx … Running as root without --no-sandbox is not supported. See https://crbug.com/638180. */
  '--no-sandbox',
  '--allow-pre-commit-input',
  '--allow-running-insecure-content',
  '--disable-background-networking',
  '--disable-background-timer-throttling',
  '--disable-backgrounding-occluded-windows',
  '--disable-breakpad',
  '--disable-client-side-phishing-detection',
  '--disable-component-extensions-with-background-pages',
  '--disable-component-update',
  '--disable-crash-reporter',
  '--disable-default-apps',
  '--disable-dev-shm-usage',
  '--disable-domain-reliability',
  '--disable-extensions',
  // eslint-disable-next-line no-secrets/no-secrets
  '--disable-features=Translate,BackForwardCache,AcceptCHFrame,MediaRouter,OptimizationHints,AudioServiceOutOfProcess,IsolateOrigins,site-per-process,ProcessPerSiteUpToMainFrameThreshold,IsolateSandboxedIframes',
  '--disable-hang-monitor',
  '--disable-infobars',
  '--disable-ipc-flooding-protection',
  '--disable-popup-blocking',
  '--disable-print-preview',
  '--disable-prompt-on-repost',
  '--disable-renderer-backgrounding',
  '--disable-search-engine-choice-screen',
  '--disable-setuid-sandbox',
  '--disable-site-isolation-trials',
  '--disable-speech-api',
  '--disable-speech-api',
  '--disable-sync',
  '--disable-web-security',
  '--disk-cache-size=33554432',
  '--enable-automation',
  '--enable-blink-features=IdleDetection',
  '--enable-features=NetworkServiceInProcess2,SharedArrayBuffer',
  '--enable-unsafe-swiftshader',
  '--export-tagged-pdf',
  '--font-render-hinting=none',
  '--force-color-profile=srgb',
  '--generate-pdf-document-outline',
  '--hide-scrollbars',
  '--ignore-gpu-blocklist',
  '--in-process-gpu',
  '--metrics-recording-only',
  '--mute-audio',
  '--no-default-browser-check',
  '--no-first-run',
  '--no-pings',
  '--no-zygote',
  '--password-store=basic',
  '--single-process',
  '--use-angle=swiftshader',
  '--use-gl=angle',
  '--use-mock-keychain',
  '--window-size=1920,1080'
]

/**
 * @type {Promise<Browser>}
 */
let getBrowserPromise

/**
 * @type {number}
 */
let browserPID

/**
 * Inbetween `@sparticuz/chromium` `108.0.1` and  `117.0.0`, the `font` function started throwing an `EEXIST` error, at
 * least on macOS, when the font you want to install already is installed. This function tries, and if the error is
 * `EEXIST`, ignores it.
 *
 * @param {UUID} flowId
 * @param {string} fontFilePath
 * @returns {Promise<string>}
 */
async function safeFontInstall(flowId, fontFilePath) {
  try {
    await chromium.font(fontFilePath)
  } catch (err) /* istanbul ignore next */ {
    if (err.code === 'EEXIST') {
      log.info(module, 'getBrowser#launchAndReport', flowId, 'FONT_ALREADY_INSTALLED', { fontFilePath, dest: err.dest })
      return err.dest
    }
    throw err
  }
}

/**
 * Robust definition of a running puppeteer browser, shared for the entire run of the lambda instance.
 *
 * The browser is never cleanly “shut down”, but will be killed implicitly if the Lambda instance stops.
 *
 * (Not declared async, because we do no want to wrap the returned Promise in a secondary Promise).
 *
 * Whether we run with or without a sandbox is determined by the environment we run in. See
 * [Puppeteer As Lambda Layer; Launching a Chromium instance](../../PuppeteerAsLambdaLayer.md).
 *
 * @param {UUID} flowId
 * @return {Promise<Browser>}
 */
function getBrowser(flowId) {
  assert(flowId)

  /**
   * Not declared async, because we do no want to wrap the returned Promise in a secondary Promise.
   *
   * @returns Promise<Browser>
   */
  function launchedBrowserNotOnLambda(tweakedArgs) {
    const { launch } = require('puppeteer')

    const launchOptions = {
      args: tweakedArgs,
      defaultViewport,
      headless
    }
    log.info(module, 'getBrowser#launchedBrowserNotOnLambda', flowId, 'LAUNCHING_BROWSER', { launchOptions })
    return launch(launchOptions)
  }

  /**
   * Not declared async, because we do no want to wrap the returned Promise in a secondary Promise.
   *
   * @returns Promise<Browser>
   */
  /* istanbul ignore next */
  function launchedBrowserOnDevelopment() {
    log.info(module, 'getBrowser#launchedBrowserOnDevelopment', flowId, 'CALLED')
    return launchedBrowserNotOnLambda(args.filter(a => a !== '--no-sandbox'))
  }

  /**
   * Not declared async, because we do no want to wrap the returned Promise in a secondary Promise.
   *
   * @returns Promise<Browser>
   */
  /* istanbul ignore next */
  function launchedBrowserOnBitbucket() {
    log.info(module, 'getBrowser#launchedBrowserOnBitbucket', flowId, 'CALLED')

    return launchedBrowserNotOnLambda(args)
  }

  /**
   * @returns Browser
   */
  /* istanbul ignore next */
  async function launchedBrowserOnLambda() {
    log.info(module, 'getBrowser#launchedBrowserOnLambda', flowId, 'CALLED')

    const { launch } = require('puppeteer-core')

    const executablePath = await chromium.executablePath()
    const launchOptions = {
      executablePath,
      // noSandbox on lambda
      args,
      defaultViewport,
      headless
    }
    log.info(module, 'getBrowser#launchedBrowserOnLambda', flowId, 'LAUNCHING_BROWSER', { launchOptions })
    return launch(launchOptions)
  }

  /**
   * @returns Promise<Browser>
   */
  async function launchAndReport() {
    try {
      log.info(module, 'getBrowser#launchAndReport', flowId, 'ADDING_FONTS', { fontFilePaths })
      const appliedFonts = await Promise.all(
        fontFilePaths.map(/* istanbul ignore next */ f => safeFontInstall(flowId, f))
      )
      log.info(module, 'getBrowser#launchAndReport', flowId, 'FONTS_ADDED', { appliedFonts })
      /* prettier-ignore */
      /* istanbul ignore next */
      const browser = await (process.env.CI
        ? launchedBrowserOnBitbucket()
        : process.env.LAMBDA_TASK_ROOT && process.env.AWS_EXECUTION_ENV
          ? launchedBrowserOnLambda()
          : launchedBrowserOnDevelopment())
      browserPID = browser.process()?.pid
      assert(browserPID !== undefined)
      const [version, userAgent] = await Promise.all([browser.version(), browser.userAgent()])
      /* Monitor disconnections of the browser, to hunt down unexpected failing tests, where the browser turns out not to
         be connected, while it should (and works) on CI. */
      browser.on('disconnected', () =>
        log.warn(module, 'getBrowser#launchAndReport', flowId, 'BROWSER_DISCONNECTED', {
          browserPID: browser.process()?.pid // browserPID variable could already have been cleared by `ensureBrowserClosed`
        })
      )
      log.info(module, 'getBrowser#launchAndReport', flowId, 'NOMINAL_END', {
        version,
        userAgent,
        connected: browser.connected,
        browserPID
      })
      return browser
    } catch (launchError) /* istanbul ignore next */ {
      log.info(module, 'getBrowser#launchAndReport', flowId, 'ERROR', { launchError })
      getBrowserPromise = undefined
      throw launchError
    }
  }

  log.info(module, 'getBrowser', flowId, 'CALLED')
  if (!getBrowserPromise) {
    getBrowserPromise = launchAndReport()
  }
  log.info(module, 'getBrowser', flowId, 'NOMINAL_END', { browserPID })
  return getBrowserPromise
}

/**
 * If a Browser was started, make sure it is closed.
 *
 * @param {UUID} flowId
 * @return {Promise<void>}
 */
async function ensureBrowserClosed(flowId) {
  /* NOTE: There is an issue here, introduced between version 108 and 117.
           See https://github.com/Sparticuz/chromium/issues/85
           ---
           The issue occurs on macOS only when the browser is not connected.
           On BB|_ it also occurs when the browser is connected.
           ---
           The `setGraphicsMode = false` mentioned does not help in any way.
           ---
           It seems to be there is a “default page” or something, that keeps the closing command from working on Linux.
           Closing all dangling pages before `await browser.close()` makes that that does resolve, but then in the
           end `mocha` does not conclude.
           ---
           As an extreme measure we give up on closing the browser responsibly, and we will just kill it in all cases,
           after closing any dangling pages.
           */
  log.info(module, 'ensureBrowserClosed', flowId, 'CALLED')
  if (getBrowserPromise) {
    log.info(module, 'ensureBrowserClosed', flowId, 'FOUND_BROWSER_PROMISE')
    const promise = getBrowserPromise
    getBrowserPromise = undefined
    browserPID = undefined
    log.info(module, 'ensureBrowserClosed', flowId, 'DISCARDED_BROWSER_PROMISE')
    const browser = await promise
    const connected = browser.connected
    const pid = browser.process()?.pid
    log.info(module, 'ensureBrowserClosed', flowId, 'PREPARING-1', {
      browserPID: pid,
      connected
    })
    if (connected) {
      const pages = await browser.pages()
      log.info(module, 'ensureBrowserClosed', flowId, 'PREPARING-2', {
        browserPID: pid,
        connected,
        nrOfPagesToClose: pages.length /* Should be 0, but often is 1. It seems to be there is a “default page” or
                                          something, that keeps the closing command from working. Close the pages
                                          first. See https://github.com/Sparticuz/chromium/issues/85. The
                                          `setGraphicsMode = false` mentioned also does not help in any way. */
      })
      await Promise.all(
        pages.map(async p => {
          try {
            await p.close()
          } catch (err) /* istanbul ignore next */ {
            if (!err.message.startsWith('Protocol error: Connection closed.')) {
              throw err
            }
            log.info(module, 'ensureBrowserClosed', flowId, 'PAGE_ALREADY_CLOSED', {
              browserPID: pid
            })
          }
        })
      )
      const pages2 = await browser.pages()
      log.info(module, 'ensureBrowserClosed', flowId, 'PAGES_CLOSED', {
        browserPID: pid,
        connected,
        nrOfPagesToClose: pages.length,
        nrOfPagesAfterClose: pages2.length
      })
      log.info(module, 'ensureBrowserClosed', flowId, 'CLOSING …', { connected, browserPID: pid })
      await browser.close()
      log.info(module, 'ensureBrowserClosed', flowId, 'CLOSED', { connected, browserPID: pid })
    } else {
      /* On macOS, `close()` does not return (the test times out after 20s), but the puppeteer chromium is stopped.
        On Linux, `close()` does not return (the test times out after 20s), but the puppeteer chromium is _not_ stopped.
        Therefor, when not connected, we kill hard. */
      try {
        log.info(module, 'ensureBrowserClosed', flowId, 'KILLING', { connected, browserPID: pid })
        const killResult = await kill(pid)
        log.info(module, 'ensureBrowserClosed', flowId, 'KILLED', { connected, browserPID: pid, killResult })
      } catch (err) /* istanbul ignore next */ {
        log.info(module, 'ensureBrowserClosed', flowId, 'KILL_FAILED', { connected, browserPID: pid })
      }
    }
  }
  log.info(module, 'ensureBrowserClosed', flowId, 'NOMINAL_END')
}

/**
 * Robust definition of a running puppeteer browser, shared for the entire run of the lambda instance.
 *
 * The browser is never cleanly “shut down”, but will be killed implicitly if the Lambda instance stops.
 */
module.exports = { getBrowser, ensureBrowserClosed }

To create a PDF, we start with asking the browser for a newPage. We are not using a separate browser context
(referring to #298,
#286, and
Frequently Asked Questions; BrowserContext isn't working properly (Target.closed)).

Edited: After that, we load content in the page (which is completely self-contained, apart from fonts, which are
loaded from disk), and ask it to generated a PDF.

// createAndStorePDF.js

const log = require('../ppwcode/simpleLog')
const { getBrowser } = require('./browser')
const storeInS3 = require('./storeInS3')
const pdfOptions = require('./pdfOptions')

/**
 * Create a PDF for the given `html`, and store it in S3 in the `s3Path`, with meta-information for download,
 * such as `cache-control` and `content-disposition = "at
 */
async function createAndStorePDF(sot, flowId, s3Path, html, cacheControl, filename) {
  log.info(module, 'createAndStorePDF', flowId, 'CALLED', { sot, s3Path, cacheControl, filename })
  const browser = await getBrowser(flowId)
  const page = await browser.newPage() // NOTE: not a new browser context (EDITED)
  log.info(module, 'createAndStorePDF', flowId, 'CHROMIUM_PAGE_CREATED')
  await page.setContent(html, { waitUntil: 'load' })
  log.info(module, 'createAndStorePDF', flowId, 'CHROMIUM_CONTENT_LOADED')
  const pdf = await page.pdf(pdfOptions) // NOTE: TargetCloseError: Protocol error (Page.printToPDF): Target closed
  log.info(module, 'createAndStorePDF', flowId, 'CHROMIUM_PDF_GENERATED')
  await Promise.all([
    (async () => {
      await page.close()
      log.info(module, 'createAndStorePDF', flowId, 'CHROMIUM_PAGE_CLOSED')
    })(),
    storeInS3(flowId, s3Path, cacheControl, filename, pdf)
  ])
  log.info(module, 'createAndStorePDF', flowId, 'NOMINAL_END')
}

module.exports = createAndStorePDF

This works on the first request, but fails consistently on the second request when browser.newPage() page.pdf is called.

Conclusion

So the problem is either

• with Chromium-on-Lambda, but there are no reports on this issues with PDF generation and the browser closing in the puppeteer repo
  (Puppeteer "newPage"), nor the Chromium issue
  tracker (Chromium "newPage",
  Chromium "ConnectionClosedError",
  Chromium "Connection closed") although version 137 is
  already out for several months;
• with the layer;
• with @sparticuz/chromium v138.0.2 on Lambda (although nothing seems to have changed regarding connections);
• or with Lambda

but there are no reports on either of this in @sparticuz/chromium, although v137.0.0 is out for almost 3 months
already. Somebody would have noticed? And there is no rationale why switching to ESM or adding AMD support in a separate
binary would make newPage page.pdf fail consistently on the second call.

We are at wit’s end. Anybody any ideas?

[jandppw]

So, continuing trying to find out what is wrong, we discovered that we missed a beat.

The tests that were consistently failing had a retry set (to deal with issues that cropped up years ago), and the reports we mentioned about the ConnectionClosedError: Connection closed. appeared on the retries, not on the first call.

What actually happens is, that on the second request, the call to page.pdf(…) fails, en the browser disconnects (we found that message in our logs too now). Each retry after that fails on browser.newPage() with ConnectionClosedError: Connection closed., which is logical, and the last retry was reported in the test logs.

The call that fails initially logs CHROMIUM_CONTENT_LOADED see above, and then throws

TargetCloseError: Protocol error (Page.printToPDF): Target closed
    at CallbackRegistry.clear (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/common/CallbackRegistry.js:81:36)
    at CdpCDPSession.onClosed (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/CdpSession.js:114:25)
    at #onClose (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Connection.js:181:21)
    at WebSocket.<anonymous> (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/node/NodeWebSocketTransport.js:48:30)
    at callListener (/var/task/node_modules/ws/lib/event-target.js:290:14)
    at WebSocket.onClose (/var/task/node_modules/ws/lib/event-target.js:220:9)
    at WebSocket.emit (node:events:518:28)
    at WebSocket.emitClose (/var/task/node_modules/ws/lib/websocket.js:272:10)
    at Socket.socketOnClose (/var/task/node_modules/ws/lib/websocket.js:1341:15)
    at Socket.emit (node:events:518:28)
    at TCP.<anonymous> (node:net:351:12)

which means it is const pdf = await page.pdf(pdfOptions) that throws.

The rest of the report still applies. Still reaching out for ideas.

(Initial report edited)

[markthiessen]

@jandppw any progress on this? I started seeing these errors recently with some larger PDFs and what is looking promising for me is increasing the ephemeral storage on my lambda. I think that chromium was crashing as the tmp folder was filling up on hot lambdas.

[jandppw]

@markthiessen No, no progress. When I wrote this, we were, as stated, at wit’s end.

Your comment is the only feedback here up until now.

Anything is possible, but that the ephemeral storage would be full always for the second document would be weird. And they are not that complex. The first one generated in the tests is often half a page. Since Chromium via puppeteer seems to be used a lot, such behavior should have been noticed by the wider community by now, I gather.

[markthiessen]

@jandppw - well, for us I think the combination of an updated chromium version possibly requiring more storage and some PDFs with some embedded images pushed us over the threshold. Since increasing storage I haven't had any more errors.

[jandppw]

Can you share your settings? I can try …