recursive use of an object detected which would lead to unsafe aliasing in rust

[mogery]

Hey, we're getting the following stack trace of this unhandled exception:

Error: recursive use of an object detected which would lead to unsafe aliasing in rust
  File "/app/node_modules/.pnpm/@dqbd+tiktoken@1.0.15/node_modules/@dqbd/tiktoken/tiktoken_bg.cjs", line 432, col 11, in module.exports.__wbindgen_throw
    throw new Error(getStringFromWasm0(arg0, arg1));
  File "wasm://wasm/01569aca:wasm-function[1197]:0xb3cba"
  File "wasm://wasm/01569aca:wasm-function[1218]:0xb3dcb"
  File "wasm://wasm/01569aca:wasm-function[456]:0x9c989"
  File "/app/node_modules/.pnpm/@dqbd+tiktoken@1.0.15/node_modules/@dqbd/tiktoken/tiktoken_bg.cjs", line 203, col 44, in TiktokenFinalization
    : new FinalizationRegistry(ptr => wasm.__wbg_tiktoken_free(ptr >>> 0));
  File "<anonymous>", in FinalizationRegistry.cleanupSome

We use tiktoken (@dqbd/tiktoken@1.0.15) in our project like so:

import { encoding_for_model } from "@dqbd/tiktoken";
import { TiktokenModel } from "@dqbd/tiktoken";

// This function calculates the number of tokens in a text string using GPT-3.5-turbo model
export function numTokensFromString(message: string, model: string): number {
  const encoder = encoding_for_model(model as TiktokenModel);

  // Encode the message into tokens
  const tokens = encoder.encode(message);

  // Free the encoder resources after use
  encoder.free();

  // Return the number of tokens
  return tokens.length;
}

Any clues what this might be related to?

[dougdellolio]

Also running into this issue. @mogery were you able to find any solution to this?

[mogery]

Not yet, unfortunately.