Custom Chunking Configuration

[aliamerj]

Description:

Currently, the file processing function uses hardcoded values for chunking—specifically, a chunkSize of 4096 and a chunkOverlap of 15% (i.e., 4096 * 0.15). This task aims to expose these options in the connection configuration so that users can customize them based on their data and needs.

Proposed Changes:

• Backend (fileProcessors/index.ts):
  Update the code to read chunkSize and chunkOverlap from the connection configuration rather than using fixed values.

• Validation & Storage:

• Zod Schema: Update the schema to include chunkSize and chunkOverlap as optional parameters with default values (4096 and 614.4, respectively).

• Database: Modify the DB schema to store these values in the connection configuration.

• User Interface: Enhance the UI to allow users to specify their preferred chunkSize and chunkOverlap values when configuring a connection. Pre-fill the fields with the default values, and validate the inputs accordingly.