InputSB support

[Stellatsuu]

Related to: #13, #619
cc: @natthan-pigoux

Changes:

• Job submission is now managed via the API component.
• Added support for InputSB when submitting a job.
  • The system will automatically create a sandbox with the local input file of the JDL and upload it to the storage backend.
  • Sandbox PFN is added to the JDL as InputSandbox = {PFN}, it replaces the previous local file.
• Updated the PATCH endpoint /api/jobs/{job_id}/sandbox/output to /api/jobs/{job_id}/sandbox/{sandbox_type}
  • Now allows assigning an InputSB to an existing job by specifying the job ID and the InputSB PFN.

Testing:

Example of JDL job.jdl:

Arguments = "jobDescription.xml -o LogLevel=INFO";
Executable = "dirac-jobexec";
JobGroup = jobGroup;
JobName = jobName;
JobType = User;
LogLevel = INFO;
InputSandbox = {
   /path/to/local/file.ext
};
OutputSandbox =
   {
       Script1_CodeOutput.log,
       std.err,
       std.out
   };
Priority = 1;
Site = ANY;
StdError = std.err;
StdOutput = std.out;

Submitting the job using the CLI:
dirac jobs submit job.jdl

Example of return values:

InputSandbox created: /S3/demo-sandboxes/diracAdmin/admin/admin/sha256:d264668aeb631a84cb5d5ac1e5c01b1fa361923eb60b1209c9d833abcf0a5c6c.tar.zst
Inserted 1 jobs with ids: 1

Assigning the InputSB to the job:

curl -X 'PATCH' \
  'DIRACX_URL/api/jobs/{job_id}/sandbox/input' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer ... \
  -H 'Content-Type: application/json' \
  -d '"/S3/demo-sandboxes/diracAdmin/admin/admin/sha256:d264668aeb631a84cb5d5ac1e5c01b1fa361923eb60b1209c9d833abcf0a5c6c.tar.zst"'

[read-the-docs-community]

Documentation build overview

📚 diracx | 🛠️ Build #30478841 | 📁 Comparing 11f5e42 against latest (927d51b)

🔍 Preview build

Show files changed (118 files in total): 📝 109 modified | ➕ 2 added | ➖ 7 deleted

File	Status
404.html	📝 modified
index.html	📝 modified
REFERENCE/index.html	📝 modified
RUN_PROD/index.html	📝 modified
SECURITY/index.html	📝 modified
SSO/index.html	📝 modified
admin/index.html	📝 modified
dev/index.html	📝 modified
roadmap/index.html	📝 modified
user/index.html	📝 modified
admin/deploy_instance/index.html	📝 modified
admin/explanations/index.html	📝 modified
admin/how-to/index.html	📝 modified
admin/manage_dependencies/index.html	📝 modified
admin/manage_release/index.html	📝 modified
admin/reference/index.html	📝 modified
admin/tutorials/index.html	📝 modified
dev/explanations/index.html	📝 modified
dev/how-to/index.html	📝 modified
dev/manage_extension/index.html	📝 modified
dev/reference/index.html	📝 modified
dev/setup_environment/index.html	📝 modified
dev/tutorials/index.html	📝 modified
dev/web-arch/index.html	📝 modified
developer/contribute/index.html	📝 modified
developer/manage_extension/index.html	📝 modified
developer/setup_environment/index.html	📝 modified
user/explanations/index.html	📝 modified
user/how-to/index.html	📝 modified
user/reference/index.html	📝 modified
user/tutorials/index.html	📝 modified
user/web/index.html	📝 modified
admin/explanations/auth-with-diracx/index.html	➖ deleted
admin/explanations/auth-with-external/index.html	➖ deleted
admin/explanations/authentication/index.html	➕ added
admin/explanations/chart-structure/index.html	📝 modified
admin/explanations/configuration/index.html	📝 modified
admin/explanations/database-management/index.html	📝 modified
admin/explanations/opentelemetry/index.html	📝 modified
admin/explanations/sandbox-store/index.html	📝 modified
admin/explanations/user-management/index.html	📝 modified
admin/how-to/debugging/index.html	📝 modified
admin/how-to/install/index.html	📝 modified
admin/how-to/rotate-a-secret/index.html	📝 modified
admin/how-to/upgrading/index.html	📝 modified
admin/reference/env-variables/index.html	➖ deleted
admin/reference/security_model/index.html	📝 modified
admin/reference/settings-and-preferences/index.html	📝 modified
admin/reference/values/index.html	📝 modified
admin/tutorials/authentication/index.html	📝 modified
admin/tutorials/run_locally/index.html	📝 modified
dev/explanations/client/index.html	➕ added
dev/explanations/components/index.html	📝 modified
dev/explanations/designing-functionality/index.html	📝 modified
dev/explanations/extensions/index.html	📝 modified
dev/explanations/run_demo/index.html	📝 modified
dev/explanations/testing/index.html	📝 modified
dev/how-to/add-a-cli-command/index.html	📝 modified
dev/how-to/add-a-db/index.html	📝 modified
dev/how-to/add-a-route/index.html	📝 modified
dev/how-to/add-a-setting/index.html	📝 modified
dev/how-to/add-a-task/index.html	📝 modified
dev/how-to/add-a-test/index.html	📝 modified
dev/how-to/add-functionality/index.html	📝 modified
dev/how-to/client-customization/index.html	📝 modified
dev/how-to/client-extension/index.html	📝 modified
dev/how-to/client-generation/index.html	📝 modified
dev/how-to/contribute/index.html	📝 modified
dev/how-to/create_application/index.html	📝 modified
dev/how-to/develop-legacy-dirac/index.html	📝 modified
dev/how-to/extend-diracx/index.html	📝 modified
dev/how-to/use-the-demo/index.html	📝 modified
dev/reference/application-state/index.html	📝 modified
dev/reference/client-metapathfinder/index.html	📝 modified
dev/reference/coding-conventions/index.html	📝 modified
dev/reference/configuration/index.html	📝 modified
dev/reference/db-transaction-model/index.html	📝 modified
dev/reference/dependency-injection/index.html	📝 modified
dev/reference/entrypoints/index.html	📝 modified
dev/reference/env-variables/index.html	➖ deleted
dev/reference/pixi-tasks/index.html	📝 modified
dev/reference/security-policies/index.html	📝 modified
dev/reference/security-properties/index.html	📝 modified
dev/reference/test-recipes/index.html	📝 modified
dev/reference/writing-tests/index.html	📝 modified
dev/tutorials/advanced-tutorial/index.html	📝 modified
dev/tutorials/develop-web/index.html	📝 modified
dev/tutorials/getting-started/index.html	📝 modified
dev/tutorials/larger-developments/index.html	📝 modified
dev/tutorials/making-changes/index.html	📝 modified
dev/tutorials/play-with-auth/index.html	📝 modified
dev/tutorials/run-locally/index.html	📝 modified
dev/tutorials/write-docs/index.html	📝 modified
user/reference/client-configuration/index.html	📝 modified
user/reference/known-installations/index.html	📝 modified
user/reference/programmatic-usage/index.html	📝 modified
user/tutorials/getting-started/index.html	📝 modified
user/web/list_and_share_applications/index.html	📝 modified
user/web/login_out/index.html	📝 modified
user/web/monitor_jobs/index.html	📝 modified
admin/how-to/install/connect/index.html	➖ deleted
admin/how-to/install/convert-cs/index.html	📝 modified
admin/how-to/install/embracing/index.html	📝 modified
admin/how-to/install/install-kubernetes/index.html	📝 modified
admin/how-to/install/installing/index.html	📝 modified
admin/how-to/install/minimal-requirements/index.html	➖ deleted
admin/how-to/install/register-a-vo/index.html	📝 modified
admin/how-to/install/register-the-admin-vo/index.html	📝 modified
dev/explanations/components/api/index.html	📝 modified
dev/explanations/components/cli/index.html	📝 modified
dev/explanations/components/client/index.html	➖ deleted
dev/explanations/components/db/index.html	📝 modified
dev/explanations/components/routes/index.html	📝 modified
dev/how-to/use-the-demo/swagger/index.html	📝 modified
dev/how-to/use-the-demo/web/index.html	📝 modified
user/reference/programmatic-usage/command-line-interface/index.html	📝 modified
user/reference/programmatic-usage/https-interface/index.html	📝 modified
user/reference/programmatic-usage/python-interface/index.html	📝 modified

[aldbr]

Same here. We should use:

• logging.info instead of print
• or an optional callback function as a parameter of submit_jobs.

Option1 is probably best in this context.
Option2 could be interesting if we would want to have a progress bar for each processed job for instance (could be implemented later).

[Stellatsuu]

Would something like this be ok for option2?:

In CLI:

@app.async_command()
async def submit(jdls: list[FileText]):
    jdls_values = [jdl.read() for jdl in jdls]

    jobs = await submit_jobs(jdls_values, print_callback=print)
    print(
        f"--- Inserted {len(jobs)} jobs with ids: {', '.join(map(str, (job.job_id for job in jobs)))}"
    )

In API:

@with_client
async def submit_jobs(jdls: list[str], *, client: AsyncDiracClient, print_callback=None):
    # Create and upload InputSandboxes from JDLs
    for i, jdl in enumerate(jdls):
       if print_callback:
            print_callback(f"--- Processing job {i + 1}/{len(jdls)}...")
       ...
       
        if class_ad_job.lookupAttribute("InputSandbox"):
            ...
            isb_log=f"InputSandbox created for job {i + 1}/{len(jdls)}: {sandboxes_pfn[13:]}\n"
            logging.info(isb_log)
            if print_callback:
                print_callback(isb_log)

    ...
    jobs = await client.jobs.submit_jdl_jobs(list(jdls))
    return jobs

Which would return:

> dirac jobs submit test.jdl test2.jdl

--- Processing job 1/2...
InputSandbox created for job 1/2: /S3/demo-sandboxes/diracAdmin/admin/admin/sha256:d264668aeb631a84cb5d5ac1e5c01b1fa361923eb60b1209c9d833abcf0a5c6c.tar.zst

--- Processing job 2/2...
InputSandbox created for job 2/2: /S3/demo-sandboxes/diracAdmin/admin/admin/sha256:984bc7b95829313ca5a5df126662485fe510f2ae4bde648d996777dad055efd4.tar.zst

--- Inserted 2 jobs with ids: 1, 2