ConcurrentModificationException thrown during serialization of LockedResourcesBuildAction#logs in org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1279) #818
Description
Jenkins and plugins versions report
Environment
Jenkins: 2.516.2
OS: Linux - 5.15.0-136-generic
Java: 17.0.16 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
allure-jenkins-plugin:2.32.0
analysis-model-api:13.8.0-902.v26f80296f743
ansible:588.v2a_a_a_f345e34f
ansicolor:1.0.6
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.5-166.v870a_96374f91
asm-api:9.8-163.vb_2a_96d3f9c3c
atlassian-bitbucket-server-integration:4.2.0
authentication-tokens:1.144.v5ff4a_5ec5c33
aws-credentials:254.v978a_5e206a_d7
aws-java-sdk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-api-gateway:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-autoscaling:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudformation:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudfront:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudwatch:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codebuild:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codedeploy:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecr:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-efs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticbeanstalk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticloadbalancingv2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-iam:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-kinesis:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-lambda:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-logs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-organizations:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-secretsmanager:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sns:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sqs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ssm:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-autoscaling:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-cloudformation:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-core:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-ec2:2.33.4-62.vc1a_8df64b_4c9
bitbucket:282.v85a_7e442f57b_
blueocean:1.27.21
blueocean-bitbucket-pipeline:1.27.21
blueocean-commons:1.27.21
blueocean-config:1.27.21
blueocean-core-js:1.27.21
blueocean-dashboard:1.27.21
blueocean-display-url:2.4.4
blueocean-events:1.27.21
blueocean-git-pipeline:1.27.21
blueocean-github-pipeline:1.27.21
blueocean-i18n:1.27.21
blueocean-jwt:1.27.21
blueocean-personalization:1.27.21
blueocean-pipeline-api-impl:1.27.21
blueocean-pipeline-editor:1.27.21
blueocean-pipeline-scm-api:1.27.21
blueocean-rest:1.27.21
blueocean-rest-impl:1.27.21
blueocean-web:1.27.21
bootstrap5-api:5.3.7-860.v1251c115c90c
bouncycastle-api:2.30.1.81-264.v95c79c0e772c
branch-api:2.1244.vf95c81f1641c
build-history-manager:1.13.0
build-token-root:151.va_e52fe3215fc
caffeine-api:3.2.2-178.v353b_8428ed56
checks-api:373.vfe7645102093
cloudbees-bitbucket-branch-source:937.0.3
cloudbees-disk-usage-simple:256.v20ec4eb_884f1
cloudbees-folder:6.1040.v8a_e6330a_54e3
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.28.0-1
commons-lang3-api:3.18.0-98.v3a_674c06072d
commons-text-api:1.14.0-194.v804a_dc3a_1b_d8
config-file-provider:994.v3d4a_5fa_f353a_
configuration-as-code:1985.vdda_32d0c4ea_b_
configuration-as-code-groovy:1.1
copyartifact:770.va_6c69e063442
credentials:1447.v4cb_b_539b_5321
credentials-binding:702.vfe613e537e88
data-tables-api:2.3.3-1383.va_5607a_a_3f3c2
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:457.v0f62a_94f11a_3
docker-workflow:621.va_73f881d9232
downstream-build-cache:1.7
durable-task:595.ve87b_f1318d67
ec2-fleet:4.2.1.515.v100267825939
echarts-api:6.0.0-1146.v5c8f3b_8f0573
eddsa-api:0.3.0.1-19.vc432d923e5ee
favorite:2.253.v9b_413168133b_
file-operations:353.vf3b_9b_a_f1f7f7
font-awesome-api:7.0.0-851.vd1feb_218a_a_63
forensics-api:3.1754.v2a_6613b_77002
git:5.7.0
git-client:6.4.0
github:1.45.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1862.v1a_fc22a_d3788
global-lockable-resources:1.3.1-RELEASE
gson-api:2.13.2-173.va_a_092315913c
handy-uri-templates-2-api:2.1.8-36.v85e4cb_234a_13
hashicorp-vault-pipeline:1.4
hashicorp-vault-plugin:371.v884a_4dd60fb_6
htmlpublisher:427
http_request:1.20
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.20.0-411.v6ef8fdee4fe9
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-3
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.21
jersey2-api:2.47-165.ve7809a_3e87e0
jfrog:1.5.10
jira:3.19
jira-steps:2.0.180.vccfe35b_5910d
jjwt-api:0.11.5-120.v0268cf544b_89
job-dsl:1.93
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.14.0-149.v1c3ce991d1b_9
jquery3-api:3.7.1-594.vb_3864f326cf0
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250517-173.v596efb_962a_31
json-path-api:2.9.0-190.veefca_05d5477
jsoup:1.21.2-66.v6ea_38164b_8a_2
junit:1355.v45e2ea_65863c
ldap:780.vcb_33c9a_e4332
lockable-resources:1412.v3f305a_fb_a_117
mailer:522.va_995fa_cfb_8b_d
matrix-auth:3.2.8
matrix-project:858.vb_b_eb_9a_7ea_99e
mercurial:1323.ve69d2a_db_8a_b_d
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
monitoring:2.5.0
nomad:0.10.0
okhttp-api:4.11.0-189.v976fa_d3379d6
opentelemetry-api:1.40.0-32.v65c59076e638
parameterized-trigger:806.vf6fff3e28c3e
pipeline-build-step:571.v08a_fffd4b_0ce
pipeline-graph-analysis:245.v88f03631a_b_21
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:534.v352f0a_e98918
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2265.v140e610fe9d5
pipeline-model-definition:2.2265.v140e610fe9d5
pipeline-model-extensions:2.2265.v140e610fe9d5
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2265.v140e610fe9d5
pipeline-stage-view:2.38
pipeline-timeline:1.0.3
pipeline-utility-steps:2.19.0
plain-credentials:199.v9f8e1f741799
plot:2.4.0
plugin-util-api:6.1167.v022176c7e0ca_
prism-api:1.30.0-609.vf0a_df102d9a_f
prometheus:784.vea_eca_f6592eb_
pubsub-light:1.19
pyenv-pipeline:2.1.2
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
role-strategy:799.v5b_e7b_ecc231e
scm-api:707.v749f968369d4
script-security:1378.vf25626395f49
snakeyaml-api:2.3-125.v4d77857a_b_402
sonar:2.18
sse-gateway:1.28
ssh-credentials:361.vb_f6760818e8c
ssh-slaves:3.1071.v0d059c7b_c555
ssh-steps:2.0.79.v1d1b_5f76dda_8
sshd:3.353.v2b_d33c46e970
structs:353.v261ea_40a_80fb_
swarm:3.51
thinBackup:2.1.3
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.209.v0e69b_c43c245
variant:70.va_d9f17f859e0
warnings-ng:12.9783.ve1cb_9f060738
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4183.v94b_6fd39da_c1
workflow-durable-task-step:1458.va_2e10a_a_b_7c4d
workflow-job:1546.v62a_c59c112dd
workflow-multibranch:811.vcd33d074c2a_0
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:706.v518c5dcb_24c0
workflow-support:976.vb_d9493c2eb_09
ws-cleanup:0.49
xml-job-to-job-dsl:0.1.13
yet-another-build-visualizer:1.17
What Operating System are you using (both controller, and any agents involved in the problem)?
The controller runs in a nomad container environment, there are windows10,windows11, ubuntu2204, ubuntu2404 agent connected to it each type has 20 instances. 80 agent total.
Reproduction steps
- execute a heavy scripted pipeline job what involves 10 agent instances from each type.
- the parallel task itself trow a java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
The whole error is this:
java.util.ConcurrentModificationException
at java.base/java.util.ArrayList$Itr.checkForComodification(Unknown Source)
at java.base/java.util.ArrayList$Itr.next(Unknown Source)
at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:285)
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:272)
Caused: java.lang.RuntimeException: Failed to serialize org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction#logs for class org.jenkins.plugins.lockableresources.actions.LockedResourcesBuildAction
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:276)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:243)
at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:228)
at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:165)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeCompleteItem(AbstractCollectionConverter.java:81)
at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:75)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:285)
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:272)
Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 7e777499-db17-467f-8c10-047de680c451
Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:276)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:243)
at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:228)
at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:165)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1307)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1296)
at com.thoughtworks.xstream.XStream.toXML(XStream.java:1269)
at hudson.util.XStream2.toXMLUTF8(XStream2.java:386)
at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34)
at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1279)
at hudson.BulkChange.commit(BulkChange.java:98)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1574)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:536)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$2(CpsVmExecutorService.java:85)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
at java.base/java.lang.Thread.run(Unknown Source)
Expected Results
The internal workflow save should not throw ConcurrentModificationException.
Actual Results
The exception is thrown from the parallel step and the job fails, the exception is logged in the catch block.
stage('Execute the jobs') {
try {
parallel(wtasks)
} catch (err) {
echo "Parallel stage had errors, the results could be partial: ${err}"
err.printStackTrace()
}
}
Anything else?
It is a multithreading issue so hard to reproduce.
Are you interested in contributing a fix?
Yes, I can beta test any possible fix.