OOM using script-exec
Reported by Michael Gibson | 2011-09-08 15:45:04 UTC | in Rundeck 1.4 (closed)
I am able to reproduce an OOM error when using the script-exec plugin and multiple threads.
To reproduce.
add to each node in the resources.yaml
node-executor: script-exec script-exec: echo ${node.hostname} --
${exec.command}
Dispatch command across 20 nodes at 20 threads.
Note: This only occurs when you have a large number of nodes in your resources.yaml
I have around 2,000 node entries in my resources.yaml.
If I trim that down to 30 or so, I cannot reproduce.
Not sure what the threshold is.
Comments and changes to this ticket
-
Alex-SF 2011-09-08 16:39:53 UTC
- Tag set to “large scale deployment, bug, performance”
-
Greg Schueler 2011-09-09 11:45:18 UTC
can you paste your JVM memory configuration? I am unable to reproduce this. I created 2000 nodes with script-exec executor as you described, but running a job with 20 threads on 20 nodes doesn't seem to produce the issue you mentioned. I will try increasing some of the parameters
using default memory configuration
-
Greg Schueler 2011-09-09 12:17:33 UTC
ok i was testing with "-XX:MaxPermSize=256m -Xmx512m -Xms256m -server ", i will try your config
-
Greg Schueler 2011-09-12 09:34:31 UTC
ah, interesting. perhaps you can send it using some free file share service like dropbox or http://ge.tt/ ?
-
Greg Schueler 2011-09-12 11:00:56 UTC
I reproduced the heap space error running rundeck 1.3. FYI i had been using development branch (1.4rc) before when I wasn't able to reproduce it.
i think it is due to using snakeyaml to parse all the nodes within the constructor of a class
-
Greg Schueler 2011-09-12 11:29:56 UTC
i wonder if just upgrading snakeyaml will fix it, we are using 1.7, but this issue was fixed for 1.8: http://code.google.com/p/snakeyaml/issues/detail?id=101
-
Greg Schueler 2011-09-12 11:35:45 UTC
also, it seems a partial workaround is to separate the contents of the .yaml file into multiple yaml "documents". Simply insert '---' on a new line at various points between the node definitions. I did this after every 500 nodes in you sample file and it mitigated the memory issue.
I believe this is the snakeyaml issue I linked. I will also test upgrading snakeyaml
-
Greg Schueler 2011-09-12 11:54:22 UTC
tested: replace snakeyaml-1.7.jar with 1.8 or 1.9 jar
rundeck 1.3 and 1.4 dev
update resources.yaml to 5000 nodes
java -Xmx512m -Xms512m5000 nodes parse correctly without the memory issue.
we need to upgrade snakeyaml dependency
-
Greg Schueler 2011-09-12 15:06:36 UTC
- Assigned user set to “Greg Schueler”
- Milestone set to “Rundeck 1.4”
- Milestone order changed from “97” to “0”
-
Greg Schueler 2011-09-12 15:07:24 UTC
- State changed from “new” to “needs_verification”
(from [d51b53c512cf92bed3d902fdbfae051c77886607]) Upgrade snakeyaml to 1.9 [#431 state:needs_verification] https://github.com/dtolabs/rundeck/commit/d51b53c512cf92bed3d902fdb...
-
Greg Schueler 2011-09-12 15:22:19 UTC
(from [240023be566dbb4c9f60dc1d8bd6e1ef2ce1685b]) Update hardcoded dependency in rpm spec [#431] https://github.com/dtolabs/rundeck/commit/240023be566dbb4c9f60dc1d8...
-
Greg Schueler 2011-09-21 09:54:28 UTC
- State changed from “needs_verification” to “resolved”
- Assigned user set to “Greg Schueler”
thanks for verifying that.
adding to 1.3.1 #446
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create new ticket
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
(DEPRECATED) Please use github issues for issue tracking at http://github.com/dtolabs/rundeck/issues
Alex-SF
Greg Schueler
