OOM using script-exec

can you paste your JVM memory configuration? I am unable to reproduce this. I created 2000 nodes with script-exec executor as you described, but running a job with 20 threads on 20 nodes doesn't seem to produce the issue you mentioned. I will try increasing some of the parameters

using default memory configuration

Currently reproducing using 1 Gig of memory. I have given it 16Gb as well and had the same result.
RDECK_JVM="$RDECK_JVM -Xmx1024m -Xms1024m"

ok i was testing with "-XX:MaxPermSize=256m -Xmx512m -Xms256m -server ", i will try your config

Is there a setting somewhere that will increase the verbosity of the stack trace? I have found a few log4j settings but they don't seem to have any effect.

Able to reproduce oom on both CentOS 5.6/x86_64/i686 and Ubuntu 9.04/xi686 both running Sun JVM 1.6.0_24 using 1024M for heap.

I am attaching the resources.yaml that I used which includes the script-exec commands for each node.

This seems to be related to the Snakeyaml parser. Upon closer inspection of the heap dump it's clear that the yaml parser is consuming a significant portion of the heap. This would explain the relationship between the resources.yaml size and the performance of the application.

Workaround: I converted the resource model to XML and have been unable to reproduce. Response times are significantly improved.

I can provide the heap dumps if needed but they are just over the 50MB limit for this system. Let me know if you have another means of delivery.

ah, interesting. perhaps you can send it using some free file share service like dropbox or http://ge.tt/ ?

I reproduced the heap space error running rundeck 1.3. FYI i had been using development branch (1.4rc) before when I wasn't able to reproduce it.

i think it is due to using snakeyaml to parse all the nodes within the constructor of a class

Nice!
I am attaching my dumpfile anyway.
I used eclipse's memory analyzer. http://www.eclipse.org/mat/
If you open the file using the Leak suspect option and then open the "Dominator" tree, all signs point to the YAML parser.

i wonder if just upgrading snakeyaml will fix it, we are using 1.7, but this issue was fixed for 1.8: http://code.google.com/p/snakeyaml/issues/detail?id=101

also, it seems a partial workaround is to separate the contents of the .yaml file into multiple yaml "documents". Simply insert '---' on a new line at various points between the node definitions. I did this after every 500 nodes in you sample file and it mitigated the memory issue.

I believe this is the snakeyaml issue I linked. I will also test upgrading snakeyaml

tested: replace snakeyaml-1.7.jar with 1.8 or 1.9 jar

rundeck 1.3 and 1.4 dev
update resources.yaml to 5000 nodes
java -Xmx512m -Xms512m

5000 nodes parse correctly without the memory issue.

we need to upgrade snakeyaml dependency

(from [240023be566dbb4c9f60dc1d8bd6e1ef2ce1685b]) Update hardcoded dependency in rpm spec [#431] https://github.com/dtolabs/rundeck/commit/240023be566dbb4c9f60dc1d8...