#178 ✓resolved
Vincent Behar

Too many open files at startup (on linux)

Reported by Vincent Behar | January 19th, 2011 @ 08:31 AM | in Rundeck 1.3 (closed)

Hi,

with the 1.1.0 and the launcher (on linux/ubuntu), when I start rundeck the first time, everything is working.
But if I shut it down, and re-start it (either using the launcher or the rundeckd script), I got the following exception :

Exception in thread "main" java.io.FileNotFoundException: /tmp/rundeck/server/exp/webapp/WEB-INF/lib/activation-1.1.jar (Too many open files)

    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
    at com.dtolabs.rundeck.ZipUtil.extractZip(Unknown Source)
    at com.dtolabs.rundeck.ZipUtil.extractZip(Unknown Source)
    at com.dtolabs.rundeck.ZipUtil.extractZip(Unknown Source)
    at com.dtolabs.rundeck.ZipUtil.extractZip(Unknown Source)
    at com.dtolabs.rundeck.ExpandRunServer.extractWar(Unknown Source)
    at com.dtolabs.rundeck.ExpandRunServer.run(Unknown Source)
    at com.dtolabs.rundeck.ExpandRunServer.main(Unknown Source)

of course it is not that the file doesn't exists, just that the jvm has too many open files.

Thanks

Comments and changes to this ticket

  • BryanWB

    BryanWB February 11th, 2011 @ 12:40 PM

    this will fix it

    $ sudo vi /etc/sysctl.conf

    increases max # of open files

    fs.file-max=800000

    $ sudo sysctl -p

    and you're there!

  • BryanWB

    BryanWB February 12th, 2011 @ 06:35 AM

    ooops, my own previous advices didn't work for me, no matter how much I increase the max # of open files the problem continues for me as well.

    Ubuntu 10.10

  • BryanWB

    BryanWB February 12th, 2011 @ 06:52 AM

    alright, this works

    $ ulimit -n 800000 # or some other large number

    the fs.file-max line in /etc/sysctl.conf may also be necessary

  • Vincent Behar

    Vincent Behar February 14th, 2011 @ 01:07 PM

    Thanks, I got it working using http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/
    - edit /etc/security/limits.conf (add "* - nofile 8000") - edit /etc/pam.d/common-session (add "session required pam_limits.so") - reboot

    perhaps someone can update the FAQ, at least to say that if we got this error we need to tweak the OS config.
    But it's strange that it works on the first run and not on the second one : the first run of the launcher is supposed to unpack a whole lot of files, and so have more "open files"...

  • Alex-SF

    Alex-SF April 27th, 2011 @ 01:33 PM

    • Tag set to documentation
    • Assigned user set to “Alex-SF”

    Making this a documentation task.

  • Alex-SF

    Alex-SF April 27th, 2011 @ 06:06 PM

    • Milestone set to Rundeck 1.3
    • Milestone order changed from “7” to “0”
  • Keith Burdis

    Keith Burdis May 14th, 2011 @ 02:01 PM

    Why does the launcher require so many file descriptors? Increasing the ulimit value seems like a band-aid to workaround a real issue of file descriptors not being freed up properly, perhaps by closing them in finally blocks. Relying on the garbage collector to clean things up is usually a bad idea.

    According to the first comment this issue wasn't present in 1.1 but I am seeing in in 1.2.0 so it appears to be something that has been added in 1.2. In my case I can't easily get the ulimit value increased on some of the hosts I'd like to run rundeck on so I need to find a proper solution.

  • Keith Burdis

    Keith Burdis May 14th, 2011 @ 02:24 PM

    Using the Hotspot JVM this issue can be worked around by forcing more frequent garbage collections by reducing the size of the "new generation" which is usually greater than 2048k:

    $ ulimit -n 1024
    $ java -XX:NewSize=1024k -jar rundeck-launcher-1.2.0.jar

    Host details:

    $ cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 8)
    $ java -version java version "1.6.0_24"
    Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
    Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

  • Keith Burdis

    Keith Burdis May 15th, 2011 @ 12:32 AM

    Similarly for the JRockit JVM:

    $ java -version java version "1.6.0_22"
    Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
    Oracle JRockit(R) (build R28.1.1-14-139783-1.6.0_22-20101206-0241-linux-x86_64, compiled mode)
    $ ulimit -n 1024
    $ java -Xns1024k -jar rundeck-launcher-1.2.0.jar

    Note that the 1024k value wasn't determined in any scientific way and is not directly related to the 'ulimit -n' value - for example the above works with a limit of 512 file descriptors too. You may need to change this value depending on the garbage collector settings for your JVM.

  • Vincent Behar

    Vincent Behar May 25th, 2011 @ 06:41 AM

    Thanks Kelth, your solution works well.
    I just set up a new rundeck instance on a new linux server, and to fix the problem (i am not root) I edited the "etc/profile" file :

    Set min/max heap size

    export RDECK_JVM="$RDECK_JVM -Xmx1024m -Xms256m -XX:NewSize=1024k"

    (add "-XX:NewSize=1024k" at the end)

    Alex, do you think you can add this parameter in the file for the next version ?

  • Alex-SF
  • Jordan Curzon

    Jordan Curzon June 14th, 2011 @ 04:30 PM

    The XX:NewSize didn't help me any, but removing the -Xms argument from etc/profile did.

  • soulonfire

    soulonfire June 30th, 2011 @ 11:43 AM

    ulimit -n 10240 >> ~/.profile fixed the problem on AIX 5.3

  • Alex-SF

    Alex-SF July 8th, 2011 @ 02:15 PM

    • State changed from “new” to “resolved”

    Added to 07-chapter1.md doc.

  • Greg Schueler
  • victorpatrick

    victorpatrick June 30th, 2023 @ 09:45 PM

    The error you're encountering indicates that the Java Virtual Machine (JVM) running Rundeck is reaching the maximum number of open files allowed by the operating system. To resolve this issue, you can increase the maximum limit for open files on your Linux/Ubuntu system. pge outage map

  • Kara Mari

    Kara Mari July 18th, 2023 @ 09:46 PM

    Each file open at startup generates a processing request to the system. When there are too many of these processing requests, the system boot time will increase, reducing performance and flexibility in the boot process. mapquest directions california

  • Jeffree Star

    Jeffree Star December 3rd, 2023 @ 06:10 PM

    Check the Rundeck documentation or community forums for known issues related to file handles and "Too many open files" errors.
    Consider upgrading to a newer version of Rundeck, as bug fixes tiki taka toe might have been implemented.

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

(DEPRECATED) Please use github issues for issue tracking at http://github.com/dtolabs/rundeck/issues

Shared Ticket Bins

Pages