#563 multipleExecutions Error - Rundeck Development

Type	To find
responsible:me	tickets assigned to you
tagged:"@high"	tickets tagged @high
milestone:next	tickets in the upcoming milestone
state:invalid	tickets with the state invalid
created:"last week"	tickets created last week
sort:number, importance, updated	tickets sorted by #, importance or updated
Combine keywords for powerful searching.
Use advanced searching »

#563 ✓resolved

multipleExecutions Error

Reported by Luiz Casey | April 25th, 2012 @ 01:50 PM | in Rundeck 1.4.3 (closed)

Issue Type: Bug
Rundeck version: 1.4.2-1

Issue:

When running a job concurrently with different options one always fails with a 500 and the error below in services.log. We tried changing the amount of threads also the strategy. All resulted in the same outcome. We are also using ldap authentication. Attached is the job.

$ xargs -P2 -n 1 wget -x -nv --load-cookies cookies.txt  < url.txt 
2012-04-25 15:03:51 URL:http://rundeck.XXXX.net/api/1/job/90569740-5b52-4737-80a6-d4de8d354604/run?argString=-host+as-d.XXX.net+-runmode+author [581] -> "rundeck.XXX.net/api/1/job/90569740-5b52-4737-80a6-d4de8d354604/run?argString=-host+as-d.XXX.net+-runmode+author" [1]
http://rundeck.XXX.net/api/1/job/90569740-5b52-4737-80a6-d4de8d354604/run?argString=-host+w-d.XXX.net+-runmode+publish:
2012-04-25 15:03:51 ERROR 500: Internal Server Error.

..rundeck/service.log

INFO  ExecutionService: updated scheduled Execution
ERROR PatchedDefaultFlushEventListener: Could not synchronize database state with session
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [ScheduledExecution#32]
    at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:1792)
    at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2435)
    at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2335)
    at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2635)
    at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:115)
    at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
    at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
    at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
    at org.codehaus.groovy.grails.orm.hibernate.events.PatchedDefaultFlushEventListener.performExecutions(PatchedDefaultFlushEventListener.java:46)
    at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
    at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
    at org.springframework.orm.hibernate3.HibernateTemplate$28.doInHibernate(HibernateTemplate.java:883)
    at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
    at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
    at org.springframework.orm.hibernate3.HibernateTemplate.flush(HibernateTemplate.java:881)
    at org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod$1.doInHibernate(SavePersistentMethod.java:58)
    at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
    at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
    at org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod.performSave(SavePersistentMethod.java:53)
    at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractSavePersistentMethod.doInvokeInternal(AbstractSavePersistentMethod.java:179)
    at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractDynamicPersistentMethod.invoke(AbstractDynamicPersistentMethod.java:59)
    at sun.reflect.GeneratedMethodAccessor326.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:188)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
    at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:54)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:132)
    at org.codehaus.groovy.grails.plugins.orm.hibernate.HibernatePluginSupport$_addBasicPersistenceMethods_closure71.doCall(HibernatePluginSupport.groovy:806)
    at sun.reflect.GeneratedMethodAccessor749.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:307)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:63)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
    at ExecutionService.createExecution(ExecutionService.groovy:1015)
    at ExecutionService$$FastClassByCGLIB$$38316c9d.invoke(<generated>)
    at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
    at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
    at ExecutionService$$EnhancerByCGLIB$$bc4d08a7.createExecution(<generated>)
    at ExecutionService$createExecution$0.call(Unknown Source)
    at ScheduledExecutionController.executeScheduledExecution(ScheduledExecutionController.groovy:2759)
    at sun.reflect.GeneratedMethodAccessor1005.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
    at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1003)
    at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:161)
    at ScheduledExecutionController$_closure41.doCall(ScheduledExecutionController.groovy:3031)
    at sun.reflect.GeneratedMethodAccessor1004.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:225)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:51)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
    at ScheduledExecutionController$_closure41.doCall(ScheduledExecutionController.groovy)
    at sun.reflect.GeneratedMethodAccessor1003.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
    at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
    at groovy.lang.Closure.call(Closure.java:282)
    at groovy.lang.Closure.call(Closure.java:277)
    at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsControllerHelper.handleAction(SimpleGrailsControllerHelper.java:368)
    at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsControllerHelper.executeAction(SimpleGrailsControllerHelper.java:232)
    at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsControllerHelper.handleURI(SimpleGrailsControllerHelper.java:190)
    at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsControllerHelper.handleURI(SimpleGrailsControllerHelper.java:129)
    at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsController.handleRequest(SimpleGrailsController.java:73)
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
    at org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(GrailsDispatcherServlet.java:292)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
    at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
    at org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:298)
    at org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:264)
    at org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:255)
    at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:183)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.obtainContent(GrailsPageFilter.java:245)
    at org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:134)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:69)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:915)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
    at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
ERROR GrailsExceptionResolver: Exception occurred when processing request: [GET] /api/1/job/90569740-5b52-4737-80a6-d4de8d354604/run
Stacktrace follows:
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [ScheduledExecution#32]
    at ExecutionService.createExecution(ExecutionService.groovy:1015)
    at ExecutionService$$FastClassByCGLIB$$38316c9d.invoke(<generated>)
    at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
    at ExecutionService$$EnhancerByCGLIB$$bc4d08a7.createExecution(<generated>)
    at ExecutionService$createExecution$0.call(Unknown Source)
    at ScheduledExecutionController.executeScheduledExecution(ScheduledExecutionController.groovy:2759)
    at ScheduledExecutionController$_closure41.doCall(ScheduledExecutionController.groovy:3031)
    at ScheduledExecutionController$_closure41.doCall(ScheduledExecutionController.groovy)
INFO  ScheduledExecutionService: scheduling immediate job run: TEMP:lcasey:32:506
INFO  ExecutionService: Execution successful: 506
INFO  ExecutionService: updated scheduled Execution

Reproduce:

Create a job and run it concurrently passing different options.

cache_flush.xml 1.5 KB

Comments and changes to this ticket

You flagged this item as spam.
Luiz Casey April 26th, 2012 @ 05:58 AM
I guess this really isn't a multiple execution issue but more of a concurrent running issue. The job runs fine when ran in series while the other is still running.
Greg Schueler April 26th, 2012 @ 02:19 PM
- Tag set to “bug, database, jobs, locking”
if you use the (default) hsqldb file-based storage, which doesn't support "pessimistic" locking, this can occur.

I think that using a relational db, or H2, will not have this issue because they do support pessimistic locking, but I haven't verified this.
You flagged this item as spam.
Luiz Casey May 4th, 2012 @ 07:42 AM
Not sure if this would be a separate ticket or not but what about upgrading to grails 2.0+ which then uses by default H2 now vs hsqldb. Then maybe can use serialized locking vs file.
Greg Schueler May 7th, 2012 @ 10:32 AM
Yes, we can add Grails 2.0 upgrade as a separate ticket. Setting h2 as the default db is also doable without upgrading grails.

I did some testing with h2 on the current codebase, but it didn't seem to help with the locking issue, so I think we have to investigate it a bit further.
You flagged this item as spam.
Luiz Casey May 11th, 2012 @ 08:48 AM
I played around with this a bit and it seemed to have worked.

1) Installed and started up h2db
2) Copied h2*.jar to $RDECK_BASE/server/lib
3) dataSource.url = jdbc:h2:/tmp/test;FILE_LOCK=SERIALIZED
4) rundeck.v14.rdbsupport=true
5) started up rundeck.

Parallel jobs worked fine coming from Jenkins which was my initial issue.
Greg Schueler May 11th, 2012 @ 04:47 PM
- State changed from “new” to “needs_verification”
(from [488b2bc4dec17e918788d20ecb50105b17ef096a]) Fix conflict due to multiple executions

Remove need to update ScheduledExecution when creating new Executions.
Handle locking conflicts on execution completion when updating
ScheduledExecution stats by retrying.
Update notifications to handle case when hibernate session has closed.
Update tests.

[#563 state:needs_verification] https://github.com/dtolabs/rundeck/commit/488b2bc4dec17e918788d20ec...
Greg Schueler May 11th, 2012 @ 04:49 PM
- Milestone set to “Rundeck 1.4.3”
- Milestone order changed from “131” to “0”
I committed a fix for this issue that works for optimistic or pessimistic locking (hsqldb as well as h2).

I removed an unnecessary data dependency (a link from executions to the jobs they belong to), and then added some retry logic for updating the Job stats when multiple executions finish at the same time.
Greg Schueler June 20th, 2012 @ 01:54 PM
- State changed from “needs_verification” to “resolved”
You flagged this item as spam.
Kara Mari July 18th, 2023 @ 09:39 PM
The information you shared in the article is really good and I believe that when reading this post, everyone must be amazed at the quality of this information. slope
You flagged this item as spam.
jonas999 October 31st, 2023 @ 09:43 AM
find the update here how do i fix connections for bluetooth audio devices and wireless displays in windows 10 have to see the batter connection.
You flagged this item as spam.
kalyl October 31st, 2023 @ 09:53 PM
Implement proper error handling Geometry Dash and reporting within your job. This can help identify the specific issue that's causing the 500 error.
You flagged this item as spam.
otis November 5th, 2023 @ 11:20 PM
Enable detailed logging or debugging pizza tower options for your service or application. This can help capture more information about the error and the steps leading up to it.
You flagged this item as spam.
victorpatrick January 2nd, 2024 @ 06:17 PM
The material you tunnel rush provided in this piece is excellent, and I think that everyone who reads it will be astounded by its caliber.
You flagged this item as spam.
johndavidd January 5th, 2024 @ 07:09 AM
Understand the frustration you may experience due to this slope technical challenge.
You flagged this item as spam.
zetisda January 31st, 2024 @ 11:23 PM
The graphics are immersive, providing a Snow Rider realistic snowy landscape that adds to the overall thrill of the experience.

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

(DEPRECATED) Please use github issues for issue tracking at http://github.com/dtolabs/rundeck/issues

Rundeck Rundeck Development → Rundeck 1.4.3