multipleExecutions Error
Reported by Luiz Casey | April 25th, 2012 @ 01:50 PM | in Rundeck 1.4.3 (closed)
- Issue Type: Bug
- Rundeck version: 1.4.2-1
Issue:
When running a job concurrently with different options one always fails with a 500 and the error below in services.log. We tried changing the amount of threads also the strategy. All resulted in the same outcome. We are also using ldap authentication. Attached is the job.
$ xargs -P2 -n 1 wget -x -nv --load-cookies cookies.txt < url.txt
2012-04-25 15:03:51 URL:http://rundeck.XXXX.net/api/1/job/90569740-5b52-4737-80a6-d4de8d354604/run?argString=-host+as-d.XXX.net+-runmode+author [581] -> "rundeck.XXX.net/api/1/job/90569740-5b52-4737-80a6-d4de8d354604/run?argString=-host+as-d.XXX.net+-runmode+author" [1]
http://rundeck.XXX.net/api/1/job/90569740-5b52-4737-80a6-d4de8d354604/run?argString=-host+w-d.XXX.net+-runmode+publish:
2012-04-25 15:03:51 ERROR 500: Internal Server Error.
..rundeck/service.log
INFO ExecutionService: updated scheduled Execution
ERROR PatchedDefaultFlushEventListener: Could not synchronize database state with session
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [ScheduledExecution#32]
at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:1792)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2435)
at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2335)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2635)
at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:115)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
at org.codehaus.groovy.grails.orm.hibernate.events.PatchedDefaultFlushEventListener.performExecutions(PatchedDefaultFlushEventListener.java:46)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
at org.springframework.orm.hibernate3.HibernateTemplate$28.doInHibernate(HibernateTemplate.java:883)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
at org.springframework.orm.hibernate3.HibernateTemplate.flush(HibernateTemplate.java:881)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod$1.doInHibernate(SavePersistentMethod.java:58)
at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:339)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.SavePersistentMethod.performSave(SavePersistentMethod.java:53)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractSavePersistentMethod.doInvokeInternal(AbstractSavePersistentMethod.java:179)
at org.codehaus.groovy.grails.orm.hibernate.metaclass.AbstractDynamicPersistentMethod.invoke(AbstractDynamicPersistentMethod.java:59)
at sun.reflect.GeneratedMethodAccessor326.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:188)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:54)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:132)
at org.codehaus.groovy.grails.plugins.orm.hibernate.HibernatePluginSupport$_addBasicPersistenceMethods_closure71.doCall(HibernatePluginSupport.groovy:806)
at sun.reflect.GeneratedMethodAccessor749.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:80)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:307)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:63)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at ExecutionService.createExecution(ExecutionService.groovy:1015)
at ExecutionService$$FastClassByCGLIB$$38316c9d.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at ExecutionService$$EnhancerByCGLIB$$bc4d08a7.createExecution(<generated>)
at ExecutionService$createExecution$0.call(Unknown Source)
at ScheduledExecutionController.executeScheduledExecution(ScheduledExecutionController.groovy:2759)
at sun.reflect.GeneratedMethodAccessor1005.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1003)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:161)
at ScheduledExecutionController$_closure41.doCall(ScheduledExecutionController.groovy:3031)
at sun.reflect.GeneratedMethodAccessor1004.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:225)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:51)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
at ScheduledExecutionController$_closure41.doCall(ScheduledExecutionController.groovy)
at sun.reflect.GeneratedMethodAccessor1003.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:282)
at groovy.lang.Closure.call(Closure.java:277)
at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsControllerHelper.handleAction(SimpleGrailsControllerHelper.java:368)
at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsControllerHelper.executeAction(SimpleGrailsControllerHelper.java:232)
at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsControllerHelper.handleURI(SimpleGrailsControllerHelper.java:190)
at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsControllerHelper.handleURI(SimpleGrailsControllerHelper.java:129)
at org.codehaus.groovy.grails.web.servlet.mvc.SimpleGrailsController.handleRequest(SimpleGrailsController.java:73)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(GrailsDispatcherServlet.java:292)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:298)
at org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:264)
at org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:255)
at org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:183)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.obtainContent(GrailsPageFilter.java:245)
at org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:134)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:69)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:915)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
ERROR GrailsExceptionResolver: Exception occurred when processing request: [GET] /api/1/job/90569740-5b52-4737-80a6-d4de8d354604/run
Stacktrace follows:
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [ScheduledExecution#32]
at ExecutionService.createExecution(ExecutionService.groovy:1015)
at ExecutionService$$FastClassByCGLIB$$38316c9d.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at ExecutionService$$EnhancerByCGLIB$$bc4d08a7.createExecution(<generated>)
at ExecutionService$createExecution$0.call(Unknown Source)
at ScheduledExecutionController.executeScheduledExecution(ScheduledExecutionController.groovy:2759)
at ScheduledExecutionController$_closure41.doCall(ScheduledExecutionController.groovy:3031)
at ScheduledExecutionController$_closure41.doCall(ScheduledExecutionController.groovy)
INFO ScheduledExecutionService: scheduling immediate job run: TEMP:lcasey:32:506
INFO ExecutionService: Execution successful: 506
INFO ExecutionService: updated scheduled Execution
Reproduce:
Create a job and run it concurrently passing different options.
Comments and changes to this ticket
-
Luiz Casey April 26th, 2012 @ 05:58 AM
I guess this really isn't a multiple execution issue but more of a concurrent running issue. The job runs fine when ran in series while the other is still running.
-
Greg Schueler April 26th, 2012 @ 02:19 PM
- Tag set to bug, database, jobs, locking
if you use the (default) hsqldb file-based storage, which doesn't support "pessimistic" locking, this can occur.
I think that using a relational db, or H2, will not have this issue because they do support pessimistic locking, but I haven't verified this.
-
Luiz Casey May 4th, 2012 @ 07:42 AM
Not sure if this would be a separate ticket or not but what about upgrading to grails 2.0+ which then uses by default H2 now vs hsqldb. Then maybe can use serialized locking vs file.
-
Greg Schueler May 7th, 2012 @ 10:32 AM
Yes, we can add Grails 2.0 upgrade as a separate ticket. Setting h2 as the default db is also doable without upgrading grails.
I did some testing with h2 on the current codebase, but it didn't seem to help with the locking issue, so I think we have to investigate it a bit further.
-
Luiz Casey May 11th, 2012 @ 08:48 AM
I played around with this a bit and it seemed to have worked.
1) Installed and started up h2db
2) Copied h2*.jar to $RDECK_BASE/server/lib
3) dataSource.url = jdbc:h2:/tmp/test;FILE_LOCK=SERIALIZED
4) rundeck.v14.rdbsupport=true
5) started up rundeck.Parallel jobs worked fine coming from Jenkins which was my initial issue.
-
Greg Schueler May 11th, 2012 @ 04:47 PM
- State changed from new to needs_verification
(from [488b2bc4dec17e918788d20ecb50105b17ef096a]) Fix conflict due to multiple executions
Remove need to update ScheduledExecution when creating new Executions.
Handle locking conflicts on execution completion when updating
ScheduledExecution stats by retrying.
Update notifications to handle case when hibernate session has closed.
Update tests.[#563 state:needs_verification] https://github.com/dtolabs/rundeck/commit/488b2bc4dec17e918788d20ec...
-
Greg Schueler May 11th, 2012 @ 04:49 PM
- Milestone set to Rundeck 1.4.3
- Milestone order changed from 131 to 0
I committed a fix for this issue that works for optimistic or pessimistic locking (hsqldb as well as h2).
I removed an unnecessary data dependency (a link from executions to the jobs they belong to), and then added some retry logic for updating the Job stats when multiple executions finish at the same time.
-
Greg Schueler June 20th, 2012 @ 01:54 PM
- State changed from needs_verification to resolved
-
Kara Mari July 18th, 2023 @ 09:39 PM
The information you shared in the article is really good and I believe that when reading this post, everyone must be amazed at the quality of this information. slope
-
jonas999 October 31st, 2023 @ 09:43 AM
find the update here how do i fix connections for bluetooth audio devices and wireless displays in windows 10 have to see the batter connection.
-
kalyl October 31st, 2023 @ 09:53 PM
Implement proper error handling Geometry Dash and reporting within your job. This can help identify the specific issue that's causing the 500 error.
-
otis November 5th, 2023 @ 11:20 PM
Enable detailed logging or debugging pizza tower options for your service or application. This can help capture more information about the error and the steps leading up to it.
-
victorpatrick January 2nd, 2024 @ 06:17 PM
The material you tunnel rush provided in this piece is excellent, and I think that everyone who reads it will be astounded by its caliber.
-
johndavidd January 5th, 2024 @ 07:09 AM
Understand the frustration you may experience due to this slope technical challenge.
-
zetisda January 31st, 2024 @ 11:23 PM
The graphics are immersive, providing a Snow Rider realistic snowy landscape that adds to the overall thrill of the experience.
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
(DEPRECATED) Please use github issues for issue tracking at http://github.com/dtolabs/rundeck/issues