node-test-pull-request workspace locked on test-mnx-ubuntu2204-x64-1

[richardlau]

It looks like the node-test-pull-request@13 workspace on node-test-pull-request is locked and preventing jobs from running in that workspace:

e,g. https://ci.nodejs.org/job/node-test-pull-request/77845/console

18:04:01  > git checkout -f 16fcfb4de5715713b7a678fd109fed6ee817987e # timeout=10
18:04:01 hudson.plugins.git.GitException: Command "git checkout -f 16fcfb4de5715713b7a678fd109fed6ee817987e" returned status code 128:
18:04:01 stdout: 
18:04:01 stderr: fatal: Unable to create '/home/iojs/build/workspace/node-test-pull-request@13/.git/index.lock': File exists.
18:04:01 
18:04:01 Another git process seems to be running in this repository, e.g.
18:04:01 an editor opened by 'git commit'. Please make sure all processes
18:04:01 are terminated then try again. If it still fails, a git process
18:04:01 may have crashed in this repository earlier:
18:04:01 remove the file manually to continue.
18:04:01 
18:04:01 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2852)
18:04:01 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:3182)
18:04:01 Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 67.158.54.159/67.158.54.159:41175
18:04:01 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1916)
18:04:01 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:384)
18:04:01 		at hudson.remoting.Channel.call(Channel.java:1108)
18:04:01 		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:155)
18:04:01 		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
18:04:01 		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
18:04:01 		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:140)
18:04:01 		at PluginClassLoader for git-client/jdk.proxy116/jdk.proxy116.$Proxy207.execute(Unknown Source)
18:04:01 		at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1360)
18:04:01 		at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
18:04:01 		at hudson.scm.SCM.checkout(SCM.java:540)
18:04:01 		at hudson.model.AbstractProject.checkout(AbstractProject.java:1254)
18:04:01 		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:648)
18:04:01 		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:84)
18:04:01 		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
18:04:01 		at PluginClassLoader for jenkins-multijob-plugin//com.tikal.jenkins.plugins.multijob.MultiJobBuild$MultiJobRunnerImpl.run(MultiJobBuild.java:131)
18:04:01 		at hudson.model.Run.execute(Run.java:1849)
18:04:01 		at PluginClassLoader for jenkins-multijob-plugin//com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:61)
18:04:01 		at hudson.model.ResourceController.execute(ResourceController.java:97)
18:04:01 		at hudson.model.Executor.run(Executor.java:456)
18:04:01 Caused: hudson.plugins.git.GitLockFailedException: Could not lock repository. Please try again
18:04:01 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:3205)
18:04:01 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:173)
18:04:01 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:164)
18:04:01 	at hudson.remoting.UserRequest.perform(UserRequest.java:225)
18:04:01 	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
18:04:01 	at hudson.remoting.Request$2.run(Request.java:391)
18:04:01 	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
18:04:01 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
18:04:01 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
18:04:01 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
18:04:01 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:312)
18:04:01 	at java.base/java.lang.Thread.run(Thread.java:1474)
18:04:01 Caused: java.io.IOException: Could not checkout 16fcfb4de5715713b7a678fd109fed6ee817987e
18:04:01 	at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1363)
18:04:01 	at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
18:04:01 	at hudson.scm.SCM.checkout(SCM.java:540)
18:04:01 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1254)
18:04:01 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:648)
18:04:01 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:84)
18:04:01 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
18:04:01 	at PluginClassLoader for jenkins-multijob-plugin//com.tikal.jenkins.plugins.multijob.MultiJobBuild$MultiJobRunnerImpl.run(MultiJobBuild.java:131)
18:04:01 	at hudson.model.Run.execute(Run.java:1849)
18:04:01 	at PluginClassLoader for jenkins-multijob-plugin//com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:61)
18:04:01 	at hudson.model.ResourceController.execute(ResourceController.java:97)
18:04:01 	at hudson.model.Executor.run(Executor.java:456)
18:04:01 Collecting metadata...

[richardlau]

AFAICT https://ci.nodejs.org/job/node-test-pull-request/77840/console failed in the middle of a git operation (and therefore likely did not release the lock):

16:47:41  > git reset --hard # timeout=10
16:47:50 Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 67.158.54.159/67.158.54.159:41175
16:47:50 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1916)
16:47:50 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:384)
16:47:50 		at hudson.remoting.Channel.call(Channel.java:1108)
16:47:50 		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:306)
16:47:50 		at PluginClassLoader for git-client/jdk.proxy116/jdk.proxy116.$Proxy200.clean(Unknown Source)
16:47:50 		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:535)
16:47:50 		at PluginClassLoader for git//hudson.plugins.git.extensions.impl.CleanCheckout.onCheckoutCompleted(CleanCheckout.java:45)
16:47:50 		at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1395)
16:47:50 		at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
16:47:50 		at hudson.scm.SCM.checkout(SCM.java:540)
16:47:50 		at hudson.model.AbstractProject.checkout(AbstractProject.java:1254)
16:47:50 		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:648)
16:47:50 		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:84)
16:47:50 		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
16:47:50 		at PluginClassLoader for jenkins-multijob-plugin//com.tikal.jenkins.plugins.multijob.MultiJobBuild$MultiJobRunnerImpl.run(MultiJobBuild.java:131)
16:47:50 		at hudson.model.Run.execute(Run.java:1849)
16:47:50 		at PluginClassLoader for jenkins-multijob-plugin//com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:61)
16:47:50 		at hudson.model.ResourceController.execute(ResourceController.java:97)
16:47:50 		at hudson.model.Executor.run(Executor.java:456)
16:47:50 hudson.plugins.git.GitException: Command "git reset --hard" returned status code 135:
16:47:50 stdout: 
16:47:50 stderr: 
16:47:50 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2852)
16:47:50 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2777)
16:47:50 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2772)
16:47:50 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:2055)
16:47:50 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.reset(CliGitAPIImpl.java:711)
16:47:50 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:1098)
16:47:50 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
16:47:50 	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
16:47:50 	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:986)
16:47:50 	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:961)
16:47:50 	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:907)
16:47:50 	at hudson.remoting.UserRequest.perform(UserRequest.java:225)
16:47:50 	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
16:47:50 	at hudson.remoting.Request$2.run(Request.java:391)
16:47:50 	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
16:47:50 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
16:47:50 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
16:47:50 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
16:47:50 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:312)
16:47:50 	at java.base/java.lang.Thread.run(Thread.java:1474)
16:47:50 Caused: java.io.IOException
16:47:50 	at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1279)
16:47:50 	at hudson.scm.SCM.checkout(SCM.java:540)
16:47:50 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1254)
16:47:50 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:648)
16:47:50 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:84)
16:47:50 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
16:47:50 	at PluginClassLoader for jenkins-multijob-plugin//com.tikal.jenkins.plugins.multijob.MultiJobBuild$MultiJobRunnerImpl.run(MultiJobBuild.java:131)
16:47:50 	at hudson.model.Run.execute(Run.java:1849)
16:47:50 	at PluginClassLoader for jenkins-multijob-plugin//com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:61)
16:47:50 	at hudson.model.ResourceController.execute(ResourceController.java:97)
16:47:50 	at hudson.model.Executor.run(Executor.java:456)
16:47:50 Collecting metadata...

This kind of looks similar to #4481.

[richardlau]

Have removed /home/iojs/build/workspace/node-test-pull-request@13/ on the server.

[richardlau]

Workspace is gone but it'll be tricky to verify completely as the new jobs starting are using lower numbered workspaces.