Fix snapshot deletion on template creation failure by harikrishna-patnala · Pull Request #9239 · apache/cloudstack

harikrishna-patnala · 2024-06-13T07:36:30Z

Description

This PR addresses an issue as part of #8946

The issue I've observed is not while taking the snapshot but while creating the template from the snapshot (stack trace also refers the same).

I could not reproduce the original issue of failed snapshot showing as backedup state rather than error (it might have already fixed after 4.17.2), but I saw another serious issue.

The issue is whenever a snapshot is used to create a template or volume and if there is failure in backing up the snapshot to the secondary store and as part of handling that failure MS is deleting the snapshot in primary storage itself.

These changes are introduced as part of the PR #5297

Create a snapshot of a volume (set snapshot.backup.to.secondary = False)
Create a template from that snapshot
As part of the creation, MS first tries to backup the snapshot to the secondary storage
I've made it fail
MS recognized the failure and as part of failure it is deleting the snapshot on the primary storage (also marking the snapshot_store_ref entry for primary store role as "Destroyed")

Here in this PR, the fix is to handle this case of not deleting the snapshot on primary storage.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

First lets check the successful scenario

Create a snapshot of a volume (set snapshot.backup.to.secondary = False)
Create a template from that snapshot
As part of the creation, MS first tries to backup the snapshot to the secondary storage
One new entry in the snapshot_store_ref table will be seen for store_role "Image" in "creating" state
After the successful creation of template, this entry will be deleted.

Now the failure scenario

Create a snapshot of a volume (set snapshot.backup.to.secondary = False)
Create a template from that snapshot
As part of the creation, MS first tries to backup the snapshot to the secondary storage
One new entry in the snapshot_store_ref table will be seen for store_role "Image" in "creating" state
Make the backup operation fail (I've played with the debugger to fail the operation)
Observe the snapshot_store_ref table, this entry will be deleted and also keeps the existing entry for store_role "Primary" as "Ready" state. (Previously this row was marked as "Destroyed")

harikrishna-patnala · 2024-06-13T07:43:56Z

@blueorangutan package

blueorangutan · 2024-06-13T07:46:04Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-06-13T07:51:24Z

Codecov Report

Attention: Patch coverage is 45.45455% with 6 lines in your changes missing coverage. Please review.

Project coverage is 14.95%. Comparing base (e2e46e3) to head (680e3c5).
Report is 173 commits behind head on 4.19.

Files with missing lines	Patch %	Lines
...org/apache/cloudstack/snapshot/SnapshotHelper.java	45.45%	5 Missing and 1 partial ⚠️

Additional details and impacted files

@@ Coverage Diff @@## 4.19 #9239 +/- ##
============================================
+ Coverage 4.29% 14.95% +10.65% - Complexity 0 11011 +11011 
============================================
Files 363 5378 +5015 Lines 29347 469941 +440594 Branches 5126 59571 +54445 ============================================
+ Hits 1261 70283 +69022 - Misses 27943 391875 +363932 - Partials 143 7783 +7640

Flag	Coverage Δ
uitests	`4.29% <ø> (-0.01%)`	⬇️
unittests	`15.66% <45.45%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2024-06-13T09:45:23Z

Packaging result [SF]: ✔️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9919

harikrishna-patnala · 2024-06-13T12:36:11Z

@blueorangutan package

blueorangutan · 2024-06-13T12:38:06Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-13T13:48:59Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9931

harikrishna-patnala · 2024-06-18T06:48:40Z

@blueorangutan package

blueorangutan · 2024-06-18T06:50:04Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland

clgtm

yadvr

LGTM didn't test it though

blueorangutan · 2024-06-18T07:55:20Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9991

DaanHoogland · 2024-06-18T08:06:49Z

- } catch (CloudRuntimeException ex) {
- logger.warn(String.format("Unable to delete the temporary snapshot [%s] on secondary storage due to [%s]. We still will expunge the database reference, consider"
- + " manually deleting the file [%s].", snapInfo.getId(), ex.getMessage(), snapInfo.getPath()), ex);
+ if (!DataStoreRole.Image.equals(snapInfo.getDataStore().getRole())) {


actually still not sure it won't be a potential NPE ; snapInfo is not null but snapInfo.getDataStore() still may be.

Datastore/Role cannot be null based on the existing method calls and it should not be null for further steps as well. Do you want me to keep a check anyways ?

maybe you are right @harikrishna-patnala (probably actually) but in this public method a snapshotInfo object may be passed that is constructed with a null-datastore. I think an extra check does not hurt here.

yadvr · 2024-06-24T15:24:15Z

@blueorangutan test ol8 kvm-ol8

blueorangutan · 2024-06-24T15:26:03Z

@rohityadavcloud [SL] unsupported parameters provided. Supported mgmt server os are: centos7, centos6, alma9, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8, ubuntu24. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-alma9, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-ubuntu24, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, vmware-80, vmware-80u1, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82

sureshanaparti · 2024-06-25T20:50:34Z

@blueorangutan test

blueorangutan · 2024-06-25T20:52:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-06-26T09:05:27Z

[SF] Trillian test result (tid-10622)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42446 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9239-t10622-kvm-centos7.zip
Smoke tests completed. 130 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	336.57	test_safe_shutdown.py

sureshanaparti · 2024-06-27T06:52:37Z

@blueorangutan package

blueorangutan · 2024-06-27T06:54:06Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-27T08:02:08Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10157

sureshanaparti · 2024-06-27T14:18:23Z

@harikrishna-patnala Unable to create volume from the snapshot, Not sure if it's related to this or different issue.

2024-06-27 09:56:52,592 DEBUG [o.a.c.s.s.SnapshotObject] (API-Job-Executor-4:ctx-e580ce2e job-54 ctx-5af933d0) (logid:a539fca6) Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
2024-06-27 09:56:52,593 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (API-Job-Executor-4:ctx-e580ce2e job-54 ctx-5af933d0) (logid:a539fca6) Failed to copy snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotObject.processEvent(SnapshotObject.java:239)
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:357)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeFromSnapshot(VolumeOrchestrator.java:565)
2024-06-27 09:56:52,598 DEBUG [o.a.c.s.SnapshotHelper] (API-Job-Executor-4:ctx-e580ce2e job-54 ctx-5af933d0) (logid:a539fca6) Expunge template for Snapshot [2] is called for primary storage role. Not expunging it, but we will still expunge the database reference of the snapshot for image storage role if any
2024-06-27 09:56:52,625 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:ctx-e580ce2e job-54) (logid:a539fca6) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.CreateVolumeCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Failed to create volume: be56269c-a6b7-4b80-b3f6-6e921eec79fe
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:1060)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:241)
...
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to copy snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:382)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
2024-06-27 09:56:52,627 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-e580ce2e job-54) (logid:a539fca6) Complete async job-54, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create volume: be56269c-a6b7-4b80-b3f6-6e921eec79fe"}

harikrishna-patnala · 2024-07-01T06:37:51Z

@sureshanaparti thanks for testing.

But I could not reproduce the issue with a new environment

I've 2 snapshots from two disks one with backup and one without

I've created 2 volumes and templates from those two snapshots

I've also created 2 templates from those two snapshots

They are successful, can you please try again testing @sureshanaparti

sureshanaparti · 2024-07-14T09:22:55Z

@blueorangutan package

blueorangutan · 2024-07-14T09:24:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-07-14T10:28:15Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10351

sureshanaparti · 2024-07-14T18:51:39Z

@blueorangutan test

blueorangutan · 2024-07-14T18:52:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

sureshanaparti · 2024-07-15T07:22:31Z

@harikrishna-patnala Unable to create template and volume from the backed up snapshot, check whether this is related to the same issue or not (seems root cause for both are the same, calling backupSnapshot() for already backed up snapshot). Able to create template and volume from the non-backed up snapshot - entry created in snapshot_store_ref table during copy and deleted after that.

Create template log =>

2024-07-15 07:17:32,499 DEBUG [o.a.c.s.s.SnapshotObject] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
2024-07-15 07:17:32,500 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Failed to copy snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotObject.processEvent(SnapshotObject.java:239)
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:357)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1681)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy236.createPrivateTemplate(Unknown Source)
at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:297)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-15 07:17:32,506 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Updating resource Type = template count for Account = 2 Operation = decreasing Amount = 1
2024-07-15 07:17:32,508 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Updating resource Type = secondary_storage count for Account = 2 Operation = decreasing Amount = (8.00 GB) 8589934592
2024-07-15 07:17:32,512 DEBUG [o.a.c.s.SnapshotHelper] (API-Job-Executor-5:ctx-24db41c0 job-51 ctx-b79ad463) (logid:e9288858) Expunge template for Snapshot [1] is called for primary storage role. Not expunging it, but we will still expunge the database reference of the snapshot for image storage role if any
2024-07-15 07:17:32,524 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-5:ctx-24db41c0 job-51) (logid:e9288858) Unexpected exception while executing org.apache.cloudstack.api.command.admin.template.CreateTemplateCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Failed to copy snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:382)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1681)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy236.createPrivateTemplate(Unknown Source)
at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:297)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-15 07:17:32,526 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-24db41c0 job-51) (logid:e9288858) Complete async job-51, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to copy snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested"}

Create volume log =>

2024-07-15 07:24:23,234 DEBUG [o.a.c.s.s.SnapshotObject] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Failed to update state:com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
2024-07-15 07:24:23,235 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Failed to copy snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotObject.processEvent(SnapshotObject.java:239)
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:357)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeFromSnapshot(VolumeOrchestrator.java:565)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy241.createVolumeFromSnapshot(Unknown Source)
at com.cloud.storage.VolumeApiServiceImpl.createVolumeFromSnapshot(VolumeApiServiceImpl.java:1082)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:1032)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:241)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy247.createVolume(Unknown Source)
at org.apache.cloudstack.api.command.user.volume.CreateVolumeCmd.execute(CreateVolumeCmd.java:226)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-15 07:24:23,247 DEBUG [o.a.c.s.SnapshotHelper] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Expunge template for Snapshot [1] is called for primary storage role. Not expunging it, but we will still expunge the database reference of the snapshot for image storage role if any
2024-07-15 07:24:23,273 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Updating resource Type = volume count for Account = 2 Operation = decreasing Amount = 1
2024-07-15 07:24:23,280 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-479e5ef7 job-52 ctx-d1ee714c) (logid:2227ec93) Updating resource Type = primary_storage count for Account = 2 Operation = decreasing Amount = (8.00 GB) 8589934592
2024-07-15 07:24:23,295 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-6:ctx-479e5ef7 job-52) (logid:2227ec93) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.CreateVolumeCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Failed to create volume: 3b6b9ce1-d7cf-45c6-94bf-1d37d70929b4
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:1060)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:241)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy247.createVolume(Unknown Source)
at org.apache.cloudstack.api.command.user.volume.CreateVolumeCmd.execute(CreateVolumeCmd.java:226)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to copy snapshot:com.cloud.utils.exception.CloudRuntimeException: Failed to update state: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state from Ready via CreateOnlyRequested
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:382)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
at org.apache.cloudstack.snapshot.SnapshotHelper.backupSnapshotToSecondaryStorageIfNotExists(SnapshotHelper.java:146)
... 45 more
2024-07-15 07:24:23,299 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-479e5ef7 job-52) (logid:2227ec93) Complete async job-52, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create volume: 3b6b9ce1-d7cf-45c6-94bf-1d37d70929b4"}

harikrishna-patnala · 2024-07-15T07:33:58Z

checking @sureshanaparti , thanks

blueorangutan · 2024-07-15T07:45:04Z

[SF] Trillian test result (tid-10831)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44824 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9239-t10831-kvm-centos7.zip
Smoke tests completed. 132 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

* Don't delete the snapshot itself on the primary storage upon any failure * Change an if condition

harikrishna-patnala added the status:needs-testing label Jun 13, 2024

harikrishna-patnala linked an issue Jun 13, 2024 that may be closed by this pull request
Failed Volume Snapshot shows State as BackedUp #8946
Closed

harikrishna-patnala mentioned this pull request Jun 13, 2024
Failed Volume Snapshot shows State as BackedUp #8946
Closed

Don't delete the snapshot itself on the primary storage upon any failure
f028a23

harikrishna-patnala force-pushed the FixSnapshotDeletionOnTemplateCreationFailure branch from cfca8f2 to f028a23Compare June 13, 2024 07:43

DaanHoogland reviewed Jun 13, 2024
View reviewed changes

Comment threadserver/src/main/java/org/apache/cloudstack/snapshot/SnapshotHelper.java Outdated

Change an if condition
680e3c5

DaanHoogland approved these changes Jun 18, 2024
View reviewed changes

yadvr approved these changes Jun 18, 2024
View reviewed changes

yadvr requested a review from shwstppr June 18, 2024 07:32

yadvr added this to the 4.19.1.0 milestone Jun 18, 2024

DaanHoogland reviewed Jun 18, 2024
View reviewed changes

sureshanaparti self-assigned this Jun 26, 2024

sureshanaparti modified the milestones: 4.19.1.0, 4.19.2.0 Jun 28, 2024

yadvr modified the milestones: 4.19.2.0, 4.19.1.0 Jul 11, 2024

harikrishna-patnala modified the milestones: 4.19.1.0, 4.19.2.0 Jul 15, 2024

vladimirpetrov self-assigned this Aug 20, 2024

vladimirpetrov removed the status:needs-testing label Aug 22, 2024

yadvr merged commit 601e9b6 into apache:4.19Sep 4, 2024

yadvr deleted the FixSnapshotDeletionOnTemplateCreationFailure branch September 4, 2024 06:44

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Sep 6, 2024
Fix snapshot deletion on template creation failure (apache#9239)
b2a4625
* Don't delete the snapshot itself on the primary storage upon any failure * Change an if condition

Uh oh!

Conversation

harikrishna-patnala commented Jun 13, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

harikrishna-patnala commented Jun 13, 2024

Uh oh!

blueorangutan commented Jun 13, 2024

Uh oh!

codecovBot commented Jun 13, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Jun 13, 2024

Uh oh!

harikrishna-patnala commented Jun 13, 2024

Uh oh!

blueorangutan commented Jun 13, 2024

Uh oh!

blueorangutan commented Jun 13, 2024

Uh oh!

Uh oh!

harikrishna-patnala commented Jun 18, 2024

Uh oh!

blueorangutan commented Jun 18, 2024

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jun 18, 2024

Uh oh!

DaanHooglandJun 18, 2024

Choose a reason for hiding this comment

Uh oh!

harikrishna-patnalaJun 18, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaanHooglandJun 19, 2024

Choose a reason for hiding this comment

Uh oh!

yadvr commented Jun 24, 2024

Uh oh!

blueorangutan commented Jun 24, 2024

Uh oh!

sureshanaparti commented Jun 25, 2024

Uh oh!

blueorangutan commented Jun 25, 2024

Uh oh!

blueorangutan commented Jun 26, 2024

Uh oh!

sureshanaparti commented Jun 27, 2024

Uh oh!

blueorangutan commented Jun 27, 2024

Uh oh!

blueorangutan commented Jun 27, 2024

Uh oh!

sureshanaparti commented Jun 27, 2024

Uh oh!

harikrishna-patnala commented Jul 1, 2024

Uh oh!

sureshanaparti commented Jul 14, 2024

Uh oh!

blueorangutan commented Jul 14, 2024

Uh oh!

blueorangutan commented Jul 14, 2024

Uh oh!

sureshanaparti commented Jul 14, 2024

Uh oh!

blueorangutan commented Jul 14, 2024

Uh oh!

sureshanaparti commented Jul 15, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

harikrishna-patnala commented Jun 13, 2024•
edited
Loading

codecovBot commented Jun 13, 2024•
edited
Loading

harikrishna-patnalaJun 18, 2024•
edited
Loading

sureshanaparti commented Jul 15, 2024•
edited
Loading