Fix foreign key constraints and the mysql procedure that is used by harikrishna-patnala · Pull Request #7381 · apache/cloudstack

harikrishna-patnala · 2023-03-30T06:58:18Z

Description

This PR fixes#7370 where same foreign key names are used for multiple tables which causes issues with different MySQL versions.

I've updated the procedure to be more precise and altered the problematic foreign keys in upgrade files. This way we can log the errors and resume the upgrade.

Also removed the mysql statements related to foreign key procedure creation and usage from 4.18.0 schema files (those are moved to 4.18.1 java upgrade files) to avoid errors caused in #7370 for certain MySQL versions.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

harikrishna-patnala · 2023-03-30T07:04:31Z

@blueorangutan package

blueorangutan · 2023-03-30T07:05:04Z

@harikrishna-patnala a Jenkins job has been kicked to build packages. It will be bundled with

SystemVM template(s). I'll keep you posted as I make progress.

blueorangutan · 2023-03-30T07:46:04Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5809

weizhouapache

@harikrishna-patnala
we also need to remove the codes in 41720to41800.sql
otherwise, the users who use mysql 5.6/5.7 will still fail to upgrade to 4.18.1.0 due to same error in #7358

codecov · 2023-03-30T08:45:23Z

Codecov Report

Merging #7381 (58c87c2) into 4.18 (03910c2) will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@ Coverage Diff @@## 4.18 #7381 +/- ##
=========================================
Coverage 12.98% 12.99% - Complexity 8984 8996 +12 
=========================================
Files 2716 2716 Lines 256329 256390 +61 Branches 39974 39984 +10 =========================================
+ Hits 33275 33307 +32 - Misses 218892 218915 +23 - Partials 4162 4168 +6

Impacted Files	Coverage Δ
...va/com/cloud/upgrade/dao/DatabaseAccessObject.java	`81.25% <0.00%> (-16.25%)`	⬇️
...ain/java/com/cloud/upgrade/dao/DbUpgradeUtils.java	`80.00% <0.00%> (-12.31%)`	⬇️
...ava/com/cloud/upgrade/dao/Upgrade41800to41810.java	`6.66% <0.00%> (-5.34%)`	⬇️

... and 5 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-advanced-security · 2023-03-30T09:04:17Z

You have successfully added a new SonarCloud configuration ``. As part of the setup process, we have scanned this repository and found no existing alerts. In the future, you will see all code scanning alerts on the repository Security tab.

harikrishna-patnala · 2023-03-30T11:13:38Z

@harikrishna-patnala we also need to remove the codes in 41720to41800.sql otherwise, the users who use mysql 5.6/5.7 will still fail to upgrade to 4.18.1.0 due to same error in #7358

Yes, I'll remove it 41720to41800.sql file

harikrishna-patnala · 2023-03-31T02:38:01Z

@blueorangutan package

blueorangutan · 2023-03-31T02:43:03Z

@harikrishna-patnala a Jenkins job has been kicked to build packages. It will be bundled with

SystemVM template(s). I'll keep you posted as I make progress.

harikrishna-patnala · 2023-03-31T05:21:23Z

@blueorangutan package

blueorangutan · 2023-03-31T05:22:03Z

@harikrishna-patnala a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2023-03-31T06:22:44Z

@harikrishna-patnala
I think we do not necessarily need a mysql procedure.

Refer to the following code block

 public void dropKey(Connection conn, String tableName, String key, boolean isForeignKey)
{
String alter_sql_str;
if (isForeignKey) {
alter_sql_str = "ALTER TABLE " + tableName + " DROP FOREIGN KEY " + key;
} else {
alter_sql_str = "ALTER TABLE " + tableName + " DROP KEY " + key;
}
try(PreparedStatement pstmt = conn.prepareStatement(alter_sql_str);)
{
pstmt.executeUpdate();
s_logger.debug("Key " + key + " is dropped successfully from the table " + tableName);
} catch (SQLException e) {
s_logger.debug("Ignored SQL Exception when trying to drop " + (isForeignKey ? "foreign " : "") + "key " + key + " on table " + tableName + " exception: " + e.getMessage());
}
}

blueorangutan · 2023-06-10T08:23:31Z

[SF] Trillian test result (tid-6717)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 49651 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7381-t6717-kvm-rocky8.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	`Failure`	598.91	test_kubernetes_clusters.py

DaanHoogland

clgtm, some malicious upgrade tests required, ... @vladimirpetrov ;)

vladimirpetrov · 2023-06-12T21:50:09Z

@blueorangutan package

blueorangutan · 2023-06-12T21:52:03Z

@vladimirpetrov a [LL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-06-12T22:58:33Z

Packaging result [LL]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6134

weizhouapache

code lgtm

need manual testing

DaanHoogland · 2023-06-20T11:45:13Z

code lgtm
need manual testing

@weizhouapache , with manual testing you mean upgrade tests, right?

weizhouapache · 2023-06-20T11:48:47Z

code lgtm
need manual testing
@weizhouapache , with manual testing you mean upgrade tests, right?

@DaanHoogland yes, probably better with mysql 5.6/5.7 which did not work when upgrade to 4.18.0.0

DaanHoogland · 2023-06-20T12:04:03Z

code lgtm
need manual testing
@weizhouapache , with manual testing you mean upgrade tests, right?
@DaanHoogland yes, probably better with mysql 5.6/5.7 which did not work when upgrade to 4.18.0.0

cc @vladimirpetrov

harikrishna-patnala · 2023-06-22T07:36:36Z

@blueorangutan package

blueorangutan · 2023-06-22T07:38:03Z

@harikrishna-patnala a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-06-22T08:32:10Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6308

yadvr · 2023-06-23T11:10:11Z

@blueorangutan test

blueorangutan · 2023-06-23T11:12:03Z

@rohityadavcloud a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

harikrishna-patnala · 2023-06-27T08:17:45Z

@blueorangutan test

blueorangutan · 2023-06-27T08:18:03Z

@harikrishna-patnala a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

harikrishna-patnala · 2023-06-28T06:34:04Z

@blueorangutan package

blueorangutan · 2023-06-28T06:36:03Z

@harikrishna-patnala a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-06-28T07:28:25Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6359

harikrishna-patnala · 2023-06-30T02:11:56Z

@blueorangutan test

blueorangutan · 2023-06-30T02:14:05Z

@harikrishna-patnala a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-06-30T13:53:46Z

[SF] Trillian test result (tid-6909)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40485 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7381-t6909-kvm-centos7.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	`Error`	79.78	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	53.44	test_vm_life_cycle.py

vladimirpetrov

LGTM based on manual testing. I tested some upgrades from 4.14, 4.15, 4.16 and 4.17 using KVM and VMware and found no issues.

harikrishna-patnala force-pushed the FixForeignKey branch from e61520b to 22948a3Compare March 30, 2023 06:59

harikrishna-patnala changed the title ~~Fix foreign key~~Fix foreign key constraints and the mysql procedure that is being usedMar 30, 2023

harikrishna-patnala changed the title ~~Fix foreign key constraints and the mysql procedure that is being used~~Fix foreign key constraints and the mysql procedure that is usedMar 30, 2023

harikrishna-patnala requested a review from weizhouapache March 30, 2023 07:01

harikrishna-patnala force-pushed the FixForeignKey branch from 22948a3 to 9292d3fCompare March 30, 2023 07:04

harikrishna-patnala added this to the 4.19.0.0 milestone Mar 30, 2023

weizhouapache reviewed Mar 30, 2023
View reviewed changes

Comment threadengine/schema/src/main/resources/META-INF/db/schema-41810to41900.sql Outdated

harikrishna-patnala force-pushed the FixForeignKey branch from 9292d3f to 601a360Compare March 30, 2023 07:21

harikrishna-patnala modified the milestones: 4.19.0.0, 4.18.1.0 Mar 30, 2023

harikrishna-patnala changed the base branch from main to 4.18March 30, 2023 07:22

boring-cyborgBot added component:agent component:api component:build component:management-server component:orchestration component:storage labels Mar 30, 2023

harikrishna-patnala force-pushed the FixForeignKey branch from 601a360 to 43b7eb1Compare March 30, 2023 07:24

weizhouapache reviewed Mar 30, 2023
View reviewed changes

Comment threadengine/schema/src/main/resources/META-INF/db/schema-41810to41900.sql Outdated

DaanHoogland approved these changes Jun 12, 2023
View reviewed changes

DaanHoogland assigned vladimirpetrov Jun 12, 2023

weizhouapache approved these changes Jun 20, 2023
View reviewed changes

weizhouapache reviewed Jun 20, 2023
View reviewed changes

Comment threadengine/schema/src/main/resources/META-INF/db/schema-41800to41810.sql Outdated

Removed unused procedure
58c87c2

weizhouapache added the Severity:BLOCKER label Jun 22, 2023

apache deleted a comment from blueorangutan Jun 30, 2023

vladimirpetrov approved these changes Jul 5, 2023
View reviewed changes

DaanHoogland merged commit 0cbe770 into apache:4.18Jul 6, 2023

Uh oh!

Conversation

harikrishna-patnala commented Mar 30, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

harikrishna-patnala commented Mar 30, 2023

Uh oh!

blueorangutan commented Mar 30, 2023

Uh oh!

Uh oh!

blueorangutan commented Mar 30, 2023

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecovBot commented Mar 30, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-advanced-securityAI commented Mar 30, 2023

Uh oh!

harikrishna-patnala commented Mar 30, 2023

Uh oh!

harikrishna-patnala commented Mar 31, 2023

Uh oh!

blueorangutan commented Mar 31, 2023

Uh oh!

harikrishna-patnala commented Mar 31, 2023

Uh oh!

blueorangutan commented Mar 31, 2023

Uh oh!

weizhouapache commented Mar 31, 2023

Uh oh!

blueorangutan commented Jun 10, 2023

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

vladimirpetrov commented Jun 12, 2023

Uh oh!

blueorangutan commented Jun 12, 2023

Uh oh!

blueorangutan commented Jun 12, 2023

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DaanHoogland commented Jun 20, 2023

Uh oh!

weizhouapache commented Jun 20, 2023

Uh oh!

DaanHoogland commented Jun 20, 2023

Uh oh!

harikrishna-patnala commented Jun 22, 2023

Uh oh!

blueorangutan commented Jun 22, 2023

Uh oh!

blueorangutan commented Jun 22, 2023

Uh oh!

yadvr commented Jun 23, 2023

Uh oh!

blueorangutan commented Jun 23, 2023

Uh oh!

harikrishna-patnala commented Jun 27, 2023

Uh oh!

blueorangutan commented Jun 27, 2023

Uh oh!

harikrishna-patnala commented Jun 28, 2023

Uh oh!

blueorangutan commented Jun 28, 2023

Uh oh!

blueorangutan commented Jun 28, 2023

Uh oh!

harikrishna-patnala commented Mar 30, 2023•
edited
Loading

codecovBot commented Mar 30, 2023•
edited
Loading