ObjectStore Framework with MinIO and Simulator plugins by kishankavala · Pull Request #7752 · apache/cloudstack

kishankavala · 2023-07-19T04:27:01Z

Description

This PR adds Object Storage feature to CloudStack.

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Object+Store+Framework

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

codecov · 2023-07-19T05:45:53Z

Codecov Report

Attention: 1481 lines in your changes are missing coverage. Please review.

Comparison is base (cb62ce6) 29.23% compared to head (22b2806) 29.15%.

Files	Patch %	Lines
...e/datastore/driver/MinIOObjectStoreDriverImpl.java	17.17%	159 Missing and 5 partials ⚠️
ui/src/components/view/ObjectStoreBrowser.vue	0.00%	139 Missing ⚠️
...loudstack/storage/object/BucketApiServiceImpl.java	12.40%	118 Missing and 2 partials ⚠️
...apache/cloudstack/api/response/BucketResponse.java	0.00%	69 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	0.00%	64 Missing ⚠️
...rc/main/java/com/cloud/usage/dao/UsageDaoImpl.java	0.00%	49 Missing ⚠️
...hema/src/main/java/com/cloud/storage/BucketVO.java	22.95%	47 Missing ⚠️
...stack/storage/datastore/db/ObjectStoreDaoImpl.java	29.85%	47 Missing ⚠️
ui/src/views/storage/UpdateBucket.vue	0.00%	43 Missing ⚠️
ui/src/views/infra/AddObjectStorage.vue	0.00%	39 Missing ⚠️
... and 42 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #7752      +/-   ##
============================================
- Coverage     29.23%   29.15%   -0.09%     
- Complexity    31180    31237      +57     
============================================
  Files          5195     5234      +39     
  Lines        366594   368348    +1754     
  Branches      53596    53739     +143     
============================================
+ Hits         107187   107391     +204     
- Misses       244708   246234    +1526     
- Partials      14699    14723      +24

Flag	Coverage Δ
simulator-marvin-tests	`25.11% <10.18%> (-0.10%)`	⬇️
uitests	`4.44% <0.33%> (-0.05%)`	⬇️
unit-tests	`14.84% <13.35%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kishankavala · 2023-07-19T06:54:47Z

@blueorangutan package

blueorangutan · 2023-07-19T06:56:03Z

@kishankavala a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-07-19T07:51:31Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6507

kishankavala · 2023-07-19T07:51:51Z

@blueorangutan test

blueorangutan · 2023-07-19T07:54:03Z

@kishankavala a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

harikrishna-patnala

Thanks for the feature @kishankavala. overall looks good, only some changes are suggested, can you please check @kishankavala

api/src/main/java/org/apache/cloudstack/api/BaseCmd.java

api/src/main/java/org/apache/cloudstack/api/ResponseGenerator.java

...rc/main/java/org/apache/cloudstack/api/command/admin/storage/UpdateObjectStoragePoolCmd.java

api/src/main/java/org/apache/cloudstack/api/command/user/bucket/CreateBucketCmd.java

api/src/main/java/org/apache/cloudstack/api/command/user/bucket/DeleteBucketCmd.java

...src/main/java/org/apache/cloudstack/storage/datastore/driver/MinIOObjectStoreDriverImpl.java

server/src/main/java/com/cloud/api/ApiResponseHelper.java

ui/public/locales/en.json

ui/src/config/section/storage.js

kishankavala · 2023-08-09T05:53:03Z

@blueorangutan package

blueorangutan · 2023-08-09T05:54:03Z

@kishankavala a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-08-09T06:47:16Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6717

kishankavala · 2023-08-09T08:24:15Z

@blueorangutan test

blueorangutan · 2023-08-09T08:26:03Z

@kishankavala a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-08-09T23:56:31Z

[SF] Trillian test result (tid-7330)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 54391 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7752-t7330-kvm-centos7.zip
Smoke tests completed. 113 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	336.84	test_safe_shutdown.py
test_01_migrate_VM_and_root_volume	`Error`	82.80	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	56.49	test_vm_life_cycle.py

vladimirpetrov · 2023-08-15T06:41:30Z

@blueorangutan package

blueorangutan · 2023-08-15T06:42:03Z

@vladimirpetrov a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-08-15T07:48:58Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6776

kishankavala · 2023-08-17T06:57:26Z

@blueorangutan package

blueorangutan · 2023-08-17T06:58:03Z

@kishankavala a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-08-17T07:51:45Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6813

github-actions · 2023-08-22T11:45:15Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

rephrasing

kishankavala · 2023-11-30T18:54:39Z

@blueorangutan package

blueorangutan · 2023-11-30T18:56:03Z

@kishankavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

kishankavala · 2023-11-30T19:53:17Z

@blueorangutan test

blueorangutan · 2023-11-30T19:54:03Z

@kishankavala a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-11-30T19:54:48Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7879

blueorangutan · 2023-12-01T12:03:05Z

[SF] Trillian test result (tid-8439)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 56645 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7752-t8439-kvm-centos7.zip
Smoke tests completed. 120 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	3604.30	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	`Failure`	3609.15	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	45.96	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	`Error`	1.25	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	59.83	test_kubernetes_clusters.py

shwstppr

LGTM - code and test results look good

shwstppr · 2023-12-01T12:20:35Z

Merging this based on reviews and testing

This PR adds Object Storage feature to CloudStack. FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+CloudStack+Object+Store

Added documentation for Object Storage feature, apache/cloudstack#7752

boring-cyborg bot added component:api component:build component:storage labels Jul 19, 2023

kishankavala changed the title ~~Csos~~ ObjectStore Framework with MinIO and Simulator plugins Jul 19, 2023

DaanHoogland marked this pull request as draft July 19, 2023 08:43

weizhouapache added this to the 4.19.0.0 milestone Jul 25, 2023

harikrishna-patnala self-assigned this Jul 25, 2023

boring-cyborg bot added the component:integration label Jul 26, 2023

DaanHoogland force-pushed the csos branch from b81298d to ba5f75a Compare July 26, 2023 08:47

harikrishna-patnala reviewed Jul 26, 2023

View reviewed changes

harikrishna-patnala removed their assignment Jul 26, 2023

kishankavala and others added 16 commits December 1, 2023 00:23

Add check for duplicate object store URL

836bdb2

Added provider name to bucket response

6139c47

Show proper error when bucket update fails for encryption and versioning

664ec8d

merge from main

6e85586

Fix missing Inject during merge

27dd747

Remove browser tab for Simulator type bucket

ce185d1

Add id to search criteria while listing object stores

2daa0f7

Add missing license and end of line

020bfe4

Fixup: show metadata key

43b542a

UI: Don't show size for directories

07c5ca1

Check minio account credetials before creating bucket

0be57cc

fix end of line

c854315

fix lint error

edb41df

Update ui/public/locales/en.json

8765ce2

rephrasing

fix smoke test failure

7f09921

fix smoke test failure

22b2806

kishankavala force-pushed the csos branch from d6cb761 to 22b2806 Compare November 30, 2023 18:53

github-actions bot removed the status:has-conflicts label Nov 30, 2023

shwstppr approved these changes Dec 1, 2023

View reviewed changes

shwstppr merged commit 5651eab into apache:main Dec 1, 2023

shwstppr pushed a commit to apache/cloudstack-documentation that referenced this pull request Dec 6, 2023

Documentation for Object Storage feature (#338)

281676a

Added documentation for Object Storage feature, apache/cloudstack#7752

weizhouapache mentioned this pull request Feb 19, 2024

UI: remove duplicated section for backup #8678

Merged

13 tasks

Conversation

kishankavala commented Jul 19, 2023 • edited by yadvr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

codecov bot commented Jul 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kishankavala commented Jul 19, 2023

Uh oh!

blueorangutan commented Jul 19, 2023

Uh oh!

blueorangutan commented Jul 19, 2023

Uh oh!

kishankavala commented Jul 19, 2023

Uh oh!

blueorangutan commented Jul 19, 2023

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kishankavala commented Aug 9, 2023

Uh oh!

blueorangutan commented Aug 9, 2023

Uh oh!

blueorangutan commented Aug 9, 2023

Uh oh!

kishankavala commented Aug 9, 2023

Uh oh!

blueorangutan commented Aug 9, 2023

Uh oh!

blueorangutan commented Aug 9, 2023

Uh oh!

vladimirpetrov commented Aug 15, 2023

Uh oh!

blueorangutan commented Aug 15, 2023

Uh oh!

blueorangutan commented Aug 15, 2023

Uh oh!

kishankavala commented Aug 17, 2023

Uh oh!

blueorangutan commented Aug 17, 2023

Uh oh!

blueorangutan commented Aug 17, 2023

Uh oh!

github-actions bot commented Aug 22, 2023

Uh oh!

kishankavala commented Nov 30, 2023

Uh oh!

blueorangutan commented Nov 30, 2023

Uh oh!

kishankavala commented Nov 30, 2023

Uh oh!

blueorangutan commented Nov 30, 2023

Uh oh!

blueorangutan commented Nov 30, 2023

Uh oh!

blueorangutan commented Dec 1, 2023

Uh oh!

shwstppr left a comment

Choose a reason for hiding this comment

Uh oh!

shwstppr commented Dec 1, 2023

kishankavala commented Jul 19, 2023 •

edited by yadvr

Loading

codecov bot commented Jul 19, 2023 •

edited

Loading