♻️ preparation for osparc migration (merging deployments) (🗃️) #7700

matusdrobuliak66 · 2025-05-19T11:59:13Z

What do these changes do?

♻️ Remove project comments resource (🗃️)
🗃️ Add missing foreign keys
🗃️ Add zzz_resource_tracker_service_runs__osparc_io_archive_202508 where we will archive osparc service runs data.zzz prefix is so it is in the end of all tables.

PLAN for index increment:

table	increment column	serial sequence name
api_keys	id	api_keys_id_seq
comp_runs	run_id	comp_runs_run_id_seq
comp_tasks	task_id	comp_tasks_task_id_seq
folders_v2	folder_id	folders_v2_folder_id_seq
group_classifiers	id	group_classifiers_id_seq
groups	gid	groups_gid_seq
payments_autorecharge	id	payments_autorecharge_id_seq
projects_nodes	project_node_id	projects_nodes_project_node_id_seq
tags	id	tags_id_seq
tokens	token_id	tokens_token_id_seq
user_to_projects	id	user_to_projects_id_seq
users	id	users_id_seq
wallets	wallet_id	wallets_wallet_id_seq
workspaces	workspace_id	workspaces_workspace_id_seq

NOTE! resource_tracker_service_runs user ID needs to be manually incremented (foreign key can not be introduced, because all anonymous users are always deleted)

For migration:

-- users
UPDATE users SET id = id + 2000000;
SELECT setval('users_id_seq', (SELECT MAX(id) FROM users));
-- user_to_projects
UPDATE user_to_projects SET id = id + 2000000;
SELECT setval('user_to_projects_id_seq', (SELECT MAX(id) FROM user_to_projects));
-- EXPLICITELY resource_usage_tracker_service_runs (It is without the foreign key)
UPDATE resource_tracker_service_runs SET user_id = user_id + 2000000;

-- wallets
UPDATE wallets SET wallet_id = wallet_id + 2000000;
SELECT setval('wallets_wallet_id_seq', (SELECT MAX(wallet_id) FROM wallets));
-- workspaces
UPDATE workspaces SET workspace_id = workspace_id + 2000000;
SELECT setval('workspaces_workspace_id_seq', (SELECT MAX(workspace_id) FROM workspaces));
-- tokens
UPDATE tokens SET token_id = token_id + 2000000;
SELECT setval('tokens_token_id_seq', (SELECT MAX(token_id) FROM tokens));
-- tags
UPDATE tags SET id = id + 2000000;
SELECT setval('tags_id_seq', (SELECT MAX(id) FROM tags));
-- projects_nodes
UPDATE projects_nodes SET project_node_id = project_node_id + 2000000;
SELECT setval('projects_nodes_project_node_id_seq', (SELECT MAX(project_node_id) FROM projects_nodes));
-- payments_autorecharge
UPDATE payments_autorecharge SET id = id + 2000000;
SELECT setval('payments_autorecharge_id_seq', (SELECT MAX(id) FROM payments_autorecharge));
-- groups
UPDATE groups SET gid = gid + 2000000 where gid != 1;
SELECT setval('groups_gid_seq', (SELECT MAX(gid) FROM groups));
-- group_classifiers
UPDATE group_classifiers SET id = id + 2000000;
SELECT setval('group_classifiers_id_seq', (SELECT MAX(id) FROM group_classifiers));
-- folders_v2
UPDATE folders_v2 SET folder_id = folder_id + 2000000;
SELECT setval('folders_v2_folder_id_seq', (SELECT MAX(folder_id) FROM folders_v2));
-- comp_tasks
UPDATE comp_tasks SET task_id = task_id + 7000000;
SELECT setval('comp_tasks_task_id_seq', (SELECT MAX(task_id) FROM comp_tasks));
-- comp_runs
UPDATE comp_runs SET run_id = run_id + 2000000;
SELECT setval('comp_runs_run_id_seq', (SELECT MAX(run_id) FROM comp_runs));
-- api_keys
UPDATE api_keys SET id = id + 2000000;
SELECT setval('api_keys_id_seq', (SELECT MAX(id) FROM api_keys));

For archiving the runs table:

INSERT INTO zzz_resource_tracker_service_runs__osparc_io_archive_202508 (
    product_name,
    service_run_id,
    wallet_id,
    wallet_name,
    pricing_plan_id,
    pricing_unit_id,
    pricing_unit_cost_id,
    pricing_unit_cost,
    simcore_user_agent,
    user_id,
    user_email,
    project_id,
    project_name,
    node_id,
    node_name,
    parent_project_id,
    root_parent_project_id,
    root_parent_project_name,
    parent_node_id,
    root_parent_node_id,
    service_key,
    service_version,
    service_type,
    service_resources,
    service_additional_metadata,
    started_at,
    stopped_at,
    service_run_status,
    modified,
    last_heartbeat_at,
    service_run_status_msg,
    missed_heartbeat_counter
)
SELECT
    product_name,
    service_run_id,
    wallet_id,
    wallet_name,
    pricing_plan_id,
    pricing_unit_id,
    pricing_unit_cost_id,
    pricing_unit_cost,
    simcore_user_agent,
    user_id,
    user_email,
    project_id,
    project_name,
    node_id,
    node_name,
    parent_project_id,
    root_parent_project_id,
    root_parent_project_name,
    parent_node_id,
    root_parent_node_id,
    service_key,
    service_version,
    service_type::text::resourcetrackerservicetypeosparciohistory,
    service_resources,
    service_additional_metadata,
    started_at,
    stopped_at,
    service_run_status::text::resourcetrackerservicerunstatusosparciohistory,
    modified,
    last_heartbeat_at,
    service_run_status_msg,
    missed_heartbeat_counter
FROM resource_tracker_service_runs;

Related issue/s

How to test

Dev-ops

codecov · 2025-05-19T12:00:58Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.00%. Comparing base (c4a6124) to head (904d22b).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7700      +/-   ##
==========================================
+ Coverage   87.51%   88.00%   +0.49%     
==========================================
  Files        1816     1545     -271     
  Lines       70506    61165    -9341     
  Branches     1144      538     -606     
==========================================
- Hits        61701    53830    -7871     
+ Misses       8493     7192    -1301     
+ Partials      312      143     -169

Flag	Coverage Δ
integrationtests	`61.95% <ø> (-2.42%)`	⬇️
unittests	`87.50% <100.00%> (+0.77%)`	⬆️

Components	Coverage Δ
api	`76.84% <ø> (ø)`
pkg_aws_library	`∅ <ø> (∅)`
pkg_dask_task_models_library	`∅ <ø> (∅)`
pkg_models_library	`93.07% <ø> (ø)`
pkg_notifications_library	`85.26% <ø> (ø)`
pkg_postgres_database	`86.91% <100.00%> (-1.67%)`	⬇️
pkg_service_integration	`∅ <ø> (∅)`
pkg_service_library	`∅ <ø> (∅)`
pkg_settings_library	`∅ <ø> (∅)`
pkg_simcore_sdk	`77.36% <ø> (-7.53%)`	⬇️
agent	`96.46% <ø> (ø)`
api_server	`91.68% <ø> (ø)`
autoscaling	`96.07% <ø> (ø)`
catalog	`92.70% <ø> (ø)`
clusters_keeper	`99.25% <ø> (ø)`
dask_sidecar	`91.67% <ø> (+0.22%)`	⬆️
datcore_adapter	`98.12% <ø> (ø)`
director	`76.78% <ø> (ø)`
director_v2	`90.55% <ø> (-0.51%)`	⬇️
dynamic_scheduler	`96.76% <ø> (ø)`
dynamic_sidecar	`90.18% <ø> (ø)`
efs_guardian	`89.79% <ø> (ø)`
invitations	`93.28% <ø> (ø)`
payments	`82.15% <ø> (-10.48%)`	⬇️
resource_usage_tracker	`86.13% <ø> (-2.79%)`	⬇️
storage	`77.75% <ø> (-9.81%)`	⬇️
webclient	`∅ <ø> (∅)`
webserver	`85.35% <ø> (-0.48%)`	⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4a6124...904d22b. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sanderegg

did you try with the DBs already?
thanks!

sanderegg · 2025-05-19T15:39:50Z

packages/postgres-database/src/simcore_postgres_database/models/file_meta_data.py

@@ -56,4 +68,5 @@
        doc="SHA256 checksum of the file content",
        index=True,
    ),
+    ###


maybe not needed

the ### comment or the change of the model? :D

GitHK · 2025-05-20T06:04:47Z

...core_postgres_database/models/zzz_resource_tracker_service_runs__osparc_io_archive_202508.py

+    ERROR = "ERROR"
+
+
+zzz_resource_tracker_service_runs__osparc_io_archive_202508 = sa.Table(


Q: I missed the point of this table, why are we adding it?

mrnicegyu11 · 2025-05-20T06:46:05Z

I dont fully get what happens in this PR (what is CASCADE?) or the introduction of the zzz_resource_tracker_service_runs__osparc_io_archive_202508 table, but I also dont see blockers. :--)

sonarqubecloud · 2025-05-20T09:10:27Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

pcrespov

thx. Please double check mycomments. Specially theone of user_id

pcrespov · 2025-05-20T09:34:47Z

packages/postgres-database/src/simcore_postgres_database/models/file_meta_data.py

+        ),
+        nullable=False,
+        doc="The user id with which the run entry is associated",
+    ),
    sa.Column("user_id", sa.String(), index=True),


why we have two columns with user_id here?

pcrespov · 2025-05-20T09:36:25Z

packages/postgres-database/src/simcore_postgres_database/models/payments_methods.py

@@ -41,13 +43,25 @@ class InitPromptAckFlowState(str, enum.Enum):
    sa.Column(
        "user_id",
        sa.BigInteger,
+        sa.ForeignKey(


Q: so that means that we stop using the "decoupled columns" for the payments-related tables, right?

pcrespov · 2025-05-20T09:37:13Z

...core_postgres_database/models/zzz_resource_tracker_service_runs__osparc_io_archive_202508.py

@@ -0,0 +1,228 @@
+"""resource_tracker_service_runs table"""


Can you explain what this table is and why it was created here?

pcrespov · 2025-05-20T09:38:15Z

api/specs/web-server/_projects_comments.py

can perhaps also delete user_to_projects table?
Double check whether there are other tables that are not used....
TIP: Just check for references to the table variable with Vscode

migration

d6ae5e6

matusdrobuliak66 changed the title ~~♻️ preparation for osparc migration (merging deployments)~~ ♻️ preparation for osparc migration (merging deployments) (🗃️) May 19, 2025

matusdrobuliak66 self-assigned this May 19, 2025

matusdrobuliak66 and others added 3 commits May 19, 2025 14:27

fix

1e70212

fix

a2cb4d5

Merge branch 'master' into preparation-for-osparc-io-migration

dc90eef

matusdrobuliak66 marked this pull request as ready for review May 19, 2025 12:30

matusdrobuliak66 requested review from pcrespov, sanderegg and GitHK as code owners May 19, 2025 12:30

matusdrobuliak66 requested review from mrnicegyu11 and odeimaiz May 19, 2025 12:31

matusdrobuliak66 added a:webserver issue related to the webserver service a:database associated to postgres service and postgres-database package labels May 19, 2025

fix

0eb5710

matusdrobuliak66 added this to the Bazinga! milestone May 19, 2025

matusdrobuliak66 and others added 4 commits May 19, 2025 16:23

Merge branch 'master' into preparation-for-osparc-io-migration

e9164ba

fix

7b4ed2a

fix

0b2edda

fix

d696f97

matusdrobuliak66 requested review from giancarloromeo and bisgaard-itis May 19, 2025 15:05

sanderegg approved these changes May 19, 2025

View reviewed changes

odeimaiz approved these changes May 19, 2025

View reviewed changes

GitHK approved these changes May 20, 2025

View reviewed changes

Merge branch 'master' into preparation-for-osparc-io-migration

904d22b

giancarloromeo approved these changes May 20, 2025

View reviewed changes

pcrespov reviewed May 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

♻️ preparation for osparc migration (merging deployments) (🗃️) #7700

♻️ preparation for osparc migration (merging deployments) (🗃️) #7700

matusdrobuliak66 commented May 19, 2025 •

edited

Loading

Uh oh!

codecov bot commented May 19, 2025 •

edited

Loading

Uh oh!

sanderegg left a comment

Uh oh!

sanderegg May 19, 2025

Uh oh!

mrnicegyu11 May 20, 2025

Uh oh!

GitHK May 20, 2025

Uh oh!

mrnicegyu11 commented May 20, 2025

Uh oh!

sonarqubecloud bot commented May 20, 2025

Uh oh!

pcrespov left a comment

Uh oh!

pcrespov May 20, 2025

Uh oh!

pcrespov May 20, 2025

Uh oh!

pcrespov May 20, 2025

Uh oh!

pcrespov May 20, 2025

Uh oh!

Uh oh!

		ERROR = "ERROR"


		zzz_resource_tracker_service_runs__osparc_io_archive_202508 = sa.Table(

♻️ preparation for osparc migration (merging deployments) (🗃️) #7700

Are you sure you want to change the base?

♻️ preparation for osparc migration (merging deployments) (🗃️) #7700

Conversation

matusdrobuliak66 commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Related issue/s

How to test

Dev-ops

Uh oh!

codecov bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sanderegg left a comment

Choose a reason for hiding this comment

Uh oh!

sanderegg May 19, 2025

Choose a reason for hiding this comment

Uh oh!

mrnicegyu11 May 20, 2025

Choose a reason for hiding this comment

Uh oh!

GitHK May 20, 2025

Choose a reason for hiding this comment

Uh oh!

mrnicegyu11 commented May 20, 2025

Uh oh!

sonarqubecloud bot commented May 20, 2025

Quality Gate passed

Uh oh!

pcrespov left a comment

Choose a reason for hiding this comment

Uh oh!

pcrespov May 20, 2025

Choose a reason for hiding this comment

Uh oh!

pcrespov May 20, 2025

Choose a reason for hiding this comment

Uh oh!

pcrespov May 20, 2025

Choose a reason for hiding this comment

Uh oh!

pcrespov May 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matusdrobuliak66 commented May 19, 2025 •

edited

Loading

codecov bot commented May 19, 2025 •

edited

Loading