Extract server apps and funcs from `sdk` into `server` #388

zrgt · 2025-05-22T21:35:45Z

This PR separates server functionalities (Repository apps, in the future also Discovery, and Registries)
from the core SDK to improve modularity and maintainability.
It also includes some refactoring to avoid the code duplication in future.

Changes:

Moved all server-related classes and functions from sdk to server
Splitted http.py into repository.py, base.py and util/converters.py
Created parent classes BaseWSGIApp and ObjectStoreWSGIApp for repository app class,
which will also be used for discovery and registry apps in the future
Added an initial pyproject.toml for the server app
Refactored object_hook() by extracting the mapping dict into _get_aas_class_parsers()
Refactored default() by extracting the mapping dict into _get_aas_class_serializers()
Refactored _create_dict()
Created a JSON_AAS_TOP_LEVEL_KEYS_TO_TYPES tuple with top-level JSON keys
and the corresponding SDK types to reuse it in JSON de-/serialization methods

This PR should be squashed.

This refactoring separates server functionalities (Discovery, Registries, Repositories) from the core SDK to improve modularity and maintainability. Changes: - Server is now a separate app - Added an initial pyproject.toml for the server app - Moved all server-related classes and functions from sdk to server - Consolidated descriptor.py, submodel_descriptor.py, and aas_descriptor.py into a single server_model.py file

Create a static method for the `aas_class_parsers` so that we can overload this method in `ServerAASFromJsonDecoder` and avoid code duplication by copy/paste `object_hook()`

Create a KEYS_TO_TYPE tuple with top-level JSON keys and the corresponding SDK types. By providing this tuple as a param for `read_aas_json_file_into` we can reuse the method in `read_server_aas_json_file_into` and avoid code duplication

Create a class method `_get_aas_class_serializers` for the mapping in `default()` so that we can overload `_get_aas_class_serializers` in `ServerAASToJsonEncoder` and avoid code duplication by copy/paste `default()`

…experimental/server_app

- Refactor `_create_dict()` - Add `JSON_AAS_TOP_LEVEL_KEYS_TO_TYPES` in `_generic` and use it for `_create_dict()` Create a class method `_get_aas_class_serializers` for the mapping in `default()` and in `read_aas_json_file_into()` - Use extended `JSON_AAS_TOP_LEVEL_KEYS_TO_TYPES` in `read_server_aas_json_file_into`

In 'repository' we keep only AAS/Submodel/CD Repository App, in `http_api_helpers` we keep all classes/funcs which will be used across discovery/repository/registry apps

All response related from http_api_helpers.py was moved to `response.py`

We created parent classes `BaseWSGIApp` and `ObjectStoreWSGIApp` for discovery/registry/repository app classes. Now we can reuse methods defined in parent classes and avoid code duplication.

…sses

Move all classes from `response.py` to `base.py`.

s-heppner

This is a really cool refactor that'll help us a lot in the long run! Thank you a lot!

Other than those small things, this already pretty much LGTM.
When the mypy issues are done, this should be ready to merge.

.github/workflows/ci.yml

s-heppner · 2025-07-02T05:23:32Z

sdk/basyx/aas/adapter/json/json_deserialization.py

+from typing import (Dict, Callable, ContextManager, TypeVar, Type,
+                    List, IO, Optional, Set, get_args, Tuple, Iterable, Any)


If you import this many, it is nicer and more readable imo, to put each of them in their own line:

from typing import ( Dict, Callable, ..., Any, )

We could but we do not do so in the rest of the project. So do you think we should from now always write each in their own line/ change the other imports in the project? I think the current style is also quite readable already.

s-heppner · 2025-07-02T05:24:33Z

sdk/basyx/aas/adapter/json/json_deserialization.py

 from .._generic import MODELLING_KIND_INVERSE, ASSET_KIND_INVERSE, KEY_TYPES_INVERSE, ENTITY_TYPES_INVERSE, \
    IEC61360_DATA_TYPES_INVERSE, IEC61360_LEVEL_TYPES_INVERSE, KEY_TYPES_CLASSES_INVERSE, REFERENCE_TYPES_INVERSE, \
-    DIRECTION_INVERSE, STATE_OF_EVENT_INVERSE, QUALIFIER_KIND_INVERSE, PathOrIO, Path
+    DIRECTION_INVERSE, STATE_OF_EVENT_INVERSE, QUALIFIER_KIND_INVERSE, PathOrIO, Path, JSON_AAS_TOP_LEVEL_KEYS_TO_TYPES


Same here. One import per line with ()

sdk/basyx/aas/adapter/json/json_deserialization.py

server/app/util/converters.py

s-heppner · 2025-07-02T05:34:32Z

server/app/util/converters.py

+    return encoded
+
+
+class Base64URLConverter(werkzeug.routing.UnicodeConverter):


Maybe a better name would be Base64IdentifierConverter?

I think Base64URLConverter is a better name since we convert from and to base64URL here. What we convert are Identifiers but I think that is less relevant here.

How about IdentifierToBase64URLConverter?
What's concerning me is that people might try to use this class for other types.
Or, alternatively, how about renaming the "to_python" method to to_identifer?

server/app/util/converters.py

moritzsommer · 2025-07-03T13:43:22Z

CI not passing properly is due to the mypy errors fixed in #399. We should rebase or merge these changes into this PR as well. server-static-analysis failed additionally, because the pyproject.toml is in ./sdk, but the working directory here is only ./server.

s-heppner · 2025-07-04T07:33:47Z

server-static-analysis failed additionally...

The SDK seems to install fine beforehand.

To be more precise, it's failing since the pyproject.toml is in /server/app/pyproject.toml, rather than in /server/, however upon closer look, shouldn't the basyx-python-sdk be in the dependencies of the server?

I think what we should do here is:

Add basyx-python-sdk (double check package name) into the dependency section of the /server/app/pyproject.toml
Adapt the server-static-analysis (and other server CI pipelines) according to this logic:

    - name: Install Python dependencies
      run: |
        python -m pip install --upgrade pip
        pip install ./app[dev]
        pip uninstall basyx-python-sdk
        pip install ../sdk

This way:

when someone installs the server via pip install to run locally it simply pulls the newest basyx-python-sdk from PyPI
when run in the CI we ensure it is run with exactly the version of the SDK in that commit we want to check

This is somehow uglily related to #386.

Frosty2500 · 2025-07-06T22:39:11Z

I do not think the basyx-python-sdk should be in the dependencies of the server. After all we directly copy it in the Dockerfile, including it in the pyproject.toml would mean also explicitly uninstalling it there. I do not see the use case.

With the current state of the refactor the working directory has to be app if one wants to run the server locally without any import errors. We could document this but since using the Dockerfile fullfills the same use case and is the intended way to use our server I see no need to explain how to run the server locally.
What does everyone else think?

s-heppner · 2025-07-07T09:02:15Z

I do not think the basyx-python-sdk should be in the dependencies of the server. After all we directly copy it in the Dockerfile, including it in the pyproject.toml would mean also explicitly uninstalling it there. I do not see the use case.

The use case would be running the Service without Docker. I thought that was our suggested way of running as a developer, or is this outdated?

Frosty2500 · 2025-07-10T20:50:54Z

I do not think the basyx-python-sdk should be in the dependencies of the server. After all we directly copy it in the Dockerfile, including it in the pyproject.toml would mean also explicitly uninstalling it there. I do not see the use case.

The use case would be running the Service without Docker. I thought that was our suggested way of running as a developer, or is this outdated?

Since we have decided to keep copying the local sdk into the docker container there is no reason to run the server locally anymore, you get all the same benefits running in docker. @moritzsommer correct me if I am wrong.

…yx#338) Previously, we used `vars()` to iterate over all attributes of an object when updating it from another object (`Referable.update_from()`). However, `vars()` only retrieves the instance's `__dict__`, e.g. the object's attributes, but does not include properties. As a result, when you use `vars(other)`, you get the `_id_short` attribute instead of the `id_short` property. This made it impossible to utilize custom setter methods we developed for certain cases, e.g. when properties should be immutable. This now adapts the `Referable.update_from()` method to utilize `dir()` instead, which iterates over all attributes and properties of the object. We skip callables and private attributes. We use `setattr()` to set attributes and properties, so that the correct setter methods will be used. Furthermore, we now raise an error if an immutable property changed between the two versions of the object. Fixes eclipse-basyx#215 --------- Co-authored-by: Sercan Sahin <s.sahin@iat.rwth-aachen.de>

…basyx#389) Previously, the build-dependencies for the sphinx autodocumentation were listed in `sdk/docs/add-requirements.txt`. This is outdated and the new state of the art is managing all dependencies via `pyproject.toml`. This refactors the dependencies from `sdk/docs/add-requirements.txt` into a new optional `[docs]` section in the `pyproject.toml` dependency definitions. Furthermore, we adapt the `.github/workflows/ci.yml` to use the new definitions accordingly. Fixes eclipse-basyx#384

server/app/pyproject.toml

s-heppner

LGTM now. I'll leave #386 open until we extract the part of running the server without Docker into its own issue. Then we can still decide if we want to support that or not.

Ornella33 and others added 30 commits March 25, 2025 13:07

Füge experimentelle Änderungen für Registry und Discovery hinzu

ce642c1

Remove test.py from repository and add it to .gitignore

7204ae2

correct discovery server implementation

25cf282

remove unused code

b68efaa

add in-memory storage and adapt README

b590e1c

change main.py and disccovery.py

1b676e7

Refactor _get_aas_class_parsers

a6577be

Create a static method for the `aas_class_parsers` so that we can overload this method in `ServerAASFromJsonDecoder` and avoid code duplication by copy/paste `object_hook()`

fix aas_descriptor construct method

11c59bc

Refactor read_aas_json_file_into

6d4aab1

Create a KEYS_TO_TYPE tuple with top-level JSON keys and the corresponding SDK types. By providing this tuple as a param for `read_aas_json_file_into` we can reuse the method in `read_server_aas_json_file_into` and avoid code duplication

Refactor default()

a34230f

Create a class method `_get_aas_class_serializers` for the mapping in `default()` so that we can overload `_get_aas_class_serializers` in `ServerAASToJsonEncoder` and avoid code duplication by copy/paste `default()`

fix method update_from

9079d82

Merge remote-tracking branch 'rwth-iat/Experimental/server_app' into …

bd2a9c7

…experimental/server_app

Remove jsonization._create_dict as not used

72297f4

Split http.py into repository and http_api_helpers

bd48dec

In 'repository' we keep only AAS/Submodel/CD Repository App, in `http_api_helpers` we keep all classes/funcs which will be used across discovery/repository/registry apps

Refactor server_model and move create interfaces folder

6fd1612

Refactor result_to_xml and message_to_xml

eba1d89

Move all response related to response.py

4acab0d

All response related from http_api_helpers.py was moved to `response.py`

Create base classes for WSGI apps

567b5f1

We created parent classes `BaseWSGIApp` and `ObjectStoreWSGIApp` for discovery/registry/repository app classes. Now we can reuse methods defined in parent classes and avoid code duplication.

Refactor http_api_helpers.py and response.py

3d15b51

Reformat code with PyCharm

cb107ed

Small fixes

4e1c647

Refactor

95b2d5a

Refactor

b65c420

Refactor some methods in registry.py and fix some typos

b0f79d6

remove xmlization for Registry and Discovery classes

7c8fbe2

change according to xmlization removal for registry and discovery cla…

eb44e8a

…sses

fix error with ServerAASToJSONEncoder

d608409

Refactor response.py

dde2499

Move all classes from `response.py` to `base.py`.

Frosty2500 added 3 commits July 1, 2025 15:54

add MyPy checking, remove codeblock checking

9d43b69

fix ci pipeline

b6a588e

adapt static-analysis

de49b3f

s-heppner marked this pull request as ready for review July 2, 2025 04:23

s-heppner requested changes Jul 2, 2025

View reviewed changes

Frosty2500 added 2 commits July 6, 2025 21:33

docstrings converter.py

db4e42e

adapt imports, ci; ignore tests for now

70d7198

Frosty2500 force-pushed the refactor/server branch from 08d9c74 to ed1de8e Compare July 10, 2025 21:52

zrgt and others added 9 commits July 11, 2025 00:43

fix MyPy issues

d52d66d

fix versioning

4502c9f

adapt lxml, ignore version file

8f2ef43

adapt versioning

1721743

exclude build from MyPy, adapt ci

c19765d

adapt ci

168f691

add lxml-stubs

7e7adfa

Frosty2500 force-pushed the refactor/server branch 2 times, most recently from d742b3d to 7e7adfa Compare July 10, 2025 23:01

clarify Base64URLConverter

d8b3015

s-heppner requested changes Jul 15, 2025

View reviewed changes

server/app/pyproject.toml Outdated Show resolved Hide resolved

Add missing newline character at the end of server/app/pyproject.toml

9bb9067

s-heppner approved these changes Jul 15, 2025

View reviewed changes

s-heppner merged commit 7438abf into eclipse-basyx:develop Jul 15, 2025
15 checks passed

s-heppner deleted the refactor/server branch July 15, 2025 07:32

		from typing import (Dict, Callable, ContextManager, TypeVar, Type,
		List, IO, Optional, Set, get_args, Tuple, Iterable, Any)

		return encoded


		class Base64URLConverter(werkzeug.routing.UnicodeConverter):

Extract server apps and funcs from sdk into server #388

Extract server apps and funcs from sdk into server #388

Uh oh!

Conversation

zrgt commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s-heppner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

s-heppner Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Frosty2500 Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

s-heppner Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

s-heppner Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Frosty2500 Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

s-heppner Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

moritzsommer commented Jul 3, 2025

Uh oh!

s-heppner commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Frosty2500 commented Jul 6, 2025

Uh oh!

s-heppner commented Jul 7, 2025

Uh oh!

Frosty2500 commented Jul 10, 2025

Uh oh!

Uh oh!

s-heppner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Extract server apps and funcs from `sdk` into `server` #388

Extract server apps and funcs from `sdk` into `server` #388

zrgt commented May 22, 2025 •

edited

Loading

s-heppner commented Jul 4, 2025 •

edited

Loading