-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: c8y operation fragments for firmware/software update and device profile #3362
base: main
Are you sure you want to change the base?
Conversation
450a075
to
ee5b094
Compare
ee5b094
to
845563f
Compare
845563f
to
0bc67d4
Compare
Codecov ReportAttention: Patch coverage is Additional details and impacted files📢 Thoughts on this report? Let us know! |
Robot Results
|
crates/extensions/c8y_mapper_ext/src/operations/handlers/software_update.rs
Outdated
Show resolved
Hide resolved
crates/extensions/c8y_mapper_ext/src/operations/handlers/firmware_update.rs
Outdated
Show resolved
Hide resolved
The |
0bc67d4
to
9af954b
Compare
9af954b
to
6f3e142
Compare
6f3e142
to
2ff567b
Compare
Updated the source ticket #3162 with some findings around the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I found some problems in this PR. It's actually not code-level but design level.
- SmartREST
116
message is the legacy software list update. Advanced software management setting is ignored. - SmartREST
115
message is to updatec8y_Firmware
fragment. This has already been done by sending totwin
topic. - Inconsistency in config update and firmware update. Firmware update publishes a message onto the
twin
topic, but config update doesn't do.
The current implementation of this PR follows what c8y guide says. However, it introduces duplicated MO updates, using legacy software list endpoint blindly, so I would say the guide does not describe 100% right behaviour.
Ok(OperationOutcome::Finished { | ||
messages: vec![c8y_notification, twin_metadata], | ||
messages: vec![twin_metadata, firmware_fragment, c8y_notification], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I understand, the 115
message updates the MO's fragment c8y_Firmware
. Here we send twin_metadata
, which is published onto te/device/<name>///twin/firmware
. (you can confirm this from unit tests)
And this twin_meatadata
message will be converted to a message that updates c8y_Firmware
fragment, JSON over MQTT message, which is purely duplicated with what 115
message does.
So, I see the issue is the order to send messages. The main branch's behavour is following:
- The MQTT message onto
te/device/<name>///twin/firmware
is published by here. - 503 message is published by here.
- Mapper converts the 1) message to JSON over MQTT message that updates
c8y_Firmware
fragment (doing same as 115).
On the other hand, the c8y guide says, updating c8y_Firmware
should be done before marking the operation successful.
What this PR does is:
- The MQTT message onto
te/device/<name>///twin/firmware
is published by here. (same) 115
message is published, that updates MO'sc8y_Firmware
fragment. <- new- 503 message is published by here. (same)
- Mapper converts the 1) message to JSON over MQTT message that updates
c8y_Firmware
fragment (doing same as 115). (same)
Truely this fits the order what userguide says, but it ends up with updating c8y_Firmware
twice.
I'm not sure which one is more important. Following the order as the guide says but two c8y_Firmware
update messages, or sending only one c8y_Firmware
update message but after 503
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure which one is more important. Following the order as the guide says but two c8y_Firmware update messages, or sending only one c8y_Firmware update message but after 503.
We'll have to follow the Cumulocity recommendation and filter out duplicate twin updates to the cloud (which is already happening in the code AFAIK).
pub fn set_c8y_firmware_fragment(name: &str, version: &str, remote_url: &str) -> SmartrestPayload { | ||
SmartrestPayload::serialize((SET_FIRMWARE, name, version, remote_url)) | ||
.expect("shouldn't put payload over size limit") | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
116
is for the legacy software list. We should not use 116
blindly. We have a control over using legacy or advanced software list, refer to the ticket #2654.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately device profile is not compatible with advanced software management. So, as per the contract, it expects the 116
message. So, I'd say let's update the software list with 116
when done as part of device profile operation and have the normal software list command which is triggered after every software update to update the advanced software list as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh that’s really bad in c8y side. But we have to careful that 116 message can easily get too big size if a lot of software are listed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@albinsuresh Technically c8y does not care about receiving the 116
message, it cares about getting the software list (116
just happens to fulfil this requirement).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rina23q We won't be using the 116
message directly but rely on the existing mapper mechanism that reacts to a software_list
command by making the equivalent HTTP call to update the c8y_SoftwareList
fragment.
I was considering the usage of the 116
message to strictly conform to the requirement of updating the software list before the software update operation is marked complete. But, since this is not easy with our current model that handles software_update
and software_list
as independent command, we've relaxed that requirement and decided to maintain status-quo (updating the software list independently after the software update operation is complete).
messages.push(MqttMessage::new( | ||
sm_topic, | ||
set_c8y_software_fragment(software.update_list), | ||
)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sneding 116
doesn't sound correct. Two reasons:
- The settings of using legacy or advanced software API is ignored. This is addressed for
software_list
commands. - Here adds
request_software_list()
message after116
message. What is the point to send software list to c8y twice?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Answered here
OperationPayload::Config(config) => { | ||
messages.push(MqttMessage::new( | ||
sm_topic, | ||
set_c8y_config_fragment(&config.config_type, &config.remote_url), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question is, don't we need to update twin
for the applied config? Sending 120
means only to change MO in C8Y. For firmware, we update the twin topic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We haven't defined any twin topics for each config type. We could add that in future, but not a requirement at the moment.
4c72d5a
to
0c5f4ac
Compare
0c5f4ac
to
383ba81
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While testing the config updates with device profile, I still noticed the UI not being happy with a Not installed on this device
warning for the config item, despite sending the 120 message. I'm guessing it's because of the URL mismatch(local proxy URL published instead of the one received in the original request).
I also noticed another odd behaviour where every time I apply a config update operation a rogue config_update
command in executing
state with the same command id was appearing after the operation completes, as follows(the last entry):
c8y/devicecontrol/notifications {"delivery":{"log":[],"time":"2025-02-06T06:54:13.127Z","status":"PENDING"},"agentId":"3523841","creationTime":"2025-02-06T06:54:13.103Z","deviceId":"3523841","id":"23853","status":"PENDING","c8y_DownloadConfigFile":{"type":"tedge-configuration-plugin","url":"https://t37070943.latest.stage.c8y.io/inventory/binaries/5019376"},"description":"Send configuration snapshot tedge-configuration-plugin of configuration type tedge-configuration-plugin to device TST_grill_convex_lose","externalSource":{"externalId":"TST_grill_convex_lose","type":"c8y_Serial"}}
c8y/s/ds 524,TST_grill_convex_lose,https://t37070943.latest.stage.c8y.io/inventory/binaries/5019376,tedge-configuration-plugin
te/device/main///cmd/config_update/c8y-mapper-23853 {"status":"init","remoteUrl":"http://127.0.0.1:8001/c8y/inventory/binaries/5019376","type":"tedge-configuration-plugin"}
te/device/main///cmd/config_update/c8y-mapper-23853 {"@version":"builtin","logPath":"/var/log/tedge/agent/workflow-config_update-c8y-mapper-23853.log","remoteUrl":"http://127.0.0.1:8001/c8y/inventory/binaries/5019376","status":"scheduled","type":"tedge-configuration-plugin"}
te/device/main///cmd/config_update/c8y-mapper-23853 {"@version":"builtin","logPath":"/var/log/tedge/agent/workflow-config_update-c8y-mapper-23853.log","remoteUrl":"http://127.0.0.1:8001/c8y/inventory/binaries/5019376","status":"executing","type":"tedge-configuration-plugin"}
c8y/s/us 504,23853
te/device/main///cmd/config_update/c8y-mapper-23853 {"@version":"builtin","logPath":"/var/log/tedge/agent/workflow-config_update-c8y-mapper-23853.log","path":"/etc/tedge/plugins/tedge-configuration-plugin.toml","remoteUrl":"http://127.0.0.1:8001/c8y/inventory/binaries/5019376","status":"successful","tedgeUrl":"http://127.0.0.1:8001/c8y/inventory/binaries/5019376","type":"tedge-configuration-plugin"}
c8y/s/us 120,tedge-configuration-plugin,http://127.0.0.1:8001/c8y/inventory/binaries/5019376,
c8y/s/us 506,23853
te/device/main///cmd/config_update/c8y-mapper-23853 (null)
te/device/main///cmd/config_snapshot {"types":["/etc/tedge/tedge.toml","system.toml","tedge-configuration-plugin"]}
te/device/main///cmd/config_update {"types":["/etc/tedge/tedge.toml","system.toml","tedge-configuration-plugin"]}
c8y/s/us 119,/etc/tedge/tedge.toml,system.toml,tedge-configuration-plugin
c8y/s/us 119,/etc/tedge/tedge.toml,system.toml,tedge-configuration-plugin
te/device/main///cmd/config_update/c8y-mapper-23853 {"status":"executing","tedgeUrl":"http://127.0.0.1:8000/tedge/file-transfer/device_main__/config_update/tedge-configuration-plugin-c8y-mapper-23853","remoteUrl":"http://127.0.0.1:8001/c8y/inventory/binaries/5019376","type":"tedge-configuration-plugin","logPath":"/var/log/tedge/agent/workflow-config_update-c8y-mapper-23853.log"}
Please look into that as well.
I'd also suggest that you update the config update and the device profile system tests to validate the uploaded fragments as well using the Device Should Have Fragments
API.
|
||
#[derive(Debug, Deserialize, Serialize, Eq, PartialEq, Clone)] | ||
#[serde(deny_unknown_fields)] | ||
pub struct Operation(OperationPayload); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question: Since this Operation
struct is flattened in DeviceProfileOperation
struct without adding any additional tag
, wondering if the OperationPayload
struct could be directly used from DeviceProfileOperation
. Or is this additional level necessary for the parsing to work the way we expect?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No actually it will parse when we call OperationPayload
from DeviceProfileOperation
. Previously, it was needed due to #[serde(deny_unknown_fields)]
but now it is just one field.
@@ -64,6 +73,7 @@ pub struct FirmwarePayload { | |||
pub remote_url: Option<String>, | |||
} | |||
|
|||
impl Jsonify for FirmwarePayload {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was this needed? Because I don't see the same done for SoftwarePayload
and ConfigPayload
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is leftover from the beginning when I tried to solve the issue with deserializing and should be removed.
Actually it is not leftover. It is done for FirmwarePayload
because it is converted to json when creating twin metadata
message. We don't need those conversions for software and config.
#[serde(default = "Default::default")] | ||
pub name: String, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why was this struct updated? Since name
is provided only in the c8y_DeviceProfile
payload and not in the regular c8y_DownloadConfigFile
payload, I expected this name
field to be added to the device_profile::ConfigPayload
struct and not here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The C8yDownloadConfigFile
struct is part of C8yDeviceProfile
. In convert.rs we convert it to ConfigPayload
. Basically, all device profile operations share those structs with their standalone equivalents.
pub fn set_c8y_config_fragment( | ||
config_type: &str, | ||
remote_url: &str, | ||
name: &str, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd take this name
field as an Option
so that it can be skipped during the config update state transition.
crates/extensions/c8y_mapper_ext/src/operations/handlers/software_update.rs
Outdated
Show resolved
Hide resolved
crates/extensions/c8y_mapper_ext/src/operations/handlers/firmware_update.rs
Outdated
Show resolved
Hide resolved
sm_topic, | ||
set_c8y_config_fragment( | ||
&command.payload.config_type, | ||
&command.payload.remote_url, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The URL can't be sent as-is to the cloud as this URL is the proxy URL, like http://127.0.0.1:8001/c8y/inventory/binaries/4119377
. The url base must be changed to point to the tenant.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Proxy conversion added in 97ad0bf
sm_topic, | ||
set_c8y_config_fragment( | ||
&config.config_type, | ||
&config.remote_url.unwrap_or_default(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment as the one made in the config_update.rs
. The local proxy URL prefix must be replaced with the tenant URL.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Proxy conversion added in 97ad0bf
383ba81
to
876aaec
Compare
Signed-off-by: Krzysztof Piotrowski <krzysztof.piotrowski@inetum.com>
…orrectly Signed-off-by: Krzysztof Piotrowski <krzysztof.piotrowski@inetum.com>
876aaec
to
4aa42e7
Compare
Signed-off-by: Krzysztof Piotrowski <krzysztof.piotrowski@inetum.com>
Signed-off-by: Krzysztof Piotrowski <krzysztof.piotrowski@inetum.com>
Signed-off-by: Krzysztof Piotrowski <krzysztof.piotrowski@inetum.com>
4aa42e7
to
28f7360
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very close, but yet one step away.
As discussed offline, even after a successful application of the device profile, the config entries still appear with a "Not installed on the device" warning, despite sending the 120
message and replacing the proxy url with the tenant URL.
I investigated and understood why this is happening. Even though I trigger the operation from my tenant albin.latest.stage.c8y.io
, the url
filed in the request payload contains the value https://t37070943.latest.stage.c8y.io/inventory/binaries/4119377
with the actual tenant id t37070943
instead of the alias albin
in it. When we send the 120
message, the URL must match what was received in the request. But, even when the proxy URL is converted using the new c8y_url_from_proxy_url
function, the host is replaced with what was configured as c8y.url
(albin.latest...
) and not the one it received in the request. This is the cause of the warning.
To fix this without any disruptive changes (by changing the remoteUrl
value itself), I'd suggest that we pass an additional cloud_url
field in the config_update
request payload with the same value from the c8y request. When the request completes, this same value can be used to send the 120 message. With such a change, we wouldn't need the c8y_url_from_proxy_url
conversion anymore.
.is_some_and(|host| host.to_string() == self.host.to_string()) | ||
&& proxy_url.port().is_some_and(|port| port == self.port) | ||
&& proxy_url.scheme() == self.protocol.as_str() | ||
&& proxy_url.path().contains("/c8y") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
&& proxy_url.path().contains("/c8y") | |
&& proxy_url.path().starts_with("/c8y") |
&command.payload.config_type, | ||
self.http_proxy | ||
.c8y_url_from_proxy_url(&command.payload.remote_url) | ||
.context("Could not retrive cumulocity url from proxy url")? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.context("Could not retrive cumulocity url from proxy url")? | |
.context("Could not retrieve cumulocity url from proxy url")? |
&config.config_type, | ||
self.http_proxy | ||
.c8y_url_from_proxy_url(&config.remote_url.unwrap_or_default()) | ||
.context("Could not retrive cumulocity url from proxy url")? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.context("Could not retrive cumulocity url from proxy url")? | |
.context("Could not retrieve cumulocity url from proxy url")? |
valid_url.set_host(Some(&self.c8y_host)).unwrap(); | ||
valid_url.set_port(None).unwrap(); | ||
valid_url.set_scheme(Protocol::Https.as_str()).unwrap(); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That /c8y
prefix added by the proxy also must be removed:
} | |
valid_url.set_path(&valid_url.path().replace("/c8y", "")); | |
} |
Proposed changes
This PR adds operation fragments messages for
device profile
and switch the order of those messages forfirmware/software update
so that they are sent before the succesful state change.Additionally, the structure of
device profile
command was changed as it was not deserializing properly. Theremote_url
field was changed toString
fromOption<String>
as it is the mandatory field (this does not affect code as it was derived from mandatory value anyway).Types of changes
Paste Link to the issue
Checklist
cargo fmt
as mentioned in CODING_GUIDELINEScargo clippy
as mentioned in CODING_GUIDELINESFurther comments