Problems receiving notifications with a NGSI-LD operator

[Blobonat]

Hi,

following the old Integrate FogFlow with Other NGSI-LD Broker tutorial I am trying to connect the Scorpio Broker with FogFlow.
I was able to create the NGSI-LD subscription (from Scorpio to FogFlow) and write my own operator using the JavaScript NGSI-LD operator template . The NGSI-LD notifications are successfully sent to FogFlow, but inside my operator I receive the notifications in NGSI-V1 style instead of the NGSI-LD format. Therefore the notifications can't be parsed by the provided ngsiagent inside the NGSI-LD operator template.

Logging the received body inside the ngsiagent:

{"subscriptionId":"a7d23742-4731-11ed-9803-0242ac120008","originator":"","contextResponses":[{"contextElement":{"entityId":{"type":"Building","id":"urn:ngsi-ld:Building:testBuilding01"},"attributes":[{"name":"airTemperature","type":"float","value":-9.5}]},"statusCode":{"code":200,"reasonPhrase":"OK"}}]}

Is there any additional configuration to be able to receive notifications in the NGSI-LD format inside my operator/fog function?

I followed step 4 of the tutorial to create my operator and the fog function. To run FogFlow I use the default docker-compose.yml from the current documentation.

---

Master log

DEBUG: 2022/10/08 17:50:10 taskMgr.go:544: [Unique Key Table]: map[Building-EntityID:[urn:ngsi-ld:Building:testBuilding01]]
DEBUG: 2022/10/08 17:50:10 taskMgr.go:243: [related groups]:  [map[Building-EntityID:urn:ngsi-ld:Building:testBuilding01]]
DEBUG: 2022/10/08 17:50:10 taskMgr.go:102: group k: Building-EntityID, v: urn:ngsi-ld:Building:testBuilding01
INFO: 2022/10/08 17:50:10 # hash = 235109731
DEBUG: 2022/10/08 17:50:10 taskMgr.go:102: group k: Building-EntityID, v: urn:ngsi-ld:Building:testBuilding01
DEBUG: 2022/10/08 17:50:10 taskMgr.go:102: group k: Building-EntityID, v: urn:ngsi-ld:Building:testBuilding01
DEBUG: 2022/10/08 17:50:10 taskMgr.go:339: new task {TaskID:Task.StateEstimation.stateEstimation.235109731 Name:stateEstimation Operator:road-state-estimator WorkerID: Status:scheduled Inputs:[{ID:urn:ngsi-ld:Building:testBuilding01 Type:Building AttributeList:[] Location:{Latitude:0 Longitude:0}}] Outputs:[0xc0003aa480]}, hashID 235109731, taskID Task.StateEstimation.stateEstimation.235109731
DEBUG: 2022/10/08 17:50:10 taskMgr.go:894: [Orchestration Action]:  ADD_TASK {Task.StateEstimation.stateEstimation.235109731 StateEstimation stateEstimation ServiceIntent.fb4ef5e8-3def-425f-a4c4-2ef9bc98b15f road-state-estimator    []  false 0 scheduled [{Building urn:ngsi-ld:Building:testBuilding01 []}] [{Out Out.235109731.1 []}]}
INFO: 2022/10/08 17:50:10 {ADD_TASK Worker.001. Master.001 {Task.StateEstimation.stateEstimation.235109731 StateEstimation stateEstimation ServiceIntent.fb4ef5e8-3def-425f-a4c4-2ef9bc98b15f road-state-estimator   MY_DOCKER_IMAGE [] Worker.001 false 0 scheduled [{Building urn:ngsi-ld:Building:testBuilding01 []}] [{Out Out.235109731.1 []}]}}
INFO: 2022/10/08 17:50:10 [Task update]:  &{Task.StateEstimation.stateEstimation.235109731 StateEstimation stateEstimation ServiceIntent.fb4ef5e8-3def-425f-a4c4-2ef9bc98b15f running}

Worker log

INFO: 2022/10/08 17:50:10 ADD_TASK
INFO: 2022/10/08 17:50:10 execute task  Task.StateEstimation.stateEstimation.235109731  with operation MY_DOCKER_IMAGE:latest
INFO: 2022/10/08 17:50:10 task configuration {ID:Task.StateEstimation.stateEstimation.235109731 TopologyName:StateEstimation TaskName:stateEstimation ServiceIntentID:ServiceIntent.fb4ef5e8-3def-425f-a4c4-2ef9bc98b15f OperatorName:road-state-estimator TaskType: FunctionCode: DockerImage:MY_DOCKER_IMAGE:latest Parameters:[] WorkerID:Worker.001 IsExclusive:false PriorityLevel:0 Status:scheduled Inputs:[{Type:RoadSegment ID:urn:ngsi-ld:RoadSegment:testRoadSegment01 AttributeList:[]}] Outputs:[{Type:Out StreamID:Out.235109731.1 Annotations:[]}]}
INFO: 2022/10/08 17:50:10 task runnable =  true
INFO: 2022/10/08 17:50:10 to execute Task [ Task.StateEstimation.stateEstimation.235109731 ] to perform Operation [ MY_DOCKER_IMAGE:latest ] with parameters [ [] ]
INFO: 2022/10/08 17:50:10 operator image MY_DOCKER_IMAGE:latest exists locally
INFO: 2022/10/08 17:50:11  task Task.StateEstimation.stateEstimation.235109731  started within container = 550ade873571c5f5471d532afcf8fc59dcb4b2700a716514322ec56dd9da5ab8
INFO: 2022/10/08 17:50:11 subscribe its input streams
INFO: 2022/10/08 17:50:11 [Create subscription]  a7d23742-4731-11ed-9803-0242ac120008
DEBUG: 2022/10/08 17:50:11 executor.go:134: ===========subID =  a7d23742-4731-11ed-9803-0242ac120008
INFO: 2022/10/08 17:50:11 register this task

[smartfog]

Hi,
all your steps are correct. At the moment FogFlow internally works only with NGSIv1 while data can still enter and exit FogFlow in NGSI-LD.

[Blobonat]

Ok, thank you for your answer. So is the NGSI-LD operator template just a placeholder for potential future updates?

[flaviocirillo]

Indeed, the operator was implemented for some testing since some functionalities to have it FogFlow fully working on NGSI-LD are implemented. Unfortunately, it is still not mature enough to have them enabled.

[Blobonat]

Thank you for the answer. I'd love to see full NGSI-LD support in the future :)

[AuboIoT]

@Blobonat I was trying to start using FogFlow for doing some comparison between cloud and edge performance.
I currently have implemented FIWARE elements with the orion-ld as broker and now I was trying to make the next step, to deploy FogFlow. Nevertheless, I've found that documentation is outdated and I can't understand how to run just a hello world. Besides that, based on what you write here, I have some doubts if FogFlow is compatible now, 2025, with NGSI-LD.

So, based on your experience, could you give me some update if it is possible to use FogFlow nowadays?? Please. Or if you @flaviocirillo could give me some update too please. Thanks in advance both of you!

[Blobonat]

Hi @AuboIoT, since the last time I worked with NGSI-LD and especially Fogflow was 2 years ago, I can't give you an update on the usability today. Sorry about that.

[flaviocirillo]

Hi @AuboIoT and @Blobonat , I am using currently FogFlow with NGSI-LD and it is working fine with Scorpio Broker as NGSI-LD Broker. I have now tested the tutorial, fixed it and updated the documentation. Can you please try again?
https://fogflow.readthedocs.io/en/latest/onepage.html#hello-world-example-ngsi-ld

[AuboIoT]

Thank you so much @flaviocirillo. I am going to give it a try.
I will inform here how everything goes. Thx!

[AuboIoT]

Hello @flaviocirillo, first off, thank you for your fast reply and for taking the time to update the documentation as fast as you did. I really appreciate it.
I was giving it a try, but I couldn't replicate the tutorial. These are some observations I've found:

1. In section 1.1 the second wget command is downloading the whole page rather than the JSON file. This is because the link that wget is poiting to is not the raw file. Hence, the link that wget should point would be:
   https://raw.githubusercontent.com/smartfog/fogflow/refs/heads/development/release/latest/cloud/config-template.json
   rather than
   https://github.com/smartfog/fogflow/blob/development/release/latest/cloud/config-template.json

2. In section 1.2 when showing how the config.json file should look like, there is a difference that I notice with the file of the config.json. In the website, appears "discovery": {... "storeOnDisk": false, ...}, whereas in the json file hosted on Github, that value is true. So, I'm not sure if that value should be true or false.

3. Also in section 1.2, a minor observation, not sure if its correct but I think that the renaming of the file could be done using mv config-template.json config.json rather than copying with cp config-template.json config.json

4. Then in Section 1.5 the only container that is missing is the one of fogflow/designer, that I understand is the web interface. Moreover, that container does not appear and it was not stopped (something that can be verified using docker ps -a.)
   Image from tutorial.

Image in my machine.

Those were my observations.
I guess that the designer container is not running because in the nginx.conf file the main path "/" that pointed to the designer container was commented.
Likewise, in the docker-compose.yml file the designer container can't be pulled and run since it is commented.

Thank you!

[flaviocirillo]

Thank you very much for trying it! I see there are multiple things that I need to fix. But I believe you can still give a try:

1. Yes, sorry for that, wrong copy-paste, will fix it

2. Keep it on "storeOnDisk": false, it is not necessary for the tutorial. This will be good to keep once you want to keep the data of FogFlow on disk (so you can kill and restart FogFlow and have all the data and subscriptions already there)

3. No big difference, as you like most

4. You are absolutely correct. Most likely I have changed it for some local testing and have wrongly pushed also that configurations. Can you try to put the designer back in the docker-file and nginx.conf and see if it works?

[AuboIoT]

Sure, I gave a try uncommenting those lines (although I left commented the lines related to "discovery" service).

I also uncomment the nginx.conf file.

But the container keeps restarting with a message:
designer-1 | exec /usr/local/bin/node: exec format error
designer-1 | exec /usr/local/bin/node: exec format error
designer-1 | exec /usr/local/bin/node: exec format error
designer-1 | exec /usr/local/bin/node: exec format error
designer-1 | exec /usr/local/bin/node: exec format error
designer-1 | exec /usr/local/bin/node: exec format error
designer-1 | exec /usr/local/bin/node: exec format error
designer-1 | exec /usr/local/bin/node: exec format error