[Bug]: GUI freezing a lot

[gertwieland]

Apache Hop version?

2.7

Java version?

OpenJDK Runtime Environment Temurin-11.0.21+9 (build 11.0.21+9)

Operating system

Windows

What happened?

Hop freezes nearly every time when a transform (especially Select) references a field that had existed before, but was removed later.

For example, let's assume a pipeline contains the field "xyz" and is used in a Select transform. If you go to an earlier transform, remove that field and run the Pipeline again, the GUI will freeze. The "DOS" window keeps working properly and even shows an appropriate error message. However, since the GUI froze, one needs to restart the application.

I'm not exaggerating here. We made significant changes to the input data of a fairly complex pipeline, and within the last two hours, I had to restart the Hop GUI at least 30 times.

Issue Priority

Priority: 1

Issue Component

Component: Hop Gui

[bamaer]

hi @gertwieland, this is often caused by database connections with a high latency.
does your pipeline read data from e.g. Azure SQL Server? if that's the case, and the metadata for a table input changes, hop will need to talk to the database server to retrieve the metadata for the new query result, which can slow down the entire pipeline up to the point where Hop Gui hangs.

[gertwieland]

Here's another scenario that leads to the GUI freezing.
When setting up a Javascript, but forgetting to set up the data type for the output fields

[gertwieland]

@bamaer Thanks Bart! Sorry, I just saw your message!
In this example, I pulled data directly from an SAP/S4 system. So yes, high latency could indeed be a factor.
I never noticed this pattern, but I'll pay attention when it happens again.

What bugs me is not that it's "slow", but it freezes entirely and never comes back. Well, "never" as in "after 15 secs or so I kill it". I'll try next time to wait even a few minutes.

The strange thing is that the "DOS Window" keeps working just fine, it does not freeze at all.

[gertwieland]

Here's another example when Hop freezes: Removing the lookup field in a "Stream Lookup" transform from the incoming stream.

[gertwieland]

It seems it's not possible to add comments here without also closing the ticket. Reopening.

[sramazzina]

@gertwieland, the things you are experiencing comes from what @bamaer already explained you. What he told you is not only valid for a single transform that is directly connected to the db but is also valid for any other transform that is in the path that ges from this input transform tol the end of the pipeline. I think you understood what I mean with this phrase.

This because any time you try to add/remove fields in a transform's configuration, to present you the set of fields you can use Hop tries to reconstruct the fieldset that is expecting will flow into the transform you are configuring. And to do this, it goes back till the very beginning of the chain and then go down. That said, if some transforms in this trip have any performance issues (becaue maybe of the connection that is missing, the fieldset recontruction is impacted, takes longer and you can suffer of these weird freezeing situations. This behavior is definitely NOT a bug in Hop but it is only due these reasons.

I hope this explanation makes the things clearer for you about how Hop works internally. Let us know if something is unclear.

[gertwieland]

Hi Sergio, Thank you very much for your reply, I appreciate that. I won't go much into details, but let me share two thoughts. I've worked with PDI for years (and I'd say, fairly extensively within my use case.) I'm not gonna say it 'never' froze, but definitely not even nearly as much as Hop. Also, you are explaining why the GUI freezes. Well, if that's not a bug, then I don't know what is. Anyway, I'll close the ticket. And every time it freezes, I'll likely swear a bit at it. But then, my love for Hop will quickly take over 😉 Cheers, Gert Feb 26, 2024 4:28:22 a.m. Sergio Ramazzina ***@***.***>:

…

@gertwieland[https://atpscan.global.hornetsecurity.com/index.php?atp_str=mrkKS65xv3lQEluQdck7-27BJjfv9oVaSUfvVUHfjsGuNiV5txlkCbT4N_KZVipk9TS9cote3UlEHGOK83JGZvlxdhPjnuPdekgSTZYO-n_cNduY3P2FfoLAS6v_U4C1rNG8BC48_vd95-fXR_QueUlSavCUO1WAej7AAxA2njn3axbWhID_3mH12ELOpAABzDbUm95AYuPKjOQFLQSjv5kWFvCsgb6J2CzQAqY37DpwUBDHN6srN6jeMB1QBqeKTmtdzq-fYHgvd4CSrCJn-2Vs-eIwIzo6I6lQSkiWfPvKKMEvxSM6OiO7pAMfqLFrNT_4oS7xU3r_], the things you are experiencing comes from what @bamaer[https://atpscan.global.hornetsecurity.com/index.php?atp_str=jG4hVnxSVmMH1nKZ5aBAG9EaQa6fN4244v9BiknELOtXUpOhQduBvrVMAjWxABDm5ZAH_TalkywIfPnWkZf11Q4l0MD_oWHgSydY3RX8sZzW6hMusSQJdnopWiyiywe4QI5Dki06EqbfuUl1djNaO85qCGUh90QM2SXeL3WGJ2DC_nJ2bw7ptTTEHPrDjz5hG6Po6LkjK_59o2op0SFgRPFW5JT_qoXh2Gg87u1v-vJvcbjjtRzMwmWdAODWtbKz7lPNiQDaTbMgW5GJOditzSM6OiOSC16rsO9jLD3gaKQjOjojx5GJ9dA_is9TTVIjrAyZnA] already explained you. What he told you he is telling you is not only valid for the single transform that is connected to the db but is also valid for any other transform that is in the path that ges from this input transform tol the end of the pipeline. I think you understood what I mean with this phrase. This because any time you try to add/remove fields in a transform's configuration, to present you the set of fields you can use Hop tries to reconstruct the fieldset that is expecting will flow into the transform you are configuring. And to do this, it goes back till the very beginning of the chain and then go down. That said, if some transforms in this trip have any performance issues (becaue maybe of the connection that is missing, the fieldset recontruction is impacted, takes longer and you can suffer of these weird freezeing situations. This behavior is definitely NOT a bug in Hop but it is only due these reasons. I hop this explanation makes the things clearer for you about how Hop works internally. Let us know if something is unclear. — Reply to this email directly, view it on GitHub[https://atpscan.global.hornetsecurity.com/index.php?atp_str=4E09IQMZadzo6UUDEcXGErw2youTD7lJDfvlv6bG5ebbwq-Q5CerN0jTI5XLadTYUDHBHaStf3xyZJHs2wVK41j0mqZKJ-G6qbioX3RtDzOzI-oHv1Ye7gE2BHBdTTWlFNzHH6D8ww7yxy9POi1J-sK5pX-hHWM3HTDmhu3aqo8TRCOx6GM12GGWAAjlGUtw7GdG_v2x-ZuuId3tisyFRv0LHqAFlXwIt3A-ymMcxB9sGlE36ScJkhrLLr77JZACkcEw1EnZGF8yM4JOIWXROP3u7QyIGY1volTdt3CEVrcvwltAHDySDrHLH9ofgFC3ypmj6EC9ezMjOjojN515Yz1KDULbeA-8Izo6I5_cO1dCS_hjeVvzWHX1yc8], or unsubscribe[https://atpscan.global.hornetsecurity.com/index.php?atp_str=z47awELsUwX2WWwsN-bIQ3YH_bMRVwcaGNr7ddXrJonTeiwSHtbeVfrujPE_PRl154-Y0nYMToO5vInBJEfnJqGHHk9PE75piX4kWNNgLGILFqjf2gn-TVt2sr4hBJexjvEzYvkPOY67MKTMMMxSBwIUGafsZxi_S4R07-ZqOqIy95fGcLbD0UtdqfO2Ln8yIpkJM0ajFNLa1JWzp5Ld7MT4EP1yV3XD2UJ3a2FVv62jB3W1FueANjFQIbQ7kWhqzZhNXCLsh1mDkVIVOGs7OD6-V5aVYnaCnHbHP0R8BEVIRuVxjqP04Mx911DPvWqxrThMBVx_xztCiNgGREacFTFT4ETJ3703FpvVIWmn91CgSPQ4KVnDNj2dW128xhHTBq5b3hrumlPzJEWpKlpYKB8j17CEsPxc365jUK8Y1kmCMFPj-aY8I93yllZc9NYfkpU0Izo6I_TKlYzGmiY8Tvyh4yM6OiPBGuXJElwLDx2HPMPCr4mg]. You are receiving this because you were mentioned. [Tracking image][https://github.com/notifications/beacon/AMK5NEZ5DFV2RQHFMUMHE53YVRIR3A5CNFSM6AAAAABDSLIO7KWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTVBMY64.gif]