Clara rules throws a Null Pointer Exception even though there is no Null values #437

stLalo · 2019-10-04T17:47:06Z

Description

I am building a ETL Software and I use clara rules to clean data by applying certain constraints (rules) to shape the data as I want. Rules always work well, until I make my ETL Software a multi-threading parallel synchronous processor. After this, I started running stress testing on the software and I was able to caught an exception that is weird because Clara is screaming at me for trying to divide a null value.

Steps to reproduce

Protocol Event

(defrecord Event [_id carrier comment tank-category  rate reference uom volume ])

Rule

(defrule convert-to-barrels
  "If the unit of measure is gallons convert it to barrels."
  [?event <- Event (= uom :ga) (some? volume)]
  =>
  (as-> (get @evnts (:_id ?event)) _
    (assoc _ :uom :barrels)
    (update _ :volume #(/ % 42))
    (update _ :rate #(/ % 42))
    (swap! evnts #(assoc % (:_id ?event) _)))

Fire rules

(defn apply-rules
  "Apply the rules in the namespace to the given events."
  [events]
  (reset! evnts (seq/seq-map->map events :_id))
  (-> events
      (mk-session 'etl.software.flowers.rules)
      (insert-all events)
      (fire-rules))
  (vals @evnts))

Actual Behavior

clojure.lang.ExceptionInfo: Exception in etl.software.flowers.rules/convert-to-barrels with bindings
 {:?event #etl.software.flowers.core.Event{:_id "733B772B94101049D1C8520FF8A9FB109CBC66BDB6A577E8A0E6311698837E12",
 :created "2019-10-02T14:36:50Z",
 :deleted false,
 :rate 48763.0,
 :reference "DFP0248029",
 :start #object[java.time.ZonedDateTime 0x2a488968 "2019-10-04T01:03Z[UTC]"],
 :stop #object[java.time.ZonedDateTime 0x6de70bdd "2019-10-04T23:00:49Z[UTC]"], 
:uom :ga,
 :volume 1071000}}
	at clara.rules.engine$fire_rules_STAR_$fn__10629$fn__10630.invoke(engine.cljc:1797)
	at clara.rules.engine$fire_rules_STAR_$fn__10629.invoke(engine.cljc:1763)
	at clara.rules.engine$fire_rules_STAR_.invokeStatic(engine.cljc:1756)
	at clara.rules.engine$fire_rules_STAR_.invoke(engine.cljc:1720)
	at clara.rules.engine.LocalSession.fire_rules(engine.cljc:1909)
	at clara.rules$fire_rules.invokeStatic(rules.cljc:44)
	at clara.rules$fire_rules.invoke(rules.cljc:29)
	at etl.software.flowers.rules$apply_rules.invokeStatic(rules.clj:159)
	at etl.software.flowers.rules$apply_rules.invoke(rules.clj:152)
	at etl.software.flowers.core$process_schedules.invokeStatic(core.clj:123)
	at etl.software.flowers.core$process_schedules.invoke(core.clj:117)
	at etl.software.ingest.io$run.invokeStatic(io.clj:79)
	at etl.software.ingest.io$run.invoke(io.clj:73)
	at etl.software.ingest.io$router.invokeStatic(io.clj:93)
	at etl.software.ingest.io$router.invoke(io.clj:84)
	at etl.software.ingest.io$lightweight_process$fn__19235$state_machine__6366__auto____19238$fn__19240.invoke(io.clj:158)
	at etl.software.ingest.io$lightweight_process$fn__19235$state_machine__6366__auto____19238.invoke(io.clj:158)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
	at etl.software.ingest.io$lightweight_process$fn__19235.invoke(io.clj:158)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
	at clojure.lang.Numbers.ops(Numbers.java:1068)
	at clojure.lang.Numbers.divide(Numbers.java:189)
	at clojure.lang.Numbers.divide(Numbers.java:3881)
	at etl.software.flowers.rules$eval53013$fn__53022$fn__53023.invoke(NO_SOURCE_FILE:55)
	at clojure.core$update.invokeStatic(core.clj:6196)
	at clojure.core$update.invoke(core.clj:6188)
	at etl.software.flowers.rules$eval53013$fn__53022.invoke(NO_SOURCE_FILE:55)
	at clara.rules.engine$fire_rules_STAR_$fn__10629$fn__10630.invoke(engine.cljc:1764)
	... 25 common frames omitted
19-10-04 15:46:29 Tardis ERROR [etl.software.ingest.io:82] - 
                                                          java.lang.Thread.run              Thread.java:  748
                            java.util.concurrent.ThreadPoolExecutor$Worker.run  ThreadPoolExecutor.java:  624
                             java.util.concurrent.ThreadPoolExecutor.runWorker  ThreadPoolExecutor.java: 1149
                                                                           ...                               
                                  etl.software.ingest.io/lightweight-process/fn                   io.clj:  158
                  clojure.core.async.impl.ioc-macros/run-state-machine-wrapped           ioc_macros.clj:  977
                          clojure.core.async.impl.ioc-macros/run-state-machine           ioc_macros.clj:  973
            etl.software.ingest.io/lightweight-process/fn/state-machine--auto--                   io.clj:  158
         etl.software.ingest.io/lightweight-process/fn/state-machine--auto--/fn                   io.clj:  158
                                                  etl.software.ingest.io/router                   io.clj:   93
                                                     etl.software.ingest.io/run                   io.clj:   79
                    etl.software.flowers.core/process-schedules                 core.clj:  123
                         etl.software.flowers.rules/apply-rules                rules.clj:  159
                                                        clara.rules/fire-rules               rules.cljc:   44
                                    clara.rules.engine.LocalSession/fire-rules              engine.cljc: 1909
                                                clara.rules.engine/fire-rules*              engine.cljc: 1756
                                             clara.rules.engine/fire-rules*/fn              engine.cljc: 1763
                                          clara.rules.engine/fire-rules*/fn/fn              engine.cljc: 1764
          etl.software.flowers.rules$eval53013$fn__53022.invoke           NO_SOURCE_FILE:   55
                                                           clojure.core/update                 core.clj: 6196
etl.software.flowers.rules$eval53013$fn__53022$fn__53023.invoke           NO_SOURCE_FILE:   55
                                                                           ...                               
java.lang.NullPointerException: 
    clojure.lang.ExceptionInfo: Exception in etl.software.flowers.rules/convert-to-barrels with bindings {:?event #etl.software.flowers.core.Event{:_id "733B772B94101049D1C8520FF8A9FB109CBC66BDB6A577E8A0E6311698837E12",
 :created "2019-10-02T14:36:50Z",
 :deleted false,
 :rate 48763.0, 
:reference "DFP0248029", 
:start #object[java.time.ZonedDateTime 0x2a488968 "2019-10-04T01:03Z[UTC]"],
 :stop #object[java.time.ZonedDateTime 0x6de70bdd "2019-10-04T23:00:49Z[UTC]"],
 :uom :ga,
 :volume 1071000}}
          batched-logical-insertions: []
             batched-rhs-retractions: []
    batched-unconditional-insertions: []
                            bindings: {:?event
                                       {:_id
                                        "733B772B94101049D1C8520FF8A9FB109CBC66BDB6A577E8A0E6311698837E12",
                                        :company-id
                                        "B18D748B4B7A2CE3FD5C18543FA5DE639F5119D5480F0B9A06D8614B7985CBFF",
                                        ...}}
                           listeners: []
                                name: "etl.software.flowers.rules/convert-to-barrels"
                                 rhs: (do
                                       (as->
                                        (get @evnts (:_id ?event))
                                        _
                                        (assoc _ :uom :barrels)
                                        (update _ :volume (fn [v] (/ v 42)))
                                        (update _ :rate (fn* [p1__18408#] (/ p1__18408# 42)))
                                        (swap! evnts (fn* [p1__18409#] (assoc p1__18409# (:_id ?event) _)))))

The text was updated successfully, but these errors were encountered:

EthanEChristian · 2019-10-07T16:48:04Z

@stLalo,
Its not recommended to do state-based manipulation in the LHS or RHS of rules.
Clara makes no guarantees on the number of times the LHS/RHS is executed, only that the steady state is factually correct, meaning that due to truth maintenance the RHS might execute many times. Additionally, the way that clara manages logical retractions would likely lead to less than ideal scenarios when using stateful actions in the RHS.

As to why the null pointer is thrown, do you have the state of the atom when the RHS is fired?
From the error it looks like that either:

evnts doesn't contain the event id in question
the event within evnts doesn't contain a : volume or :rate value.

stLalo · 2019-10-07T17:18:16Z

@EthanEChristian
During the weekend, I explore more my situation and I figured out the problem. You said,

evnts doesn't contain the event id in question
And this is correct.

As the ETL Software is a Parallel Synchronous multi-thread processor, the state was lost somewhere in between threads. I think my proper solution could be the following steps:

For every rule, insert the Event into Clara rules in-memory database
Query for the events that fitted the Clara rules
Apply corresponding changes for each Clara rule type
Repeat step 1 to 4 until there is no more Records to apply the rules too.

Please bare with me as I am refactoring legacy code and it is the first time encountering this problem.

EthanEChristian · 2019-10-07T18:58:47Z

@stLalo,
I agree with:

For every rule, insert the Event into Clara rules in-memory database

Query for the events that fitted the Clara rules

Without knowing the constraints of your codebase I can't say for sure, but I feel that all events could be processed in the same session for:

Apply corresponding changes for each Clara rule type

Repeat step 1 to 4 until there is no more Records to apply the rules too.

But regardless of 3&4, I think that the Insert & Query pattern should achieve what you need.

stLalo · 2019-10-07T19:11:02Z

So what I have in mind would be something like

Insert all my events in one session

(-> events
      (mk-session 'some.namespace.rules)
      (insert-all events)
      (fire-rules))

Matching rules will be inserted such as this

(defrule do-something-coll
[?event <- Event (= type :tulip)]
=>
 (insert! (->Event ?event))

(defquery get-tulips
[]
(Event (= :tulip (:type ?event))))

Apply changes to type = tulip

(defn thingamagiq
 [Event]
 (assoc Event :howdy "howdy"))

Repeat the same steps with other rules and queries.
I based my approach from this example

EthanEChristian · 2019-10-08T13:42:51Z

(defrule do-something-coll
[?event <- Event (= type :tulip)]
=>
 (insert! (->Event ?event))

Rules that bind and insert the same fact type can be dangerous as they are prone to infinite looping.

For queries, you could use parameters to be more flexible in the event that you need other event types:

(defquery get-events-by-type
[:?type]
[?event <- Event (= ?type type)])

stLalo · 2019-10-08T17:44:01Z

So to continue with this thread, I went back and did the following to my rules.

Create a new Record with similar attributes from the main Event Record.

  (mk-session 'some.namespace.rules)
  (insert-all events)
  (fire-rules))

 (defrule do-something-coll
        [?event <- Event (= type :tulip)]
       =>
         (insert! (map->NewEvent (assoc ?event :color "blue"))

 (defquery get-events
    []
     [?newevents <- NewEvent])

This works wonderful for only one rule. However, I have cases where Event Records will be true for several Rules. Inserting! the true fact into NewEvent, will cause the creation of duplicates of the same events but with different values here and there.
I was thinking about using Retract! on the rules after the first one, but it seems not to return the previous NewEvent

EthanEChristian · 2019-10-08T18:34:42Z

@stLalo,
I think by modeling this slightly differently you could get away from the need to have explicit retractions. Again retractions can lead to unintended looping behavior.

From some of the things that you posted so far, it looks like you are trying to apply a series of changes to the original model. The way I was thinking would be something like this:

Facts:

(defrecord Event [id type color stem-length])
(defrecord FinalEvent [id type color stem-length])
(defrecord Correction [id field val])

Correction Rules:

(defrule correction-one 
  [?event <- Event (= type :tulip) (= ?id id)]
  =>
  (insert! (->Correction ?id :color "blue")))

(defrule correction-two 
  [?event <- Event (= type :tulip) 
    (= ?id id)
    (> stem-length 15)]
  =>
  (insert! (->Correction ?id :stem-length 15)))

Consolidation:

(defrule final-event
  [?event <- Event (= ?id id)]
  [?corrections <- (acc/all) :from [Correction (= id ?id)]]
  =>
  (let [final-event (reduce (fn [final correction]
                              (assoc final (:field correction) (:val correction)))
                            ?event
                            ?corrections)]
    (insert! (map->FinalEvent final-event))))

Query:

(defquery get-events
    []
   [?final-event <- FinalEvent])

This side steps the duplicate event behavior by adding an accumulator and consolidation rule.

EthanEChristian · 2019-10-10T13:49:39Z

I suppose if the Session had enough scope it might include rules that wanted to correct the same field. That might require a hierarchy of what corrections need to be applied in what order.

The approach above could also be done where the correction fact would contain a fn instead of key/value pairs.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clara rules throws a Null Pointer Exception even though there is no Null values #437

Clara rules throws a Null Pointer Exception even though there is no Null values #437

stLalo commented Oct 4, 2019 •

edited

Loading

EthanEChristian commented Oct 7, 2019

stLalo commented Oct 7, 2019 •

edited

Loading

EthanEChristian commented Oct 7, 2019

stLalo commented Oct 7, 2019 •

edited

Loading

EthanEChristian commented Oct 8, 2019

stLalo commented Oct 8, 2019

EthanEChristian commented Oct 8, 2019

EthanEChristian commented Oct 10, 2019

Clara rules throws a Null Pointer Exception even though there is no Null values #437

Clara rules throws a Null Pointer Exception even though there is no Null values #437

Comments

stLalo commented Oct 4, 2019 • edited Loading

Description

Steps to reproduce

Actual Behavior

EthanEChristian commented Oct 7, 2019

stLalo commented Oct 7, 2019 • edited Loading

EthanEChristian commented Oct 7, 2019

stLalo commented Oct 7, 2019 • edited Loading

EthanEChristian commented Oct 8, 2019

stLalo commented Oct 8, 2019

EthanEChristian commented Oct 8, 2019

EthanEChristian commented Oct 10, 2019

stLalo commented Oct 4, 2019 •

edited

Loading

stLalo commented Oct 7, 2019 •

edited

Loading

stLalo commented Oct 7, 2019 •

edited

Loading