Deduping mechanism in pinot

[pradeepgv42]

Is there a way to dedup similar rows? Will help with dealing with duplicate rows during kafka consumption and if partial data is ingested through batch mechanism (dedup the overlapping segment with the data consumed through realtime)

[kishoreg]

@npawar
This can be a good transformer to add to filter duplicate events.

[mcvsubbu]

There is a table config setting to aggregate metrics. If all your dimension values are same, then the metric values are added, and the rows are squished. Is this what you mean by "similar?"

There are some limitations, but give it a try

[mcvsubbu]

https://docs.pinot.apache.org/basics/components/table#tableindexconfig

[pradeepgv42]

I could use aggregate and unique, but I am thinking of following case which I believe wouldn't return accurate results, correct me if I am wrong or if there is an alternate solution.

With "aggregated" (meaning all column values are same including the timestamp and assume timestamp is in microseconds) if there are two rows of same value, count would become 2. Usually for querying purposes we would be interested in a slightly higher time-granularity (across some set of columns, for example 5m) then the overall counts we get would be doubled rite?

If we are looking at individual rows, we can get away with aggregate/unique sort of approach.

[pradeepgv42]

nm, I could actually just do count(*) instead of sum(count column) 🤦

[cad]

What is the suggested solution to this problem at the moment?

[Jackie-Jiang]

Dedup feature is added in #8708