Fix: spark bit_count function

[kazantsev-maksim]

Which issue does this PR close?

Closes #18225

Rationale for this change

After adding the bit_count function in Comet, we got different results from Spark. (apache/datafusion-comet#2553)

Are these changes tested?

Tested with existing unit tests

[kazantsev-maksim]

Spark supports only signed int types

[alamb]

This comment seems misplaced as the code adds support for boolean 🤔

[alamb]

I also think this code will now panic if you pass in a Boolean array

[kazantsev-maksim]

I wanted to note that Spark only supports signed integer and boolean types as input.

[kazantsev-maksim]

Added an additional test for BooleanArray.

[pepijnve]

Sorry for reviving this one. We had a need for bit_count in our codebase as well, so I was looking at the Spark implementation. This test seems really bizarre:

#[test]
fn test_bit_count_int32() {
    // Test bit_count on Int32Array
    let result =
        spark_bit_count(&[Arc::new(Int32Array::from(vec![0i32, 1, 255, 1023, -1]))])
            .unwrap();

    let arr = result.as_primitive::<Int32Type>();
...
    assert_eq!(arr.value(4), 64); // -1 in two's complement = all 32 bits set
}

Counting the bits of a 32-bit value and getting back 64 is not what you would expect at all. Is that really how Spark works? It's a bit underspecced (or at least ambiguous) in the Spark documentation.

Just for comparison, here's DuckDB's answer to the same question

D select bit_count(cast(-1 as int));
┌────────────────────────────────┐
│ bit_count(CAST(-1 AS INTEGER)) │
│              int8              │
├────────────────────────────────┤
│               32               │
└────────────────────────────────┘

[pepijnve]

Figured it out in the meantime. Spark's implementation (like, for instance also SQL Server's) always operates on BIGINT (i.e. 64-bit). I logged a PR to simplify the implementation.