Fengttt agg skl

[fengttt]

User description

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #22761

Move agg state from AoS to SoA

What this PR does / why we need it:

This supercedes #23420

---

PR Type

Enhancement, Bug fix

---

Description

• Refactor aggregation state storage from Array-of-Structures to Structure-of-Arrays (SoA) layout

• Implement skiplist-based argument storage for distinct aggregations

• Reimplement sum and avg functions using new aggregation framework

• Add support for decimal types in sum/avg operations

• Fix memory management and spill handling in group operations

---

Diagram Walkthrough

flowchart LR
  A["Old AoS State<br/>Generic Type S"] -->|Refactor| B["New SoA State<br/>Vector-based"]
  B -->|For Distinct| C["Skiplist Arena<br/>Argument Storage"]
  B -->|For Non-Distinct| D["Direct Vectors<br/>Count + Sum"]
  E["Sum/Avg Functions"] -->|Use| B
  E -->|Support| F["Int/Float/Decimal<br/>Types"]

Loading

File Walkthrough

	Relevant files
Enhancement	12 files aggState.go Refactor aggregation state to SoA with skiplist storage    +544/-381 sumavg2.go Implement sum and avg using new framework                                +487/-0  count2.go Update count functions for new state layout                            +37/-46  types.go Register sum and avg in special aggregations                          +4/-0      register.go Add registration functions for sum and avg                              +12/-0    encoding.go Add int16/uint16/uint32 read/write helpers                              +61/-16  decimal.go Add decimal conversion helper functions                                    +21/-1    vector.go Add null append and improve vector free handling                  +10/-1    ptrlen.go Add Replace and Free methods to PtrLen                                      +34/-0    unsafe.go Add UnsafeFromBytes helper function                                            +4/-0      list_agg.go Update sum/avg to use new framework functions                        +20/-6    special.go Add registration wrappers for sum and avg                                +8/-0
Bug fix	2 files types.go Remove unused int32/uint32 conversion functions                    +6/-6      helper.go Fix spill memory release timing                                                    +3/-4
Tests	9 files mpool_test.go Add test for PtrLen Replace functionality                                +26/-0    count2_test.go Add skiplist validation checks to tests                                    +4/-1      time_window.result Update test results for new aggregation behavior                  +23/-9    window.result Update window function test results                                            +2/-12    window.sql Add issue markers to window tests                                                +3/-1      boundary_comprehensive.result Update overflow error message format                                          +2/-2      func_datetime_utc_timestamp.result Update timestamp calculation test result                                  +1/-1      table_func_metadata_scan.result Update overflow error message format                                          +1/-1      bit.result Update bit type aggregation test results                                  +2/-4
Documentation	1 files repl.go Update copyright year to 2026                                                        +1/-1
Additional files	9 files encoding_test.go +0/-9      reuse.go +0/-75    avg.go +0/-275  size_test.go +0/-15    sum.go +0/-336  sum_test.go +0/-605  types.go +0/-4      sum.go +0/-118  sum_test.go +0/-48

---

[qodo-code-review]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
⚪	Unbounded length allocation Description: ReadSizeBytesToBuf allocates a new buffer based directly on the int32 length read from an io.Reader without validating that the size is non-negative and within a sane upper bound, enabling memory-exhaustion or panic (negative/overflow length) denial-of-service if the reader is influenced by untrusted/corrupted input. encoding.go [801-821] Referred Code func ReadSizeBytesToBuf(r io.Reader, buf []byte, offset int32) (int32, []byte, error) { sz, err := ReadInt32(r) if err != nil || sz == 0 { return 0, buf, err } if sz+offset <= int32(cap(buf)) { buf = buf[:offset+sz] if _, err := io.ReadFull(r, buf[offset:offset+sz]); err != nil { return 0, buf, err } return sz, buf, nil } else { newbuf := make([]byte, sz+offset) copy(newbuf, buf[:offset]) if _, err := io.ReadFull(r, newbuf[offset:offset+sz]); err != nil { return 0, newbuf, err } return sz, newbuf, nil } }
	Panic on invalid input Description: checkAggStateMagic panics on read errors or mismatched magic, which can crash the process and become a denial-of-service vector if intermediate aggregation state bytes can be corrupted or attacker-influenced (e.g., via spill files or distributed exchange). aggState.go [659-664] Referred Code func checkAggStateMagic(reader io.Reader) { magic, err := types.ReadUint64(reader) if err != nil || magic != magicNumber { panic(moerr.NewInternalErrorNoCtxf("invalid magic number, got %d, %v", magic, err)) } }
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
🔴	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Panic-based failures: New logic uses multiple panic(...) paths for malformed/intermediate data (e.g., magic mismatch and arg count mismatches), which can crash the process instead of returning actionable errors. Referred Code func checkAggStateMagic(reader io.Reader) { magic, err := types.ReadUint64(reader) if err != nil || magic != magicNumber { panic(moerr.NewInternalErrorNoCtxf("invalid magic number, got %d, %v", magic, err)) } Learn more about managing compliance generic rules or creating your own custom rules
⚪	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Internal details in panic: Panic messages include internal state details (e.g., magic values and mismatch counters) and it is unclear from the diff whether these panics can surface to user-facing error paths. Referred Code func checkAggStateMagic(reader io.Reader) { magic, err := types.ReadUint64(reader) if err != nil || magic != magicNumber { panic(moerr.NewInternalErrorNoCtxf("invalid magic number, got %d, %v", magic, err)) } Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Missing length validation: UnsafeFromBytes dereferences bs[0] without checking len(bs), which can panic if any caller passes an empty slice due to corrupted/intermediate state data. Referred Code func UnsafeFromBytes[T any](bs []byte) *T { return (*T)(unsafe.Pointer(&bs[0])) } Learn more about managing compliance generic rules or creating your own custom rules
Update

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[ouyuanning]

可以先判断 vectors[0].hasNull。弄2个分支，可能对于没有null值的会快一点。
if vectors[0].hasNull{
//当前代码
} else {
//当前代码去掉125行的判断
}