Migrating DatabaseSessionService to use JSON serialization

[DeanChensj]

Problem Statement

Pickle Serialization Causes Breaking Changes

The current DatabaseSessionService stores the EventActions object using pickle serialization.
When the EventActions Pydantic model schema changes (e.g., adding new fields like compaction, agent_state, or rewind_before_invocation_id), existing pickled data in the database cannot be properly deserialized. This creates a breaking change that requires database migration.

Inconsistent storage schema with LlmResponse

Adding new fields to LlmResponse necessitates explicit inclusion in the Events table. This has historically led to discrepancies between the database schema and the corresponding Python objects.

Incompatible schema for different language

EventAction is stored in pickle, which can not be shared between languages

• Python-Specific: pickle serializes internal Python bytecode and object structures. Languages like Java, JavaScript, or Go cannot understand or reconstruct them.
• Security Risk: Unpickling data is insecure and can allow arbitrary code execution if the database is compromised.

Proposed Solution: JSON Representation

The solution is to refactor StorageEvent to use JSON serialization instead of pickle, similar to how other fields like content are already handled. For key fields like ID, timestamp, invocation_id, we will still keep dedicated columns for them.

Implementation Pattern

The conversion from Event to StorageEvent already uses JSON serialization for complex fields. And deserialization uses a utility function. The decode_model utility function handles JSON-to-Pydantic conversion.

ADK team will provide a one-time migration script to update existing databases to the new schema, compatible with both PostgreSQL and MySQL.

Benefits of JSON Representation

• Forward Compatibility: New fields in EventActions won't break existing data
• Human Readable: JSON is inspectable and debuggable in the database
• Database Agnostic: Works consistently across PostgreSQL, MySQL, and SQLite using the existing DynamicJSON type.
• Consistent Pattern: Aligns with how all other Event fields are already stored

We are planning to deploy this migration in the near future. We would greatly appreciate any feedback, insights, or potential concerns from the community regarding this proposed change. Please feel free to share your comments or feedback within this thread.

APPENDIX

• Database Session Management : Migration errors. #3520
• Migration script generates invalid Python syntax with MySQL/PostgreSQL #3501
• Update database schema automatically #3343
• Need migration to update Database on new releases #456
• AttributeError in Events Compaction with DatabaseSessionService #3530
• DatabaseSessionService class: set a more explicit way to decide the schema #2217
• Specify Schema when using DatabaseSessionService #2115

[Pranav-Bhatlapenumarthi]

One thing worth discussing is the performance overhead—are there any anticipated impacts on read/write speed or storage size compared to pickle? Also, regarding complex nested objects or custom types, will additional validation or conversion be required?

Overall, this is a much-needed and progressive change. Cheers !

[DeanChensj]

To overcome the query speed overhead, we can introduce index on the fields that we are interested in, e.g. timestamp, invocation_id.

Regarding nested objects, we should make sure they are JSON-serializable.