BUG-001: Anonymous Type Materialization Fails

[improveTheWorld]

BUG-001: Anonymous Type Materialization Fails

Summary

Anonymous types cannot be materialized when using ToList(), ToArray(), or First() on projected queries. This affects both Spark and Snowflake connectors.

Error Message

System.ArgumentException: Expression must be writeable (Parameter 'left')
   at System.Linq.Expressions.Expression.RequiresCanWrite(Expression expression, String paramName)
   at System.Linq.Expressions.Expression.Assign(Expression left, Expression right)

Affected Components

• DataFlow.Framework.ObjectMaterializer (OSS)
• DataFlow.Spark v1.2.0
• DataFlow.Snowflake v1.2.0

Root Cause

Location: MemberMaterializationPlan.cs:300 in DataFlow.Framework.ObjectMaterializer

Anonymous types in C# have readonly properties (init-only backed by readonly fields). The MemberMaterializationPlan.CompileSetterForField() uses Expression.Assign() which requires a writable target. Anonymous type properties have no public setter, causing the assignment to fail.

// MemberMaterializationPlan.cs line 300
var assign = Expression.Assign(memberAccess, convertedValue); // FAILS for anonymous types

Reproduction Steps

Spark

using DataFlow.Spark;

var context = Spark.Connect();
var orders = context.Read.Parquet<Order>("path/to/orders");

// This FAILS:
var results = orders
    .Select(o => new { o.Id, o.CustomerName })  // Anonymous type
    .ToList();  // Throws ArgumentException

Snowflake

using DataFlow.Snowflake;

var context = Snowflake.Connect("connection-string");
var orders = context.Read.Table<Order>("ORDERS");

// This FAILS:
var results = orders
    .Select(o => new { o.Id, o.Amount })  // Anonymous type
    .ToList();  // Throws ArgumentException

Failing Tests

Project	Test
SparkPackageAudit	BUG001_Select_AnonymousType_ToList_Fails
SparkPackageAudit	BUG001_GroupBy_AnonymousType_ToList_Fails
SparkPackageAudit	BUG001_Join_AnonymousType_ToList_Fails
IntegrationTests	AutoUdf_CustomStaticMethod_IsAutoRegistered
IntegrationTests	AutoUdf_StringTransformation_WorksCorrectly
SnowflakeIntegrationTests	Distinct_WithAnonymousType
SnowflakeIntegrationTests	Select_ProjectsColumns

Current Workarounds

Workaround 1: Use concrete DTO classes

public class OrderSummary  // Concrete class with settable properties
{
    public int Id { get; set; }
    public string CustomerName { get; set; }
}

var results = orders
    .Select(o => new OrderSummary { Id = o.Id, CustomerName = o.CustomerName })
    .ToList();  // Works!

Workaround 2: Use Count() for existence checks

var count = orders
    .Select(o => new { o.Id, o.CustomerName })
    .Count();  // Works - no materialization needed

Proposed Fix

Detect anonymous types in MemberMaterializationPlan and use constructor injection instead of property assignment:

// Pseudo-code for the fix
if (IsAnonymousType(typeof(T)))
{
    // Use constructor: new { Id = x, Name = y } compiles to new AnonymousType(x, y)
    var constructorParams = GetConstructorParameters(typeof(T));
    var newExpression = Expression.New(constructor, constructorParams);
    return Expression.Lambda<Func<object[], T>>(newExpression, rowParameter).Compile();
}
else
{
    // Existing logic: property assignment
}

Impact

• Severity: HIGH
• Frequency: Common (anonymous types are idiomatic in LINQ)
• User Impact: Forces users to create DTOs for simple projections

Labels

bug, high-priority, oss-dependency, spark, snowflake, materialization