Summary

This issue tracks priorities and discussions around DataFrame improvements based on issues and feedback.

Microsoft.Data.Analysis Open Issue Query: https://github.com/dotnet/machinelearning/issues?q=is%3Aopen+is%3Aissue+label%3AMicrosoft.Data.Analysis

Work Items

• DataFrame - Add KeyType Column Support #6499
• DataFrame - Support loading / saving to SQL #6498
• DataFrame - Drop multiple columns #6497
• DataFrame - Rename columns #6496
• Fix reported Merge operation bugs #6493
• Fix reported Join operation bugs #6494
• Fix reported Group operation bugs #6495

Related Issues

Create DataFrames

Data Formatting

• [DataFrame] can't handle separators in data #5647
• [DataFrame] Allow quote escaping #6238

Data Sources

• DataFrame LoadSql Method #5662
• Add parquet support for importing and exporting data to/from DataFrame. #5972
• DataFrame enhancements #6088
• DataFrame.LoadCsv should support URLs #5905
• Dataframe load data from excel and query using sql #5646

Other

• addIndexColumn in DataFrame.LoadCsv doesn't work #5968

Reshape DataFrames

• Pivot / Transpose
• Create columns
  • DataFrame feature suggestions #5670
• Reset indices
  • [DataFrame][Request] Multi-dimensional Index #5658
• Rename columns
  • Microsoft.Data.Analysis DataFrameColumns.SetName method is weird #6129
  • Changing the name of a column doesn't change the name you use to access the column #5916

Filter / Sort DataFrames

• Filter
  • DataFrameColumn: Unable to Filter by Null via ElementWiseXYZ Methods (System.NotSupportedException) #6110

Combine DataFrames

• Merge
  • DataFrame merge results in column datatype problem #6127
• Join
  • Microsoft.Data.Analysis DataFrame.Join infinite loop #6128
  • Microsoft.Data.Analysis.DataFrame Join should produce a result with a single joined column #6133
• Concat
  • Adding Insert() method to DataColumns to allow prepending data #5932
  • Add ability to concatenate 2 IDataViews. #6134

Group DataFrames

• Grouping
  • Enable multi column group by for Dataframe #5724
  • Improve support for Null values in DataFrame GroupBy #5691
• Aggregate Functions
  • Add Sort to GroupBy in DataFrame  #5681

Summarize DataFrames

• Descriptive statistics
• PrimitiveDataFrameColumnComputations produce wrong result for Min/Max functions #5759

Handle Missing Data

• Drop missing data
• Impute / Fill missing data

DataTypes

Array / Vector / VBuffer

• DataFrame - add support for vbuffer #5872
• DataFrame needs to support vector columns. #5721
• DataFrame should support Array data #5746
• Support "Sparse" and "Dense" vector typed columns in the DataFrame  #5690

DateTime

• Cannot run Transform on DataFrame with DateTime type #6213
• Add missing implementation for datetime relevant arrow types #6201

Other

• Let DataFrame support UInt32 columns to be compatible with MapValueToKey #5898
• Improve error experience when LoadCsv throws #5656

Misc

• LINQ / EFCore?
• DateTime operations