Support pseudo columns #1203
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Haven't see that anywhere in the code, so please let me know if I overlooked it.
Spark supports pseudo columns, a pseudo column is a column that has a single value for an entire partition, and can be used to filter data without reading any physical files.
Describe the solution you'd like
reading basepath which contains two directories foo=a and foo=b gives me a dataframe with the utf8 column foo merged into the schema, and a partition for each value.
If I filter on that column, for instance select * from basepathtable where foo = "a" only scans the files in that directory.
Describe alternatives you've considered
Currently AFAIK, datafusion requires you to know the full list of partition directories and read from each individually.
Additional context
Example pyspark code where basepath contains directories like dt={*}
df = self.spark.read.option("basePath", basepath).format("parquet").load(path)
filtered = df.filter((col('dt') >= start) & (col('dt') <= end))