Python: Finish filesystem bindings

[wjones127]

Description

The PyArrow filesystem bindings are only partially implemented: https://github.com/delta-io/delta-rs/blob/main/python/deltalake/fs.py

Use Case

We will need the rest of the methods implemented for the PyArrow writer (write_deltalake()) to work.

Related Issue(s)

Blocks: #542

[wjones127]

In addition, it probably makes sense to also add in support for Windows and "older" Linux. See

delta-rs/rust/src/storage/file/rename.rs

Lines 96 to 114 in d54e952

	unsafe fn platform_specific_rename(from: *const libc::c_char, to: *const libc::c_char) -> i32 {
	cfg_if::cfg_if! {
	if #[cfg(all(target_os = "linux", target_env = "gnu"))] {
	cfg_if::cfg_if! {
	if #[cfg(glibc_renameat2)] {
	libc::renameat2(libc::AT_FDCWD, from, libc::AT_FDCWD, to, libc::RENAME_NOREPLACE)
	} else {
	// target has old glibc (< 2.28), we would need to invoke syscall manually
	unimplemented!()
	}
	}
	} else if #[cfg(target_os = "macos")] {
	libc::renamex_np(from, to, libc::RENAME_EXCL)
	} else {
	unimplemented!()
	}
	}
	}
	}

[wjones127]

I am working on this one.

[wjones127]

FYI I am working on this, but will take a while. The route I am taking is working upstream to get the ObjectStore trait to a place where it can be wrapped as a PyArrow filesystem. Then we can use that in the dataset writer as well as in the delta log writer. See:

• ObjectStore write support apache/datafusion#2185
• Add write and delete operations to ObjectStore apache/datafusion#2246
• Implement write and delete for S3FileSystem datafusion-contrib/datafusion-objectstore-s3#54

[wjones127]

Update: current plan is to use object-store-rs (soon to be moved into arrow-rs) and then wrap that as a PyArrow filesystem for the writer.

So blocked on #610 for now.

[wjones127]

We can now pass ObjectStores into PyArrow writer. But we need to be able to provide PyArrow / fsspec filesystems into DeltaTable.

fs = S3Filesystem()
dt = DeltaTable("s3://path/to/table", filesystem=fs)
dt.to_pyarrow_table() # Should use fs
write_deltalake(dt, pa.table({'x': [1,2,3]})) # should use fs
write_deltalake("s3://path/to/table", pa.table({'x': [1,2,3]}), filesystem=fs)  # should use fs