Skip to content

GeoDataFrame init #909

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ import org.jetbrains.kotlinx.jupyter.api.libraries.JupyterIntegration
import org.jetbrains.kotlinx.jupyter.api.libraries.resources
import kotlin.reflect.KClass
import kotlin.reflect.KProperty
import kotlin.reflect.KType
import kotlin.reflect.full.isSubtypeOf

/** Users will get an error if their Kotlin Jupyter kernel is older than this version. */
Expand All @@ -70,29 +69,6 @@ internal class Integration(private val notebook: Notebook, private val options:

val version = options["v"]

private fun KotlinKernelHost.execute(codeWithConverter: CodeWithConverter, argument: String): VariableName? {
val code = codeWithConverter.with(argument)
return if (code.isNotBlank()) {
val result = execute(code)
if (codeWithConverter.hasConverter) {
result.name
} else {
null
}
} else {
null
}
}

private fun KotlinKernelHost.execute(
codeWithConverter: CodeWithConverter,
property: KProperty<*>,
type: KType,
): VariableName? {
val variableName = "(${property.name}${if (property.returnType.isMarkedNullable) "!!" else ""} as $type)"
return execute(codeWithConverter, variableName)
}

private fun KotlinKernelHost.updateImportDataSchemaVariable(
importDataSchema: ImportDataSchema,
property: KProperty<*>,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
package org.jetbrains.kotlinx.dataframe.jupyter

import org.jetbrains.kotlinx.dataframe.codeGen.CodeWithConverter
import org.jetbrains.kotlinx.jupyter.api.KotlinKernelHost
import org.jetbrains.kotlinx.jupyter.api.VariableName
import kotlin.reflect.KProperty
import kotlin.reflect.KType

internal fun KotlinKernelHost.execute(codeWithConverter: CodeWithConverter, argument: String): VariableName? {
val code = codeWithConverter.with(argument)
return if (code.isNotBlank()) {
val result = execute(code)
if (codeWithConverter.hasConverter) {
result.name
} else {
null
}
} else {
null
}
}

internal fun KotlinKernelHost.execute(
codeWithConverter: CodeWithConverter,
property: KProperty<*>,
type: KType,
): VariableName? {
val variableName = "(${property.name}${if (property.returnType.isMarkedNullable) "!!" else ""} as $type)"
return execute(codeWithConverter, variableName)
}
6 changes: 6 additions & 0 deletions dataframe-geo/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
## :dataframe-geo

This module, published as `dataframe-geo`, contains all logic and tests for DataFrame to be able to work
with geographical data.

Experimental.
71 changes: 71 additions & 0 deletions dataframe-geo/build.gradle.kts
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
import org.jetbrains.kotlin.gradle.tasks.BaseKotlinCompile
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

plugins {
with(libs.plugins) {
alias(kotlin.jvm)
alias(publisher)
alias(jupyter.api)
alias(ktlint)
alias(dataframe)
alias(ksp)
}
}

group = "org.jetbrains.kotlinx"

repositories {
// geo repository should come before Maven Central
maven("https://repo.osgeo.org/repository/release")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it true, that these maven repo links will be hidden in the module, and the final user don't need to refer to them somehow in its Gradle script? (just for self checking)

Copy link
Collaborator Author

@AndreiKingsley AndreiKingsley Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, the user will need to add the repository. However, I believe we can fix this with shadow jar.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please check whether you can do that before this is merged

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd leave it that way for now, then I'll try to get rid of it. When using a descriptor in notebook, the user won't need to do this.

mavenCentral()
}

// https://stackoverflow.com/questions/26993105/i-get-an-error-downloading-javax-media-jai-core1-1-3-from-maven-central
// jai core dependency should be excluded from geotools dependencies and added separately
fun ExternalModuleDependency.excludeJaiCore() = exclude("javax.media", "jai_core")

dependencies {
api(project(":core"))

implementation(libs.geotools.main) { excludeJaiCore() }
implementation(libs.geotools.shapefile) { excludeJaiCore() }
implementation(libs.geotools.geojson) { excludeJaiCore() }
implementation(libs.geotools.referencing) { excludeJaiCore() }
implementation(libs.geotools.epsg.hsql) { excludeJaiCore() }

implementation(libs.jai.core)

implementation(libs.jts.core)
implementation(libs.jts.io.common)

implementation(libs.ktor.client.core)
implementation(libs.ktor.client.cio)
implementation(libs.ktor.client.content.negotiation)
implementation(libs.ktor.serialization.kotlinx.json)

testImplementation(kotlin("test"))
}

tasks.withType<KotlinCompile>().configureEach {
val friendModule = project(":core")
val jarTask = friendModule.tasks.getByName("jar") as Jar
val jarPath = jarTask.archiveFile.get().asFile.absolutePath
(this as BaseKotlinCompile).friendPaths.from(jarPath)
}

kotlinPublications {
publication {
publicationName = "dataframeGeo"
artifactId = "dataframe-geo"
description = "GeoDataFrame API"
packageName = artifactId
}
}

tasks.processJupyterApiResources {
libraryProducers = listOf("org.jetbrains.kotlinx.dataframe.jupyter.IntegrationGeo")
}

tasks.test {
useJUnitPlatform()
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
package org.jetbrains.kotlinx.dataframe.geo

import org.geotools.api.referencing.crs.CoordinateReferenceSystem
import org.geotools.geometry.jts.JTS
import org.geotools.referencing.CRS
import org.jetbrains.kotlinx.dataframe.DataFrame
import org.jetbrains.kotlinx.dataframe.api.update
import org.jetbrains.kotlinx.dataframe.api.with

/**
* A data structure representing a geographical DataFrame, combining spatial data with
* an optional Coordinate Reference System (CRS).
*
* @param T The type parameter extending `WithGeometry`, indicating the presence of a geometry column.
* @property df The underlying `DataFrame` containing geometries.
* @property crs The coordinate reference system associated with the data, if any.
*/
class GeoDataFrame<T : WithGeometry>(val df: DataFrame<T>, val crs: CoordinateReferenceSystem?) {
/**
* Creates a new `GeoDataFrame` with the modified underlying DataFrame.
*
* @param block The block defining the transformations to be applied to the DataFrame.
* @return A new `GeoDataFrame` instance with updated dataframe and the same CRS.
*/
inline fun modify(block: DataFrame<T>.() -> DataFrame<T>): GeoDataFrame<T> = GeoDataFrame(df.block(), crs)

/**
* Transforms the geometries to a specified Coordinate Reference System (CRS).
*
* This function reprojects the geometry data from the current CRS to a target CRS.
* If no target CRS is specified and the `GeoDataFrame` has no CRS, WGS 84 is used by default.
*
* @param targetCrs The target CRS for transformation.
* @return A new `GeoDataFrame` with reprojected geometries and the specified CRS.
*/
fun applyCrs(targetCrs: CoordinateReferenceSystem): GeoDataFrame<T> {
if (crs == null) {
return GeoDataFrame(df, targetCrs)
}
if (targetCrs == this.crs) return this
// Use WGS 84 by default TODO
val sourceCRS: CoordinateReferenceSystem = this.crs
val transform = CRS.findMathTransform(sourceCRS, targetCrs, true)
return GeoDataFrame(
df.update { geometry }.with { JTS.transform(it, transform) },
targetCrs,
)
}

override fun equals(other: Any?): Boolean {
if (this === other) return true
if (javaClass != other?.javaClass) return false

other as GeoDataFrame<*>

if (df != other.df) return false

return when {
crs == null && other.crs == null -> true
crs == null || other.crs == null -> false
else -> CRS.equalsIgnoreMetadata(crs, other.crs)
}
}

override fun hashCode(): Int {
var result = df.hashCode()
result = 31 * result + (crs?.hashCode() ?: 0)
return result
}

override fun toString(): String = "GeoDataFrame(df=$df, crs=$crs)"

companion object {
val DEFAULT_CRS = CRS.decode("EPSG:4326", true)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be ENUM of different constants

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it even exists in some form in GeoTools (see DefaultEngineeringCRS). But here is a more global question - whether it is necessary to use geotools API directly or should we wrap it completely.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest not wrap for now, we could not avoid the learning of that

}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
package org.jetbrains.kotlinx.dataframe.geo

import org.jetbrains.kotlinx.dataframe.annotations.DataSchema
import org.locationtech.jts.geom.Geometry
import org.locationtech.jts.geom.LineString
import org.locationtech.jts.geom.MultiLineString
import org.locationtech.jts.geom.MultiPoint
import org.locationtech.jts.geom.MultiPolygon
import org.locationtech.jts.geom.Point
import org.locationtech.jts.geom.Polygon

@DataSchema
interface WithGeometry {
val geometry: Geometry
}

@DataSchema
interface WithPolygonGeometry : WithGeometry {
override val geometry: Polygon
}

@DataSchema
interface WithMultiPolygonGeometry : WithGeometry {
override val geometry: MultiPolygon
}

@DataSchema
interface WithPointGeometry : WithGeometry {
override val geometry: Point
}

@DataSchema
interface WithMultiPointGeometry : WithGeometry {
override val geometry: MultiPoint
}

@DataSchema
interface WithLineStringGeometry : WithGeometry {
override val geometry: LineString
}

@DataSchema
interface WithMultiLineStringGeometry : WithGeometry {
override val geometry: MultiLineString
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package org.jetbrains.kotlinx.dataframe.geo

import org.geotools.geometry.jts.ReferencedEnvelope
import org.jetbrains.kotlinx.dataframe.api.asIterable
import org.jetbrains.kotlinx.dataframe.geo.jts.computeBounds

/**
* Computes the bounding envelope for all geometries in a `GeoDataFrame`,
* considering the specified coordinate reference system (CRS).
*
* @receiver The `GeoDataFrame` containing the geometries for which to compute bounds.
* @return The bounding envelope that includes all geometries,
* associated with the CRS of the `GeoDataFrame`.
*/
fun GeoDataFrame<*>.bounds(): ReferencedEnvelope = ReferencedEnvelope(df.geometry.asIterable().computeBounds(), crs)
Loading