Possible GTFS Loader Bugs

[zackAemmer]

Two possible bugs with loading and embedding GTFS data with GTFS2VEC loader and embedder:

• GTFSLoader uses df.pivot_table() to calculate the hourly embedding features from the static feed, which gives NaN values for hours/stops that don't have trips. For _load_trips this is filled with 0, but for _load_directions it should be an empty set, which as far as I can tell is not possible using df.pivot_table(). I handled this in GTFS2VecEmbedder by filtering NaN values as they are reduced. I also added an initial value because in a few cases there were hours with no trips at all.
• GTFS2VecEmbedder expects a features GeoDataFrame with an index that matches the joint GeoDataFrame, which is checked in _validate_indexes. GTFSLoader assigns the features GeoDataFrame an index of None. I just changed it to assign the FEATURE_ID constant.

from pathlib import Path
from srai.embedders import GTFS2VecEmbedder
from srai.joiners import IntersectionJoiner
from srai.loaders import GTFSLoader, download_file
from srai.neighbourhoods.h3_neighbourhood import H3Neighbourhood
from srai.regionalizers import H3Regionalizer
import geopandas as gpd
from shapely.geometry import Polygon
from srai.constants import WGS84_CRS

# Load GTFS from example notebook
wroclaw_gtfs = Path().resolve() / "files" / "example.zip"
gtfs_url = "https://transitfeeds.com/p/mpk-wroc-aw/663/20221221/download"
download_file(gtfs_url, wroclaw_gtfs.as_posix())
gtfs_loader = GTFSLoader()
features = gtfs_loader.load(wroclaw_gtfs)

print(features.index.name) # None

# Get H3 embedding regions covering the GTFS bounding box, join with features
min_x, min_y = features.geometry.bounds[['minx', 'miny']].min()
max_x, max_y = features.geometry.bounds[['maxx', 'maxy']].max()
geo = Polygon((
    (min_x, min_y),
    (min_x, max_y),
    (max_x, max_y),
    (max_x, min_y),
    (min_x, min_y)
))
area = gpd.GeoDataFrame(
    {'region_id': ['Wroclaw_test'],
     'geometry': [geo]},
     crs=WGS84_CRS
)
area.set_index('region_id', inplace=True)
regionalizer = H3Regionalizer(resolution=8)
joiner = IntersectionJoiner()
regions = regionalizer.transform(area)
neighbourhood = H3Neighbourhood(regions_gdf=regions)
joint = joiner.transform(regions, features)

# Fit embedder
embedder = GTFS2VecEmbedder(hidden_size=2, embedding_size=4)
embedder.fit(regions, features, joint)
embeddings_gtfs = embedder.transform(regions, features, joint)

# ValueError: features_gdf must have a named index.

features.index.name = 'feature_id'
embedder = GTFS2VecEmbedder(hidden_size=2, embedding_size=4)
embedder.fit(regions, features, joint)
embeddings_gtfs = embedder.transform(regions, features, joint)

# TypeError: descriptor 'union' for 'set' objects doesn't apply to a 'float' object

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (0b8e4ea) 88.76% compared to head (7a49246) 88.25%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #427      +/-   ##
==========================================
- Coverage   88.76%   88.25%   -0.52%     
==========================================
  Files          66       66              
  Lines        2928     2928              
==========================================
- Hits         2599     2584      -15     
- Misses        329      344      +15     

Flag	Coverage Δ
macos-latest-python3.11	87.67% <100.00%> (-0.45%)	⬇️
ubuntu-latest-python3.10	87.56% <100.00%> (-0.45%)	⬇️
ubuntu-latest-python3.11	87.63% <100.00%> (-0.38%)	⬇️
ubuntu-latest-python3.9	87.61% <100.00%> (-0.38%)	⬇️
windows-latest-python3.11	87.73% <100.00%> (-0.45%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[piotrgramacki]

Hi @zackAemmer

Thanks a lot for finding and fixing those bugs. It's a great contribution!

Everything looks good to me. I added the CHANGELOG entry and will merge and ship those changes with the next release.

[piotrgramacki]

LGTM!

[zackAemmer]

Awesome!