Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scene.save_datasets with GeoTIFF writes valid (but black) output file despite compute=False — is this expected? #2267

Open
gerritholl opened this issue Nov 10, 2022 · 1 comment
Open

Scene.save_datasets with GeoTIFF writes valid (but black) output file despite compute=False — is this expected? #2267

gerritholl opened this issue Nov 10, 2022 · 1 comment
Labels
component:writers question

Comments

@gerritholl
Copy link
Collaborator

I'm not sure if this is a bug or if this behaviour is expected.

Describe the bug

When I call Scene.save_datasets(..., compute=False) with the GeoTIFF writer, Satpy still writes a valid output file.

To Reproduce

# Your code here
from satpy import Scene
from satpy.utils import debug_on; debug_on()

f = "/media/nas/x21308/scratch/AVHRR/Tharthar/AVHR_xxx_1B_M02_20210421061303Z_20210421061603Z_N_O_20210421074726Z"

sc = Scene(filenames=[f], reader=["avhrr_l1b_eps"])
sc.load(["3a"])
ls = sc.resample("iraq")
ls.save_datasets(filename="/tmp/test.tif", compute=False)

Expected behavior

I expect no valid output file, because I passed compute=False.

Actual results

Satpy produces an output file /tmp/test.tif:

/data/gholl/checkouts/satpy/satpy/_config.py:125: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for entry_point in entry_points().get(name, []):
[DEBUG: 2022-11-10 09:59:28 : satpy.readers.yaml_reader] Reading ('/data/gholl/checkouts/satpy/satpy/etc/readers/avhrr_l1b_eps.yaml',)
[DEBUG: 2022-11-10 09:59:28 : satpy.readers.yaml_reader] Assigning to avhrr_l1b_eps: ['/media/nas/x21308/scratch/AVHRR/Tharthar/AVHR_xxx_1B_M02_20210421061303Z_20210421061603Z_N_O_20210421074726Z']
[DEBUG: 2022-11-10 09:59:28 : satpy.composites.config_loader] Looking for composites config file avhrr-3.yaml
[DEBUG: 2022-11-10 09:59:28 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2022-11-10 09:59:28 : satpy.readers.eps_l1b] Reading /media/nas/x21308/scratch/AVHRR/Tharthar/AVHR_xxx_1B_M02_20210421061303Z_20210421061603Z_N_O_20210421074726Z
[DEBUG: 2022-11-10 09:59:29 : satpy.readers.yaml_reader] No coordinates found for DataID(name='longitude', resolution=1050, modifiers=())
[DEBUG: 2022-11-10 09:59:29 : satpy.readers.yaml_reader] No coordinates found for DataID(name='latitude', resolution=1050, modifiers=())
[DEBUG: 2022-11-10 09:59:29 : satpy.scene] Resampling DataID(name='3a', wavelength=WavelengthRange(min=1.58, central=1.61, max=1.64, unit='µm'), resolution=1050, calibration=<calibration.reflectance>, modifiers=())
[INFO: 2022-11-10 09:59:29 : satpy.scene] Not reducing data before resampling.
[INFO: 2022-11-10 09:59:29 : satpy.resample] Using default KDTree resampler
[DEBUG: 2022-11-10 09:59:29 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2022-11-10 09:59:29 : satpy.resample] Resampling truediv-8a8421c38b07fc3f9b63f3e8f281a132
[DEBUG: 2022-11-10 09:59:29 : satpy.writers] Reading ['/data/gholl/checkouts/satpy/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2022-11-10 09:59:29 : rasterio.session] Could not import boto3, continuing with reduced functionality.
/data/gholl/checkouts/satpy/satpy/_config.py:125: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for entry_point in entry_points().get(name, []):
[DEBUG: 2022-11-10 09:59:29 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/generic.yaml
[DEBUG: 2022-11-10 09:59:29 : satpy.writers] Adding enhancement configuration from file: /home/gholl/checkouts/pytroll-dwd-config/enhancements/generic.yaml
/data/gholl/checkouts/satpy/satpy/_config.py:125: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for entry_point in entry_points().get(name, []):
[DEBUG: 2022-11-10 09:59:29 : satpy.writers] Data for DataID(name='3a', wavelength=WavelengthRange(min=1.58, central=1.61, max=1.64, unit='µm'), resolution=1050, calibration=<calibration.reflectance>, modifiers=()) will be enhanced with options:
        [{'name': 'linear_stretch', 'method': <function stretch at 0x7f0087deb520>, 'kwargs': {'stretch': 'crude', 'min_stretch': 0.0, 'max_stretch': 100.0}}, {'name': 'gamma', 'method': <function gamma at 0x7f0087fd0790>, 'kwargs': {'gamma': 1.5}}]
[DEBUG: 2022-11-10 09:59:29 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': 0.0, 'max_stretch': 100.0}
[DEBUG: 2022-11-10 09:59:29 : trollimage.xrimage] Applying gamma 1.5
[DEBUG: 2022-11-10 09:59:29 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f0087ddbe80>
[DEBUG: 2022-11-10 09:59:29 : rasterio.env] Starting outermost env
[DEBUG: 2022-11-10 09:59:29 : rasterio.env] No GDAL environment exists
[DEBUG: 2022-11-10 09:59:29 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f0087dda7a0> created
[DEBUG: 2022-11-10 09:59:29 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2022-11-10 09:59:29 : rasterio._env] PROJ_LIB found in environment.
[DEBUG: 2022-11-10 09:59:29 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x7f0087dda7a0>.
[DEBUG: 2022-11-10 09:59:29 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f0087ddbe80>
[DEBUG: 2022-11-10 09:59:29 : rasterio._io] Path: UnparsedPath(path='/tmp/test.tif'), mode: w, driver: GTiff
[DEBUG: 2022-11-10 09:59:29 : rasterio._io] Option: ('COMPRESS', b'DEFLATE')
[DEBUG: 2022-11-10 09:59:29 : rasterio._io] Option: ('ZLEVEL', b'6')
[DEBUG: 2022-11-10 09:59:29 : rasterio._io] Option: ('TILED', b'TRUE')
[DEBUG: 2022-11-10 09:59:29 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2022-11-10 09:59:29 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000
[DEBUG: 2022-11-10 09:59:29 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f0087ddbe80>
[DEBUG: 2022-11-10 09:59:29 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f0087dda7a0> options
[DEBUG: 2022-11-10 09:59:29 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f0087dda7a0>.
[DEBUG: 2022-11-10 09:59:29 : rasterio.env] Exiting outermost env
[DEBUG: 2022-11-10 09:59:29 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f0087ddbe80>

$ ls -l /tmp/test.tif
-rw-rw-r-- 1 gholl o16217 3.8K Nov 10 09:59 /tmp/test.tif
$ gdalinfo /tmp/test.tif
Driver: GTiff/GeoTIFF
Files: /tmp/test.tif
Size is 1143, 955
Coordinate System is:
PROJCRS["unknown",
    BASEGEOGCRS["unknown",
        DATUM["Unknown_based_on_WGS84_ellipsoid",
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1],
                ID["EPSG",7030]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]],
    CONVERSION["Equidistant Cylindrical",
        METHOD["Equidistant Cylindrical",
            ID["EPSG",1028]],
        PARAMETER["Latitude of 1st standard parallel",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting",east,
            ORDER[1],
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]],
        AXIS["northing",north,
            ORDER[2],
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]]
Data axis to CRS axis mapping: 1,2
Origin = (4309326.187589963898063,4161295.542300541885197)
Pixel Size = (999.630905511811079,-999.570517015706855)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_DATETIME=2021:04:21 06:13:03
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  ( 4309326.188, 4161295.542) ( 38d42'40.81"E, 37d22'53.59"N)
Lower Left  ( 4309326.188, 3206705.699) ( 38d42'40.81"E, 28d48'22.78"N)
Upper Right ( 5451904.313, 4161295.542) ( 48d58'31.04"E, 37d22'53.59"N)
Lower Right ( 5451904.313, 3206705.699) ( 48d58'31.04"E, 28d48'22.78"N)
Center      ( 4880615.250, 3684000.620) ( 43d50'35.93"E, 33d 5'38.19"N)
Band 1 Block=256x256 Type=Byte, ColorInterp=Gray
  Mask Flags: PER_DATASET ALPHA
Band 2 Block=256x256 Type=Byte, ColorInterp=Alpha

Screenshots

The resulting image is black:

test

Environment Info:

  • OS: openSUSE 15.3
  • Satpy Version: v0.37.1-201-g05419d70

Additional context

When I write use a different writer, such as simple_image or cf, nothing gets written. When I pass the results of save_dataset to compute_writer_results, a colourful image is written (as expected).

Maybe this is all expected, but I was surprised, because I thought that with compute=False, everything is postponed until I call compute_writer_results.
@djhoese
Copy link
Member

djhoese commented Nov 10, 2022

With the current implementation, this is expected. This could change when we switch to rioxarray. As a short explanation, we are limited by what geotiff/GDAL/rasterio is able to do. To get around some of these limitations we currently create a file with the basic metadata during call time, but don't write image data until dask computation time where we are able to write chunks in parallel as long as we use a lock.

Most other writers don't have this limitation as they either support multiple writers (ex. NetCDF) or they are being written from one single worker/task (PNG via PIL).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component:writers question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gerritholl @djhoese