Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
25c32eb
feat!(core): STAC formatted properties
sbrunato Jul 9, 2025
ec56670
fix: hide private properties
sbrunato Jul 10, 2025
793f396
fix: remove product-type properties from product
sbrunato Jul 10, 2025
4f2410f
feat: sort product properties
sbrunato Jul 10, 2025
94fbaf0
docs: complementary_url_key better explained
sbrunato Jul 10, 2025
ae05461
refactor: STAC properties in stac-providers conf
sbrunato Jul 10, 2025
f85f127
refactor: stac-providers conf shorten
sbrunato Jul 10, 2025
3bce7b4
fix: product-types mapping and _collection usage
sbrunato Jul 10, 2025
b66a073
refactor: typing
sbrunato Jul 17, 2025
4ec2646
refactor: instruments property mapping
sbrunato Jul 17, 2025
302f3e2
refactor: order:status values
sbrunato Jul 17, 2025
2999f46
refactor: productType to collection and collection to _collection/pro…
sbrunato Sep 5, 2025
9399a11
refactor: product types to collection in conf
sbrunato Sep 5, 2025
c839292
refactor: product-types to collection in qssearch
sbrunato Sep 5, 2025
62cdcac
refactor: product-types to collection
sbrunato Sep 5, 2025
0d73a78
refactor: product_type_def_params to collection_def_params
sbrunato Sep 5, 2025
7d1d728
refactor: product-types to collection in buildc_search_results and co…
sbrunato Sep 5, 2025
f564918
refactor: provider_product_type to provider_collection
sbrunato Sep 5, 2025
2051e22
refactor: product-types to collection in csw
sbrunato Sep 5, 2025
ec01c97
refactor: product-types to collection
sbrunato Sep 5, 2025
1b60bff
refactor: product-types to collection in tests
sbrunato Sep 5, 2025
2156879
refactor: *TimeFromAscendingNode replaced with start/end_datetime
sbrunato Sep 5, 2025
a4c781f
refactor: replace *ProductType with *Collection in classes names
sbrunato Sep 5, 2025
20b5cb6
refactor: replace product_type with collection in methods names, para…
sbrunato Sep 5, 2025
575cdb8
docs: replace product type with collection in code
sbrunato Sep 5, 2025
c064523
docs: replace product type with collection in doc and ci
sbrunato Sep 5, 2025
3440bd2
refactor: renamed *product_type* files with *collection*
sbrunato Sep 5, 2025
8530017
refactor: removed DEFAULT_METADATA_MAPPING
sbrunato Sep 10, 2025
be2acbb
feat: prepend unknown properties with provider as stac prefix
sbrunato Sep 11, 2025
37b94d4
feat: convert_not_available in metadata_mapping
sbrunato Sep 11, 2025
624af20
refactor: serialization eodag prefix
sbrunato Sep 11, 2025
85c8ada
refactor: collections conf description & instruments
sbrunato Sep 11, 2025
75d7a42
refactor: collections conf keywords
sbrunato Sep 11, 2025
5cbeb30
refactor: collections conf constellation level platform
sbrunato Sep 11, 2025
08ce297
refactor: collections conf temporal extent
sbrunato Sep 11, 2025
891cc39
refactor: replace productType by collection in tests
sbrunato Sep 11, 2025
7e7fe22
test: prepend unknown properties with provider as stac prefix
sbrunato Sep 11, 2025
42233a0
refactor: GENERIC_PRODUCT_TYPE to GENERIC COLLECTION
sbrunato Sep 12, 2025
2ca3e65
refactor: PRODUCT_TYPE to COLLECTION in constants and env vars
sbrunato Sep 12, 2025
ff6a7f5
refactor: generic_product_type to generic_collection
sbrunato Sep 12, 2025
4780d75
refactor: remaining product_type to collection
sbrunato Sep 12, 2025
b8d675d
test: queryables - check id instead on collection
sbrunato Sep 12, 2025
95416c2
refactor: storageStatus to order:status
sbrunato Sep 12, 2025
af288b4
fix: dates parameters for cop_ providers
sbrunato Sep 15, 2025
923b6e0
fix: alias in properties
sbrunato Sep 15, 2025
bc95bfa
refactor: queryables to stac
sbrunato Sep 15, 2025
7ccf511
fix: restore ecmwf:product_type
sbrunato Sep 15, 2025
e9068ad
fix: undo collection remove at qssearch query beginning
sbrunato Sep 15, 2025
0dbb5bd
fix: aws_eos search
sbrunato Sep 15, 2025
6ef518a
fix: discovered collections key
sbrunato Sep 15, 2025
e335b37
fix: discovered collections keywords
sbrunato Sep 15, 2025
920ff51
test: wekeo product_type fix
sbrunato Sep 15, 2025
7d553cb
refactor: collections mission dates
sbrunato Sep 16, 2025
a78ead4
refactor: collections metadata
sbrunato Sep 16, 2025
9c4967d
refactor: collections metadata dates
sbrunato Sep 16, 2025
ccad5ab
refactor: EODAGSearch
sbrunato Sep 16, 2025
3e428b4
refactor: cli
sbrunato Sep 16, 2025
9c236ef
fix: collections and cli metadata
sbrunato Sep 16, 2025
bc0709d
fix: get_collection_dates import
sbrunato Sep 16, 2025
a4f340e
fix: get_collection_dates
sbrunato Sep 16, 2025
29a326f
refactor: tested dates in json search results
sbrunato Sep 16, 2025
f1e1c5f
refactor: post rebase rename product types
sbrunato Sep 20, 2025
ab58b62
refactor: mundi old conf
sbrunato Sep 20, 2025
1baa513
test: onda and removed props from collections
sbrunato Sep 20, 2025
4dc4583
fix: search plugins tests, wekeo mapping & queryables
sbrunato Sep 21, 2025
258d732
fix: grid-code parsing for earth_search
sbrunato Sep 21, 2025
7e07bbc
fix: EOProduct collection from kwargs or properties
sbrunato Sep 22, 2025
08976c7
fix: imported stac item collection
sbrunato Sep 22, 2025
674af4f
test: tests updates
sbrunato Sep 22, 2025
31a5fad
fix: new product types formatting
sbrunato Sep 22, 2025
df6b429
fix: mtd mapping and sortables
sbrunato Oct 6, 2025
1bbe421
refactor: collections.yml temporal extent
sbrunato Oct 6, 2025
4614644
test: mtd mapping fix and guess collection
sbrunato Oct 6, 2025
d4e96eb
fix: PostJsonSearch and tests
sbrunato Oct 6, 2025
3f29241
fix: collection in queryables
sbrunato Oct 6, 2025
fc9bd72
refactor: post rebase fixes
sbrunato Oct 6, 2025
544f227
refactor: properties as list, creodias and geodes updates
sbrunato Oct 8, 2025
7b8cedd
refactor: providers property
sbrunato Oct 8, 2025
cceaa48
refactor: s1 & s2 extensions in creodias
sbrunato Oct 9, 2025
1d30ee0
refactor: deserialization and ValidationError raise
sbrunato Oct 9, 2025
61ffb61
feat: replace colons with _ in env var conf
sbrunato Oct 10, 2025
58f23a1
refactor: downloadLink to download_link
sbrunato Oct 10, 2025
91d88aa
refactor: usgs metadata
sbrunato Oct 10, 2025
f63b572
refactor: renamed orderLink orderId orderStatusLink searchLink
sbrunato Oct 10, 2025
e24efe4
refactor: renamed combinedOrderId status message
sbrunato Oct 10, 2025
5d87247
refactor: renamed orderStatus request_params percent download_id
sbrunato Oct 10, 2025
9fe850f
refactor: renamed metaDownloadLink
sbrunato Oct 10, 2025
9cb8436
refactor: renamed thumbnail quicklook
sbrunato Oct 10, 2025
ee62318
refactor: renamed sensorType
sbrunato Oct 10, 2025
f66cd5d
refactor: missionStartDate in collection discovery
sbrunato Oct 10, 2025
90b0943
refactor: renamed defaultGeometry
sbrunato Oct 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions .github/workflows/fetch.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Fetch product types
name: Fetch collections

on:
workflow_dispatch:
Expand All @@ -9,7 +9,7 @@ on:

jobs:
fetch-product-types:
name: Fetch providers for new product types
name: Fetch providers for new collections
runs-on: ubuntu-latest
steps:
- name: Checkout the repo
Expand All @@ -28,22 +28,22 @@ jobs:
uv venv
source .venv/bin/activate
uv pip install ".[all-providers]"
- name: Fetch and update external product types reference
- name: Fetch and update external collections reference
env:
EODAG__WEKEO_MAIN__AUTH__CREDENTIALS__USERNAME: ${{ secrets.EODAG__WEKEO_MAIN__AUTH__CREDENTIALS__USERNAME }}
EODAG__WEKEO_MAIN__AUTH__CREDENTIALS__PASSWORD: ${{ secrets.EODAG__WEKEO_MAIN__AUTH__CREDENTIALS__PASSWORD }}
run: |
source .venv/bin/activate
export JSON_OUTPUT_FILE="eodag/resources/ext_product_types.json"
export JSON_REF_FILE=$(python -c "import eodag; print(eodag.config.EXT_PRODUCT_TYPES_CONF_URI)")
export JSON_OUTPUT_FILE="eodag/resources/ext_collections.json"
export JSON_REF_FILE=$(python -c "import eodag; print(eodag.config.EXT_COLLECTIONS_CONF_URI)")
eodag -vvv discover --storage ${JSON_OUTPUT_FILE}
# add empty line at end of file
sed -i -e '$a\' ${JSON_OUTPUT_FILE}
git config user.name "github-actions[bot]"
git config user.email "'github-actions[bot]@users.noreply.github.com"
git add "${JSON_OUTPUT_FILE}"
git commit -m "chore: update external product types reference" || exit 0
echo "Update external product types reference from daily fetch. See [Python API User Guide / Product types discovery](https://eodag.readthedocs.io/en/latest/notebooks/api_user_guide/2_providers_products_available.html#Product-types-discovery)" >> $GITHUB_STEP_SUMMARY
git commit -m "chore: update external collections reference" || exit 0
echo "Update external collections reference from daily fetch. See [Python API User Guide / Collections discovery](https://eodag.readthedocs.io/en/latest/notebooks/api_user_guide/2_providers_products_available.html#Product-types-discovery)" >> $GITHUB_STEP_SUMMARY
echo '### Changed files' >> $GITHUB_STEP_SUMMARY
echo 'May be truncated.' >> $GITHUB_STEP_SUMMARY
COMMIT_SHA=$(git rev-parse HEAD)
Expand All @@ -63,7 +63,7 @@ jobs:
with:
branch: external-product-types-ref-update
delete-branch: true
title: 'chore: update external product types reference'
title: 'chore: update external collections reference'
body: ${{ env.UPDATE_SUMMARY }}
labels: |
automated pr
6 changes: 3 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,9 @@ docs/notebooks/tutos/*workspace*
examples/eodag_workspace
tests/resources/user_conf.yml
search_results.geojson
ext_product_types.json
!eodag/resources/ext_product_types.json
!tests/resources/ext_product_types.json
ext_collections.json
!eodag/resources/ext_collections.json
!tests/resources/ext_collections.json

# IDE
# Pycharm
Expand Down
10 changes: 5 additions & 5 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,9 @@ EODAG (Earth Observation Data Access Gateway) is a command line tool and a plugi
aggregating results and downloading remote sensed images while offering a unified API for data access regardless of the
data provider. The EODAG SDK is structured around three functions:

* List product types: list of supported products and their description
* List collections: list of supported products and their description

* Search products (by product type or uid): searches products according to the search criteria provided
* Search products (by collection or uid): searches products according to the search criteria provided

* Download products: download product “as is"

Expand Down Expand Up @@ -139,7 +139,7 @@ Start playing with the CLI:

eodag search --productType S2_MSI_L1C --box 1 43 2 44 --start 2021-03-01 --end 2021-03-31

The request above searches for ``S2_MSI_L1C`` product types in a given bounding box, in March 2021. It saves the results in a GeoJSON file (``search_results.geojson`` by default).
The request above searches for ``S2_MSI_L1C`` collections in a given bounding box, in March 2021. It saves the results in a GeoJSON file (``search_results.geojson`` by default).

Results are paginated, you may want to get all pages at once with ``--all``, or search products having 20% of maximum coud cover with ``--cloudCover 20``. For more information on available options::

Expand All @@ -153,11 +153,11 @@ Start playing with the CLI:

eodag download --quicklooks --search-results search_results.geojson

- To list all available product types and supported providers::
- To list all available collections and supported providers::

eodag list

- To list available product types on a specified supported provider::
- To list available collections on a specified supported provider::

eodag list -p creodias

Expand Down
2 changes: 1 addition & 1 deletion docs/_static/params_mapping_offline_infos.json

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions docs/add_product_type.rst → docs/add_collection.rst
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
.. _add_product_type:
.. _add_collection:

Add a product type
==================
Add a collection
================

A product type in `eodag` represents a specific kind of Earth Observation data product, defined by its characteristics and metadata.
A collection in `eodag` represents a specific kind of Earth Observation data product, defined by its characteristics and metadata.
By following the steps outlined below, you can extend EODAG's capabilities to support additional product types from various providers.

Add product type definition
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Add collection definition
^^^^^^^^^^^^^^^^^^^^^^^^^

To add a new product type to EODAG, you must add an entry to the file ``eodag/resources/product_types.yml``. Here is an
To add a new collection to EODAG, you must add an entry to the file ``eodag/resources/collections.yml``. Here is an
example:

.. code-block:: yaml
Expand All @@ -34,12 +34,12 @@ and underscores to separate words. This name will be used when searching for
products of the corresponding type.

The following lines need to be indented because they make a dictionary of
configuration information for the product type we are defining. Each bit of
configuration information for the collection we are defining. Each bit of
information can usually be found on the provider's catalog. Note how the value
used for the ``keywords`` entry brings together values from other entries such
as ``instrument``, ``processingLevel``, ``platform``, etc.

Add product type to a provider
Add collection to a provider
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In the file ``eodag/resources/providers.yml``, add the product-type to the ``products``
Expand All @@ -60,7 +60,7 @@ entry of a provider:

Then for each product-type listed under the ``products`` entry, you may
specify default parameters that will be used when searching for products of this
product type:
collection:

.. code-block:: yaml

Expand All @@ -82,7 +82,7 @@ Each of those parameters can be overridden when performing an actual search. Not
parameters have to be named following the common model used in EODAG (see
`Parameters mapping <params_mapping.rst>`_). Part of the provider search metadata
mapping can also be overridden per product-type, by adding a ``metadata_mapping``
section to the product type definition:
section to the collection definition:

.. code-block:: yaml

Expand All @@ -99,7 +99,7 @@ section to the product type definition:
previewBaseName: '{$.sceneID#replace_str("_L4","")}'
thumbnail: 'https://s3.amazonaws.com/cbers-meta-pds/{awsPath}/{previewBaseName}_small.jpeg'

In the example above, we can see that the metadata mapping for the product type
In the example above, we can see that the metadata mapping for the collection
in the context of this provider can be specified in two ways:

- ``metadata_mapping_from_product`` will include an existing metadata mapping
Expand Down
6 changes: 3 additions & 3 deletions docs/add_provider.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ provide the new provider's configuration in a ``YAML`` format. The following exa
products:
S2_MSI_L1C:
productType: sentinel-2-l1c
GENERIC_PRODUCT_TYPE:
GENERIC_COLLECTION:
productType: '{productType}'
download:
type: AwsDownload
Expand Down Expand Up @@ -97,13 +97,13 @@ The plugin structure is reflected in the internal providers configuration file.
provider_name:
priority: 1
products:
# List of supported product types
# List of supported collections
# This is a mapping containing all the information required by the search plugin class to perform its job.
# The mapping is available in the config attribute of the search plugin as config['products']
S2_MSI_L1C:
a-config-key-needed-by-search-plugin-to-search-this-product-type: value
another-config-key: another-value
# Whether this product type is partially supported by this provider (the provider does not contain all the
# Whether this collection is partially supported by this provider (the provider does not contain all the
# products of this type)
partial: True
...
Expand Down
18 changes: 9 additions & 9 deletions docs/api_reference/core.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,18 +24,18 @@ Configuration
EODataAccessGateway.set_preferred_provider
EODataAccessGateway.get_preferred_provider
EODataAccessGateway.update_providers_config
EODataAccessGateway.update_product_types_list
EODataAccessGateway.update_collections_list

Catalog
-------

.. autosummary::

EODataAccessGateway.available_providers
EODataAccessGateway.list_product_types
EODataAccessGateway.guess_product_type
EODataAccessGateway.fetch_product_types_list
EODataAccessGateway.discover_product_types
EODataAccessGateway.list_collections
EODataAccessGateway.guess_collection
EODataAccessGateway.fetch_collections_list
EODataAccessGateway.discover_collections

Search
------
Expand Down Expand Up @@ -78,14 +78,14 @@ Misc
.. autosummary::

EODataAccessGateway.group_by_extent
EODataAccessGateway.guess_product_type
EODataAccessGateway.guess_collection
EODataAccessGateway.list_queryables
EODataAccessGateway.available_sortables
EODataAccessGateway.import_stac_items

.. autoclass:: eodag.api.core.EODataAccessGateway
:members: add_provider, set_preferred_provider, get_preferred_provider, update_providers_config, list_product_types,
:members: add_provider, set_preferred_provider, get_preferred_provider, update_providers_config, list_collections,
available_providers, search, search_all, search_iter_page, crunch, download, download_all, serialize,
deserialize, deserialize_and_register, group_by_extent, guess_product_type, get_cruncher,
update_product_types_list, fetch_product_types_list, discover_product_types, list_queryables,
deserialize, deserialize_and_register, group_by_extent, guess_collection, get_cruncher,
update_collections_list, fetch_collections_list, discover_collections, list_queryables,
available_sortables, import_stac_items
2 changes: 1 addition & 1 deletion docs/api_reference/eoproduct.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Constructor

.. autoattribute:: provider
.. autoattribute:: properties
.. autoattribute:: product_type
.. autoattribute:: collection
.. autoattribute:: geometry
.. autoattribute:: search_intersection
.. autoattribute:: location
Expand Down
30 changes: 15 additions & 15 deletions docs/cli_user_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Search
--all \
--storage my_search.geojson

The request above searches for `S2_MSI_L1C` product types in a given bounding box, in January 2018. The command fetches
The request above searches for `S2_MSI_L1C` collections in a given bounding box, in January 2018. The command fetches
internally all the products that match these criteria. Without ``--all``, it would only fetch the products found on the
first result page. It finally saves the results in a GeoJSON file.

Expand All @@ -69,7 +69,7 @@ string search sent to the provider. For instance, if you want to add foo=1 and b
--cruncher-args FilterOverlap minimum_overlap 10 \
--query "foo=1&bar=2"

* If the product type is not known, it can also be guessed by EODAG during the search based on parameters in the search
* If the collection is not known, it can also be guessed by EODAG during the search based on parameters in the search
request. The possible parameters are:

- `instrument` (e.g. MSI)
Expand All @@ -79,9 +79,9 @@ string search sent to the provider. For instance, if you want to add foo=1 and b
- `sensorType` (e.g. OPTICAL)
- `keywords` (e.g. SENTINEL2 L1C SAFE), which is case insensitive and ignores `-` or `_` characters

For example, the following search request will first search for a product type for platform SENTINEL2 and
processingLevel L1 (there are several product types matching these criteria, e.g., `S2_MSI_L1C`) and then use this
product type to execute the actual search.
For example, the following search request will first search for a collection for platform SENTINEL2 and
processingLevel L1 (there are several collections matching these criteria, e.g., `S2_MSI_L1C`) and then use this
collection to execute the actual search.

.. code-block:: console

Expand Down Expand Up @@ -114,32 +114,32 @@ Download

eodag search --productType S2_MSI_L1C --bbox 1 43 2 44 --start 2025-03-01 download

Product Types
Collections
-------------

.. command-output:: eodag list --help

* To list all available product types and supported providers:
* To list all available collections and supported providers:

.. code-block:: console

eodag list

* To list available product types on a specified supported provider:
* To list available collections on a specified supported provider:

.. code-block:: console

eodag list -p creodias

* By default, ``list`` command will also fetch for new product types, which may be slow depending on the network status.
* By default, ``list`` command will also fetch for new collections, which may be slow depending on the network status.
To skip fetching, use the following option:

.. code-block:: console

eodag list --no-fetch

* EODAG can fetch providers (all or only a given one) to discover available product types, using the following command.
It will store result in a JSON file (defaults to `ext_product_types.json`):
* EODAG can fetch providers (all or only a given one) to discover available collections, using the following command.
It will store result in a JSON file (defaults to `ext_collections.json`):

.. command-output:: eodag discover --help

Expand All @@ -149,10 +149,10 @@ Examples:

eodag discover
eodag discover -p planetary_computer
eodag discover -p planetary_computer --storage my_product_types_conf.json
eodag discover -p planetary_computer --storage my_collections_conf.json

This file can then be used in EODAG using the environment variable ``EODAG_EXT_PRODUCT_TYPES_CFG_FILE``.
This file can then be used in EODAG using the environment variable ``EODAG_EXT_COLLECTIONS_CFG_FILE``.

Please note that if you did not customize EODAG with new providers settings, this command should not be useful.
For more information on the product types discovery mechanism, please see
`Python API User Guide / Providers and products / Product types discovery <notebooks/api_user_guide/2_providers_products_available.html#Product-types-discovery>`_.
For more information on the collections discovery mechanism, please see
`Python API User Guide / Providers and products / Collections discovery <notebooks/api_user_guide/2_providers_products_available.html#Collections-discovery>`_.
Loading
Loading