[RFC] Integrating Looker projects with Dagster

[rexledesma]

Summary

As part of our dagster==1.8.0 release on August 8, 2024, we’re releasing dagster-looker==0.24.0 to support creating Looker views, explores, and dashboards as external assets.

We would like to use this document to gather feedback and answer questions about the new APIs. If you have any comments or use cases that are not addressed, please participate in the discussion!

The main interface we are releasing as part of this release is:

• build_looker_asset_specs, a function for users to define Dagster asset specs from their Looker project.
• DagsterLookerTranslator, a class for customizing the metadata of your Dagster asset specs.

Requirements

• To use the experimental APIs, you must use dagster-looker>=0.24.0, which is scheduled to be released on August 8, 2024.
• To integrate with dagster-looker, your Looker project must be represented in LookML and backed by git.

Use cases

• How do I represent my Looker views, explores, and dashboards as assets in Dagster?
• How do I configure Dagster metadata on my Looker views, explores and dashboards?
• How do I connect my Looker asset graph to my broader Dagster asset graph?

Examples

In all examples, we use the Looker marketing_demo project as a sample project to represent Looker views, explores, and dashboards.

We also use the following folder structure locally:

├── dagster_with_looker
│   ├── dagster_with_looker
│   │       ├── __init__.py
│   │       ├── assets_for_looker.py
│   │       ├── definitions.py
│   ├── pyproject.toml
│   ├── setup.py
├── marketing_demo

Getting started

Here, we load the Looker project as a set of Dagster assets using build_looker_asset_specs.

# assets_for_looker.py

from pathlib import Path

from dagster_looker import build_looker_asset_specs

project_dir = Path(__file__).joinpath("..", "..", "..", "marketing_demo").resolve()

marketing_demo_looker_assets = build_looker_asset_specs(project_dir=project_dir)

# definitions.py

from dagster import Definitions

from .assets_for_looker import marketing_demo_looker_assets

defs = Definitions(assets=[*marketing_demo_looker_assets])

After running dagster dev, we can visualize the Dagster assets.

Customizing the metadata

Here, we customize the metadata of the looker assets we just loaded. To do this, we subclass DagsterLookerTranslator and implement the method of the metadata we want to customize. In this example, we set the Looker structure type (e.g. view, explore, dashboard) as the asset’s group name.

# assets_for_looker.py

from pathlib import Path
from typing import Any, Mapping, Optional, Tuple

from dagster_looker import (
    DagsterLookerTranslator,
    LookMLStructureType,
    build_looker_asset_specs,
)

project_dir = Path(__file__).joinpath("..", "..", "..", "marketing_demo").resolve()

class CustomDagsterLookerTranslator(DagsterLookerTranslator):
    def get_group_name(
        self,
        lookml_structure: Tuple[Path, LookMLStructureType, Mapping[str, Any]],
    ) -> Optional[str]:
        _, lookml_structure_type, _ = lookml_structure

        return lookml_structure_type

marketing_demo_looker_assets = build_looker_asset_specs(
    project_dir=project_dir,
    dagster_looker_translator=CustomDagsterLookerTranslator(),
)

Next steps

Again, we want to use this document to gather feedback and answer questions about the new APIs. If you have any comments or use cases that are not addressed, please participate in the discussion!