Auto-Import Dashboards

[mccahan]

Hello! I've had my PW3 for a week, found this pretty quick, and love the project.

My pitch for a slightly easier onboarding process, asking the user whether they would like to import the dashboards into Grafana automatically, and whether they would like to set a dashboard to show when they first log in. Coupled with my other PR for anonymous access (#583), you can effectively skip through the manual "import a JSON" file process, and after setup just go to http://localhost:9000 and have a dashboard ready and waiting for you.

Here they are in the Grafana dashboard list:

Notes:
import-dashboards.sh has a little bit of search-replace nonsense happening

• I opted to do some search-replacing on the files you have to clean them up for the auto-import so that I didn't modify them for whatever process you've figured out for the manual imports
• To handle the auto-provisioned data sources, you just pass the name of the source instead of the JSON object with the type and the variable
• There's a bug in Grafana where it will show a "Invalid dashboard UID in the request" error if you don't disable the built-in Grafana annotation query
• Swapping out the variables driven from the environment with some values (these are still configurable in the same place inside Grafana)
• I was having an annoying time getting sed to work right on macOS with all the variable interpolation, so I ended up with Perl here but that might not work on Windows?

I chose to be a bit opinionated and make importing the dashboards the default, since they go into a separate "Imported" folder in Grafana anyway and I think it might be a bit more tedious for users if they have to go back later to try it again, but happy to make the default not automatically importing.

[jasonacox]

We have had issues with sed -i across different platforms. Since we know sed -i.bak works for our supported platforms, I would prefer to use that in these cases. I think the right setup would be:

    # Replace variables in dashboards
    for file in ./grafana/dashboards/*.json; do
        # Replace data source objects with just the name of the source
        sed -i.bak -E 's|"datasource":\s*\{[^}]*DS_INFLUXDB[^}]*\}|"datasource": "InfluxDB (auto provisioned)"|g' "$file"
        sed -i.bak -E 's|"datasource":\s*\{[^}]*DS_SUN_AND[^}]*\}|"datasource": "DS_SUN_AND MOON"|g' "$file"

        # Disable the built-in annotation
        sed -i.bak "s|\"enable\": true|\"enable\": false|g" "$file"

        # Timezone variable
        sed -i.bak "s|\${VAR_TZ}|\${tz}|g" "$file"
        CURRENT=$(cat tz)
        sed -i.bak "s|\"query\": \"\${tz}\"|\"query\": \"$CURRENT\"|g" "$file"

        # Cost variables
        sed -i.bak "s|\${VAR_AVG_BUY_PER_KWH}|\${avg_buy_per_kwh}|g" "$file"
        sed -i.bak "s|\"query\": \"\${avg_buy_per_kwh}\"|\"query\": \"0.19\"|g" "$file"

        sed -i.bak "s|\${VAR_AVG_SELL_PER_KWH}|\${avg_sell_per_kwh}|g" "$file"
        sed -i.bak "s|\"query\": \"\${avg_sell_per_kwh}\"|\"query\": \"0.19\"|g" "$file"

        sed -i.bak "s|\${VAR_AVG_PER_KWH}|\${avg_per_kwh}|g" "$file"
        sed -i.bak "s|\"query\": \"\${avg_per_kwh}\"|\"query\": \"0.19\"|g" "$file"

        sed -i.bak "s|0.19|${avg_price_per_kwh}|g" "$file"

        # Remove backup files
        rm -f "$file.bak"
    done

[mccahan]

The multiline sed commands for the data sources weren't working for me here, so I took a small liberty and updated them in the dashboards/*.json files themselves: c0e4838

I tested this with both manual and automatic import and it worked fine. Assuming most (all?) users don't bother renaming those data sources, even on a manual import this would skip the step where you have to select the data sources.

Let me know if you have any strong objections 😅

[jasonacox]

Similar to the comment on the other PR, I think we need to assume that people will run setup.sh to change their configuration so import-dashboard.sh should be able to run against existing installations (or even standalone). For this case, I think you could probably just have it check for imported dashboards and prompt the user to see if they want to overwrite the existing versions.

[jasonacox]

Thanks @mccahan - very cool. I think this would be a nice addition. I want to think through complications.

I would generally prefer not to keep creating separate helper scripts in the main repo. It adds some confusion. I wonder if this should just be in setup.sh. If not, this particular one could go into the grafana or dashboards directory.

[mccahan]

Moving it would make sense! I split it into a separate file just to keep from polluting your setup.sh too much

[mccahan]

@jasonacox appreciate the feedback!

Updates:

• Moved the import script to grafana/import-dashboards.sh
• Running it every time the user runs setup.sh
• Checks for existing dashboards and confirms overwriting
• Remove default dashboard setting when the user updates it
• Possibly-overstretching update to the original dashboard JSON files (see Auto-Import Dashboards #584 (comment))

[jasonacox]

Thinking through all the use cases that users may have...

There was a specific reason why we added the variables #512 - How will this work for existing users who didn't auto-provision? All my test boxes are auto-provisioned at this point so I need to see if I can recreate what may be in the wild.

[jasonacox]

Possibly-overstretching update to the original dashboard JSON files (see

Same concern. If we move from sed to perl for those multi-line changes, could we make it work without adjusting? I spot tested perl on most of the platform we support and it does seem to be installed by default.