Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refine Atmos Configuration Handling to Support Imports, Remote Imports, and Configuration Directories #808

Closed
wants to merge 177 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
177 commits
Select commit Hold shift + click to select a range
892cfeb
feat: add support for importing configurations from local and remote …
haitham911 Nov 24, 2024
052673b
refactor: streamline configuration initialization and enhance import …
haitham911 Nov 24, 2024
63ffce1
fix: sort full matches instead of initial matches in GetGlobMatches f…
haitham911 Nov 24, 2024
000e439
fix: add json and mapstructure tags to Import field in CliConfiguration
haitham911 Nov 24, 2024
4493540
fix: handle unexpected errors when checking for atmos.d path in InitC…
haitham911 Nov 24, 2024
d1903c4
fix: update atmos.yaml to include wildcard import for YAML files and …
haitham911 Nov 24, 2024
d9a3517
fix: remove obsolete configuration files and update atmos.yaml for cl…
haitham911 Nov 24, 2024
089a82f
fix: validate import paths to prevent directory traversal in InitCliC…
haitham911 Nov 24, 2024
c020d63
fix: enhance downloadRemoteConfig to return temporary directory and h…
haitham911 Nov 24, 2024
31c844c
fix: remove environment variable from extra-config.yaml for cleaner c…
haitham911 Nov 30, 2024
d70f66f
feat: update YAML configurations to define commands and enhance compo…
haitham911 Dec 3, 2024
512e02e
fix: replace path package with filepath for consistent path handling
haitham911 Dec 3, 2024
c22624f
Merge branch 'main' into DEV-1534
osterman Dec 4, 2024
750f4b5
feat: enhance CLI configuration loading logic to prioritize ATMOS_CLI…
haitham911 Dec 7, 2024
8eeda00
feat: reorganize configuration files and enhance YAML support for log…
haitham911 Dec 11, 2024
a8d3528
feat: rename environment variable for clarity and improve test handli…
haitham911 Dec 11, 2024
d8a00ef
feat: update remote configuration URL in atmos.yaml for improved acce…
haitham911 Dec 11, 2024
8050589
feat: restructure configuration files and remove deprecated logging s…
haitham911 Dec 11, 2024
52b170d
feat: migrate custom import configuration files to a new structure fo…
haitham911 Dec 11, 2024
f85d53e
feat: remove deprecated custom import configuration files and update …
haitham911 Dec 11, 2024
73d9255
feat: add URL validation for remote imports to enhance error handling
haitham911 Dec 12, 2024
9913b54
feat: update configuration documentation to clarify CLI config loadin…
haitham911 Dec 12, 2024
cdb9fa1
fix: correct example paths in CLI configuration documentation for cla…
haitham911 Dec 12, 2024
7090975
fix: clarify environment variable usage in CLI configuration document…
haitham911 Dec 12, 2024
3dfb2e0
Update examples/demo-atmos-cli-imports/configs.d/vendor.yaml
osterman Dec 12, 2024
cdafe20
enhance CLI configuration documentation with detailed import examples…
haitham911 Dec 13, 2024
b702884
fix: improve atmos.d path validation and add support for .atmos.d dir…
haitham911 Dec 13, 2024
8704668
fix: update command in demo configuration and improve URL validation …
haitham911 Dec 13, 2024
989cd9e
Merge branch 'main' into DEV-1534
haitham911 Dec 17, 2024
a7414bb
add env demo
haitham911 Dec 17, 2024
3a24ee6
expose env variable ATMOS_CLI_CONFIG_PATH and ATMOS_BASE_PATH before…
haitham911 Dec 18, 2024
061ba9c
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
e57aacb
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
e6a3200
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
fa5f25e
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
86e2505
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
95cdf74
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
bc2c4bc
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
8e6998a
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
87d4b0f
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
31d71c1
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
e64d2eb
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
378897c
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
e8d8507
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
6e4bc4d
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
930fde6
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
94e99d6
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
85957a4
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
b75c251
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
a742412
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
30c82d2
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
0609a78
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
1c02902
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
792ff0a
Apply suggestions from code review
osterman Dec 20, 2024
33da5d0
Apply suggestions from code review
osterman Dec 20, 2024
885bd00
Apply suggestions from code review
osterman Dec 20, 2024
b933331
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
50cd50e
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 30, 2024
aafdf16
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 30, 2024
3bb33fa
Update examples/demo-env/stacks/deploy/prod.yaml
osterman Dec 30, 2024
a55f6f2
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 30, 2024
66c0a59
Apply suggestions from code review
osterman Dec 30, 2024
27d3acd
Apply suggestions from code review
osterman Dec 30, 2024
1d46232
revise the docs for intended behavior
osterman Dec 30, 2024
9a27195
Make it more clear
osterman Dec 30, 2024
bce6c4d
Merge branch 'main' into DEV-1534
haitham911 Jan 5, 2025
f45cf0d
fix missed from marge main branch
haitham911 Jan 5, 2025
1d33e16
change to atmosConfig
haitham911 Jan 5, 2025
f9e44cd
replace cliConfig with atmosConfig
haitham911 Jan 5, 2025
086c8c6
add mermaid for configuration
haitham911 Jan 5, 2025
bf02e59
edit mermaid doc
haitham911 Jan 5, 2025
eb443e3
refactor updates
haitham911 Jan 16, 2025
8f1bad1
handel import recursive
haitham911 Jan 19, 2025
8aeb10d
minor fix
haitham911 Jan 19, 2025
a89ddb9
Merge branch 'main' into DEV-1534
haitham911 Jan 19, 2025
4c02adf
go mod
haitham911 Jan 19, 2025
58d2baa
Persistent Flag base-path
haitham911 Jan 20, 2025
f284aa7
add base-path , config flag
haitham911 Jan 22, 2025
6662a10
add BasePathComputing
haitham911 Jan 23, 2025
aab3982
add todo
haitham911 Jan 23, 2025
687a00d
improve BasePathComputing func
haitham911 Jan 26, 2025
e8c8c8c
fix relative path
haitham911 Jan 26, 2025
adbc84c
fix log
haitham911 Jan 26, 2025
224dbbe
update system load config
haitham911 Jan 26, 2025
2dd72cf
fix base base
haitham911 Jan 26, 2025
d90e2ed
add loadAtmosConfigFromEnv
haitham911 Jan 26, 2025
cb72208
config loader refactor
haitham911 Jan 28, 2025
d6bc7d7
fix get log level from arg
haitham911 Jan 29, 2025
1e67a35
fix load file from --config
haitham911 Jan 29, 2025
aa811fc
fix --config nil
haitham911 Jan 29, 2025
e575831
fix import
haitham911 Jan 29, 2025
848b986
add log for merged files
haitham911 Jan 29, 2025
9b3c31b
go mod
haitham911 Jan 29, 2025
be0d4d0
add const AtmosGitRootFunc
haitham911 Jan 29, 2025
8727747
Merge branch 'main' into DEV-1534
haitham911 Feb 5, 2025
5438111
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 5, 2025
4f6fa94
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 5, 2025
df92b1d
fix log
haitham911 Feb 5, 2025
f37d1b8
Merge remote-tracking branch 'origin/DEV-1534' into DEV-1534
haitham911 Feb 5, 2025
719109e
move work dir up to git root
haitham911 Feb 5, 2025
04d6e9f
fix debugLogging
haitham911 Feb 5, 2025
44e7415
fix %s
haitham911 Feb 5, 2025
597e7db
modify embedded file
haitham911 Feb 5, 2025
5a4cf74
add Atmos Yaml Func GitRoot
haitham911 Feb 6, 2025
e0d3cb7
remove TestBasePathComputingWithRepoRoot
haitham911 Feb 6, 2025
4dac65a
Merge branch 'main' into DEV-1534
haitham911 Feb 6, 2025
87b4826
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 6, 2025
fe0094f
fix load git repo
haitham911 Feb 6, 2025
7ce4f8f
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 6, 2025
e267fd4
viper set defaults
haitham911 Feb 6, 2025
58c5361
modify snapshots stdout
haitham911 Feb 6, 2025
2088f36
configuration.mdx
haitham911 Feb 6, 2025
4e12998
use log debug
haitham911 Feb 6, 2025
0f0f841
fix test Valid_Log_Level_in_Config_File
haitham911 Feb 6, 2025
616328e
fix TestCLICommands/Valid_Log_Level_in_Config_File
haitham911 Feb 6, 2025
f3bf4e1
fix snapshots
haitham911 Feb 6, 2025
4a07688
fix snapshots
haitham911 Feb 7, 2025
c3a75e6
fix load defaults
haitham911 Feb 7, 2025
12a388b
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 7, 2025
e7b6143
fix test snapshots
haitham911 Feb 7, 2025
1abd9e7
fix test stdout
haitham911 Feb 7, 2025
33eb064
update snapshot for test
haitham911 Feb 7, 2025
93fca1a
update empty dir stdout
haitham911 Feb 7, 2025
4979f46
fix test vendor pull
haitham911 Feb 7, 2025
e2aa858
fix test atmos and vendor pull
haitham911 Feb 7, 2025
bbf9e97
fix TestCLICommands/atmos
haitham911 Feb 7, 2025
b63015e
fix test atmos
haitham911 Feb 7, 2025
d41aa85
fix snapshots atmos test
haitham911 Feb 7, 2025
87160de
fix atmos snapshot
haitham911 Feb 7, 2025
929f784
fix test config loader
haitham911 Feb 7, 2025
4b691f9
fix TestBasePathComputingWithBasePathSetInConfiguration
haitham911 Feb 7, 2025
23b4305
fix TestBasePathComputingWithBasePathSetInConfiguration
haitham911 Feb 7, 2025
5e0bbf7
update base-path help
haitham911 Feb 8, 2025
d278ed5
using charmbracelet logger
haitham911 Feb 8, 2025
f340882
small fix
haitham911 Feb 8, 2025
7f1cd32
fix return nil
haitham911 Feb 8, 2025
edbd513
update snapshots
haitham911 Feb 8, 2025
72a2e07
improve log
haitham911 Feb 8, 2025
6b03048
update snapshot
haitham911 Feb 8, 2025
e01fde3
fix error formatting directives
haitham911 Feb 8, 2025
5105cb8
fix sortFilesByDepth
haitham911 Feb 8, 2025
67c6df3
improve log key
haitham911 Feb 8, 2025
8516156
update snapshots
haitham911 Feb 9, 2025
da24379
add test for import config
haitham911 Feb 9, 2025
a4284f6
update snapshots
haitham911 Feb 9, 2025
186b49a
fix import test
haitham911 Feb 9, 2025
bd15949
update import test snapshot
haitham911 Feb 9, 2025
4e4c6e3
add atmos-configuration to test
haitham911 Feb 9, 2025
848d667
move demo env to /tests/fixtures/scenarios
haitham911 Feb 9, 2025
f7e1db4
move components/terraform/example folder to tests/fixtures/component…
haitham911 Feb 9, 2025
ba84a12
modify infraBasePath
haitham911 Feb 9, 2025
5a07a83
refactor and resolve small update
haitham911 Feb 9, 2025
7648313
add log
haitham911 Feb 9, 2025
f64ca12
update snapshots
haitham911 Feb 9, 2025
16eb6de
modify env test case
haitham911 Feb 9, 2025
c754426
Merge branch 'main' into DEV-1534
haitham911 Feb 9, 2025
0ee1363
add test env
haitham911 Feb 9, 2025
bba19c4
fix regex for env test
haitham911 Feb 10, 2025
61875bc
remove sync.RWMutex
haitham911 Feb 10, 2025
3694cfc
call sanitizeOutput on verifyOutput
haitham911 Feb 10, 2025
375a027
fix collapseExtraSlashes
haitham911 Feb 10, 2025
ca63a83
update snapshot
haitham911 Feb 10, 2025
cb0a36f
fix which test
haitham911 Feb 10, 2025
81fdc5b
commit to test terraform_output_function
haitham911 Feb 10, 2025
4ea3455
uncommit sanitizeOutput
haitham911 Feb 10, 2025
5177e3c
add more log to verifyOutput
haitham911 Feb 10, 2025
a6ea4c9
fix verifyOutput sanitize Output
haitham911 Feb 10, 2025
7eecb7f
add stderr to verifyOutput
haitham911 Feb 10, 2025
4bc9e9c
test verifyOutput without sanitizeOutput
haitham911 Feb 10, 2025
9370fb6
use sanitizeOutput on verifyOutput
haitham911 Feb 10, 2025
67bdc56
update which atmos
haitham911 Feb 11, 2025
454bb85
fix which atmos test
haitham911 Feb 11, 2025
8345062
Merge branch 'main' into DEV-1534
haitham911 Feb 11, 2025
83051df
refactor inferBasePath
haitham911 Feb 11, 2025
86c1034
refactor code SearchAtmosConfig
haitham911 Feb 11, 2025
52e4779
improve logs
haitham911 Feb 12, 2025
6e401a9
improve logs
haitham911 Feb 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make it more clear
  • Loading branch information
osterman committed Dec 30, 2024
commit 9a27195b1f3e66029f9309ba3f04cbdb0e1860d4
22 changes: 11 additions & 11 deletions website/docs/cli/configuration/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import Screengrab from '@site/src/components/Screengrab'
import Terminal from '@site/src/components/Terminal'
import File from '@site/src/components/File'
import Intro from '@site/src/components/Intro'
import Admonition from '@theme/Admonition';

# CLI Configuration

Expand All @@ -23,7 +24,9 @@ Think of this file as where you [bootstrap the settings or configuration of your

## Configuration File (`atmos.yaml`)

The Atmos configuration is processed in 3 stages, enabling you to define settings at different levels of precedence.
The Atmos configuration is processed in three stages, allowing you to define settings at different levels of precedence.

As part of this process, Atmos updates the `ATMOS_CLI_CONFIG_PATH` environment variable to include a list of the absolute configuration paths in the order they were processed. These paths are separated by the system's standard path separator (`:` on Unix-like systems and `;` on Windows). This ensures that subshells and Terraform providers like `terraform-provider-utils` can consistently access and process the merged configurations across different operating systems.

### Stage 1: Load Explicit or System-Wide Configurations

Expand All @@ -35,15 +38,18 @@ Atmos will search for the configuration file in the following locations, in orde
<dd>
The `--config` flag lets you provide a relative or absolute path to a valid configuration file or a directory that only contains Atmos configuration files. If a relative path is supplied, it will be relative to your current working directory. The `--config` flag has the highest priority, meaning it overrides all other configuration sources.

When you use the `--config` flag, any other search paths, like those in `XDG_CONFIG_HOME`, are ignored, making it the flag the exclusive configuration source.
<Admonition type="important">
When you use the `--config` flag, all subsequent stages are skipped, and only the configuration files specified by the `--config` flag are loaded.
</Admonition>

You can pass the `--config` flag multiple times in a single command; the configurations will be deep merged in the order you provide them. The first config you specify is the lowest priority, and the last one is the highest, allowing later configs to override settings from earlier ones.

For example, to load multiple configuration files, you would run:

```bash
atmos --config /path/to/config1.yaml --config /path/to/config2.yaml ...
```

Additionally, Atmos will update the `ATMOS_CLI_CONFIG_PATH` environment variable to contain a list of the specified configuration paths in order, separated by the system's standard path separator (`:` on Unix-like systems and `;` on Windows). This ensures that subshells and Terraform providers like `terraform-provider-utils` can access and process the merged configurations consistently across different operating systems.
</dd>

<dt>
Expand All @@ -59,8 +65,6 @@ Atmos will search for the configuration file in the following locations, in orde
**On Linux or macOS:**
- `/usr/local/etc/atmos/atmos.yaml`
- `/etc/atmos/atmos.yaml`


</dd>
</dl>

Expand All @@ -77,18 +81,14 @@ Atmos will search for additional configuration files in the following locations,
`ATMOS_CLI_CONFIG_PATH` environment variable
</dt>
<dd>
The environment variable `ATMOS_CLI_CONFIG_PATH` should point to a folder (or a specific file). This is mutually exclusive with the `--config` parameter. If `--config` is specified, then the `ATMOS_CLI_CONFIG_PATH` is ignored.
The environment variable `ATMOS_CLI_CONFIG_PATH` should point to a folder (or a specific file). Multiple paths can be specified, separated by the system's standard path separator (`:` on Unix-like systems and `;` on Windows).

Atmos will only search for the configuration file in the specified folder. Note that `.yaml` and `.yml` file extensions are treated interchangeably and are loaded in lexicographical order. Atmos will search for the following files, in order, relative to the `ATMOS_CLI_CONFIG_PATH` environment variable.

Atmos processes each path specified by the `ATMOS_CLI_CONFIG_PATH` environment variable in the order they are listed, separated by the system's standard path separator (`:` on Unix-like systems and `;` on Windows).

If a path is a directory, Atmos will search for the following files in the directory, in order: (in order from highest to lowest priority):

- `atmos.yaml`
- `.atmos.yaml`
- `atmos.d/**/*`
- `.atmos.d/**/*`
</dd>

<dt>
Expand Down Expand Up @@ -120,7 +120,7 @@ Atmos will search for additional configuration files in the following locations,
If the `--config` flag is used, this stage is skipped.
:::

Additionally, after processing all the configuration files, Atmos will apply user preferences. Since user preferences are applied last, they have the highest precedence.
After processing all configuration files, Atmos applies user preferences. Since user preferences are applied last, they have the highest precedence. This allows you to override any settings from previous stages to suit your specific needs, such as adjusting logging levels, configuring aliases, tweaking colors, or even adding your own custom commands.

Preferences are deep-merged, meaning they override any conflicting sections defined in subsequent locations:

Expand Down