Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refine Atmos Configuration Handling to Support Imports, Remote Imports, and Configuration Directories #808

Closed
wants to merge 177 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
177 commits
Select commit Hold shift + click to select a range
892cfeb
feat: add support for importing configurations from local and remote …
haitham911 Nov 24, 2024
052673b
refactor: streamline configuration initialization and enhance import …
haitham911 Nov 24, 2024
63ffce1
fix: sort full matches instead of initial matches in GetGlobMatches f…
haitham911 Nov 24, 2024
000e439
fix: add json and mapstructure tags to Import field in CliConfiguration
haitham911 Nov 24, 2024
4493540
fix: handle unexpected errors when checking for atmos.d path in InitC…
haitham911 Nov 24, 2024
d1903c4
fix: update atmos.yaml to include wildcard import for YAML files and …
haitham911 Nov 24, 2024
d9a3517
fix: remove obsolete configuration files and update atmos.yaml for cl…
haitham911 Nov 24, 2024
089a82f
fix: validate import paths to prevent directory traversal in InitCliC…
haitham911 Nov 24, 2024
c020d63
fix: enhance downloadRemoteConfig to return temporary directory and h…
haitham911 Nov 24, 2024
31c844c
fix: remove environment variable from extra-config.yaml for cleaner c…
haitham911 Nov 30, 2024
d70f66f
feat: update YAML configurations to define commands and enhance compo…
haitham911 Dec 3, 2024
512e02e
fix: replace path package with filepath for consistent path handling
haitham911 Dec 3, 2024
c22624f
Merge branch 'main' into DEV-1534
osterman Dec 4, 2024
750f4b5
feat: enhance CLI configuration loading logic to prioritize ATMOS_CLI…
haitham911 Dec 7, 2024
8eeda00
feat: reorganize configuration files and enhance YAML support for log…
haitham911 Dec 11, 2024
a8d3528
feat: rename environment variable for clarity and improve test handli…
haitham911 Dec 11, 2024
d8a00ef
feat: update remote configuration URL in atmos.yaml for improved acce…
haitham911 Dec 11, 2024
8050589
feat: restructure configuration files and remove deprecated logging s…
haitham911 Dec 11, 2024
52b170d
feat: migrate custom import configuration files to a new structure fo…
haitham911 Dec 11, 2024
f85d53e
feat: remove deprecated custom import configuration files and update …
haitham911 Dec 11, 2024
73d9255
feat: add URL validation for remote imports to enhance error handling
haitham911 Dec 12, 2024
9913b54
feat: update configuration documentation to clarify CLI config loadin…
haitham911 Dec 12, 2024
cdb9fa1
fix: correct example paths in CLI configuration documentation for cla…
haitham911 Dec 12, 2024
7090975
fix: clarify environment variable usage in CLI configuration document…
haitham911 Dec 12, 2024
3dfb2e0
Update examples/demo-atmos-cli-imports/configs.d/vendor.yaml
osterman Dec 12, 2024
cdafe20
enhance CLI configuration documentation with detailed import examples…
haitham911 Dec 13, 2024
b702884
fix: improve atmos.d path validation and add support for .atmos.d dir…
haitham911 Dec 13, 2024
8704668
fix: update command in demo configuration and improve URL validation …
haitham911 Dec 13, 2024
989cd9e
Merge branch 'main' into DEV-1534
haitham911 Dec 17, 2024
a7414bb
add env demo
haitham911 Dec 17, 2024
3a24ee6
expose env variable ATMOS_CLI_CONFIG_PATH and ATMOS_BASE_PATH before…
haitham911 Dec 18, 2024
061ba9c
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
e57aacb
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
e6a3200
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
fa5f25e
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
86e2505
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
95cdf74
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
bc2c4bc
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
8e6998a
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
87d4b0f
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
31d71c1
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
e64d2eb
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
378897c
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
e8d8507
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
6e4bc4d
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
930fde6
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
94e99d6
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
85957a4
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
b75c251
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
a742412
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
30c82d2
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
0609a78
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
1c02902
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
792ff0a
Apply suggestions from code review
osterman Dec 20, 2024
33da5d0
Apply suggestions from code review
osterman Dec 20, 2024
885bd00
Apply suggestions from code review
osterman Dec 20, 2024
b933331
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 20, 2024
50cd50e
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 30, 2024
aafdf16
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 30, 2024
3bb33fa
Update examples/demo-env/stacks/deploy/prod.yaml
osterman Dec 30, 2024
a55f6f2
Update website/docs/cli/configuration/configuration.mdx
osterman Dec 30, 2024
66c0a59
Apply suggestions from code review
osterman Dec 30, 2024
27d3acd
Apply suggestions from code review
osterman Dec 30, 2024
1d46232
revise the docs for intended behavior
osterman Dec 30, 2024
9a27195
Make it more clear
osterman Dec 30, 2024
bce6c4d
Merge branch 'main' into DEV-1534
haitham911 Jan 5, 2025
f45cf0d
fix missed from marge main branch
haitham911 Jan 5, 2025
1d33e16
change to atmosConfig
haitham911 Jan 5, 2025
f9e44cd
replace cliConfig with atmosConfig
haitham911 Jan 5, 2025
086c8c6
add mermaid for configuration
haitham911 Jan 5, 2025
bf02e59
edit mermaid doc
haitham911 Jan 5, 2025
eb443e3
refactor updates
haitham911 Jan 16, 2025
8f1bad1
handel import recursive
haitham911 Jan 19, 2025
8aeb10d
minor fix
haitham911 Jan 19, 2025
a89ddb9
Merge branch 'main' into DEV-1534
haitham911 Jan 19, 2025
4c02adf
go mod
haitham911 Jan 19, 2025
58d2baa
Persistent Flag base-path
haitham911 Jan 20, 2025
f284aa7
add base-path , config flag
haitham911 Jan 22, 2025
6662a10
add BasePathComputing
haitham911 Jan 23, 2025
aab3982
add todo
haitham911 Jan 23, 2025
687a00d
improve BasePathComputing func
haitham911 Jan 26, 2025
e8c8c8c
fix relative path
haitham911 Jan 26, 2025
adbc84c
fix log
haitham911 Jan 26, 2025
224dbbe
update system load config
haitham911 Jan 26, 2025
2dd72cf
fix base base
haitham911 Jan 26, 2025
d90e2ed
add loadAtmosConfigFromEnv
haitham911 Jan 26, 2025
cb72208
config loader refactor
haitham911 Jan 28, 2025
d6bc7d7
fix get log level from arg
haitham911 Jan 29, 2025
1e67a35
fix load file from --config
haitham911 Jan 29, 2025
aa811fc
fix --config nil
haitham911 Jan 29, 2025
e575831
fix import
haitham911 Jan 29, 2025
848b986
add log for merged files
haitham911 Jan 29, 2025
9b3c31b
go mod
haitham911 Jan 29, 2025
be0d4d0
add const AtmosGitRootFunc
haitham911 Jan 29, 2025
8727747
Merge branch 'main' into DEV-1534
haitham911 Feb 5, 2025
5438111
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 5, 2025
4f6fa94
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 5, 2025
df92b1d
fix log
haitham911 Feb 5, 2025
f37d1b8
Merge remote-tracking branch 'origin/DEV-1534' into DEV-1534
haitham911 Feb 5, 2025
719109e
move work dir up to git root
haitham911 Feb 5, 2025
04d6e9f
fix debugLogging
haitham911 Feb 5, 2025
44e7415
fix %s
haitham911 Feb 5, 2025
597e7db
modify embedded file
haitham911 Feb 5, 2025
5a4cf74
add Atmos Yaml Func GitRoot
haitham911 Feb 6, 2025
e0d3cb7
remove TestBasePathComputingWithRepoRoot
haitham911 Feb 6, 2025
4dac65a
Merge branch 'main' into DEV-1534
haitham911 Feb 6, 2025
87b4826
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 6, 2025
fe0094f
fix load git repo
haitham911 Feb 6, 2025
7ce4f8f
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 6, 2025
e267fd4
viper set defaults
haitham911 Feb 6, 2025
58c5361
modify snapshots stdout
haitham911 Feb 6, 2025
2088f36
configuration.mdx
haitham911 Feb 6, 2025
4e12998
use log debug
haitham911 Feb 6, 2025
0f0f841
fix test Valid_Log_Level_in_Config_File
haitham911 Feb 6, 2025
616328e
fix TestCLICommands/Valid_Log_Level_in_Config_File
haitham911 Feb 6, 2025
f3bf4e1
fix snapshots
haitham911 Feb 6, 2025
4a07688
fix snapshots
haitham911 Feb 7, 2025
c3a75e6
fix load defaults
haitham911 Feb 7, 2025
12a388b
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 7, 2025
e7b6143
fix test snapshots
haitham911 Feb 7, 2025
1abd9e7
fix test stdout
haitham911 Feb 7, 2025
33eb064
update snapshot for test
haitham911 Feb 7, 2025
93fca1a
update empty dir stdout
haitham911 Feb 7, 2025
4979f46
fix test vendor pull
haitham911 Feb 7, 2025
e2aa858
fix test atmos and vendor pull
haitham911 Feb 7, 2025
bbf9e97
fix TestCLICommands/atmos
haitham911 Feb 7, 2025
b63015e
fix test atmos
haitham911 Feb 7, 2025
d41aa85
fix snapshots atmos test
haitham911 Feb 7, 2025
87160de
fix atmos snapshot
haitham911 Feb 7, 2025
929f784
fix test config loader
haitham911 Feb 7, 2025
4b691f9
fix TestBasePathComputingWithBasePathSetInConfiguration
haitham911 Feb 7, 2025
23b4305
fix TestBasePathComputingWithBasePathSetInConfiguration
haitham911 Feb 7, 2025
5e0bbf7
update base-path help
haitham911 Feb 8, 2025
d278ed5
using charmbracelet logger
haitham911 Feb 8, 2025
f340882
small fix
haitham911 Feb 8, 2025
7f1cd32
fix return nil
haitham911 Feb 8, 2025
edbd513
update snapshots
haitham911 Feb 8, 2025
72a2e07
improve log
haitham911 Feb 8, 2025
6b03048
update snapshot
haitham911 Feb 8, 2025
e01fde3
fix error formatting directives
haitham911 Feb 8, 2025
5105cb8
fix sortFilesByDepth
haitham911 Feb 8, 2025
67c6df3
improve log key
haitham911 Feb 8, 2025
8516156
update snapshots
haitham911 Feb 9, 2025
da24379
add test for import config
haitham911 Feb 9, 2025
a4284f6
update snapshots
haitham911 Feb 9, 2025
186b49a
fix import test
haitham911 Feb 9, 2025
bd15949
update import test snapshot
haitham911 Feb 9, 2025
4e4c6e3
add atmos-configuration to test
haitham911 Feb 9, 2025
848d667
move demo env to /tests/fixtures/scenarios
haitham911 Feb 9, 2025
f7e1db4
move components/terraform/example folder to tests/fixtures/component…
haitham911 Feb 9, 2025
ba84a12
modify infraBasePath
haitham911 Feb 9, 2025
5a07a83
refactor and resolve small update
haitham911 Feb 9, 2025
7648313
add log
haitham911 Feb 9, 2025
f64ca12
update snapshots
haitham911 Feb 9, 2025
16eb6de
modify env test case
haitham911 Feb 9, 2025
c754426
Merge branch 'main' into DEV-1534
haitham911 Feb 9, 2025
0ee1363
add test env
haitham911 Feb 9, 2025
bba19c4
fix regex for env test
haitham911 Feb 10, 2025
61875bc
remove sync.RWMutex
haitham911 Feb 10, 2025
3694cfc
call sanitizeOutput on verifyOutput
haitham911 Feb 10, 2025
375a027
fix collapseExtraSlashes
haitham911 Feb 10, 2025
ca63a83
update snapshot
haitham911 Feb 10, 2025
cb0a36f
fix which test
haitham911 Feb 10, 2025
81fdc5b
commit to test terraform_output_function
haitham911 Feb 10, 2025
4ea3455
uncommit sanitizeOutput
haitham911 Feb 10, 2025
5177e3c
add more log to verifyOutput
haitham911 Feb 10, 2025
a6ea4c9
fix verifyOutput sanitize Output
haitham911 Feb 10, 2025
7eecb7f
add stderr to verifyOutput
haitham911 Feb 10, 2025
4bc9e9c
test verifyOutput without sanitizeOutput
haitham911 Feb 10, 2025
9370fb6
use sanitizeOutput on verifyOutput
haitham911 Feb 10, 2025
67bdc56
update which atmos
haitham911 Feb 11, 2025
454bb85
fix which atmos test
haitham911 Feb 11, 2025
8345062
Merge branch 'main' into DEV-1534
haitham911 Feb 11, 2025
83051df
refactor inferBasePath
haitham911 Feb 11, 2025
86c1034
refactor code SearchAtmosConfig
haitham911 Feb 11, 2025
52e4779
improve logs
haitham911 Feb 12, 2025
6e401a9
improve logs
haitham911 Feb 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: enhance downloadRemoteConfig to return temporary directory and h…
…andle cleanup on error
  • Loading branch information
haitham911 committed Nov 24, 2024
commit c020d636661a9c25618674da4f2bc8cf1843fec2
19 changes: 12 additions & 7 deletions pkg/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"path/filepath"
"runtime"
"strings"
"time"

"github.com/fatih/color"
"github.com/hashicorp/go-getter"
Expand Down Expand Up @@ -415,12 +416,14 @@ func processImports(cliConfig schema.CliConfiguration, v *viper.Viper) error {

if strings.HasPrefix(importPath, "http://") || strings.HasPrefix(importPath, "https://") {
// Handle remote URLs
tempFile, err := downloadRemoteConfig(importPath)
tempDir, tempFile, err := downloadRemoteConfig(importPath)
if err != nil {
u.LogWarning(cliConfig, fmt.Sprintf("Warning: failed to download remote config '%s': %v", importPath, err))
continue
}
resolvedPaths = []string{tempFile}
defer os.RemoveAll(tempDir)

} else {
impWithExt := importPath
ext := filepath.Ext(importPath)
Expand Down Expand Up @@ -458,22 +461,24 @@ func processImports(cliConfig schema.CliConfiguration, v *viper.Viper) error {
return nil
}

func downloadRemoteConfig(url string) (string, error) {
func downloadRemoteConfig(url string) (string, string, error) {
tempDir, err := os.MkdirTemp("", "atmos-import-*")
if err != nil {
return "", err
return "", "", err
}
tempFile := filepath.Join(tempDir, "config.yaml")

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
client := &getter.Client{
Ctx: context.Background(),
Ctx: ctx,
Src: url,
Dst: tempFile,
Mode: getter.ClientModeFile,
}
err = client.Get()
if err != nil {
return "", fmt.Errorf("failed to download remote config: %w", err)
os.RemoveAll(tempDir)
return "", "", fmt.Errorf("failed to download remote config: %w", err)
}
return tempFile, nil
return tempDir, tempFile, nil
}