Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add E2E test for tools #3325

Merged
merged 42 commits into from
Nov 30, 2023
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
3563118
Update new.feature
SajidAlamQB Nov 17, 2023
2c672cd
Update new.feature
SajidAlamQB Nov 21, 2023
b199e0f
Merge branch 'develop' into dev/e2e-test-for-add-ons
SajidAlamQB Nov 21, 2023
849f77e
Update new.feature
SajidAlamQB Nov 23, 2023
38c4766
add-ons e2e test
SajidAlamQB Nov 23, 2023
7295a0e
fix minor errors
SajidAlamQB Nov 23, 2023
0978bf6
Merge branch 'develop' into dev/e2e-test-for-add-ons
SajidAlamQB Nov 27, 2023
f91c6a6
update e2e add-ons
SajidAlamQB Nov 27, 2023
523dbab
Merge branch 'dev/e2e-test-for-add-ons' of https://github.com/kedro-o…
SajidAlamQB Nov 27, 2023
e482787
add tests for pyspark and viz
SajidAlamQB Nov 27, 2023
745e973
install dependencies
SajidAlamQB Nov 27, 2023
88e5e69
lint
SajidAlamQB Nov 27, 2023
65653c9
add hadoop
SajidAlamQB Nov 28, 2023
1d18ed9
Update cli_steps.py
SajidAlamQB Nov 28, 2023
b0c4613
Update e2e-tests.yml
SajidAlamQB Nov 28, 2023
dfc3954
add java to ENV
SajidAlamQB Nov 28, 2023
6e6d320
use java github action
SajidAlamQB Nov 28, 2023
d10dec5
remove hadoop
SajidAlamQB Nov 28, 2023
189ccfa
dummy hadoop
SajidAlamQB Nov 28, 2023
6593a0a
fix version
SajidAlamQB Nov 28, 2023
5b965d6
Merge branch 'develop' into dev/e2e-test-for-add-ons
SajidAlamQB Nov 28, 2023
19479d2
add VC++
SajidAlamQB Nov 28, 2023
5abf064
Update e2e-tests.yml
SajidAlamQB Nov 28, 2023
d19b053
Merge branch 'dev/e2e-test-for-add-ons' of https://github.com/kedro-o…
SajidAlamQB Nov 28, 2023
e425b6e
Update e2e-tests.yml
SajidAlamQB Nov 28, 2023
15ccb80
Update e2e-tests.yml
SajidAlamQB Nov 28, 2023
966be35
use only choco
SajidAlamQB Nov 28, 2023
3836091
Update e2e-tests.yml
SajidAlamQB Nov 28, 2023
5644186
hadoop_home attempt 2
SajidAlamQB Nov 28, 2023
bfe1f3d
revert choco installs
SajidAlamQB Nov 28, 2023
6de192f
try hadoop 2.7.1 and try starters hadoop fix
SajidAlamQB Nov 28, 2023
9581ae7
Update e2e-tests.yml
SajidAlamQB Nov 28, 2023
ec29275
Update e2e-tests.yml
SajidAlamQB Nov 28, 2023
eb59c96
add file check for all addons
SajidAlamQB Nov 28, 2023
2eca95a
lint
SajidAlamQB Nov 28, 2023
cf4d1fb
update add-ons e2e test
SajidAlamQB Nov 28, 2023
312ff97
Merge branch 'develop' into dev/e2e-test-for-add-ons
SajidAlamQB Nov 30, 2023
d094c02
rename to tools
SajidAlamQB Nov 30, 2023
b67361f
Update cli_steps.py
SajidAlamQB Nov 30, 2023
67f44af
Merge branch 'develop' into dev/e2e-test-for-add-ons
SajidAlamQB Nov 30, 2023
e1159bb
Merge branch 'develop' into dev/e2e-test-for-add-ons
SajidAlamQB Nov 30, 2023
053f981
changes based on review
SajidAlamQB Nov 30, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/e2e-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,21 @@ jobs:
- name: Add MSBuild to PATH
if: inputs.os == 'windows-latest'
uses: microsoft/setup-msbuild@v1
- name: Set Java environment variables
if: inputs.os == 'windows-latest'
run: |
echo "JAVA_HOME=C:\OpenJDK8U\openjdk-8u252-b09" | Out-File -FilePath $env:GITHUB_ENV -Append
echo "Path=${env:Path};C:\OpenJDK8U\openjdk-8u252-b09\bin" | Out-File -FilePath $env:GITHUB_ENV -Append
shell: powershell
SajidAlamQB marked this conversation as resolved.
Show resolved Hide resolved
- name: Set up Hadoop binary
if: inputs.os == 'windows-latest'
run: |
$ProgressPreference = "SilentlyContinue"
Invoke-WebRequest "https://github.com/steveloughran/winutils/raw/master/hadoop-2.6.3/bin/winutils.exe" -OutFile "winutils.exe" -UseBasicParsing
New-Item -ItemType directory -Path "C:\hadoop\bin" -Force
Move-Item -Path ".\winutils.exe" -Destination "C:\hadoop\bin"
echo "HADOOP_HOME=C:\hadoop\" | Out-File -FilePath $env:GITHUB_ENV -Append
shell: powershell
SajidAlamQB marked this conversation as resolved.
Show resolved Hide resolved
- name: Install dependencies
run: |
pip install --upgrade pip
Expand Down
38 changes: 38 additions & 0 deletions features/addons.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
Feature: New Kedro project with add-ons

Scenario: Create a new Kedro project without any add-ons
Given I have prepared a config file with add-ons "none"
When I run a non-interactive kedro new without starter
Then the expected add-on directories and files should be created

Scenario: Create a new Kedro project with all add-ons except 'viz' and 'pyspark'
Given I have prepared a config file with add-ons "1,2,3,4,5"
When I run a non-interactive kedro new without starter
Then the expected add-on directories and files should be created
Given I have installed the project dependencies
When I execute the kedro command "run"
Then I should get a successful exit code

Scenario: Create a new Kedro project with all add-ons
Given I have prepared a config file with add-ons "all"
When I run a non-interactive kedro new without starter
Then the expected add-on directories and files should be created
Given I have installed the project dependencies
When I execute the kedro command "run"
Then I should get a successful exit code

Scenario: Create a new Kedro project with only 'pyspark' add-on
Given I have prepared a config file with add-ons "6"
When I run a non-interactive kedro new without starter
Then the expected add-on directories and files should be created
Given I have installed the project dependencies
When I execute the kedro command "run"
Then I should get a successful exit code

Scenario: Create a new Kedro project with only 'viz' add-on
Given I have prepared a config file with add-ons "7"
When I run a non-interactive kedro new without starter
Then the expected add-on directories and files should be created
Given I have installed the project dependencies
When I execute the kedro command "run"
Then I should get a successful exit code
37 changes: 37 additions & 0 deletions features/steps/cli_steps.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,31 @@ def create_config_file(context):
yaml.dump(config, config_file, default_flow_style=False)


@given('I have prepared a config file with add-ons "{addons}"')
def create_config_file_with_add_ons(context, addons):
"""Behave step to create a temporary config file
(given the existing temp directory) and store it in the context.
It takes a custom add-ons list and sets example prompt to `y`.
"""

addons_str = addons if addons != "none" else ""

context.config_file = context.temp_dir / "config.yml"
context.project_name = "project-dummy"
context.root_project_dir = context.temp_dir / context.project_name
context.package_name = context.project_name.replace("-", "_")
config = {
"add_ons": addons_str,
"example_pipeline": "y",
"project_name": context.project_name,
"repo_name": context.project_name,
"output_dir": str(context.temp_dir),
"python_package": context.package_name,
}
with context.config_file.open("w") as config_file:
yaml.dump(config, config_file, default_flow_style=False)


@given("I have installed the project dependencies")
def pip_install_dependencies(context):
"""Install project dependencies using pip."""
Expand Down Expand Up @@ -453,6 +478,18 @@ def is_created(name):
assert is_created(path)


@then("the expected add-on directories and files should be created")
def check_created_project_structure_from_addons(context):
"""Behave step to check the subdirectories created by kedro new with add-ons."""

def is_created(name):
"""Check if path exists."""
return (context.root_project_dir / name).exists()

for path in ("README.md", "src", "pyproject.toml", "requirements.txt"):
assert is_created(path)


@then("the logs should show that {number} nodes were run")
def check_one_node_run(context, number):
expected_log_line = f"Completed {number} out of {number} tasks"
Expand Down
Loading