create docker cpu fullload command failed #233

jianxingzhe · 2019-11-12T02:21:30Z

Issue Description

Type: bug report

Describe what happened (or what feature you want)

why the command response the follow error and what the error meaning?

./blade create docker cpu fullload --cpu-percent 80 --blade-tar-file /root/chaosblade-0.4.0.tar.gz --container-id b9f364c73dae
{"code":801,"success":false,"error":"\u0002\u0000\u0000\u0000\u0000\u0000\u0000%sh: /opt/chaosblade/blade: not found"}

root@test7:~# ls -l /root/chaosblade-0.4.0.tar.gz 
-rw-r--r-- 1 root root 87858816 Nov 12 09:53 /root/chaosblade-0.4.0.tar.gz

operator logs:

time="2019-11-11T08:52:10Z" level=info msg="oldVersion:29954747, newVersion: 29954748"
time="2019-11-11T08:52:10Z" level=info msg="get pods by names python-789fdcf879-r4kts, len is 1"
time="2019-11-11T08:52:10Z" level=info msg="matched container: b9f364c73dae015efbe373f2793bdedd8754fbad24e83cdb7cea82c2560e65d5, pod: python-789fdcf879-r4kts, node: 172.24.5.7"
time="2019-11-11T08:52:10Z" level=info msg="start exec, target: cpu, action: fullload"
time="2019-11-11T08:52:10Z" level=info msg="nodeNameUidMap: map[172.24.5.7:]"
time="2019-11-11T08:52:10Z" level=info msg="experimentIdentifiers: [{Id: Uid:b9f364c73dae015efbe373f2793bdedd8754fbad24e83cdb7cea82c2560e65d5 Name:python Command:/opt/chaosblade/blade create docker cpu fullload  --cpu-percent=100 --blade-tar-file /opt/chaosblade-0.4.0.tar.gz --container-id b9f364c73dae015efbe373f2793bdedd8754fbad24e83cdb7cea82c2560e65d5}]"
time="2019-11-11T08:52:10Z" level=info msg="exec command in pod, command: /opt/chaosblade/blade create docker cpu fullload  --cpu-percent=100 --blade-tar-file /opt/chaosblade-0.4.0.tar.gz --container-id b9f364c73dae015efbe373f2793bdedd8754fbad24e83cdb7cea82c2560e65d5, container: chaosblade-tool"
time="2019-11-11T08:52:10Z" level=warning msg="invoke exec command err, command terminated with exit code 1"
time="2019-11-11T08:52:10Z" level=info msg="err: {\"code\":801,\"success\":false,\"error\":\"\\u0002\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000%sh: /opt/chaosblade/blade: not found\"}; out: "
time="2019-11-11T08:52:10Z" level=info msg="success: false, statuses: [{Id: Uid:b9f364c73dae015efbe373f2793bdedd8754fbad24e83cdb7cea82c2560e65d5 Name:python State:Error Kind:container Error:\x02\x00\x00\x00\x00\x00\x00%sh: /opt/chaosblade/blade: not found Success:false NodeName:172.24.5.7}]"
time="2019-11-11T08:52:10Z" level=info msg="trigger update event"

Describe what you expected to happen

the command should success

How to reproduce it (as minimally and precisely as possible)

./blade create docker cpu fullload --cpu-percent 80 --blade-tar-file /root/chaosblade-0.4.0.tar.gz --container-id $container_id

Tell us your environment

docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:09:54 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:07:56 2018
  OS/Arch:          linux/amd64
  Experimental:     false

chaosblade version

operator image: registry.cn-hangzhou.aliyuncs.com/chaosblade/chaosblade-operator:0.0.1
chaosblade-tool image: registry.cn-hangzhou.aliyuncs.com/chaosblade/chaosblade-tool:0.4.0

Anything else we need to know?

The text was updated successfully, but these errors were encountered:

tsingloon · 2019-11-13T11:41:24Z

i have the same problem. operator log:
time="2019-11-13T11:29:44Z" level=info msg="get pods by names deployment-example-8675bff4b5-4fqrm, len is 1"
time="2019-11-13T11:29:44Z" level=info msg="matched container: 32e67b8f197c945cabba423aa036bf5473304741c871b8aa95b3c10b20428223, pod: deployment-example-8675bff4b5-4fqrm, node: 192.168.1.84"
time="2019-11-13T11:29:44Z" level=info msg="start exec, target: cpu, action: fullload"
time="2019-11-13T11:29:44Z" level=info msg="nodeNameUidMap: map[192.168.1.84:]"
time="2019-11-13T11:29:44Z" level=info msg="experimentIdentifiers: [{Id: Uid:32e67b8f197c945cabba423aa036bf5473304741c871b8aa95b3c10b20428223 Name:nginx Command:/opt/chaosblade/blade create docker cpu fullload --cpu-percent=50 --blade-tar-file /opt/chaosblade-0.4.0.tar.gz --container-id 32e67b8f197c945cabba423aa036bf5473304741c871b8aa95b3c10b20428223}]"
time="2019-11-13T11:29:44Z" level=info msg="exec command in pod, command: /opt/chaosblade/blade create docker cpu fullload --cpu-percent=50 --blade-tar-file /opt/chaosblade-0.4.0.tar.gz --container-id 32e67b8f197c945cabba423aa036bf5473304741c871b8aa95b3c10b20428223, container: chaosblade-tool"
time="2019-11-13T11:29:45Z" level=warning msg="invoke exec command err, command terminated with exit code 1"
time="2019-11-13T11:29:45Z" level=info msg="err: {"code":801,"success":false,"error":"\u0002\u0000\u0000\u0000\u0000\u0000\u0000%sh: /opt/chaosblade/blade: not found"}; out: "
time="2019-11-13T11:29:45Z" level=info msg="success: false, statuses: [{Id: Uid:32e67b8f197c945cabba423aa036bf5473304741c871b8aa95b3c10b20428223 Name:nginx State:Error Kind:container Error:\x02\x00\x00\x00\x00\x00\x00%sh: /opt/chaosblade/blade: not found Success:false NodeName:192.168.1.84}]"

jianxingzhe · 2019-11-14T01:55:55Z

if the blade tool is not in the target pod, it just return nil instead of copying it into the target pod.
@xcaspar it's a bug?

func (r *RunCmdInContainerExecutorByCP) DeployChaosBlade(ctx context.Context, containerId string, srcFile string, override bool) error {
	// check if the blade tool exists
	output, err := r.Client.execContainer(containerId, fmt.Sprintf("ls %s", BladeBin))
	logrus.Debugf("output: %s, %v", output, err)
	if err == nil && !strings.Contains(output, "cannot access") && !override {
		return nil
	}

xcaspar · 2019-11-14T03:10:54Z

if the blade tool is not in the target pod, it just return nil instead of copying it into the target pod.
@xcaspar it's a bug?

func (r *RunCmdInContainerExecutorByCP) DeployChaosBlade(ctx context.Context, containerId string, srcFile string, override bool) error {
	// check if the blade tool exists
	output, err := r.Client.execContainer(containerId, fmt.Sprintf("ls %s", BladeBin))
	logrus.Debugf("output: %s, %v", output, err)
	if err == nil && !strings.Contains(output, "cannot access") && !override {
		return nil
	}

You can execute blade command with --debug, for example, ./blade create docker cpu fullload --cpu-percent 80 --blade-tar-file /root/chaosblade-0.4.0.tar.gz --container-id b9f364c73dae --debug, then view the output log in chaosblade.log under chaosblade/logs path. Please check the output messages.

tsingloon · 2019-11-14T03:31:04Z

if the blade tool is not in the target pod, it just return nil instead of copying it into the target pod.
@xcaspar it's a bug?
func (r *RunCmdInContainerExecutorByCP) DeployChaosBlade(ctx context.Context, containerId string, srcFile string, override bool) error {
	// check if the blade tool exists
	output, err := r.Client.execContainer(containerId, fmt.Sprintf("ls %s", BladeBin))
	logrus.Debugf("output: %s, %v", output, err)
	if err == nil && !strings.Contains(output, "cannot access") && !override {
		return nil
	}
You can execute blade command with --debug, for example, ./blade create docker cpu fullload --cpu-percent 80 --blade-tar-file /root/chaosblade-0.4.0.tar.gz --container-id b9f364c73dae --debug, then view the output log in chaosblade.log under chaosblade/logs path. Please check the output messages.

add debug, logs/chaosblade.log error as follows：

time="2019-11-14T11:42:39.176617314+08:00" level=info msg="execute command: sh -c ls /opt/chaosblade/blade"
time="2019-11-14T11:42:39.258289182+08:00" level=debug msg="output: \x02\x00\x00\x00\x00\x00\x005ls: /opt/chaosblade/blade: No such file or directory, "
time="2019-11-14T11:42:39.258348526+08:00" level=info msg="execute command: sh -c /opt/chaosblade/blade create cpu fullload --cpu-percent=50 --debug=true --uid f8f6de35d58a509b"
time="2019-11-14T11:42:39.322641925+08:00" level=warning msg="decode \x02\x00\x00\x00\x00\x00\x00%sh: /opt/chaosblade/blade: not found err, return default value, {"code":513,"success":false,"error":"unmarshal \u0002\u0000\u0000\u0000\u0000\u0000\u0000%sh: /opt/chaosblade/blade: not found err: invalid character '\\x02' looking for beginning of value"}"

xcaspar · 2019-11-14T03:40:48Z

if the blade tool is not in the target pod, it just return nil instead of copying it into the target pod.
@xcaspar it's a bug?
func (r *RunCmdInContainerExecutorByCP) DeployChaosBlade(ctx context.Context, containerId string, srcFile string, override bool) error {
	// check if the blade tool exists
	output, err := r.Client.execContainer(containerId, fmt.Sprintf("ls %s", BladeBin))
	logrus.Debugf("output: %s, %v", output, err)
	if err == nil && !strings.Contains(output, "cannot access") && !override {
		return nil
	}
You can execute blade command with --debug, for example, ./blade create docker cpu fullload --cpu-percent 80 --blade-tar-file /root/chaosblade-0.4.0.tar.gz --container-id b9f364c73dae --debug, then view the output log in chaosblade.log under chaosblade/logs path. Please check the output messages.
add debug, logs/chaosblade.log error as follows：

time="2019-11-14T03:34:44.167937033Z" level=info msg="execute command: sh -c ls /opt/chaosblade/blade"
time="2019-11-14T03:34:44.320055116Z" level=debug msg="output: \x02\x00\x00\x00\x00\x00\x005ls: /opt/chaosblade/blade: No such file or directory, "
time="2019-11-14T03:34:44.320175157Z" level=info msg="execute command: sh -c /opt/chaosblade/blade create cpu fullload --debug=true --cpu-percent=50 --uid 78e52ddd8c54adaf"
time="2019-11-14T03:34:44.402556285Z" level=warning msg="decode \x02\x00\x00\x00\x00\x00\x00%sh: /opt/chaosblade/blade: not found err, return default value, {"code":513,"success":false,"error":"unmarshal \u0002\u0000\u0000\u0000\u0000\u0000\u0000%sh: /opt/chaosblade/blade: not found err: invalid character '\x02' looking for beginning of value"}"

@tsingloon @jianxingzhe Thank you very much. The problem occurred in the judgment of the return value of the ls command. It is indeed a bug. Can you provide a pr?

xcaspar · 2019-11-14T03:44:24Z

@jianxingzhe @tsingloon You can add --blade-override flag to avoid the problem currently.

jianxingzhe · 2019-11-15T06:38:26Z

if the blade tool is not in the target pod, it just return nil instead of copying it into the target pod.
@xcaspar it's a bug?
func (r *RunCmdInContainerExecutorByCP) DeployChaosBlade(ctx context.Context, containerId string, srcFile string, override bool) error {
	// check if the blade tool exists
	output, err := r.Client.execContainer(containerId, fmt.Sprintf("ls %s", BladeBin))
	logrus.Debugf("output: %s, %v", output, err)
	if err == nil && !strings.Contains(output, "cannot access") && !override {
		return nil
	}
You can execute blade command with --debug, for example, ./blade create docker cpu fullload --cpu-percent 80 --blade-tar-file /root/chaosblade-0.4.0.tar.gz --container-id b9f364c73dae --debug, then view the output log in chaosblade.log under chaosblade/logs path. Please check the output messages.
add debug, logs/chaosblade.log error as follows：
time="2019-11-14T03:34:44.167937033Z" level=info msg="execute command: sh -c ls /opt/chaosblade/blade"
time="2019-11-14T03:34:44.320055116Z" level=debug msg="output: \x02\x00\x00\x00\x00\x00\x005ls: /opt/chaosblade/blade: No such file or directory, "
time="2019-11-14T03:34:44.320175157Z" level=info msg="execute command: sh -c /opt/chaosblade/blade create cpu fullload --debug=true --cpu-percent=50 --uid 78e52ddd8c54adaf"
time="2019-11-14T03:34:44.402556285Z" level=warning msg="decode \x02\x00\x00\x00\x00\x00\x00%sh: /opt/chaosblade/blade: not found err, return default value, {"code":513,"success":false,"error":"unmarshal \u0002\u0000\u0000\u0000\u0000\u0000\u0000%sh: /opt/chaosblade/blade: not found err: invalid character '\x02' looking for beginning of value"}"
@tsingloon @jianxingzhe Thank you very much. The problem occurred in the judgment of the return value of the ls command. It is indeed a bug. Can you provide a pr?

In our pod，the ls command is a link to /bin/busbox， such as:

/usr/src/app # ls -l /bin/ls
lrwxrwxrwx    1 root     root            12 Jan 30  2019 /bin/ls -> /bin/busybox
/usr/src/app # ls  /tmp/test
ls: /tmp/test: No such file or directory

and when the file is not exist, ls command does not return 'cannot access'. Just using the No such file or directory as the judgment maybe more suitable. @xcaspar

func (r *RunCmdInContainerExecutorByCP) DeployChaosBlade(ctx context.Context, containerId string, srcFile string, override bool) error {
	// check if the blade tool exists
	output, err := r.Client.execContainer(containerId, fmt.Sprintf("ls %s", BladeBin))
	logrus.Debugf("output: %s, %v", output, err)
	if err == nil && !strings.Contains(output, "No such file or directory") && !override {
		return nil
	}

xcaspar · 2019-11-15T07:29:39Z

@jianxingzhe The ls command returns different values in different Linux images, so we use other commands to check whether the file exists, such as [ -e /opt/chaosblade/blade ] && echo True || echo False. It exists if the output message contains True, otherwise it does not exist.

xcaspar added the kind/bug label Nov 14, 2019

xcaspar added this to the 0.4.0 milestone Nov 14, 2019

jianxingzhe mentioned this issue Nov 19, 2019

create pod network delay failed #235

Closed

xcaspar mentioned this issue Dec 3, 2019

bugfix: fix chaosblade deployed in docker chaosblade-io/chaosblade-exec-docker#2

Merged

xcaspar closed this as completed Dec 5, 2019

xcaspar added type/bug Something isn't working chaosblade-exec-docker chaosblade-exec-docker project labels Feb 18, 2020

xcaspar self-assigned this Feb 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

create docker cpu fullload command failed #233

create docker cpu fullload command failed #233

jianxingzhe commented Nov 12, 2019

tsingloon commented Nov 13, 2019

jianxingzhe commented Nov 14, 2019

xcaspar commented Nov 14, 2019

tsingloon commented Nov 14, 2019 •

edited

Loading

xcaspar commented Nov 14, 2019

xcaspar commented Nov 14, 2019

jianxingzhe commented Nov 15, 2019

xcaspar commented Nov 15, 2019 •

edited

Loading

create docker cpu fullload command failed #233

create docker cpu fullload command failed #233

Comments

jianxingzhe commented Nov 12, 2019

Issue Description

Describe what happened (or what feature you want)

Describe what you expected to happen

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Anything else we need to know?

tsingloon commented Nov 13, 2019

jianxingzhe commented Nov 14, 2019

xcaspar commented Nov 14, 2019

tsingloon commented Nov 14, 2019 • edited Loading

xcaspar commented Nov 14, 2019

xcaspar commented Nov 14, 2019

jianxingzhe commented Nov 15, 2019

xcaspar commented Nov 15, 2019 • edited Loading

tsingloon commented Nov 14, 2019 •

edited

Loading

xcaspar commented Nov 15, 2019 •

edited

Loading