Update cdn config (Remove hypnotoad section and update listen field) #7450

rimashah25 · 2023-04-20T14:58:43Z

Remove hypnotoad section and move cert and key fields to traffic_ops_golang section. The cert and key fields are required to run traffic_ops.

Which Traffic Control components are affected by this PR?

Documentation
Traffic Ops
Automation (Ansible, GH Actions, CIAB)

What is the best way to verify this PR?

Ensure that you are able to run traffic_ops either locally or with CIAB and run a few API to ensure they give the expected results. Will need to change your local cdn conf (traffic_ops_golang section) if you run traffic_ops locally).

If this is a bugfix, which Traffic Control versions contained the bug?

N/A

PR submission checklist

This PR has tests
This PR has documentation
This PR has a CHANGELOG.md entry
This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

codecov · 2023-05-01T19:23:17Z

Codecov Report

Merging #7450 (2163668) into master (daa7d21) will decrease coverage by 2.97%.
The diff coverage is 52.17%.

@@             Coverage Diff              @@
##             master    #7450      +/-   ##
============================================
- Coverage     33.31%   30.34%   -2.97%     
  Complexity       98       98              
============================================
  Files           499      790     +291     
  Lines         50832    82623   +31791     
  Branches        851      851              
============================================
+ Hits          16937    25076    +8139     
- Misses        32655    55435   +22780     
- Partials       1240     2112     +872

Flag	Coverage Δ
golib_unit	`48.58% <ø> (?)`
grove_unit	`4.60% <ø> (?)`
t3c_unit	`5.32% <ø> (?)`
traffic_monitor_unit	`21.28% <ø> (?)`
traffic_ops_integration	`69.42% <ø> (ø)`
traffic_ops_unit	`23.44% <52.17%> (+<0.01%)`	⬆️
traffic_portal_v2	`74.98% <ø> (ø)`
traffic_stats_unit	`10.14% <ø> (?)`
unit_tests	`27.43% <52.17%> (-1.41%)`	⬇️
v3	`57.79% <ø> (ø)`
v4	`79.18% <ø> (ø)`
v5	`78.63% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...affic_ops/traffic_ops_golang/traffic_ops_golang.go	`0.59% <0.00%> (+<0.01%)`	⬆️
traffic_ops/traffic_ops_golang/config/config.go	`36.20% <60.00%> (+0.22%)`	⬆️

... and 292 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

.github/actions/to-integration-tests/cdn.json

traffic_ops/install/bin/_postinstall.py

zrhoffman

Just some unused variables to clean up, besides that it all looks good!

infrastructure/ansible/roles/traffic_ops/templates/cdn.conf.j2

zrhoffman

It doesn't have to be in this PR, but in the future, we should remove this postinstall question, since it serves no purpose.

zrhoffman

Postinstall fails in CDN in a Box. Opening the ciab-logs artifact and inspecting trafficops.log:

trafficops_1       | 2023-05-12T18:22:47.518203240Z ++ /opt/traffic_ops/install/bin/postinstall --debug -a --cfile /opt/traffic_ops/install/bin/input.json -n --no-restart-to
trafficops_1       | 2023-05-12T18:22:47.591300943Z INFO:root:Starting postinstall
trafficops_1       | 2023-05-12T18:22:47.591504548Z INFO:root:Debug is on
trafficops_1       | 2023-05-12T18:22:47.591640751Z INFO:root:Running in automatic mode
trafficops_1       | 2023-05-12T18:22:47.591876057Z INFO:root:Using input file /opt/traffic_ops/install/bin/input.json
trafficops_1       | 2023-05-12T18:22:47.592441670Z INFO:root:Adding question 'Port to serve on?' with default answer 443
trafficops_1       | 2023-05-12T18:22:47.592617774Z INFO:root:Adding question 'ldap.conf location?' with default answer /opt/traffic_ops/app/conf/ldap.conf
trafficops_1       | 2023-05-12T18:22:47.592754277Z INFO:root:Adding question 'LDAP server hostname' with default answer 
trafficops_1       | 2023-05-12T18:22:47.592908081Z INFO:root:Adding question 'LDAP Admin Password' with default answer
trafficops_1       | 2023-05-12T18:22:47.593045684Z INFO:root:Adding question 'LDAP Search Query' with default answer 
trafficops_1       | 2023-05-12T18:22:47.593192587Z INFO:root:Adding question 'LDAP Skip TLS verify' with default answer 
trafficops_1       | 2023-05-12T18:22:47.593316090Z INFO:root:Adding question 'LDAP Timeout Seconds' with default answer 
trafficops_1       | 2023-05-12T18:22:47.593457794Z INFO:root:File sanity check complete - found 7 differences
trafficops_1       | 2023-05-12T18:22:47.593586597Z INFO:root:===========/opt/traffic_ops/app/conf/production/database.conf===========
trafficops_1       | 2023-05-12T18:22:47.594050207Z INFO:root:Database configuration has been saved
trafficops_1       | 2023-05-12T18:22:47.594551519Z INFO:root:===========/opt/traffic_ops/app/conf/production/tv.conf===========
trafficops_1       | 2023-05-12T18:22:47.594916027Z INFO:root:Database configuration has been saved
trafficops_1       | 2023-05-12T18:22:47.595348438Z INFO:root:Not setting up ldap
trafficops_1       | 2023-05-12T18:22:47.595526342Z INFO:root:===========/opt/traffic_ops/install/data/json/users.json===========
trafficops_1       | 2023-05-12T18:22:47.679911908Z INFO:root:===========/opt/traffic_ops/install/data/json/openssl_configuration.json===========
trafficops_1       | 2023-05-12T18:22:47.680141114Z INFO:root:===========/opt/traffic_ops/install/data/json/profiles.json===========
trafficops_1       | 2023-05-12T18:22:47.680560923Z INFO:root:Not generating openssl certification
trafficops_1       | 2023-05-12T18:22:47.680722927Z INFO:root:===========/opt/traffic_ops/app/conf/cdn.conf===========
trafficops_1       | 2023-05-12T18:22:47.683847700Z Traceback (most recent call last):
trafficops_1       | 2023-05-12T18:22:47.683861500Z   File "/opt/traffic_ops/install/bin/_postinstall.py", line 994, in generate_cdn_conf
trafficops_1       | 2023-05-12T18:22:47.683865500Z     existing_conf = json.load(conf_file)
trafficops_1       | 2023-05-12T18:22:47.683868800Z   File "/usr/lib64/python3.6/json/__init__.py", line 299, in load
trafficops_1       | 2023-05-12T18:22:47.683872300Z     parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
trafficops_1       | 2023-05-12T18:22:47.683875301Z   File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
trafficops_1       | 2023-05-12T18:22:47.683878501Z     return _default_decoder.decode(s)
trafficops_1       | 2023-05-12T18:22:47.683914801Z   File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
trafficops_1       | 2023-05-12T18:22:47.683929002Z     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
trafficops_1       | 2023-05-12T18:22:47.683932102Z   File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
trafficops_1       | 2023-05-12T18:22:47.683935102Z     raise JSONDecodeError("Expecting value", s, err.value) from None
trafficops_1       | 2023-05-12T18:22:47.683938102Z json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)
trafficops_1       | 2023-05-12T18:22:47.683941002Z 
trafficops_1       | 2023-05-12T18:22:47.683943702Z The above exception was the direct cause of the following exception:
trafficops_1       | 2023-05-12T18:22:47.683946702Z 
trafficops_1       | 2023-05-12T18:22:47.683949402Z Traceback (most recent call last):
trafficops_1       | 2023-05-12T18:22:47.683952202Z   File "/opt/traffic_ops/install/bin/_postinstall.py", line 1519, in <module>
trafficops_1       | 2023-05-12T18:22:47.683955702Z     ARGS.no_database
trafficops_1       | 2023-05-12T18:22:47.683958602Z   File "/opt/traffic_ops/install/bin/_postinstall.py", line 1322, in main
trafficops_1       | 2023-05-12T18:22:47.683961603Z     postgresTV = generate_cdn_conf(user_input[CDN_CONF_FILE], CDN_CONF_FILE, automatic, root_dir)
trafficops_1       | 2023-05-12T18:22:47.683964503Z   File "/opt/traffic_ops/install/bin/_postinstall.py", line 998, in generate_cdn_conf
trafficops_1       | 2023-05-12T18:22:47.683967503Z     raise exception
trafficops_1       | 2023-05-12T18:22:47.683970303Z ValueError: invalid existing cdn.config at /opt/traffic_ops/app/conf/cdn.conf: Expecting value: line 2 column 1 (char 1)
trafficops_1       | 2023-05-12T18:22:47.690645158Z Error on line 38 of /opt/traffic_ops/install/bin/postinstall
trafficops_1       | 2023-05-12T18:22:47.691045368Z +++ echo 'Error on line 216 of /run-go.sh'
trafficops_1       | 2023-05-12T18:22:47.691131170Z Error on line 216 of /run-go.sh
trafficops_1       | 2023-05-12T18:22:47.691199471Z +++ exit 1

rimashah25 · 2023-05-19T05:05:48Z

It doesn't have to be in this PR, but in the future, we should remove this postinstall question, since it serves no purpose.

What question is it?

zrhoffman · 2023-05-19T05:22:10Z

It doesn't have to be in this PR, but in the future, we should remove this postinstall question, since it serves no purpose.

What question is it?

Whoops, meant to link the Number of workers? question

trafficcontrol/traffic_ops/install/bin/_postinstall.py

Line 246 in 7390acc

Question("Number of workers?", "12", "workers"),

rimashah25 · 2023-05-19T05:29:40Z

Expecting value: line 2 column 1 (char 1)

I looked at the traceback and the line (998) where the error is being generated and it is not informative.
Expecting value: line 2 column 1 (char 1)

zrhoffman · 2023-05-19T18:03:19Z

Expecting value: line 2 column 1 (char 1)

I looked at the traceback and the line (998) where the error is being generated and it is not informative. Expecting value: line 2 column 1 (char 1)

The error is happening at the top line of the stack trace:

  File "/opt/traffic_ops/install/bin/_postinstall.py", line 994, in generate_cdn_conf
    existing_conf = json.load(conf_file)
  File "/usr/lib64/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

so _postinstall.py", line 994, in generate_cdn_conf

trafficcontrol/traffic_ops/install/bin/_postinstall.py

Line 994 in 7390acc

existing_conf = json.load(conf_file)

…ield since invalidationjobs (L671) uses URL field of Config struct.

zrhoffman

Looks great!

rimashah25 force-pushed the refactor/hypnotoad branch from ea75812 to 3987e17 Compare April 24, 2023 16:05

rimashah25 changed the title ~~Updated Hypnotoad~~ Update cdn config (Remove hypnotoad section and update listen field) May 1, 2023

rimashah25 marked this pull request as ready for review May 1, 2023 19:18

rimashah25 added the Traffic Ops related to Traffic Ops label May 2, 2023

ocket8888 added documentation related to documentation low impact affects only a small portion of a CDN, and cannot itself break one tech debt rework due to choosing easy/limited solution labels May 2, 2023

rimashah25 force-pushed the refactor/hypnotoad branch from b2a3606 to 1b02b8d Compare May 4, 2023 19:14

rimashah25 requested a review from zrhoffman May 9, 2023 15:53

zrhoffman requested changes May 10, 2023

View reviewed changes

.github/actions/to-integration-tests/cdn.json Outdated Show resolved Hide resolved

traffic_ops/install/bin/_postinstall.py Outdated Show resolved Hide resolved

rimashah25 force-pushed the refactor/hypnotoad branch 2 times, most recently from 91f5a2f to ab2cd04 Compare May 12, 2023 04:27

rimashah25 requested a review from zrhoffman May 12, 2023 18:08

zrhoffman requested changes May 16, 2023

View reviewed changes

infrastructure/ansible/roles/traffic_ops/templates/cdn.conf.j2 Show resolved Hide resolved

zrhoffman reviewed May 16, 2023

View reviewed changes

zrhoffman requested changes May 16, 2023

View reviewed changes

rimashah25 force-pushed the refactor/hypnotoad branch from 7e2b3bf to 7390acc Compare May 19, 2023 05:17

rimashah25 force-pushed the refactor/hypnotoad branch 2 times, most recently from 116eb96 to 2163668 Compare May 31, 2023 18:38

rimashah25 added 6 commits May 31, 2023 13:59

Removed Listen field and added key and cert fields.

24f3683

Removed Listen field and added key and cert fields-1

228839f

Corrected path in cdn.json files

74be1b4

Added listen section in traffic_ops_golang and removed key and cert f…

628b36d

…ield since invalidationjobs (L671) uses URL field of Config struct.

Updated CHANGELOG.md

75319c2

Fixed indentation.

94906a7

rimashah25 added 5 commits May 31, 2023 13:59

Addressed review comments.

52e94c2

Fixed traffic_ops_golang.go

fd50a31

Fixed postinstall issues.

f4b0579

Removed unused variable

cbcdc2d

adding comma and double quotes

c332c53

rimashah25 force-pushed the refactor/hypnotoad branch from 2163668 to c332c53 Compare May 31, 2023 19:59

rimashah25 requested a review from zrhoffman May 31, 2023 20:00

zrhoffman approved these changes May 31, 2023

View reviewed changes

zrhoffman merged commit d967201 into apache:master May 31, 2023

rimashah25 deleted the refactor/hypnotoad branch May 31, 2023 21:00

zrhoffman mentioned this pull request Jun 2, 2023

Postinstall unit tests fail: Incorrect key file #7508

Closed

mitchell852 added the configuration related to configuration - not limited to any one component label Jun 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update cdn config (Remove hypnotoad section and update listen field) #7450

Update cdn config (Remove hypnotoad section and update listen field) #7450

rimashah25 commented Apr 20, 2023 •

edited

Loading

codecov bot commented May 1, 2023 •

edited

Loading

zrhoffman left a comment

zrhoffman left a comment

zrhoffman left a comment

rimashah25 commented May 19, 2023

zrhoffman commented May 19, 2023

rimashah25 commented May 19, 2023 •

edited

Loading

zrhoffman commented May 19, 2023

zrhoffman left a comment

Update cdn config (Remove hypnotoad section and update listen field) #7450

Update cdn config (Remove hypnotoad section and update listen field) #7450

Conversation

rimashah25 commented Apr 20, 2023 • edited Loading

Which Traffic Control components are affected by this PR?

What is the best way to verify this PR?

If this is a bugfix, which Traffic Control versions contained the bug?

PR submission checklist

codecov bot commented May 1, 2023 • edited Loading

Codecov Report

zrhoffman left a comment

Choose a reason for hiding this comment

zrhoffman left a comment

Choose a reason for hiding this comment

zrhoffman left a comment

Choose a reason for hiding this comment

rimashah25 commented May 19, 2023

zrhoffman commented May 19, 2023

rimashah25 commented May 19, 2023 • edited Loading

zrhoffman commented May 19, 2023

zrhoffman left a comment

Choose a reason for hiding this comment

rimashah25 commented Apr 20, 2023 •

edited

Loading

codecov bot commented May 1, 2023 •

edited

Loading

rimashah25 commented May 19, 2023 •

edited

Loading