-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME
1678 lines (933 loc) · 103 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# NOTE: for implementation specifics, commands, directories and script details see notes.txt on the local VSCode repo or the EC2 controller repo. The notes.txt are not publishied to the github repo.
# Tools and process used in building this infrastructure include the following:
Base server: VPS server (linode) running latest Archlinux
Ownership of a domain. Mine is on Google Cloud DNS and NS will be delegated to linode domains so that it can be used for this project (similar to Route 53)
Linode VPS will be configured with a storage volume as well.
For network details etc. see notesx.txt file on EC2 controller
## High level list of Appication tools used in this project:
Linode VPS
SSL and TLS tools: certbot/letsencrypt for LXC containers non-HTTPS (like mail) and Traefik/letsencrypt for the Traefik HTTPS termination for all the containers (LXC and docker) (Web Admin on nginx in the containers)
Lots of tools installed on the VPS: docker engine and lxc to name a few.
Ansible is the deployment tool for all of the apps
Wireguard on iPhone and WIndows10 client. Mac is outdated and not supported with Wireguard client.
The VPN is necessary as an added layer of security.
Mariadb (mysql) on VPS. Note some of the containers have their own mysql db and others utilize the
mariadb on the host (note: ip tables has to be configured accordingly)
iRedMail (LXC container)
Zulip messaging (LXC container) Zulip is similar to slack
Traefik (docker container) reverse proxy and SSL termination point for all HTTP/HTTPS traffic going into the LXC and docker containers. Note for iRedMail native mail protocols (IMAP and SMTP) certbot/letsencrypt is used to generate the certs; on Traefik letsencrypt is exclusively used without certbot.
pi-hole (docker container) as a DNS resolver on all Wireguard clients when the VPN tunnel is up. This filters out ads, etc.
Nextcloud (docker container) for doc sharing, and much more
checkmk (docker container) for monitoriing of the VPS and emailing status through an SMTP
Borg and Borgmatic
Gitlab and Gitlab runner
Python backup-checker
Cronjobs are used for backups for all of the LXC and docker containers as well as the VPC mariadb. An addional 40GB volume for backups has been acquired from linode for this purpose. (/mnt/storage)
## git repository architecture
Use the local VSCode on Mac as local repo (this does not have security senstive files as it is a pull from github)
Github repo (this does not have security sensitive files)
EC2 controller. Running ansible-playbook and venv from here to provision the VPS. This has the full repository source code including sensitive files and folders.
Use EC2 controller only, to provision the VPS. Once changes are tested push to github and then pull the changes to local VSCode on mac for local record keeping.
## Ansible:
Primary tool for provisioning the apps onto the linode VPS will be ansible (ansible-playbook will have respective roles for the various apps and services added to the setup.yml in the root. See github repo)
I will be running the ansible client from an EC2 ubunut controller. Mac as well would be fine, but i prefer the controller as it has many other devops tools that may be required. It is fully AWS configured as well but we will not be using and AWS, GCP or Azure for this project
## Running virtual environment on the EC2 controller:
KEY NOTE: run all of the ansible from a virtual environment. I will be using venv. Set up the python env in the terminal that will run the venv prior to creating and running the venv. The venv will inherit the python version which is critical for the ansible to work with the latest Archlinux VPS on linode.
The ansible matrix is located here:
https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#ansible-core-support-matrix
For the latest Archlinux: it is running python 3.12.5. This is the ansible target
One possible ansible client/python version to run on the venv on the EC2 controller is ansibile 10.3.0/core 2.17.3 with python 3.12.5 running in the venv.
To install the proper ansible version and python version in the venv use the following process. I prefer this over using penv and virtualenv which was causing issues with python3 and python versiions in the virtualenv not being aligned.
## Setting up the venv:
on EC2 controller (ansible client):
mkdir tmp_ansible && cd $_
pyenv install -l
pyenv install 3.12.5
pyenv shell 3.12.5
pyenv local 3.12.5
python -V should show 3.12.5 in this terminal
python -m venv latestenv
source latestenv/bin/activate
which python should point to venv path
which pip should point to venv path
python -V should state 3.12.5
pip install ansible==10.3.0 OR just pip install ansible <<< it will get latest which is 10.3.0
pip freeze should show same as you have
This ansible version and python version runs very well with latest Archlinux linode VPS (python version is 3.12.5 as well)
There are other combinations that will work in the venv but the above is ideal since it is the latest ansible version GA.
## ansible playbook files:
The ansible playbook and the roles as indicated in setup.yml are included in the github repository.
There are some files that have been intentially gitignored because that are clear text passwords or keys. Also ansible encrypted password files (ansible-vault was used) are also not included in this githup repo. The complete files and folders are on the EC2 contoller.
## Ansible essentials:
this includes provioning the base linux packages on the Archlinux VPS
installing cronie for the cron jobs that will be used throughout
Configuring the network bride interface on the VPS that will be used for the LXC linux container communication with the host VPS
Configuring the loopback interace on the VPS that will be used for docker container to host VPS network communication and some other things
## Basic VPS security (iptables firewall):
iptables are used to lock down the VPS to SSH and some other stuff as required as the applications and services and servers are added. SSH is running on a proprietary port number to further mitigate attacks. 9473 is addded for the wireguard VPN tunnel (see below) which is required to run secure client to server communication to native services and servers like mariadb that do not support native encryption. Wireguard will use private ip addresses to communcate with the WireGuard server on the VPS inside the tunnel. These private ip addresses can then be used on the iptables and thus are allowed through to the VPS server. This is important.
Note that the nat rules in iptables will be added dynamically for docker containers that are brought up on the VPS in accordance with the private address space on the VPS (for docker outbound connections, i.e. MASQUERADE and docker inbound connections based upon the docker assigned ip address)
Kernel params on the Archlinux require tweaking for the ip forwarding for docker containers to use the aforementioned loopback address for docker container to host network communicatiion.
## NOTE on LXC linux containers vs. docker containers usage:
The docker containers will be used for mostly HTTP/HTTPS (nginx) stuff like hosting the gitlab whereas the LXC linux containers will be used for non-HTTP services like iRedMail and Zulip (collaboration tool similar to Slack for notifcations, for example of pipeline results)
LXC linux conatiners will also run HTTP as well (nginx) if the application reqiures full fledged linux file system for installation.
## Wireguard security and Traefik:
Wireguard plays a critical role in the security as noted above. Wireguard server on the VPS will be run in conjunction with Traefik for traffic steering to the backend docker containers (and HTTP LXC containers running HTTP/HTTPS services. Encryption (HTTPS) will run to the Traefik middleware and this middlewhere will whitelist the private ip address wireguard clients (Windows and iPhone for me; some using Mac) and then run the native HTTP to the docker containers or LXC containers. There is no need for backend encryption at this time.
Traefik is an HTTPS reverse proxy with HTTP on the backend to the containers. It is a TLS termination point. The containers that it will steer traffic to (docker and LXC linux containers) will be running nginx.
Traefik has an ACME client built into it so the Let's Encrypt TLS certificates can be generated as needed.
## Wireguard VPN packet flow design, example:
VPS Wireguard server tunnel endpoint will have a private ip address
Wireguard clients will have private ip addresses on this same network
Public VPS linode server adddress is the public Wireguard VPN endpoint
The Public client tunnel endpoint will be the Comcast router public IP address of my home network
iptables on the VPS needs to have explicit rules added for the private ip addresses of the clients as source allow all. The tunnel is shaved off and then iptables kicks in, so iptables will see the private client ip addresses and these need to be allowed accordingly.
Example traffic flow from Wireguard client to gitlab docker container service:
Example Gitlab will be running in a native docker container (not LXC that is for linux containers)
Traefik will be the endpoint of the TLS to the backend docker containers (HTTP and HTTPS)
Traefik will have a whitelist middleware to allow only certain ip addresses to access gitlab web site. For this example 10.100.94.11 and 10.100.94.12 VPN clients
DNS will be configured for gitlab.linode.cloudnetworktesting.com
This will be the public ip address of the VPS. In my case 173.230.155.238
On the PC will set up a wireguard rule that for gitlab.linode.cloudnetworktesting.com will use the VPN tunnel.
The process flow is gitlab.linode.*****cloudnetworktesitng*****.com (my dowmain) in the PC browser
It resolves to the ip of the VPS
The PC network configuration with rule for VPN will use the wireguard tunnel for this gitlab.linode.*********.com
Once tunneled and then deencapsulated by the Wireguard server on the VPS (linode) the packet will be forwarded to the HTTPS reverse proxy, traefik, to terminate the HTTPS connection and send the native HTTP to the backend Gitlab that is running on the docker container on the VPS (docker containers will use the loopback interface, not the bridge. The bridge is for native linux containers running on LXC)
Traefik is used for all HTTP/HTTPS communication. Non-HTTP will use certbot certificate wildcard.
Traeik sees packet sourced from private IP client wireguard address. It will check the ip whitelist middleware for that domain, sees the source ip is present and lets the traffic through to the docker container running gitlab.
## certbot, ACME protocol, wildcard TLS certs:
Certbot will be used for non-HTTP services like iRedMail and Zulip. Certbot can be used to generate wildcard certs for my domain and this makes secure communication with these types of services and servers more practical. Traefik will not be used to terminate the TLS for these servcies and servers.
Certbot is an ACME client and Let' Encrypt will challenge it with a long string and certbot will be able to respond by creating the DNS entry for the domain because it will be given access to do.
To update Linode DNS via certbot we will need Linode’s personal access token so certbot has access to Linode DNS settings and configuration to make the change and answer the challenge from Let's Encrypt.
## Mariadb:
Mariadb is installed now and running on the linode VPS server. Running mariadb from wireguard clients is the way to do this. Do not run mariadb from non-Wireguard client like EC2 controller because the mariadb server on the VPS does not support native TLS by default. For security I am using ansible vault for the mariadb root password and database user password so that ansible can be used to configure this server (automated)
Mariadb will be used by backup-checker to do the backups on a cronjob.
For details on commands, networking, etc, see notesx.txt
## LXC and Linux containers:
As stated above some servers and applications like iRedMail and Zulip will need to be installed and run on LXC linux containers rather than docker containers. Although there are solutions now with docker containers, the native full file system linux available in LXC linux containers will make installation of these servers and running these servers more practical.
We can use certbot TLS certs to securitize the communication with remote hosts.
The ansible LXC role will actually install the lxc application on the VPS so that LXC containers can be created for iRedMail and Zulip
The ansible role is called lxc in the setup.yml
Creating and Provisioning the LXC containers will be done manually. Once the LXC container is created there is some additional configuration for assigning the ip address to the container from the bridge network as well as configuring /etc/resolv.conf (with 1.1.1.1 cloudfare.com as the nameserver) so that DNS resolution can be done from the mail.linode.cloudnetworktesting.com container. DNS is required in order for iRedMail application to work properly.
For network details see notesx.txt on the EC2 controller. The LXC containers will use the bridge interface br0 for the subnet on which to create the LXC containers. The iRedMail will use 10.100.100.11/24 and the Zulip will use 10.100.100.12/24
notesx.txt has all the details on the full process, directory paths for configuration on the VPS and the containers themselves, etc.....
## This completes the basic infrastructure setup
## For SSH network details and configuration and files see notesx.txt
Use the ProxyJump or the ProxyCommand in the ~/.ssh/config file on the EC2 controller and add the public EC2 VPS cert to the mail.linode.cloudnetworktesting.com container in the ~/.ssh/authorized_keys. This will make ssh directly from the EC2 controller into the iRedMail LXC container possible.
## Other servers, apps and services required to get the DevOps infra up and running (these will be used by end users):
These include: (as mentioned earlier)
iRedMail
Zulip collaboration tool
Traefik as mentioned above
Pi-Hole for DNS
Nextcloud
checkmk
Borg and Borgmatic
Gitlab and Gitlab runner
Python backup-checker
Each of these are cited below in terms of the infrastructre design and pipeline architecture.
## iRedMail:
The installation of this is complex. There is an LXC mariadb that is not associated with the linode VPS. There is also a nginx service to serve the webmail application. The mail can be accessed through the web via either Roundcube or Web admin panel (iRedAdmin). THe URLS are in notesx.txt file.
The bundle also includes iRedAPD - Postfix Policy Server
It uses Dovecot an open-source email server that acts as a mail storage server, mail proxy server, or lightweight mail user agent (MUA)
The notesx.txt file has the complete installation process including how to configure the LXC container prior to installing iRedMail and configuring DNS for the mail server as well as a TLS cert for the service via certbot.
### Additonial details about the implementation:
Use a certbot certificate as this is for non-HTTP traffic. The HTTP/HTTPS traffic will use Traefik and those certs will be installed on there.
For this generate a new certbot cert for mail.linode.cloudnetworktesting.com iRedMail domain
Once the certs are created and in the letsencrypt/archive and live directories on the VPS they then have to be tranferred onto the iRedMai LXC container
An lxc-mount must be added to the lxc config file on the VPS for both live and archive directories
Reboot the LXC container and verify that the certs are on the container
See notes.txt for specific directory details, etc.
Next edit the Postfix and Dovecot services on the LXC container to use these certificates
Restart both services so that the certs are loaded.
### User configuration and iRedMail Web Admin console
At this point we should be able to hit the Web Admin site for iRedMail through the browser on TLS/HTTPS
It fails from non Wireguard VPN because VPS does not have a webserver (nginx) yet to forward the traffic to the LXC container
However putting the LXC container address in the browser to the iRedMail LXC container at LXC private container address should work
But the Wireguard VPN has to be up on the Windows client
First add the LXC container entire private subnet to the Wireguard allowed IP list on the Windows client
This works because the iRedMail container is already running the nginx web server on it
The self signed cert is fine to accept
Must use firefox. Micrsoft edge and chrome will not accept the self signed certbot cert
Log into the iRedMail Web Admin console using the Web Admin url and the private ip address of the LXC container. The user is the postmaster user.
Add users to the email domain linode.cloudnetworktesting.com
This will include several users that will be used for notifications, personal email, reports.
NOTE that the MX record will redirect linode.cloudnetworktesting.com to mail.linode.cloudnetworktesting.com
### iptables
iptables nat rules need to be added for 139, 587 and 25
The NAT rules are required to redirect the traffic coming into the VPS to the iRedMail LXC container.
Run the ansible-playbook security role once the iptables have been added to the playbook
Verify that the iptables nat rules have been added to the VPS with iptables -t nat -nvL (iptables -nvL will just show the base rules)
Once the rules are allowed run telnet tests from EC2 controller or mac laptop to ports 587, 139 and 25
NOTE: for port 25 need to raise support ticket to open port 25 up. They block outbound 25 by default for spam protection
Even with the port 25 opened, the telnet still fails because Comcast/Xfinity and AWS EC2 both block inbound and outbund port 25
The way to test port 25 is to use the Windows Wireguard VPN client and tunnel the port 25 telent through the blocks.At the VPS the packet will be de-encapsulated and nat'ed to the LXC private container address. The source ip will be a private wireguard ip address and this is ALLOWED all by the iptables as well.
The packet to the private ip LXC container ip is routed throiugh a bridge interface on the VPS routing table to the LXC container.
The bridge interface is used for all LXC containers on the VPS (loopback address on the VPS is used for docker containers)
### Thunderbird clients and testing the email services with inter-domain sending and receiving of email
Add Thunderbird clients to the Mac and windows laoptops and configure the notifications user on the Mac and a personal email user on the Windows.
The users should be able to be verified as they were configured earlier on the iRedMail via the Web console above.
Send the email from notifications acccount to check-auth@verifier.port25.com
If sucessful will get Authentication Report:
This checks the DKIM DNS record as well as the SPF (TXT) DNS records and verified the inter-domain email reception on the notifications Thunderbird account.
Summary of Results
SPF check: pass
"iprev" check: pass
DKIM check: pass
At this point the email server is fully functional
Ran additional tests with inter-domain emails to gmail account (to and from) the linode.cloudnetworktesting.com account and it is verified.
## Zulip collaboration:
NOTE: can use the Windows10 client and iphone as wireguard clients to test this out.
All traffic must be done on wireguard clients. The zulip.linode.cloudnetworktesting.com site is not reachable outside of the VPN tunnel. Iphone and window10 client both have wireguard client installed on them.
Zulip is installed as an LXC linux container and not a docker container. See the notes above on the LXC container architecture and design for this project, including the networking considerations. Of note is that the LXC containers use the bridge interface br0 on the VPS rather than the loopback ip address for routing betweeen the LXC containers and the VPS host. (The docker containers use the loopback ip address on the VPS)
The installation and design for teh Zulip is very similar to iRedMail given that both use LXC containers for deployment. Both iRedMail and Zulip have container addresses on the private interface address 10.100.100.x/24. The installation directories for both of these containers is in the /root/lxc folder. Each container has its own dedicated folder (FQDN of the container). lxc-create command is used to create the container.
Once the ip address is assigned, the lxc-start starts the container. The lxc-attach is required as some of the configuration (as in iRedMail) needs to be done in the container itself. The DNS configuration, as with iRedMail, is done manually. (not done through ansible). LXC containers ar difficult to fully automate, unlike docker containers.
For this zulip installation the OS of the LXC container needs to be upgraded. openssh-server needs to be installed as well. With the DNS set up we will be able to ssh into the LXC container directly from the EC2 controller using the ProxyJump command in the ~/.ssh/config file on the LXC container. Similar to what we did for iRedMail. The proxy in this cases is the VPS. So EC2 controller to proxy VPS to the LXC container on the VPS. (See the secction on ssh configuration in the notes.txt file on the EC2 controller and local VSCode repo. It is not pushed to github for security reasons).
The backup script to /mnt/storage and the cronjob are set up in the ansbile playbook zulip role much in the same manner that was used for iRedMail. The shell script is esssentially the same.
Now that the ansible scripts are done, the installation of zulip must be manual. We have ssh directly to the LXC set up, so this can be done easily.
NOTE: at this stage in the project we do not have Traefik docker container up (with nginx) so there is no https daemon running on the VPS itself. But zulip service has a bundled http server in it. For now, with Wireguard VPN point the FQDN for zulip to the private LXC container ip adddress to bring up the Web Admin console so that we can continue configuration of the zulip server. We did the same for iRedMail Web Admin console. However, with zulip there is Host header checking so the /etc/hosts file on the Windows Wireguard VPN client needs to temporarily point to the private ip for the FQDN of zulip.
The email admin and notifications needs to be set up: one for the zulip administrator and one for no-reply. See the word doc for specifics on the configuration for this and the changes required in settings.py There are also changes on the iredmail LXC container conifguration that need to be made for zulip emails so that iredmail can receive emails from no-reply emails from zulip.
## Traefik:
### Traefik, PART1:
#### TLS and HTTPS high level security for this project:
Traefik is a reverse proxy and TLS termination proxy and also can be used as a loadbalancer
It is critical on this setup because it is the only way to access HTTPS into the VPS
It will be deployed as a docker container and not an LXC container so it will use the loopback address on the VPS rather than the bridge interface
It will route the traffic the appropriate docker container or LXC container on the backend.
Traefik:
For zulp and for iRedMail it will be used so that the web admin consoles for each can be reached through the public interface. Right now we are getting the web console of both via going to the private ip address. The certs used for this are not the certbot certs created for iRedMail for example, but the default nginx TLS certs that are with the zulip and iRedMail nginx instances (they are both self signed certs)
For the actual mail the iRedMail cert uses the certbot cert
For webadmin is it using a self signed cert from GuangDong (see above) for now until Traefik is up
For zulip it is also a self signed cert for the nginx (webadmin) (see above) until Traefik is up
For zulip, traefik will also be used for client to server communication for the actual messaging.(unlike iRedMail which uses certbot cert for the actual mail sending and receiving)
Traefick will have own ssl/TLS certs and will expose the public ip address
#### Docker container for Traefik
Traefik will run as a docker container on the VPS
the loopback address on the VPS will be used to route the traffic coming into the public VPS interace to the the traefik docker container.
The good thing about docker is that the iptables rules will automatically be added to the iptables as the docker container is configured and run. So the port 80 traffic and the 443 traffic will be allowed through the VPS interface and into the loopback interface to the docker container.
When the container is started and run we will map 80 on VPS to 80 on the container and 443 on the VPS to 443 on the container. See below.
[root@vps ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d8e96b83e26c traefik:latest "/entrypoint.sh --lo…" 17 hours ago Up 17 hours 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp traefik.linode.cloudnetworktesting.com
NOTE: the traefik is a reverse proxy and TLS termination point and will be used to redirect incoming HTTPS traffic to the VPS to the appropriate LXC container below using the file provider. More on this below (PART3)
#### Traefik docker container design:
From the traefik docker-compose.yml file:
#the commands below will be passed to the traefik binary. If running traefik command from the command line these would be the
#arguments and flgs to start the traefik service on the docker container
#enable the dashboard
#set up http and https entrypoints on ports 80 and 443
#enable both docker providers (for redirection to the gitlab docker container) and file providers (redirect to LXC containers
#like iRedMail and zulip
#Set up automatic certs with let's encrypt
#The ports are the -p in docker run, i.e. mapping host 80 to docker container 80 and host 443 to docker container 443
#docker.sock is used so docker can communicate with docker engine and monitor for docker labels. These are required for the
#docker provider redirection by traefik to docker containers (using docker providers)
#acme.json is for cert storage
#./configs is for the file provider which is used to redirect to LXC linux containers like iRedMail and zulip
#This traefik container is part of the docker web network
#Docker labels section below:
#the last part is the setup for the lables that will be used by the docker provider: basically set up a router to match a
#certain hostname, then redirect http to https, set up lets encrypt as the TLS resolver,
#Finally set up the whitelist and basic auth. Since the release of Traefik 3.0, 'ipwhitelist' is deprecated and is now called
#'ipallowlist' This has been corrected below. Also '${service}-whitelist' has been renamed '${service}-allowlist', and
#'${whitelist}' should become '${ip_allowlist}' All of these corrections have been done below and are in the latest git repo.
#Note in Traefik 3.0 and up the hostregexp is '.+' which is already corrected below
#NOTE: the file providers directory is specified in the docker-compose.yml file (see providers.file.directory=/configs/)
#so that the container knows which directory is for the file providers. The /configs directory is copied over from source code
#to the container as part of the ansible plabyook. Finally, in the docker-compose.yml file note that the /configs directory
#from source is mounted to the traefik docker container as a volume.
#### Routing to LXC containers:
Full network diagrams are available in the word doc.
See below. Traefik will be used to route the traffic to the zulip LXC container
Traefik providers route traffic to endpoints
File providers are used specifically for linux LXC containers. We will use these to route traffic to the web admin servers for LXC iRedMail and LXC zulip and zulip can use this also for HTTPS messaging traffic as well (client to server communication)
The file provider will tell it for zulip, for example, route the traffic to the LXC at 10.100.100.12
#### Routing to Docker containers:
For gitlab it will be a bit different (docker container)
A docker provider instead of a traefik provider will be used
Docker labels will be used in traefik to add to the traefik config so that it can route the traffic to the gitlab docker container
### Traefik, PART2:
This involves adding the traefik role to the ansible-playbook. This part of the ansible playbook will deploy the traefik docker instance as noted above and below:
[root@vps ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d8e96b83e26c traefik:latest "/entrypoint.sh --lo…" 17 hours ago Up 17 hours 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp traefik.linode.cloudnetworktesting.com
An A record for the docker container, traefik.linode.cloudnetworktesting.com must be added to the linode Domains manager. The FQDN will be used in the browser for the Traefik Web Admin console access. Note that ansible playbook will also configure the password (add this .env file to the .gitignore, even though the password is entered in encoded format)
Note that the docker network for this docker container is named 'web' as this traefik docker container is for all HTTP/HTTPS traffic that needs to get into the private VPS network.
The docker-compose.yml file is used to bring up the traefik docker container. See the file on the github repo for comments regarding details of the code in this file.
/traefik/.env will have all the values of the variables used in the docker-compose.yml file.
Note that the ip_allowlist (formerly whitelist) covers the entire private ip subnet range.
Once the docker container is up after running the playbook. the Traefik Web UI Admin console should be able to be reached from the Windows Wireguard VPN client using the dummy A record created above with the private IP. This is just a temporary workaround to get into the Traefik web admin console until the console is reachable via the public ip and FQDN (see PART3 below)
#### Detailed packet flow with Traefik
When we enter https://traefik.linode.cloudnetworktesitng.com it resolves to public ip of VPS
Packet hits VPS and 443 is a match for the docker container on the VPS listening on 443. See below from VPS
[root@vps traefik]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d8e96b83e26c traefik:latest "/entrypoint.sh --lo…" 20 minutes ago Up 20 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp traefik.linode.cloudnetworktesting.com
the 443 host is mapped to 443 on traefik docker container.
The packet is forwarded using the loopback interface to the container on 443.
The third router below is activated because host header has traefik.linode.cloudnetworktesting.com
Click on the router in the web console and you can see the below
traffic@docker is the router
Then the 2 middlewares are hit: first the ipallowlist. The source must be from the 10.100.100.x subnet (whichi it is if from wireguard VPN on windows). If this fails you will get Forbidden and no basic auth challenge. See above when I tried going in on nonVPN client (Mac).
Next basic auth which is the admin user and ******* password.
Finally the service api@internal is hit. This will route the packet to the final destination. For example if there is redirection to the iRedMail LXC container or the zulip LXC container. These will be handled in next section (PART3) under the ansible-playbook configs folder under traefik directory. LXC containers use the file provider (whereas redirection to docker containers use the docker provider and labels. Gitlab will use this)
Once we get the redirection up on to iRedMail and Zulip we will be able to web Admn in via the public ip addresses (but still must use VPN because of the traefik whitelist)
### Traefik, PART3:
#### Introduction:
Using the file provider to redirect traefik traffic flows to the appropriate LXC container based upon the file provider in Traefik. More on this later. NOTE: the docker provider will be used to redirect traffic flow to docker containers using labels. This will be done for a gitllab docker container. For now the focus is on the 2 LXC containers for iRedMail and Zulip.
[root@vps ~]# lxc-ls -f
NAME STATE AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
mail.linode.cloudnetworktesting.com RUNNING 0 - 10.100.100.11 - false
zulip.linode.cloudnetworktesting.com RUNNING 0 - 10.100.100.12 - false
[root@vps ~]#
This is all about getting public access to the LXC containers. Right now, we are going directly to the 10.100.100.x private ip addresses in the browser through the VPN
Since traefik whitelists 10.100.100.x only, we still have to have the VPN up but we will now be able to get to the Web Admin of iRedMail and Zulip through the FQDN and public ip address.
Note the SSL/TLS certs for this HTTPS traffic is in Traefik and NOT certbot. Certbot is only used for the native apps like iRedMail mail protocols.
All web traffic passes through the Traefik as SSL termination point. The Traefik will redirect the cleartext HTTP traefik to the proper LXC container. The backend can be done through HTTPS but if the network is that compromised, there are other security problems. In general the backend does not need to be encrypted.
The file providers are in the traefik/traefik/configs/ part of the ansible-playbook as mail.yml for redirection to the iRedMail LXC container. and zulip.yml for redirecton to the Zulip LXC container.
#### Ansible plabyook for the file providers:
Currently attempting to get to https://mail.linode.cloudnetworktesitng.com admin page will result in 404 because traefik dose not know how to route the traffic for this Hostname URL.
This is the purpose of the file providers section in ansible, to configure traefik with the redirection configuration to do this.
For example, the mail.yml has the routing for traefik to the backend iRedMail LXC container. Note that it is Host header based and the redirection endpoint is provided at the bottom as services: mail: loadbalancer: servers: url: with the private ip of the iRedMail LXC container.
Once this is configured on the traefik docker container this will permit public ip address HTTPS traffic for the Web Admin URL to get redirected to the iRedMail container. Note that the source ip is still whitelisted so the Wireguard VPN must be up.
NOTE: the file providers directory is specified in the docker-compose.yml file so that the container knows which directory is for the file providers. The /configs directory is copied over from source code to the container as part of the ansible plabyook. Finally, in the docker-compose.yml file note that the /configs directory from source is mounted to the traefik docker container as a volume.
NOTE: Important note that this is specifically for the public access to the iRedMail web admin via HTTPS. This is not for sending and receiving email.That is through mail protocols 587, 143 and 25 (SMTP and IMAP). This email is encrypted with certbot cert and not the SSL certs on traefik
After running the ansible playbook the https://mail.linode.cloudnetworktesting.com admin page now works
#### The same needs to be done for the Zulip admin web page
File provider needs to be added under traefik/traefik/configs/zulip.yml for the Zulip web admin traffic redirection
## Pi-Hole:
Pi-Hole is a network ad and tracking blocking application
We will use the Pi-Hole as DNS server for wireguard. So when VPN is up **all** DNS requests going through the tunnel will use Pi-Hole as DNS Resolver
All ad related URLs will get 0.0.0.0 response from Pi-Hole (blocked) and all legitimate will get forwarded to the DNS server configured in Pi-Hole (1.1.1.1 cloudflare.com) and resolved
1.1.1.1 is a public DNS resolver operated by Cloudflare that offers a fast and private way to browse the Internet. Unlike most DNS resolvers, 1.1.1.1 does not sell user data to advertisers. In addition, 1.1.1.1 has been measured to be the fastest DNS resolver available.
In linode Admin console need to first add an A record for pi-hole: pi-hole.linode.cloudnetworktesting.com
For ansible playbook, in pi-hole/tasks/main.yml copy over the source from the EC2 controller pi-hole directory to the /root/service/pi-hole directory on VPS so VPS can create the pi-hole docker container. We will use a docker container for pi-hole thus routing to it will use the loopback interface on the VPS and not the bridge interface. See earlier sections above for differences between LXC containers and docker containers packet flow and configuration.
The main.yml also has the backup shell script lcoation and the cronjob configuration for the docker container's backup. All LXC and docker containers are backed up as well as mariadb dump to the backup. /mnt/storage is 40GB external volume with linode that has all the backups.
The docker-compose.yml is under pi-hole/pi-hole/docker-compose.yml
Notes on the docker configuration and design are below:
### Pi-hole Docker container design:
#the labels below are how traefik routes or steers the traffic for this pi-hole docker container when it comes to it on the public VPS interface
#The ${} are variables and will be defined in the pi-hole/pi-hole/.env file (note this is added to .gitignore)
#pi-hole container will be in the docker network web just like traefik is.
#Recall that the VPS uses the loopback ip address 10.36.7.11 to route the traffic to docker containers that are on another different private network (like 172.21.x.x)
#The pi-hole DNS ports are 53 tcp and udp and 67 udp, but 67 only used if pi-hole is acting as a dhcp server as well
#These ports are mapped from VPS to the container through the loopback ip address
#The etc-pihole and etc-dnsmasqd and etc-hosts directories will be mapped to volumes for persistency
etc-hosts and /etc/hosts files:
#The etc-hosts has a unique function in this setup: pi-hole container hostname set to same name as the A record that maps to the
#public VPS ip adddress (in linode domains: pi-hole.linode.cloudnetworktesting.com) then there will be a problem
#The docker container will assign the private docker container ip address 172.21.x.x to the hostname.
#Need a custom /etc/hosts file mounted as a docker volume from the /etc-hosts on the container to specify the host to ip mapping
#using the public VPS address. If we do not create our own /etc/hosts then pi-hole will create its own /etc/hosts inside the container
#(not in a volume) and it will use the private ip container address and not the public VPS address
#If pi-hole is set as the DNS server (as with Wireguard clients), when asked to resolve pi-hole.linode.cloudnetworktesitng.com, if
#there is no /etc/hosts volume from the /etc-hosts with public VPS, it will use its local /etc/hosts that has the private container
#ip address. This will cause the wireguard clients to fail because the private address cannot be routed
#With the volume /etc/hosts from the /etc-hosts with the public VPS ip address, it will not create its own /etc/hosts and this volume
#/etc/hosts will be used to resolve pi-hole.linode.cloudnetworktesting.com to the proper public VPS ip address.
#This will resolve as if it used the upstream name server (linode domain server). This is all so that we can use traefik to route
#to the public ip address via HTTPS through the whitelist Wireguard. This won't work unless the public ip is used.
### docker labels for traefik based routing:
#The labels below will be used by traefik to route the incoming HTTPS443 packets to the pi-hole docker container based upon the
#Host header specified as Host below. Very similar as the file providers that were used for LXC containers iRedMail and zulip,
#except here the docker provider is used by traefik along with these labels, so there are no explicit config file provider files to
#be added to ansible. It is all done on the fly for docker containers based upon the labes below.
#The port 80 is the webserver port on the pi-hole docker container running the webadmin GUI.
### /pi-hole/pi-hole/etc-hosts file in the ansible-playbook:
#The etc-hosts has a unique function in this setup: pi-hole container hostname set to same name as the A record that maps to the
#public VPS ip adddress (in linode domains: pi-hole.linode.cloudnetworktesting.com) then there will be a problem
#The docker container will assign the private docker container ip address 172.21.x.x to the hostname.
#Need a custom /etc/hosts file mounted as a docker volume from the /etc-hosts on the container to specify the host to ip mapping
#using the public VPS address. If we do not create our own /etc/hosts then pi-hole will create its own /etc/hosts inside the container
#(not in a volume) and it will use the private ip container address and not the public VPS address
#If pi-hole is set as the DNS server (as with Wireguard clients), when asked to resolve pi-hole.linode.cloudnetworktesitng.com, if
#there is no /etc/hosts volume from the /etc-hosts with public VPS, it will use its local /etc/hosts that has the private container
#ip address. This will cause the wireguard clients to fail because the private address cannot be routed
#With the volume /etc/hosts from the /etc-hosts with the public VPS ip address, it will not create its own /etc/hosts and this volume
#/etc/hosts will be used to resolve pi-hole.linode.cloudnetworktesting.com to the proper public VPS ip address.
#This will resolve as if it used the upstream name server (linode domain server). This is all so that we can use traefik to route
#to the public ip address via HTTPS through the whitelist Wireguard. This won't work unless the public ip is used.
### Traefik http router for pi-hole HTTP/HTTPS Web Admin traffic:
Once the ansible playbook is executed a new HTTP router will be configured on Traefik (See Traefik web console through the Wireguard VPN)
With this in place we can log into the p-hole Web Admin console.
The Upstream DNS servers should show the Cloudflare DNS server because that was configured through the ansible playbook.(.env file)
from the docker-compose.yml file note the traefik http/https trafifc routing for Web Admin:
#NOTE the port 80 in traefik labels below:
#NOTE that there is no iptables rule for port 80. This is in the docker-compose.yml file for the pi-hole. Traefik uses the labels # on the containers to route the traffic for docker containers for HTTP/HTTPS. The 53 DNS traffic for pi-hole on the other hand# needs the iptables nat rules to redirect from loopback to the docker container pi-hole private ip address.
#But this port 80 is allowed through (initial pi-hole dashboard) and there is a redirection on the pi-hole nginx web server to redirect to port 443. Once the # redirect to 443 occurs the HTTP router for pi-hole then challenges the user for basic auth and routes the traffic to port 80 and container ip address of pi-hole based # upon the Host header in the 443 packet.. The 80 below is just for the initial hit to the pi-hole through Traefik. The redirection # is where the traefik HTTP router kicks in based upon the Host header.
### Detailed packet flow of DNS requests with Wireguard clients configured to use pi-hole as DNS resolver:
For testing the pi-hole as DNS resolver must do this on a Wireguard VPN client because regular clients will not use pi-hole as DNS resolver
For example on EC2 controller metrics.icloud.com will show several answers to drill metrics.icloud.com
Going onto to the Windows client without VPN up an nslookup on metrics.icloud.com will show the same.
Next in the Wireguard configuration we set DNS=10.36.7.11 which is the VPS lookback. Recall that the loopback is how VPS routes traffic to the docker containers. 10.36.7.11 will only be reachable through the wireguard VPN. There are iptables nat rules in the iptables configured on VPS so that the DNS traffic to 10.36.7.11 is routed to the pi-hole docker container.
3 156 DNAT 6 -- !br-3db65a09fb68 * 0.0.0.0/0 10.36.7.11 tcp dpt:53 to:172.21.1.3:53
5407 447K DNAT 17 -- !br-3db65a09fb68 * 0.0.0.0/0 10.36.7.11 udp dpt:53 to:172.21.1.3:53
0 0 DNAT 17 -- !br-3db65a09fb68 * 0.0.0.0/0 10.36.7.11 udp dpt:67 to:172.21.1.3:67
Once this is set up, running nslookup metrics.icloud.com on Windows10 client with VPN up will show 0.0.0.0, meaning the ads are effectively blocked
Note the ip tables nat rules above. These were installed automatically by docker when the docker container was created. This is why docker containers are ideal for certain apps like pi-hole
The Wireguard tunnels the inner packet with source 10.100.94.11 and dest 10.36.7.11 to the VPS. It is decapsulated and the 10.36.7.11 is then nat’ed to the pi-hole container ip address
This is similar to the iRedMail rules above this which would route traffic from public VPS ip on those ports to the LXC container ip addres 10.100.100.11.
## NextCloud:
Nextcloud will be deployed as another docker container (in addition to pi-hole, traefik, checkmk, etc) on the VPS
First add the A record for nextcloud.linode.cloudnetworktesting.com to the linode Domains configuration.
Ansible: The nextcloud/tasks/main.yml has the copy over to /root/services/nextcloud of the source code from the EC2 controller onto the VPS so that the docker container can be created. Main.yml also has the backup scripts for the nextcloud docker container and this is configured as a cronjob
[root@vps config]# crontab -l
#Ansible: mariadb dump all databases
01 3 * * * /root/scripts/mariadb-dump-all-databases.sh > /dev/null 2>&1
#Ansible: lxc mail.linode.cloudnetworktesting.com backup
01 1 * * * /root/scripts/lxc-backup-mail-linode-cloudnetworktesting-com.sh > /dev/null 2>&1
#Ansible: lxc zulip.linode.cloudnetworktesting.com backup
11 1 * * * /root/scripts/lxc-backup-zulip-linode-cloudnetworktesting-com.sh > /dev/null 2>&1
#Ansible: backup pi-hole
01 2 * * * /root/services/pi-hole/backup.sh > /dev/null 2>&1
#Ansible: backup nextcloud
06 2 * * * /root/services/nextcloud/backup.sh > /dev/null 2>&1
#Ansible: nextcloud background jobs
*/5 * * * * docker exec --user www-data nextcloud.linode.cloudnetworktesting.com php cron.php
Since the nextcloud docker container is using the VPS mariadb the nextcloud/nextcloud/docker-compose.yml will have the mariadb nextcloud mysql user's name and password so that the container has access to the mariadb thorugh this nextcloud mariadb user. Nextcloud uses the mariadb on the VPS host. See implementation details in "Nextcloud total packet flow" below.
In addition the docker-compose.yml will have all the necessary labels so that traefik can properly terminate and route the packets destined for the nextcloud docker container.
The new mariadb user "nextcloud" is created by ansible in the mariadb role. The users are added to the mariadb/tasks/users.yml file as shown below:
- name: check for database user 'nextcloud@172.21.0.0/16'
community.mysql.mysql_user:
login_user: root
login_password: "{{ mariadb_root_password }}"
name: nextcloud
password: "{{ mariadb_nextcloud_password }}"
host: "172.21.%.%"
priv: 'nextcloud.*:ALL'
state: present
no_log: true
All the passwords are encrypted with ansible-vault. All sensitive files are in .gitignore file and not present in the github repo. (mariadb/defaults/main.yml)
The iptables in the ansible role security must be updated with the 3306 ip tables Allow rule as indicated above so that the nextcloud docker container can access mariadb on the VPS. This is done in security/files/iptables.rules
The source subnets are based upon the /etc/docker/daemon.json on the VPS which has the 172 subnets that all docker containers will use by the docker engine on the VPS when creating docker containers. Thus the iptables must have rules to allow 3306 from any of these predefined docker private ip container address subnets.
Prior to running the nextcloud role on ansible-playbook, run the mariadb and the security roles on ansible-playbook to update the mariadb and iptables with the above information.
Note this will restart all docker containers and ip tables prior to running the nextcloud ansible role.
The users will be created as shown below on the VPS
[root@vps zulip.linode.cloudnetworktesting.com]# mariadb -e "select user,host from mysql.user;"
+---------------+-------------+
| User | Host |
+---------------+-------------+
| PUBLIC | |
| backup_script | 10.100.94.% |
| nextcloud | 10.100.94.% |
| nextcloud | 172.21.%.% |
| nextcloud | 172.22.%.% |
| nextcloud | 172.23.%.% |
| nextcloud | 172.24.%.% |
| nextcloud | 172.25.%.% |
| backup_script | localhost |
| mariadb.sys | localhost |
| mysql | localhost |
| nextcloud | localhost |
| root | localhost |
On traefik Web Admin there will be a new HTTP router for nextcloud
On the traafik HTTP router in middleware in addtion to the ipallowlist (basic auth challenge) there is a redirect for CalDAV and CardDAV for other client integration with the nextcloud contacts and calendars. I tried to get this to work on Windows10 and could not because it does not natively support nextcloud "other", but just the standards like google and icloud integration. Mac has an "other" and one is able to share calendar and contacts in nextcloud with the calendar and contacts on Mac and viseversa.
### Nextcloud docker container design:
#since nextcloud is being created as a docker container, need to set labels so that traefik can route the HTTPS traffic coming into# public interface to the docker container ip address which is routed through the VPS loopback interface
#Very similar to the pi-hole docker-compose.yml file, this will use .env file for the values of the variables below
#The volume ./html/var/www/html must be created as noted in the nextcloud documentation
#This container will also be a part of the docker network as well.
#All environment vars are in the nextloud website user docs. We only need to set the database stuff
#The db that is used will target the mariadb running on the VPS
#An alternative is to run a new mysql db inside the container, but alternatively the mariadb on host VPS can be accessed
#by the nextcloud docker container via docker networking and this is the approach taken here.
### Docker labels for traefik based routing:
#For the labels the first 6 are similar to those used for the pi-hole container insofar as traefik is concerned.
#Traefik uses these labels to create the HTTP router that will route the traffic from the VPS public to the nextcloud
#container ip address.
#"traefik.http.routers.${service}.middlewares=${service}-allowlist,${service}-redirects,${service}-sts"
#The line above adds the middleware but also configures redirects and sts, or strict transfer security header
#The use of the redirects: for CalDAV and CarDAV to work we have to have the redirects configured as below:
#"traefik.http.middlewares.${service}-redirects.redirectregex.permanent=true"
#"traefik.http.middlewares.${service}-redirects.redirectregex.regex=https://(.*)/.well-known/(card|cal)dav"
#For sts, this is not absolutely necessary but will remove a warning in the nextcloud admin console. The sts labels
#are the last 3 lines in the labels section below
### Nextcloud total packet flow using Traefik HTTP router (diagrams in word doc):
Nextcloud uses the mariadb on the VPS host. To do this there must be iptables that allow 3306 from the docker container networks (172 networks)
Nextcloud will route the packet out of the container to the VPS mariadb using the 10.36.7.11 loopback
Iptables firewall rule will allow traffic from docker to the VPS
The connection to 3306 mariadb on the VPS will be from one of the nextcloud configured users (all docker private subnets are added by the ansible playbook to mariadb so any nextcloud container will be able to connect to mariadb on 3306).
From wireguard client to nextcloud.linode.cloudnetworktesting.com
First it needs to resolve it to an ip address.
Pi-hole is the dns resolver for all these wireguard clients
The ip address will resolve to the public ip of the VPS
The public ip address in wireguard config is in the IP allowed list
The route is through the VPN tunnel
The packet is then forwarded through the tunnel. The inner packet is 443 and this is forwarded to the traefik TLS termination service on the VPS that has an HTTP router for routing the traffic to the nextcloud docker container based upon the docker labels in the docker-compose.yml for nextcloud
Traefik routes based upon the Host header in the packet
Traefik is a TLS termination point (certs are issued by letsencrypt) so traefik can look at the decrypted HTTP packet headers to do this routing.
Traefik has an ip whitelist and the private ip client addreses assigned by the the VPN (to the Windows or Mac client) is in the whitelist so the packet can be routed through traefik. Before that basic authentication challenge to log in to the nextcloud docker container.
### Nextcloud Web Admin GUI
In the Web Admin GUI make sure to install all the default apps.
When bringing this up the first time there are lot of errors and warnings.
The reverse proxy error and the HTTPS error can be addressed by adding code to the /root/services/nextcloud/config/config.php file (see word doc for details on this)
For the maintenance window warning add this to the config.php file at the end:
'maintenance_window_start' => 1,
For the occ warning run this command in the VPS shell. It will add all the missing indices
docker exec -it --user www-data nextcloud.linode.cloudnetworktesting.com php occ db:add-missing-indices
Also the email needs to be configured on the nextcloud Admin so that nextcloud can send updates and admin information to the user. Add the email to the account admin as well.
Also can add the iRedMail email info to share the iRedMail with the nextcloud mail app. Make sure to use StartTLS for both IMAP and SMTP and port 143 for the dovecot IMAP
### versions 29.0.8 and 30.0.0 issues
The latest codes unfortunately are throwing many more warnings than before. These will be addressed in 29.0.8 and 30.0.1, but these patches are not avaiable at the time of this writing.
Will have to downgrade to a 28 version for now until it is patched. The process for downgrading involves wiping out the old docker configuration: drop the mariadb database table "nextcloud" and delete the /root/services/nextcloud/html folder configuration on the VPS. Also put in the older version in the .env file for the nextcloud role: docker_image_tag and update this value in the running /root/services/nextcloud/.env file on the VPS. Then bring back up the container and it should be downgraded.
## checkmd:
### The 3 agent files and adding EC2 public ip to the traefik whitelist for checkmk:
IMPORTANT NOTE: checkmk requires 3 agent components and these can be downloaded from the Web Admin console above
The URL is from the Web UI Console and wget can be performed on the ansible controller to get the file
The file is being downloaded from the checkmk docker container.
In order to get access to the docker container URL to download these 3 agent files, one must be either on a Wireguard client
(the middleware on checkpoint traefik allows 10.100.94.x wireguard client ips)
OR one must add the public IP of the address of the ansible controller. In my case this is an EC2 controller that does not have wireguard installed.
Add that public EC2 IP address to the ip_allowlist for traefik label whitelist for the checkmk container in the .env file
- "traefik.http.middlewares.${service}-allowlist.ipallowlist.sourcerange=${ip_allowlist}"
### S3 bucket (optional):
I have added access to an S3 bucket for the EC2 ansible controller for any files that are not easily able to be downloaded to the EC2 controller. Did not have to use this here but this is good to have.
Ensure that the aws cli on the ansible controller is set to the proper profile
aws configure
aws configure list
export AWS_PROFILE=
~/.aws/config and ~/.aws/credentials
### Notes on the shell scripts used for this docker container and operational deployment of host/agent and adding the VPS public ip to the traefik whitelist for checkmk for the REST API packets from VPS to checkmk docker container:
The setup_host_non_REST_API.sh is used if adding the agent/host to checkmk with Web Admin GUI. . This script just puts the 3 agent files in the following directories on the VPS: /etc/systemd/system and /usr/bin. These are agent files
The setup_host.sh is used if adding the agent/host to checkmk with the REST API. This has an additional 3 lines to copy over the following files to the VPS:
add_host.sh and .env to the /root/services/checkmk/agent directory. The agent directory is a new directory created on the VPS for this REST API host adding.
The last line in this file also runs the add_host.sh which invokes the REST API to add the VPS host to checkmk.
NOTE: the add_host.sh is run from the VPS itself after it has been copied to the VPS with the setup_host.sh. Because of this the VPS public ip needs to be added to the checkmk middleware (traefik) ip_allowlist whitelist. The REST API packets are https packets so they are sent to traefik and as such they have to be whitelisted to be reverse proxied to the checkmk docker container.
The setup_host.sh needs to be run from the ansible controller. The syntax is:
./setup_host.sh linode.********.com
This will place the 3 agent files onto the VPS and place the add_host.sh on the VPS and then execute the add_host.sh on the VPS
The add_host.sh will sequentially execute the 3 curl REST API actions to create the host on the checkmk and then service discovery and then apply service discovery to the host on the Web Admin console.
There was an issue between the second curl REST API invocation and the third curl REST API invocation due to the service discovery not being complete prior to the apply (the third curl REST API). Adding a sleep of 10 seconds between second and third curl resolved the issue. A sleep of 3 seconds was inadequate to fully resolve the issue.
See the notes.txt for more detail on the script and the exact nature of the fix.
### Important note on iptables configuration:
In order for the docker checkmk container on the VPS to monitor the VPS host itself we need to add iptables rules for allowing traffic from docker container networks (172 nets) where checkmk docker container is running to the VPS host on port 6556 (the agent listening port), the checkmk protocol service port.
This can be effectively tested with the nsenter command and telnet from the VPS since the checkmk container does not have telnet installed. See notes.txt file for extensive details on how to do this.
The command is below to be run on the VPS itself:
nsenter -t $(docker inspect --format {{.State.Pid}} checkmk.linode.*********.com) -n telnet vps.linode.******************.com 6556
This cmmand gets the docker pid and runs a docker inspect on the container of that pid (checkmk docker container) and then initiates a telnet FROM the docker container to the host (VPS) by utlizing the telnet command binary on the VPS itself, to port 6556
This will fail prior to the iptables urles being added and will work once the iptables rules are added. See notes.txt for the exact rules and implementation details.
In short all the docker 172 container nets need to be allowed to the VPS public ip on port 6556
### Docker container design for checkmk service and mailer service (smtp relay service):
The checkmk setup consists of 2 containers: one for checkmk server and the other for an smtp relay using iRedMail as desitination. The udpates are important becuase checkmk as a monitoring tool requires a lot of updates. It works very well during testing.
#2 services on this docker compose: checkmk and mailer
#For checkmk, tmpfs is suggested in checkmk docs for better performance
#For the volumes, the /etc/localtime on VPS is mapped to /etc/localtime on the container to match the timezones
#For the second volume ./sites is /root/services/checkmk/sites on VPS to /omd/sites to persist the data the checkmk creates on the container
#The MAIL_RELAY_HOST is section 2.4 in checkmk docker documumentation link: sending notifications
#https://docs.checkmk.com/latest/en/managing_docker.html
#the host in the container cannot send mail directly but has to send to an smtp relay server (smarthost)
#which is this mailer docker service (second service name below), which will then forward to iRedMai LXC container mail server
#The labels for checkmk are for the traefik http/https router as was done for the other container. Letsencrypt TLS cert will be
#used for the HTTPS termination.
#Note that in the labels, the webserver on checkmk listens on port 5000
#Mailer service notes:
#The mailer service (smtp relay on the checkmk docker container) is below
#The mailer will use the .env variables below. Checkmk mail will go to this mailer and the mailer will then
#relay the mail to our LXC mail container on the VPS and the mail will then reach its final destination
#Note that both mailer and checkmk services are part of the same docker network web so they can innately communicate
#with one another.
### Docker labels for Traefik based routing for the checkmk docker service:
The labels are similar to the other traefik based labels for the docker containers in this project. Note that the http daemon for checkmd is running on 5000 and also note that middleware ip_allow list is enforced. To disable the whitelist simply comment out the last 2 lines of the lables. This is not recommended. All traffic will be passed through to the container through the public URL for checkmd. Letsencrypt certs are used as with all the other docker containers running HTTP/HTTPS.
## Borg and borgmatic:
Borg is just installed as a server/service on the VPS. It is not deployed as a separate container.
Borg needs to be installed on both the source (what is being backed up, in this case the VPS) and the destination repository (in this case we are using the block storage volumne for our linode VPS at /mnt/storage and borg backups will be made to /nnt/storage/backups/borg directory)
As such, there are only tasks and files and not a borg/borg directory with a docker-compose.yml file
In the tasks the main.yml will install borg, borgmatic and python-llfuse packages to the archlinux VPS using pacman ansible module. Python-llfuse is required to mount the backups to our local file system at /mnt/storage
The tasks will create the borgmatic directory at /etc/borgmatic. Ansible copies all of the shell scirpts the config.yaml and the .env. Borgmatic is an API hook command application to make borg backup administration much more finer tuned and detailed. We can run some tests with this.
Prior to this, make the first backup. NOTE the backups will be added to the cronjobs and done daily thereafter. First the backups directory must be initialized. Once intialized the first backup will be done manually so that we can run some tests with it.
We will use borgmatic to do some intentional file deletion and restore tests using the following commands:
borgmatic extract --archive latest --path usr/bin/chmod
and method 2 a mount: borgmatic mount --archive latest --mount-point /mnt/borg --path usr/bin
## Gitlab and the Gitlab runner:
### repository design FOR GITLAB PROJECTS:
See the first gitlab project in the section below "GITLAB PROJECT1: Basic website application test"
### Introduction:
When configuring this make sure that the VPS ip address is included in the ip_allowlist (whitelist) for traefik for the gitlab container. This is because the gitlab-runner is running on VPS as well and the source of the packet when it registers with gitlab docker container is the VPS public ip address. Otherwise the gitlab-runner registration will fail.
NOTE the gitlab-runner can be installed on a standalone separate VPS server if throughput transactions need to be scaled up.
### detailed packet flow for gitlab docker container itself:
NOTE: for details of specific iptables rules etc. see notes.txt file.
The network diagrams are in the Word doc.
NOTE: give that gitlab is essentially a repository driven based CI/CD tool, we must consider both HTTP/HTTPS packet flow and SSH packet flow analysis
NOTE: this analysis is not for the deployment of an application packet flow. That analysis follows in the next section
#### HTTP/HTTPS:
When connecting to gitlab.linode.*****.com or when doing a git clone https://.... for example then this packet flow will kick in.
Traefik’s labels (NOT gitlab container traefik labels) are below. Note the 80 is immediately redirected by the traefik middleware.
labels:
- "traefik.enable=true"
- "traefik.http.routers.${service}.rule=Host(`${hostname}`)"
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
- "traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true"
- "traefik.http.routers.redirects.rule=hostregexp(`.+`)"
- "traefik.http.routers.redirects.entrypoints=http"
- "traefik.http.routers.redirects.middlewares=redirect-to-https"
- "traefik.http.routers.${service}.tls.certresolver=letsencrypt"
- "traefik.http.routers.${service}.service=api@internal"
- "traefik.http.routers.${service}.middlewares=${service}-allowlist,${service}-auth"