new pf rule design does not work

[tniethammer]

Describe the bug
a fresh install on FreeBSD 13.3 with toaster version 20241019 all jails are installed propably but i have an issue with pf ruleset. I find the new way to place the rules in the data folder an great idear, but it does not work on my installation. if i set up everything and start an jail there are no rules or rdr installed into the runing pf ruleset or tables..

I found out that tables which are not created with the persist attribute are removed when they where empty anyway I also did not find any allow rule or rdr rule, maybe I do sth wrong but i did not find anything, I tried to add the things manaly with pfctl as your pfrule.sh script is doing

if i do:
pfctl -a allow/haproxy -f /data/haproxy/etc/pf.conf.d/allow.conf -vv
No ALTQ support in kernel
ALTQ related functions disabled
table <http_servers> { 10.50.128.105 172.16.15.12 fd7a:e5cd:1fc1:c7cc:dead:beef:cafe:c }
@0 pass in quick proto tcp from any to <http_servers:0> port = http flags S/SA keep state
@1 pass in quick proto tcp from any to <http_servers:0> port = https flags S/SA keep state
pfctl: warning: namespace collision with <http_servers> global table.

I created the table http_servers with the persist tag so that it is shown up in PF.

I could add Ips to the table if i do:
pfctl -t http_server -T add 10.50.128.105 172.16.15.12 fd7a:e5cd:1fc1:c7cc:dead:beef:cafe:000c
1 table created.
3/3 addresses added.

the anchors are also not installed
pfctl -a rdr/haproxy -sr
returns nothing.

in the way you mentioned with loading PF rules in every Jail Start, does not work

To Reproduce
flush pf
restart pf
start jail haproxy
pfctl -sa

no rules or rdr's shown up
http_servers table does not exist so there are also no ip's listed

Expected behavior
Start Jail and the rules and rdr are getting loaded and shown in pfctl -sa

Server (please complete the following information):

• OS: FreeBSD 13.3 RELEASE-p6
  -Mailtoaster: 20241019

[msimerson]

I found out that tables which are not created with the persist attribute are removed when they where empty

Yep, that's how PF works, and why the 'persist' attribute exists.

the anchors are also not installed

Tables and anchors need to be defined in /etc/pf.conf, which is done for you with the default /etc/pf.conf installed by the host script. Within each anchor, additional PF rules can be added, which is what pfrule.sh does when jails are started and stopped.

[rricker1]

How can we verify that anchor rules are being loaded? I found this suggestion: pfctl -vsA

result:
# pfctl -vsA
No ALTQ support in kernel
ALTQ related functions disabled
allow
nat
rdr

[msimerson]

Good tip @rricker1 . The output looks like this on a default toaster build while the jails are running:

# pfctl -vsA
No ALTQ support in kernel
ALTQ related functions disabled
  allow
  allow/dcc
  allow/dovecot
  allow/webmail
  nat
  rdr
  rdr/dcc
  rdr/dovecot
  rdr/haraka
  rdr/webmail

[rricker1]

Indeed. I had not completed the build when I ran the above. Once jails load that have rules to report, they are reported as expected.

# pfctl -vsA
No ALTQ support in kernel
ALTQ related functions disabled
allow
allow/dcc
allow/dovecot
nat
rdr
rdr/dcc
rdr/dovecot
rdr/haraka