Sounds interesting. I always wanted to use a Raspberry PI as router (to have one as backup in case the OpenWRT Linksys goes down), but couldn't wrap my head around properly how to overcome the single network port (I think the usual recommendation is to use an extra USB network card/adapter). Can you elaborate more about this VLAN stuff (you would put your modem, your router, and all your machines on the switch... and in the switch you tell the router connection to double use the connection for WAN and LAN separated via VLANs? And put the modem into the "WAN VLAN" too?)
Ideally the PI also should to what the extra DSL Modem does… but I guess that's where the dram must stop. :D
The TL;DR is to have two vlans on the cable from your switch (called a "trunk"), "lan" and "wan", carrying the respective LAN and WAN networks. Then, on the Pi, create two vlans on the underlying Ethernet interface. Then those two VLAN interfaces can be configured just like the LAN and WAN interfaces of the router. On the switch, you’d dedicate one port to the WAN by adding it to the WAN VLAN without tagging, and the other interfaces do the LAN VLAN, also untagged.
I've created a blog post about this: https://louwrentius.com/raspberry-pi-as-a-router-using-a-sin...
When you setup your single ethernet port (let's call it eth0) as a VLAN trunk port, you'll get the ability to configure multiple virtual interfaces off of it. How many virtual interfaces you get depends on how many VLANs you want to tag traffic for. For example, if you have 2 VLANS with ids 100 and 200 (100 being your public Internet-facing traffic, and 200 being your LAN traffic), you would then have interfaces eth0.100 and eth0.200 to work with that you can then use in your firewall scripts as if they were two separate, physical interfaces.
This of course means you need a VLAN-aware switch that this single ethernet port can plug into, configured as a VLAN trunk (in Cisco terms) port. You would then want to configure one of the other switch ports as a VLAN access port assigned to VLAN 100 (untagged). This is the port you would plug your cable modem into. Then (in the simplest example) you could assign all the rest of the switch ports to VLAN 200 (untagged), and you would plug all your LAN devices into them.