An L2 switch with two VLANs and SVIs
This page builds a small switch by hand, using only the diag shell for the switch chip: two untagged VLANs with two ports each, and a switched virtual interface (SVI, an IP address on the VLAN) in each VLAN that hosts can ping and use as their gateway. It is a good way to learn how the SDK’s VLAN, L2, L3 and KNET objects fit together, and a way to test the chip without SONiC’s configuration in between.
VLAN |
Ports (untagged) |
SVI address |
|---|---|---|
10 |
|
192.0.2.1/24 |
20 |
|
203.0.113.1/24 |
Both SVIs use the router MAC 02:00:00:00:01:00.
Warning
Everything here is configured behind SONiC’s back. Use ports that SONiC
does not use (no VLAN, port channel or IP address on them), VLAN IDs and
addresses that SONiC does not use, and remove it all again when you are
done (see Removing it again). A syncd restart also removes it.
1. Check that the ports are free
The ports must not be in a trunk (port channel) or in another VLAN:
Hardware-validated
drivshell> trunk show
trunk 1: (front panel, 0 ports)
drivshell> pvlan show xe0-xe3
Port xe0 default VLAN is 1
Port xe1 default VLAN is 1
Port xe2 default VLAN is 1
Port xe3 default VLAN is 1
An empty trunk and PVID 1 (the default VLAN) mean the ports are free.
2. VLANs
Create each VLAN with its member ports, all untagged, with
vlan create, and make
it the ports’ default VLAN (PVID) with pvlan so that untagged
frames are classified into it:
Hardware-validated
drivshell> vlan create 10 PortBitMap=xe0,xe1 UntagBitMap=xe0,xe1
drivshell> vlan create 20 PortBitMap=xe2,xe3 UntagBitMap=xe2,xe3
drivshell> pvlan set xe0,xe1 10
drivshell> pvlan set xe2,xe3 20
drivshell> pvlan show xe0-xe3
Port xe0 default VLAN is 10
Port xe1 default VLAN is 10
Port xe2 default VLAN is 20
Port xe3 default VLAN is 20
drivshell> stg show
STG 0: contains 0 VLANs
Disable: ce,xe
STG 1: contains 10 VLANs (1,10,20,902,3999,4095,28673,29574,32671,45055)
Forward: ce,xe
New VLANs join spanning-tree group 1, where every port is in Forward
state, so nothing more is needed for spanning tree.
3. Port settings
On a SONiC switch, ports outside any VLAN discard untagged frames and do not
learn MAC addresses. ps shows this as Untag in the discrd column
and F (forward only) in the lrn ops column. Turn off the discard and
turn on learning with port: LeaRN=5 is hardware learning (1)
plus forwarding (4).
Hardware-validated
drivshell> port xe0-xe3 DISCard=None LeaRN=5
drivshell> ps xe0-xe3
ena/ speed/ link auto STP lrn inter max cut loop
port link Lns duplex scan neg? state pause discrd ops face frame thru? back encap
xe0( 9) up 1 10G FD SW No Forward None FA XFI 9122 No IEEE
xe1( 10) up 1 10G FD SW No Forward None FA XFI 9122 No IEEE
xe2( 11) up 1 10G FD SW No Forward None FA XFI 9122 No IEEE
xe3( 12) up 1 10G FD SW No Forward None FA XFI 9122 No IEEE
A in FA means the port learns. Until this step, the hosts’ frames
were counted as receive drops (RDBGC0) and nothing was learned.
4. Check L2 forwarding
Hosts in the same VLAN can now reach each other, and l2 show
lists the learned addresses:
Hardware-validated
drivshell> l2 show
mac=74:83:ef:3f:2c:fd vlan=20 GPORT=0xb modid=0 port=11/xe2 Hit
mac=74:83:ef:3f:2c:fd vlan=10 GPORT=0x9 modid=0 port=9/xe0 Hit
mac=e0:39:d7:88:3b:00 vlan=1 GPORT=0x0 modid=0 port=0/cpu0 Static CPU
The same MAC address can be learned in both VLANs; each VLAN has its own address table.
5. SVIs in the switch chip
An SVI needs four things in the chip:
a station entry for the router MAC, so that packets addressed to it are routed instead of switched (
l2 station add);an L3 interface per VLAN with that MAC (
l3 intf add);an egress object per neighbour: its MAC, L3 interface and port (
l3 egress add);a host entry per neighbour IP pointing at its egress object (
l3 l3table add).
Hardware-validated
drivshell> l2 station add MACaddress=02:00:00:00:01:00 MACaddressMask=ff:ff:ff:ff:ff:ff IPv4=true
Created SID=0x02000003
drivshell> l3 intf add VLAN=10 MAC=02:00:00:00:01:00
drivshell> l3 intf add VLAN=20 MAC=02:00:00:00:01:00
drivshell> l3 intf show
Free L3INTF entries: 16375
Unit Intf VRF Group VLAN Source Mac MTU TTL Tunnel InnerVlan NATRealm
------------------------------------------------------------------
0 1 0 0 29574 e0:39:d7:88:3b:00 9100 0 0 0 0
0 8192 0 0 1 00:00:00:00:00:01 16383 0 0 0 0
0 8193 -3 0 1 00:00:01:00:00:01 16383 0 0 0 0
0 8194 -3 0 1 e0:39:d7:88:3b:00 9100 0 0 0 0
0 8195 -3 0 902 e0:39:d7:88:3b:00 9100 0 0 0 0
0 8196 -3 0 10 02:00:00:00:01:00 16383 0 0 0 0
0 8197 -3 0 20 02:00:00:00:01:00 16383 0 0 0 0
l3 intf add does not print the new interface’s ID; find it in l3 intf
show by VLAN and MAC (8196 and 8197 here). It refused an explicit
INTF=10 with Invalid parameter, so let the SDK choose the ID.
Next an egress object for each host. The shell prints the new object’s
index and also stores it in the shell variable egr_object_id:
Hardware-validated
drivshell> l3 egress add Mac=74:83:ef:3f:2c:fd INtf=8196 Port=xe0
New egress object index: 132770
Environment variable (egr_object_id) was set
drivshell> l3 egress add Mac=74:83:ef:3f:2c:fd INtf=8197 Port=xe2
New egress object index: 132771
Environment variable (egr_object_id) was set
l3 egress add does not check that the L3 interface exists; with a wrong
INtf it creates an object that routes nowhere. Check l3 egress show:
the Vlan column must show the interface’s VLAN.
Then the host entries. The chip does not resolve ARP by itself, so every neighbour is added by hand:
Hardware-validated
drivshell> l3 l3table add VRF=0 IP=192.0.2.11 Mac=74:83:ef:3f:2c:fd INtf=132770 Port=xe0
drivshell> l3 l3table add VRF=0 IP=203.0.113.11 Mac=74:83:ef:3f:2c:fd INtf=132771 Port=xe2
drivshell> l3 l3table show
Unit 0, free L3 table entries: 16379
Entry VRF IP address Mac Address INTF MOD PORT CLASS HIT H/W Index
1 0 203.0.113.11 00:00:00:00:00:00 132771 0 0 0 n 2560
2 0 198.51.100.1 00:00:00:00:00:00 132807 0 0 1 n 7744 (LOCAL ROUTE)
3 0 192.0.2.11 00:00:00:00:00:00 132770 0 0 0 n 8108
Hosts in VLAN 10 can now reach hosts in VLAN 20 through the chip, if they
know the router MAC for their gateway (for example as a static ARP entry;
the next step makes the SVIs answer ARP). The HIT column of l3 l3table
show turns y once an entry is used, and a ping crosses the switch with
its TTL lowered by one:
switch# ping vrf howto-a 203.0.113.11 repeat 5 timeout 1
PING 203.0.113.11 (203.0.113.11) 72(100) bytes of data.
80 bytes from 203.0.113.11: icmp_seq=1 ttl=63 time=0.184 ms
...
5 packets transmitted, 5 received, 0% packet loss, time 0ms
6. SVI addresses on the CPU
To answer ARP and ping, the SVI addresses must reach Linux on the switch CPU. That takes:
an egress object and a host entry that send each SVI address to the CPU port,
cpu0;the CPU port as a member of each VLAN, so that ARP broadcasts reach it;
a KNET netif per VLAN, the Linux network interface, and a KNET filter that sends the VLAN’s packets to it (see KNET and the CPU port).
Hardware-validated
drivshell> l3 egress add Mac=02:00:00:00:01:00 INtf=8196 Port=cpu0
New egress object index: 132772
Environment variable (egr_object_id) was set
drivshell> l3 egress add Mac=02:00:00:00:01:00 INtf=8197 Port=cpu0
New egress object index: 132773
Environment variable (egr_object_id) was set
drivshell> l3 l3table add VRF=0 IP=192.0.2.1 Mac=02:00:00:00:01:00 INtf=132772 Port=cpu0
drivshell> l3 l3table add VRF=0 IP=203.0.113.1 Mac=02:00:00:00:01:00 INtf=132773 Port=cpu0
drivshell> vlan add 10 PortBitMap=cpu0
drivshell> vlan add 20 PortBitMap=cpu0
drivshell> knetctrl netif create Type=Vlan Vlan=10 IFName=svi10
Interface ID 37: name=svi10 type=Vlan vlan=10 port=n/a
drivshell> knetctrl netif create Type=Vlan Vlan=20 IFName=svi20
Interface ID 38: name=svi20 type=Vlan vlan=20 port=n/a
drivshell> knetctrl filter create DestType=NetIF DestID=37 Vlan=10 PRIOrity=50 StripTag=yes DESCription=svi10
Filter ID 37: prio=50 dest=NetIF(37) desc='svi10' striptag vlan=10
drivshell> knetctrl filter create DestType=NetIF DestID=38 Vlan=20 PRIOrity=50 StripTag=yes DESCription=svi20
Filter ID 38: prio=50 dest=NetIF(38) desc='svi20' striptag vlan=20
Filters are matched by priority, lowest first. SONiC’s own filters, one per
port, use priority 100, so these VLAN filters at 50 are checked first.
StripTag=yes delivers the packets to Linux untagged.
The last step is the only one outside the diag shell: give the new Linux interfaces the router MAC and the SVI addresses, on the switch’s host:
$ sudo ip link set svi10 address 02:00:00:00:01:00
$ sudo ip link set svi20 address 02:00:00:00:01:00
$ sudo ip addr add 192.0.2.1/24 dev svi10
$ sudo ip addr add 203.0.113.1/24 dev svi20
$ sudo ip link set svi10 up
$ sudo ip link set svi20 up
Hosts now resolve the gateway with ARP and can ping it:
switch# ping vrf howto-a 192.0.2.1 repeat 3 timeout 1
80 bytes from 192.0.2.1: icmp_seq=1 ttl=64 time=0.351 ms
...
3 packets transmitted, 3 received, 0% packet loss, time 0ms
switch# show ip arp vrf howto-a
Address Age (sec) Hardware Addr Interface
192.0.2.1 0:00:08 0200.0000.0100 Ethernet7/1
tcpdump -ni svi10 on the switch shows the requests arriving, with the
router MAC as source because the chip routed them to the CPU. Linux answers
through the same interface; the chip switches its replies in VLAN 10. The L2
table now also holds the router MAC as a static CPU entry in both VLANs:
Hardware-validated
drivshell> l2 show
mac=74:83:ef:3f:2c:fd vlan=20 GPORT=0xb modid=0 port=11/xe2 Hit
mac=74:83:ef:3f:2c:fd vlan=10 GPORT=0x9 modid=0 port=9/xe0 Hit
mac=e0:39:d7:88:3b:00 vlan=1 GPORT=0x0 modid=0 port=0/cpu0 Static CPU
mac=02:00:00:00:01:00 vlan=10 GPORT=0x0 modid=0 port=0/cpu0 Static Hit CPU
mac=02:00:00:00:01:00 vlan=20 GPORT=0x0 modid=0 port=0/cpu0 Static Hit CPU
Hosts that are not in the host table are not reached: routed packets to
unknown neighbours are dropped. A full router also adds a subnet route per
VLAN that sends unresolved destinations to the CPU so that software can ARP
for them (l3 defip add); that is not shown here.
Removing it again
Undo the steps in reverse order: first Linux, then the KNET objects, the L3
objects, the station, the L2 entries, the VLANs, and last the port settings.
Look up the IDs you used with the show commands first.
$ sudo ip addr flush dev svi10; sudo ip addr flush dev svi20
$ sudo ip link set svi10 down; sudo ip link set svi20 down
Hardware-validated
drivshell> knetctrl filter destroy 37
drivshell> knetctrl filter destroy 38
drivshell> knetctrl netif destroy 37
drivshell> knetctrl netif destroy 38
drivshell> l3 l3table destroy VRF=0 IP=192.0.2.1
drivshell> l3 l3table destroy VRF=0 IP=203.0.113.1
drivshell> l3 l3table destroy VRF=0 IP=192.0.2.11
drivshell> l3 l3table destroy VRF=0 IP=203.0.113.11
drivshell> l3 egress delete 132770
drivshell> l3 egress delete 132771
drivshell> l3 egress delete 132772
drivshell> l3 egress delete 132773
drivshell> l3 intf destroy INTF=8196
drivshell> l3 intf destroy INTF=8197
drivshell> l2 station delete ID=0x02000003
drivshell> l2 clear Vlanid=10
L2: Deleting static and non-static addresses by VLAN
drivshell> l2 clear Vlanid=20
L2: Deleting static and non-static addresses by VLAN
drivshell> pvlan set xe0-xe3 1
drivshell> vlan destroy 10
drivshell> vlan destroy 20
drivshell> port xe0-xe3 DISCard=Untag LeaRN=4
The last command restores the SONiC defaults for ports outside a VLAN:
untagged frames discarded, learning off. Compare ps, vlan show, l3
intf show, l3 egress show and knetctrl filter show with what they
showed before you started.