KNET and the CPU port

SONiC’s EthernetN interfaces are KNET netdevs created by the linux-bcm-knet kernel module. The ports live on the switch ASIC, not on the host CPU; KNET forwards packets between the CPU’s packet DMA and these netdevs. Only packets the ASIC sends to the CPU (cpu0) are seen, so tools like tcpdump do not see transit traffic (see Capture port traffic with tcpdump).

$ ethtool --driver Ethernet92
driver: bcm-knet

KNET state and debug controls are under /proc/bcm/knet. Writing a debug mask enables logging, including full packet dumps of all TX/RX traffic:

# cat /proc/bcm/knet/debug
# echo 'debug=0x10304' > /proc/bcm/knet/debug

Router MACs and the CPU

Changing an interface’s MAC address in Linux does not make the ASIC send that MAC to the CPU. The ASIC traps routed traffic to the CPU by matching the L3 interface MAC (l3 intf show), and needs an L2 entry pointing the MAC at cpu0 (l2 add). On SONiC, config_db.json can set the interface MAC (INTERFACE|EthernetN|mac_addr) but, at the time the original note was written, did not add the L2 entry.

On recent SONiC builds the switch MAC is already there: l2 show lists it on VLAN 1 as a static cpu0 entry, and l2 station show ID=1 shows it as the router MAC. The workaround is only needed for MACs set per interface. It was:

#!/bin/bash
set -euo pipefail
for mac in $(bcmcmd 'l3 intf show' | awk '/:[0-9a-f][0-9a-f]:[0-9a-f][0-9a-f]:/ {print $6}' | sort | uniq)
do
  if ! bcmcmd 'l2 show' | grep CPU | grep -q "${mac}"; then
    echo "Adding missing MAC: ${mac}"
    bcmcmd "l2 add mac=${mac} port=cpu0 Static=true ReplacePriority=false"
  fi
done

Related: sonic-swss #814, #1833, #1847.

Inspecting KNET from the diag shell

The netifs and filters that syncd created can be listed without changing anything:

drivshell> knetctrl netif show
drivshell> knetctrl filter show

knetctrl netif show lists one Interface ID …: name=… line per netdev, with its type, VLAN and port. knetctrl filter show lists the filters in match order, with their destination (a netif, the user-space receive layer, or Null to drop) and their match fields. In both, the keywords must be written in full (knetctrl netif sh is rejected).

On a SONiC switch there is one Port netif per front-panel port, named after the SONiC interface, and one filter per port that sends everything received for the CPU on that port to its netif:

Interface ID 1: name=Ethernet0 type=Port vlan=0 port=ce0 keeprxtag
Filter ID 1: prio=100 dest=NetIF(1) desc='Ethernet0' ingport=ce0

This is also the quickest way to map diag-shell port names (ce0, xe0) to SONiC interface names.

Most other commands in this area change state that belongs to syncd, and syncd is not told about the change:

knetctrl netif destroy and knetctrl filter destroy remove a netdev or filter that SONiC created. The port’s or trap’s packets then stop reaching Linux.
RXMon stop and PacketWatcher stop stop packet reception for the whole switch, including SONiC’s own. Avoid RXInit as well.
RXCfg with any option other than SPPS sets the receive rate limit of every CPU queue. Queues without an explicit value get 100 packets per second.
knetctrl debugpkt and PacketWatcher debugpkt copy every packet that KNET handles to user space as well.

To see packets on a port, prefer tcpdump on the EthernetN netdev, together with a mirror to the CPU if needed (Capture port traffic with tcpdump), over PacketWatcher.