L3

Manage L3 (IP) addresses

L3

The L3 (routing) tables: router interfaces, egress objects (next hops), ECMP/multipath groups, host entries, IPv4 and IPv6 routes, tunnels, DSCP maps, the IPv6 prefix map and NAT.

A command is l3 <table> <sub-command> [Key=<value> …]. The table and sub-command words must be written out in full; case does not matter. l3 egress show and L3 EGRESS SHOW work, but l3 eg sh does not. destroy and delete mean the same thing everywhere. An unknown sub-command prints the usage. The Key=<value> arguments can be abbreviated as usual: the upper-case letters of a key are the ones you must type.

The built-in help differs from what the shell accepts in a few places:

  • l3 l3table untag is in the help, but prints the usage: there is no such sub-command.

  • l3 l3table sanity only prints Command only valid for XGS2 switches.

  • The help shows Gateway= for l3 defip add, but only l3 defip destroy accepts it.

  • Some keys are accepted but not shown: InnerVlan and NATrealm (l3 intf add), IPMC (route add), DSCPMap (l3 tunnel_init set), RemoteTerminate and TunnelID (l3 tunnel_term add), and TrieTree (l3 alpm show). The help spells InternelDebug; the key is InternalDebug.

  • l3 defipinfo (not in the help) does nothing on Trident 3.

  • l3 route test perf takes keys the help does not list; see l3 route test perf.

When an L3 call fails, the shell prints an L3: or ERROR L3: line with the SDK’s reason, followed by Failed to execute the diagnostic command. Error: Operation failed.; bcmcmd still exits 0.

What a SONiC switch shows: SONiC’s router interfaces (for example Vlan902) appear in l3 intf show with their VLAN and the switch MAC. Egress objects are numbered from 132768. Two of them carry most routes: one that sends to the CPU (INTF 16383), used by connected subnets and the switch’s own addresses, and one that drops, used by the default route until SONiC learns a real one. With ALPM off, routes are in the plain route TCAM shown by l3 defip show.

Commands that only show (show, info, find, hash, conflict) are safe on a running switch. SONiC’s own generate_dump runs l3 intf show, l3 egress show, l3 defip show, l3 l3table show, l3 ecmp egress show, l3 multipath show, l3 ip6host show and l3 ip6route show on Broadcom XGS switches (scripts/generate_dump (show techsupport collector)).

Warning

Every add, update, destroy/delete, clear, set, insert, remove and age changes the hardware behind SONiC’s back. SONiC is not told and its view goes stale. l3 init, l3 detach and l3 route test perf disrupt a running switch.

The same commands are shown on other Broadcom-based systems in Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 9.3(x) – Troubleshooting VLANs, FBOSS: BcmSwitch gatherSdkState/rcload, validated shell commands, diag shell stress test, vendor-diag-shell skill, debugging doc, Edgecore AS5610-52X reverse engineering: Cumulus 2.5 baseline captures (Trident+) and Dell Networking OS10 Routes are Not Installed (KB 000322394) (Trident 3-X7).

Hardware-validated — Full words are required, case is ignored

drivshell> L3 EGRESS SHOW
Entry  Mac                 Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC
132768  00:00:00:00:00:01    1 8192     0t   0        -1   no   no    0  yes
132769  00:00:00:00:00:01    1 8192     0t   0        -1   no   no    0   no
132770  00:00:00:00:00:00    0   -1    12    0        -1   no   no    1   no
132771  ff:ff:ff:ff:ff:ff    0 8195    12    0        -1   no   no    0  yes
132806  e0:39:d7:88:3b:00    1 8194     0    0        -1   no  yes    2   no
132807  e0:39:d7:88:3b:00    0 16383     0    0        -1   no   no    4   no
l3 init

Initialise the L3 module. On a running SONiC switch this resets L3 state underneath syncd. Do not use it on a production switch.

Unvalidated

drivshell> l3 init
l3 detach

Detach the L3 module, removing all L3 state. Do not use it on a production switch.

Unvalidated

drivshell> l3 detach
l3 info

Show the size and use of the L3 tables (hosts, routes, next hops, ECMP groups) as <name> : <used> / <max> lines, plus two ECMP defragmentation counters. Safe. On Trident 3 with ALPM off there is room for 16384 hosts, 16384 routes, 65536 next hops and 4096 ECMP groups. Dell Networking OS10 Routes are Not Installed (KB 000322394) uses it on a Trident 3-X7 switch to find a full route table.

Hardware-validated

drivshell> l3 info
hosts            :      3 /  16384
routes           :      6 /  16384
nexthops         :      7 /  65536
ecmp_groups      :      0 /   4096
overlay_ecmp_defrag  :      0
underlay_ecmp_defrag :      0
l3 defipinfo

Not in the built-in help. Does nothing on Trident 3.

Hardware-validated

drivshell> l3 defipinfo
l3 intf show [INtf=<id>]

List the L3 interfaces (router interfaces), or one of them: VRF, VLAN, source MAC and MTU, after a Free L3INTF entries: <n> line. Safe.

On a SONiC switch a VLAN interface such as Vlan902 shows up with its VLAN, the switch MAC and SONiC’s MTU (9100). Interfaces 8192 and up are created by the SAI for its own use.

Hardware-validated — output abridged

drivshell> l3 intf show
Free L3INTF entries: 16377
Unit  Intf  VRF Group VLAN    Source Mac     MTU TTL Tunnel InnerVlan  NATRealm
------------------------------------------------------------------
0     1     0     0     29574 e0:39:d7:88:3b:00  9100 0    0     0     0
0     8192  0     0     1    00:00:00:00:00:01  16383 0    0     0     0
0     8193  -3    0     1    00:00:01:00:00:01  16383 0    0     0     0
0     8194  -3    0     1    e0:39:d7:88:3b:00  9100 0    0     0     0
0     8195  -3    0     902  e0:39:d7:88:3b:00  9100 0    0     0     0

Hardware-validated — A running SONiC switch, output abridged

drivshell> l3 intf show
Free L3INTF entries: 16377
Unit  Intf  VRF Group VLAN    Source Mac     MTU TTL Tunnel InnerVlan  NATRealm
------------------------------------------------------------------
0     1     0     0     29574 e0:39:d7:88:3b:00  9100 0    0     0     0
0     8192  0     0     1    00:00:00:00:00:01  16383 0    0     0     0
0     8193  -3    0     1    00:00:01:00:00:01  16383 0    0     0     0
0     8194  -3    0     1    e0:39:d7:88:3b:00  9100 0    0     0     0
0     8195  -3    0     902  e0:39:d7:88:3b:00  9100 0    0     0     0
l3 intf destroy INTF=<id>

Delete an L3 interface. l3 intf delete is the same command. The interface must exist (see l3 intf show); otherwise the shell prints L3: Error (Entry not found) destroy interface (<id>). Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 intf destroy intf=2
l3 intf add VLAN=<vid> MAC=<mac> INTF=<id> [VRF=<vrf>] [MTU=<mtu>] [Group=<n>] [InnerVlan=<vid>] [NATrealm=<n>]

Create an L3 interface. The VLAN must exist and the interface ID must be free; the example below also needs VRF 2 to be usable. If not, the shell prints L3: Error creating entry to L3 Intf table: Invalid parameter. The built-in help does not show InnerVlan or NATrealm. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 intf add Vlan=4095 Mac=76:c0:f8:4b:c8:04 Intf=2 vrf=2 mtu=9100
l3 intf clear

Delete all L3 interfaces.

Warning

This removes every router interface SONiC created.

Unvalidated

drivshell> l3 intf clear
l3 egress show [<entry>]

List the L3 egress objects (next hops), or one by ID. Egress interface 16383 sends to the CPU, Drop yes marks a drop object, and a t after the port number (0t) means a trunk. Safe. On Trident 3 the IDs start at 132768; take the IDs for the other egress, multipath and ecmp commands from this list.

Hardware-validated — output abridged

drivshell> l3 egress show
Entry  Mac                 Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC
132768  00:00:00:00:00:01    1 8192     0t   0        -1   no   no    0  yes
132769  00:00:00:00:00:01    1 8192     0t   0        -1   no   no    0   no
132770  00:00:00:00:00:00    0   -1    12    0        -1   no   no    1   no
132771  ff:ff:ff:ff:ff:ff    0 8195    12    0        -1   no   no    0  yes
132806  e0:39:d7:88:3b:00    1 8194     0    0        -1   no  yes    2   no
132807  e0:39:d7:88:3b:00    0 16383     0    0        -1   no   no    4   no

Hardware-validated — A running SONiC switch, output abridged

drivshell> l3 egress show
Entry  Mac                 Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC
132768  00:00:00:00:00:01    1 8192     0t   0        -1   no   no    0  yes
132769  00:00:00:00:00:01    1 8192     0t   0        -1   no   no    0   no
132770  00:00:00:00:00:00    0   -1    12    0        -1   no   no    1   no
132771  ff:ff:ff:ff:ff:ff    0 8195    12    0        -1   no   no    0  yes
132806  e0:39:d7:88:3b:00    1 8194     0    0        -1   no  yes    2   no
132807  e0:39:d7:88:3b:00    0 16383     0    0        -1   no   no    4   no
l3 egress add Mac=<mac> INtf=<id> Port=<port> [Trunk=<tgid>] [MOdule=<modid>] [L2tocpu=<bool>] [L3MC=<bool>] [MplsLabel=<label>] [Drop=<bool>] [CopyToCpu=<bool>]

Create an egress object and print its ID. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 egress add Mac=00:00:5e:00:53:01 INtf=2 Port=xe0
l3 egress update EgrId=<id> Mac=<mac> INtf=<id> Port=<port> [Trunk=<tgid>] [MOdule=<modid>] [L2tocpu=<bool>] [L3MC=<bool>] [MplsLabel=<label>]

Replace an existing egress object. Takes the same keys as l3 egress add plus EgrId. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 egress update EgrId=100005 Mac=00:00:5e:00:53:02 INtf=2 Port=xe0
l3 egress delete <entry>

Delete an egress object by ID. l3 egress destroy is the same command. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 egress delete 100005
l3 egress find Mac=<mac> INtf=<id> Port=<port> [Trunk=<tgid>] [MOdule=<modid>] [L2tocpu=<bool>] [L3MC=<bool>] [MplsLabel=<label>]

Print the ID of the egress object with these attributes. Safe.

Port= takes a port name (cpu0, xe0); a bare number such as Port=0 is rejected with Invalid port "0", as the first example shows.

Hardware-validated

drivshell> l3 egress find Mac=3c:2c:30:78:5b:80 INtf=1 Port=0
Invalid port "0"
Examples: fe0 ge1 3 cpu0 any
L3: Error: Unknown option: Port=0

Failed to execute the diagnostic command. Error: Operation failed.

Hardware-validated — Look up SONiC’s to-CPU egress object

drivshell> l3 egress find Mac=e0:39:d7:88:3b:00 INtf=16383 Port=cpu0
Egress object interface id: 132807
Environment variable (egr_object_id) was set
l3 multipath show [<entry>]

List the multipath (ECMP) egress objects, or one by ID. Safe. It prints nothing when there are none, as on a SONiC switch without ECMP routes. Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 9.3(x) – Troubleshooting VLANs shows l3 multipath show 200264 on a Nexus 9000.

Hardware-validated

drivshell> l3 multipath show
l3 multipath add Size=<n> [Intf0=<egr>] [Intf1=<egr>] [Intf2=<egr>] [Intf3=<egr>]

Create a multipath egress object from up to four egress objects. The built-in help writes the keys as INtf0…; the shell accepts Intf0…Intf3. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 multipath add Size=2 Intf0=100002 Intf1=100003
l3 multipath maxadd Size=<n> [MaxGrpSize=<n>] [Intf0=<egr>] [Intf1=<egr>] [Intf2=<egr>] [Intf3=<egr>]

Like add, but sets a maximum group size so the group can grow later. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 multipath maxadd Size=2 MaxGrpSize=8 Intf0=100002 Intf1=100003
l3 multipath update EgrId=<id> Size=<n> [Intf0=<egr>] [Intf1=<egr>] [Intf2=<egr>] [Intf3=<egr>]

Replace the members of a multipath object. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 multipath update EgrId=200001 Size=1 Intf0=100002
l3 multipath delete <entry>

Delete a multipath object. destroy is the same. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 multipath delete 200001
l3 multipath insert EgrId=<id> INtf=<egr>

Add one egress object to a multipath object. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 multipath insert EgrId=200001 INtf=100003
l3 multipath remove EgrId=<id> INtf=<egr>

Remove one egress object from a multipath object. The built-in help shows a bare <Intf>; the shell expects INtf=. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 multipath remove EgrId=200001 INtf=100003
l3 multipath find Size=<n> [Intf0=<egr>] [Intf1=<egr>] [Intf2=<egr>] [Intf3=<egr>]

Print the multipath object with exactly these members. Safe. If a member is not an existing egress object, the shell prints L3: Error finding mpath egress object entry: Invalid parameter, as in the example, whose IDs do not exist on Trident 3.

Hardware-validated

drivshell> l3 multipath find Size=2 Intf0=100002 Intf1=100003
L3: Error finding mpath egress object entry: Invalid parameter

Failed to execute the diagnostic command. Error: Operation failed.
l3 ecmp egress show [<entry>]

List the ECMP groups, or one by ID. The word egress is required: l3 ecmp show prints Unsupported command. With no ECMP groups it prints nothing. Safe. SONiC’s generate_dump runs this command (scripts/generate_dump (show techsupport collector)).

Hardware-validated

drivshell> l3 ecmp egress show
l3 ecmp egress add Size=<n> [Intf0=<egr>] [Intf1=<egr>] [Intf2=<egr>] [Intf3=<egr>]

Create an ECMP group. The other l3 ecmp egress sub-commands work like their multipath counterparts and take the same keys: maxadd, update, delete/destroy, insert EgrId=<id> INtf=<egr>, remove EgrId=<id> INtf=<egr> and find. All except find change the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 ecmp egress add Size=2 Intf0=100002 Intf1=100003

Unvalidated — Remove a member

drivshell> l3 ecmp egress remove EgrId=200001 INtf=100003
l3 l3table show [<entry>]

List the IPv4 host entries, or the entry with this index, after a free L3 table entries line. INTF is the egress object. Safe.

On a SONiC switch the switch’s own interface addresses are listed as (LOCAL ROUTE) entries pointing at the to-CPU egress object, and IPv4 neighbours appear as entries with their MAC and port.

Hardware-validated

drivshell> l3 l3table show
Unit 0, free L3 table entries: 16381
Entry VRF IP address       Mac Address           INTF MOD PORT    CLASS HIT    H/W Index
1     0    198.51.100.1     00:00:00:00:00:00  132807    0    0         1 n      7744       (LOCAL ROUTE)

Hardware-validated — No IPv4 neighbours installed

drivshell> l3 l3table show
Unit 0, free L3 table entries: 16381
Entry VRF IP address       Mac Address           INTF MOD PORT    CLASS HIT    H/W Index
1     0    198.51.100.1     00:00:00:00:00:00  132807    0    0         1 n      7744       (LOCAL ROUTE)
l3 l3table add VRF=<vrf> IP=<addr> Mac=<mac> INtf=<egr> Port=<port> [MOdule=<modid>] [Trunk=<tgid>] [Hit=<bool>] [PRI=<n>] [RPE=<bool>] [Untag=<bool>] [L2tocpu=<bool>] [HOST_AS_ROUTE=<bool>] [Replace=<bool>] [Group=<n>]

Add an IPv4 host entry. The built-in help shows LookupClass=; the shell accepts Group= instead. Changes the hardware behind SONiC’s back (the neighbour table).

Unvalidated

drivshell> l3 l3table add VRF=0 IP=192.0.2.10 Mac=00:00:5e:00:53:01 INtf=100002 Port=xe0
l3 l3table destroy VRF=<vrf> IP=<addr>

Delete an IPv4 host entry. delete is the same. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 l3table destroy VRF=0 IP=192.0.2.10
l3 l3table clear

Delete all IPv4 host entries.

Warning

This removes every IPv4 neighbour SONiC installed.

Unvalidated

drivshell> l3 l3table clear
l3 l3table age

Age the host table: clear the hit bits and delete entries that were not hit. SONiC does not expect its neighbours to disappear this way.

Unvalidated

drivshell> l3 l3table age
l3 l3table hash VRF=<vrf> IP=<addr> [SrcIP=<addr>] [SipKey=<n>] [VID=<vid>] [Hash=<type>]

Print the hash bucket of an IPv4 host key, as Hash(UC)[<hash>] of key 0x<key> is bucket 0x<n> (<n>). l3 l3table ip6hash IP=<addr6> … does the same for IPv6. Safe.

Hardware-validated

drivshell> l3 l3table hash VRF=0 IP=192.0.2.10
Hash(UC)[2] of key 0x00010500 is bucket 0xea4 (3748)
l3 l3table conflict VRF=<vrf> DIP=<addr> [SIP=<addr>] [DIP6=<addr6>] [SIP6=<addr6>] [IP6=<bool>] [IPMC=<bool>] [VLAN=<vid>]

Show the entries that hash into the same buckets as the given key. It prints nothing when there are none. Safe.

Hardware-validated

drivshell> l3 l3table conflict VRF=0 DIP=192.0.2.10
l3 l3table sanity

The built-in help shows Start=/End= options, but on Trident 3 this only prints Command only valid for XGS2 switches.

Unvalidated

drivshell> l3 l3table sanity
l3 defip show [<start> [<count>]]

List the IPv4 routes (the DEFIP table), all or a range. INTF is an egress object; see l3 egress show. Safe.

The first line gives the table size (Total Number of DEFIP entries: 32768 on Trident 3 with ALPM off); the # column is the TCAM index. On a SONiC switch, connected subnets point at the to-CPU egress object and the default route points at a drop object until SONiC installs a real default route.

Hardware-validated — output abridged

drivshell> l3 defip show
Unit 0, Total Number of DEFIP entries: 32768
#     VRF     Net addr             Next Hop Mac        INTF MODID PORT PRIO CLASS HIT VLAN
10240 0        198.51.100.0/24      00:00:00:00:00:00 132807    0     0     0    2 n
12288 0        0.0.0.0/0            00:00:00:00:00:00 132806    0     0     0    0 n

Hardware-validated — No IPv4 routes installed

drivshell> l3 defip show
Unit 0, Total Number of DEFIP entries: 32768
#     VRF     Net addr             Next Hop Mac        INTF MODID PORT PRIO CLASS HIT VLAN
10240 0        198.51.100.0/24      00:00:00:00:00:00 132807    0     0     0    2 n
12288 0        0.0.0.0/0            00:00:00:00:00:00 132806    0     0     0    0 n
l3 defip add VRF=<vrf> IP=<addr> MaSk=<mask> INtf=<egr> [Mac=<mac>] [Port=<port>] [MOdule=<modid>] [Trunk=<tgid>] [Untag=<bool>] [Local=<bool>] [ECMP=<bool>] [Hit=<bool>] [VLAN=<vid>] [DstDiscard=<bool>] [PRI=<n>] [RPE=<bool>] [Replace=<bool>] [MplsLabel=<label>] [TunnelOpt=<n>] [LookupClass=<n>] [IPMC=<bool>]

Add an IPv4 route. The built-in help shows Gateway=, but this command does not accept it. IPMC is accepted but not shown. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 defip add VRF=0 IP=198.51.100.0 MaSk=255.255.255.0 INtf=100002
l3 defip destroy VRF=<vrf> IP=<addr> MaSk=<mask> [ECMP=<bool>] [Mac=<mac>] [Gateway=<addr>] [INtf=<egr>] [Port=<port>] [MOdule=<modid>] [VLAN=<vid>]

Delete an IPv4 route. delete is the same. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 defip destroy VRF=0 IP=198.51.100.0 MaSk=255.255.255.0
l3 defip clear

Delete all IPv4 routes.

Warning

This removes every IPv4 route SONiC installed.

Unvalidated

drivshell> l3 defip clear
l3 defip age

Age the IPv4 routes: routes that were not hit are deleted. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 defip age
l3 defip ecmp MAX=<n>

Set the maximum number of ECMP paths. This overrides a global setting made by SONiC.

Unvalidated

drivshell> l3 defip ecmp MAX=64
l3 defip ecmphash DIP=<addr> SIP=<addr> COUNT=<n> [SIP6=<addr6>] [DIP6=<addr6>] [IP6=<bool>] [L4_SRC_PORT=<n>] [L4_DST_PORT=<n>] [EHash=<sel>] [Hash=<type>]

Print which ECMP member a flow hashes to, as ECMP hash is <n>. COUNT is the number of members minus one. The calculation needs no ECMP group to exist. Safe.

Hardware-validated

drivshell> l3 defip ecmphash DIP=198.51.100.1 SIP=192.0.2.1 COUNT=3
ECMP hash is 2
l3 alpm show [brief | all] [VRF=<vrf>] [Global-VRF=<bool>] [Pivot=<bool>] [Bucket=<bool>] [Counter=<bool>] [BucketuSage=<bool>] [InternalDebug=<bool>] [TrieTree=<bool>]

Dump the ALPM (algorithmic longest-prefix match) route structures. brief takes Bucket=<n> and Index=<n>. The l3 alpm commands work only when routes are stored in ALPM (config property l3_alpm_enable). Otherwise they print ALPM is unsupported or disabled!. The built-in help spells InternelDebug; the key is InternalDebug. Safe. See also the separate ALPM command.

On a SONiC Trident 3 switch running with ALPM off, every l3 alpm command prints only ALPM is unsupported or disabled!; use l3 defip show to see the routes.

Hardware-validated

drivshell> l3 alpm show brief
ALPM is unsupported or disabled!
l3 alpm find IP4=<addr> | IP6=<addr6> [VRF=<vrf>] [Length=<len>]

Show the route a longest-prefix-match lookup of the address hits, as Matching <vrf, addr/len> ==> …. Safe.

Hardware-validated

drivshell> l3 alpm find VRF=0 IP4=198.51.100.1
ALPM is unsupported or disabled!
l3 alpm sanity [Memory=<mem>] [Index=<n>] [Sw=<0|1>]

Check the ALPM route tables for consistency, all of them or one memory.

Unvalidated

drivshell> l3 alpm sanity
l3 ip6host show

List the IPv6 host entries. Safe. On a SONiC switch the switch’s link-local address appears as a (LOCAL ROUTE) entry.

Hardware-validated

drivshell> l3 ip6host show
Unit 0, free IPv6 table entries: 8190
Entry VRF IP address                                   Mac Address        INTF MOD PORT     CLASS HIT     H/W Index
1     0    fe80:0000:0000:0000:e239:d7ff:fe88:3b00     00:00:00:00:00:00 132807    0     0         1 n      1116
      (LOCAL ROUTE)
l3 ip6host add VRF=<vrf> IP=<addr6> Mac=<mac> INtf=<egr> Port=<port> [MOdule=<modid>] [Trunk=<tgid>] [Hit=<bool>] [PRI=<n>] [RPE=<bool>] [Replace=<bool>] [HOST_AS_ROUTE=<bool>] [Group=<n>]

Add or update an IPv6 host entry. It takes the same keys as l3 l3table add. The other forms are delete VRF=<vrf> IP=<addr6>, clear (deletes all IPv6 hosts) and age. All change the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 ip6host add VRF=0 IP=2001:db8::10 Mac=00:00:5e:00:53:01 INtf=100002 Port=xe0
l3 ip6route show [<start> [<count>]]

List the IPv6 routes, all or a range. Safe. The header gives the table size split into /64 and /128 entries, and the maximum ECMP width (128 on Trident 3). On a SONiC switch fe80::/10 points at the to-CPU egress object and ::/0 at the drop object.

Hardware-validated

drivshell> l3 ip6route show
Unit 0, Total Number of IPv6 entries: 4096 (IPv6/64 0, IPv6/128 4096)
Max number of ECMP paths 128
Free IPv6 entries available: 4096 (IPv6/64 0, IPv6/128 4096)
#     VRF     Net addr                                     Next Hop Mac        INTF MODID PORT PRIO CLASS HIT VLAN
13312 0        fe80:0000:0000:0000:0000:0000:0000:0000/10  00:00:00:00:00:00 132807    0     0     0    2 n
14336 0        0000:0000:0000:0000:0000:0000:0000:0000/0  00:00:00:00:00:00 132806    0     0     0    0 n

Hardware-validated — No IPv6 routes installed

drivshell> l3 ip6route show
Unit 0, Total Number of IPv6 entries: 4096 (IPv6/64 0, IPv6/128 4096)
Max number of ECMP paths 128
Free IPv6 entries available: 4096 (IPv6/64 0, IPv6/128 4096)
#     VRF     Net addr                                     Next Hop Mac        INTF MODID PORT PRIO CLASS HIT VLAN
13312 0        fe80:0000:0000:0000:0000:0000:0000:0000/10  00:00:00:00:00:00 132807    0     0     0    2 n
14336 0        0000:0000:0000:0000:0000:0000:0000:0000/0  00:00:00:00:00:00 132806    0     0     0    0 n
l3 ip6route add VRF=<vrf> IP=<addr> MaskLen=<len> MAC=<mac> INTF=<egress>

Add an IPv6 route. INTF must be an existing egress object (see l3 egress show) and the VRF must be usable; otherwise the shell prints L3: Error adding route table: Invalid parameter. It accepts the same optional keys as l3 defip add, with MaskLen instead of MaSk. Changes the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 ip6route add vrf=3 IP=2a10:11c0:: masklen=32 mac=00:09:0f:09:bc:07 intf=100008
l3 ip6route delete VRF=<vrf> IP=<addr6> MaskLen=<len> [ECMP=<bool>] [Mac=<mac>] [INtf=<egr>] [Port=<port>] [MOdule=<modid>]

Delete an IPv6 route. l3 ip6route clear deletes all IPv6 routes, and l3 ip6route age clears their hit bits. All change the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 ip6route delete VRF=3 IP=2a10:11c0:: MaskLen=32
l3 tunnel_init show INtf=<id>

Show the tunnel initiator of an L3 interface. Safe. If the interface has none, or does not exist, the shell prints ERROR L3: getting tunnel initiator for <id> Entry not found. SONiC uses no tunnels in a plain configuration, so this is what a SONiC switch prints.

Hardware-validated

drivshell> l3 tunnel_init show INtf=2
ERROR L3: getting tunnel initiator for 2 Entry not found

Failed to execute the diagnostic command. Error: Operation failed.
l3 tunnel_init set INtf=<id> TYpe=<n> TTL=<n> Mac=<mac> DIP=<addr> SIP=<addr> [DSCPSel=<n>] [DSCPV=<n>] [DFSEL4=<n>] [DFSEL6=<n>] [SIP6=<addr6>] [DIP6=<addr6>] [DSCPMap=<id>]

Attach a tunnel initiator to an L3 interface. DSCPMap is accepted but not shown in the built-in help. l3 tunnel_init clear INtf=<id> removes it. Both change tunnel encapsulation (for example VXLAN or IP-in-IP) behind SONiC’s back.

Unvalidated

drivshell> l3 tunnel_init clear INtf=2
l3 tunnel_term show TYpe=<n> VRF=<vrf> DIP=<addr> SIP=<addr> DipMaskLen=<len> SipMaskLen=<len> [DIP6=<addr6>] [SIP6=<addr6>] [L4DstPort=<n>] [L4SrcPort=<n>]

Look up one tunnel terminator by its key. There is no way to list them all. Safe. If there is no match, the shell prints ERROR L3: adding tunnel term Entry not found (the message says “adding” even for show).

Hardware-validated

drivshell> l3 tunnel_term show TYpe=1 VRF=0 DIP=192.0.2.1 SIP=0.0.0.0 DipMaskLen=32 SipMaskLen=0
ERROR L3: adding tunnel term Entry not found

Failed to execute the diagnostic command. Error: Operation failed.
l3 tunnel_term add TYpe=<n> VRF=<vrf> DIP=<addr> SIP=<addr> DipMaskLen=<len> SipMaskLen=<len> [DIP6=<addr6>] [SIP6=<addr6>] [L4DstPort=<n>] [L4SrcPort=<n>] [OuterTTL=<bool>] [OuterDSCP=<n>] [InnerDSCP=<n>] [TrustDSCP=<bool>] [VLanid=<vid>] [PBMP=<pbmp>] [AllowIPv6Payload=<bool>] [AllowIPv4Payload=<bool>] [Replace=<bool>] [RemoteTerminate=<n>] [TunnelID=<id>]

Add a tunnel terminator. RemoteTerminate and TunnelID are accepted but not shown in the built-in help. l3 tunnel_term delete takes only the key fields. Both change the hardware behind SONiC’s back.

Unvalidated

drivshell> l3 tunnel_term delete TYpe=1 VRF=0 DIP=192.0.2.1 SIP=0.0.0.0 DipMaskLen=32 SipMaskLen=0
l3 dscp_map show DscpMapId=<id>

Show a tunnel DSCP map. l3 dscp_map add creates one and prints its ID, destroy DscpMapId=<id> deletes it, and set DscpMapId=<id> Priority=<n> Color=<c> DSCPValue=<n> sets one entry. Only show is safe. For a map that does not exist, show prints the DSCP Map <id>: heading and then L3: Error L3 accessing: Invalid parameter; SONiC creates no tunnel DSCP maps, so map 0 does not exist there.

Hardware-validated

drivshell> l3 dscp_map show DscpMapId=0

DSCP Map 0:

-----------
L3: Error L3 accessing: Invalid parameter

Failed to execute the diagnostic command. Error: Operation failed.
l3 dscp_map_port show Port=<port>

Show the per-port tunnel DSCP map: a DSCP Map for port <n>: heading (the logical port number) and one line for each of the 8 priorities and 3 colours. On a SONiC switch every value is 0. l3 dscp_map_port set Port=<port> Priority=<n> Color=<c> DSCPValue=<n> changes it, and add means the same as set. Only show is safe.

Hardware-validated

drivshell> l3 dscp_map_port show Port=xe0

DSCP Map for port 9:

-----------
 Priority = 0, Color = Green,  DSCP = 0
 Priority = 0, Color = Yellow, DSCP = 0
 Priority = 0, Color = Red,    DSCP = 0
 Priority = 1, Color = Green,  DSCP = 0
 Priority = 1, Color = Yellow, DSCP = 0
 Priority = 1, Color = Red,    DSCP = 0
 Priority = 2, Color = Green,  DSCP = 0
 Priority = 2, Color = Yellow, DSCP = 0
 Priority = 2, Color = Red,    DSCP = 0
 Priority = 3, Color = Green,  DSCP = 0
 Priority = 3, Color = Yellow, DSCP = 0
 Priority = 3, Color = Red,    DSCP = 0
 Priority = 4, Color = Green,  DSCP = 0
 Priority = 4, Color = Yellow, DSCP = 0
 Priority = 4, Color = Red,    DSCP = 0
 Priority = 5, Color = Green,  DSCP = 0
 Priority = 5, Color = Yellow, DSCP = 0
 Priority = 5, Color = Red,    DSCP = 0
 Priority = 6, Color = Green,  DSCP = 0
 Priority = 6, Color = Yellow, DSCP = 0
 Priority = 6, Color = Red,    DSCP = 0
 Priority = 7, Color = Green,  DSCP = 0
 Priority = 7, Color = Yellow, DSCP = 0
 Priority = 7, Color = Red,    DSCP = 0
l3 prefix_map show [<entry>]

Show the table that maps IPv6 prefixes to IPv4 entries, under an Entry  IP6_ADDR header. When the table is empty, as on a SONiC switch, it prints nothing at all. add IP=<addr6>, delete IP=<addr6> and clear change it. Any other sub-command prints Unsupported command.

Hardware-validated

drivshell> l3 prefix_map show
l3 nat_ingress show

Show the NAT ingress rules (columns ID IP VRF L4Port IPProto NextHop). This and l3 nat_egress exist only on chips with NAT support; on others they print the usage. Trident 3 has them: a SONiC switch without NAT rules prints the header and the empty SNAT, DNAT and DNAT Pool sections. SONiC’s generate_dump runs them only if bcmcmd 'show feature' lists NAT. Safe.

Hardware-validated

drivshell> l3 nat_ingress show
IDX   ID    IP               VRF   L4Port IPProto NextHop
*****SNAT Entries*****
*****DNAT Entries*****
*****DNAT Pool Entries*****
l3 nat_egress add DNAT=<0|1> DIP=<addr> SIP=<addr> DIPMASK=<mask> SIPMASK=<mask> SRCPORT=<n> DSTPORT=<n>

Add a NAT egress translation and print NAT ID: <n>. l3 nat_egress show lists them and is safe. add changes the hardware.

Hardware-validated

drivshell> l3 nat_egress show
IDX   TYPE  ID    IP               MASK             L4Port RefCount
l3 route test perf [v4v6=<1-4>] [loops=<n>] [nsets=<n>] [UseApi=<0|1>] [OPs=<bitmap>] [GblPri=<n>]

A route add/update/lookup/delete benchmark. None of its keys is in the built-in help. v4v6: 1 IPv4, 2 IPv6, 3 IPv4 then IPv6, 4 IPv6 then IPv4. OPs bits: 0 add, 1 update, 2 lookup, 3 delete.

Warning

Never run this on a production switch. It turns off l2mode, linkscan, counter, memscan and sramscan, creates and then deletes L3 interface 1 and its own egress objects, and turns the background tasks back on at the end. On SONiC this deletes SONiC’s interface 1 and pauses link and counter handling while it runs.

Unvalidated

drivshell> l3 route test perf v4v6=1 OPs=7

Built-in help

Usages:
      l3 init    - Init L3 function.
      l3 detach  - Detach L3 function.
      l3 info    - Display the capacity and status of the L3 resources.

      l3 intf add <Vlan>=<val> <Mac>=<val> [<INtf>=<val>] [<VRF>=<val>]
                       [<Mtu>=<val>] [<Group>=<val>]
                       - Add an entry into l3 interface table
      l3 intf destroy <INtf>=<val>
                       - Destroy an entry in l3 interface table
      l3 intf clear    - Destroy all intf
      l3 intf show     - [<INtf>=<val>]
                         Display all Intfs or a single one

      l3 egress add <Mac>=<val> <INtf>=<val> [<Trunk>=<val>]
                      <Port>=<val> [<MOdule>=<val>] [<L2tocpu>=<val>]
                      [<L3MC>=<val>] [<MplsLabel>=<val>] [<Drop>=<val>]
                      [<CopyToCpu>=<val>]
                      - Create egress object entry
      l3 egress update <EgrId>=<val> <Mac>=<val> <INtf>=<val>
                      [<Trunk>=<val>] <Port>=<val> [<MOdule>=<val>]
                      [<L2tocpu>=<val>] [<L3MC>=<val>] [<MplsLabel>=<val>]
                      - Update egress object entry
      l3 egress delete <entry>
                      - Delete egress object entry
      l3 egress show [entry]
                      -Show egress object entry
      l3 egress find <Mac>=<val> <INtf>=<val> [<Trunk>=<val>]
                      <Port>=<val> [<MOdule>=<val>] [<L2tocpu>=<val>]
                      [<L3MC>=<val>] [<MplsLabel>=<val>]
                      - Find egress object entry
      l3 multipath add <Size>=<val> [<INtf0>=<val>] [<INtf1>=<val>]
                      [<INtf2>=<val>] [<INtf3>=<val>]
                      - Create multipath egress object entry
      l3 multipath maxadd <Size>=<val> [<MaxGrpSize>=<val>] [<INtf0>=<val>] [<INtf1>=<val>]
                      [<INtf2>=<val>] [<INtf3>=<val>]
                      - Create multipath egress object entry(dynamic grp size)
      l3 multipath update <EgrId>=<val> <Size>=<val> [<INtf0>=<val>]
                      [<INtf1>=<val>] [<INtf2>=<val>] [<INtf3>=<val>]
                      - Update multipath egress object entry
      l3 multipath delete <entry>
                      - Delete multipath egress object entry
      l3 multipath show [entry]
                      -Show multipath egress object entry
      l3 multipath insert <EgrId>=<val> <Intf>=<val>
                      -Add interface to multipath egress object entry
      l3 multipath remove <EgrId>=<val> <Intf>
                      -Remove interface from multipath egress object entry
      l3 multipath find <Size>=<val> [<INtf0>=<val>] [<INtf1>=<val>]
                      [<INtf2>=<val>] [<INtf3>=<val>]
                      - Find multipath egress object entry
      l3 ecmp egress add <Size>=<val> [<INtf0>=<val>] [<INtf1>=<val>]
                      [<INtf2>=<val>] [<INtf3>=<val>]
                      - Create ecmp egress object entry
      l3 ecmp egress maxadd <Size>=<val> [<MaxGrpSize>=<val>] [<INtf0>=<val>] [<INtf1>=<val>]
                      [<INtf2>=<val>] [<INtf3>=<val>]
                      - Create ecmp egress object entry(dynamic grp size)
      l3 ecmp egress update <EgrId>=<val> <Size>=<val> [<INtf0>=<val>]
                      [<INtf1>=<val>] [<INtf2>=<val>] [<INtf3>=<val>]
                      - Update ecmp egress object entry
      l3 ecmp egress delete <entry>
                      - Delete ecmp egress object entry
      l3 ecmp egress show [entry]
                      -Show ecmp egress object entry
      l3 ecmp egress insert <EgrId>=<val> <Intf>=<val>
                      -Add interface to ecmp egress object entry
      l3 ecmp egress remove <EgrId>=<val> <Intf>
                      -Remove interface from ecmp egress object entry
      l3 ecmp egress find <Size>=<val> [<INtf0>=<val>] [<INtf1>=<val>]
                      [<INtf2>=<val>] [<INtf3>=<val>]
                      - Find ecmp egress object entry
      l3 l3table add <VRF>=<val> <IP>=<val> <Mac>=<val> <INtf>=<val>
           [<Hit>=<val>] [<Trunk>=<val>] [MOdule=<val>] <Port>=<val>
           <PRI>=<val> <RPE>=<val> [<Untag>=<val>] [<L2tocpu>=<val>]
           [<HOST_AS_ROUTE>=<val>] [Replace=<bool>] <LookupClass>=<val>
      l3 l3table destroy <VRF>=<val> <IP>=<val>
                       - Destroy an entry in l3 table
      l3 l3table clear - Destroy all l3 table
      l3 l3table show  - Display all l3 table or a single one
      l3 l3table age   - Age L3 table
      l3 l3table sanity <Start=<index> End=<index>
                       - Check l3x tables for (san)ity in the given range
      l3 l3table hash <VRF>=<val> <IP>=<val> [<SrcIP>=<val>] [<SipKey>=<val>]
                        [<VID>=<val>] [<Hash>=<hash_type>]
                       - Calculate l3 hash index
      l3 l3table ip6hash <IP>=<val> [<SrcIP>=<val>] [<SipKey>=<val>]
                        [<VID>=<val>] [<Hash>=<hash_type>]
                       - Calculate l3 hash index for IPv6 entry
      l3 l3table untag  - Recalculate untagged bits for L3

      l3 l3table conflict <VRF>=<val> <DIP>=<val> <DIP6>=<val>
                         <SIP>=<val> <SIP6>=<val> <IP6>=<val> <IPMC>=<val>
                         <VLAN>=<val>
                         - Get conflicting l3 entries.

      l3 defip add <VRF>=<val> <IP>=<val> <MaSk>=<val> <Mac>=<val>
           <INtf>=<val> <Gateway>=<next hop Ip address> <Port>=<val>
           [MOdule=<val>] [<Untag>=<val>] [<Local>=<val>] [<ECMP>=<1/0>]
           [<Hit>=<val>] [<VLAN>=<val>](BCM5695 def route 0.0.0.0/0 only)
           [<DstDiscard>=<val>] <PRI>=<val> <RPE>=<val> [Replace=<bool>]
           [<MplsLabel>=<val>] <TunnelOpt>=<val> <LookupClass>=<val>
                       - Add an IPv4 entry into DEFIP table
      l3 defip destroy <VRF>=<val> <IP>=<val> <MaSk>=<val>
           [<ECMP=<1/0> <Mac>=<val> <Gateway>=<next hop Ip address>
            <INtf>=<val> <Port>=<val> [MOdule=<val>]
           [<VLAN>=<val>](BCM5695 def route 0.0.0.0/0 only)
                       - Destroy an IPv4 entry in def IP table
      l3 defip clear   - Destroy all IPv4 DEFIP table
      l3 defip age     - Age DEFIP(LPM) table(XGS Switch only)
      l3 defip show [<START> [<COUNT>]]
                       - Display all IPv4 DEFIP table or
                         the entires in a specified range
      l3 defip ecmp <max>=<val> - Set MAX number of ECMP paths allowed
      l3 defip ecmphash <DIP>=<val> <SIP>=<val> <SIP6>=val <DIP6>=val
                        <COUNT>=<val> <IP6>=val  <L4_SRC_PORT>=val
                        <L4_DST_PORT>=val [<EHash>=<ECMP hash Sel>]
                        [<Hash>=<L3 hash type>]
                       - Calculate ECMP hash index (COUNT is zero based)
      l3 alpm show [all] [<VRF>=<val>] [<Global-vrf>=<bool>]
           [<Pivot>=<bool>] [<Bucket>=<bool>] [<Counter>=<bool>]
           [BucketuSage=<bool>] [InternelDebug=<bool>
                       - Show ALPM debug info

      l3 alpm show brief [<Bucket>=<val>] [<Index>=<val>]
                       - Show ALPM brief debug info

      l3 alpm find [<VRF>=<val>] <IP4>=<val> <IP6>=<val> [<Length>=<val>]
                       - Find hit longest prefix match route

      l3 alpm sanity [<memory>=<val>] [<index>=<val>] [<sw>=<0/1>]
                       - Process sanity check for ALPM tables

      l3 ip6host add <VRF>=<val> <IP>=<val> <Mac>=<val> [MOdule=<val>]
           <Port>=<val> [<Trunk>=<val>] <INtf>=<val> [<Hit>=<val>]
           <PRI>=<val> <RPE>=<val> [<Replace>=<bool>]
           [<HOST_AS_ROUTE>=<bool>] <LookupClass>=<val>
                       - Add/Update an IPv6 entry
      l3 ip6host delete <VRF>=<val> <IP>=<val>
                       - Delete an IPv6 entry
      l3 ip6host clear - Delete all IPv6 host entries
      l3 ip6host age   - Age IPv6 host entries
      l3 ip6host show  - Show all IPv6 host entries

      l3 ip6route add  <VRF>=<val> <IP>=<val> <MaskLen>=<val> <Mac>=<val>
                       [MOdule=<val>] <Port>=<val> [<Trunk>=<val>]
                       <INtf>=<val> <VLan>=<val> [<Hit>=<val>]
                       [<ECMP>=<1/0>] [<DstDiscard>=<val>] <PRI>=<val>
                       <RPE>=<val> [Replace=<bool>] [<MplsLabel>=<val>]
                       <TunnelOpt>=<val> <LookupClass>=<val>
                        - Add IPv6 route
      l3 ip6route delete <VRF>=<val> <IP>=<val> <MaskLen>=<val>
           [<ECMP=<1/0> <Mac>=<val> <INtf>=<val> <Port>=<val> [MOdule=<val>]
                        - Delete an IPv6 route
      l3 ip6route clear - Delete all IPv6 routes
      l3 ip6route age   - Clear all IPv6 route HIT bits
      l3 ip6route show [<START> [<COUNT>]]
                        - Display all IPv6 routes or
                          the routes in a specified range
      l3 tunnel_init set <INtf>=<val> <TYpe>=<val> <TTL>=<val>
           <Mac>=<val> <DIP>=<val> <SIP>=<val> <DSCPSel>=<val> <DSCPV>=<val>
           <DFSEL4>=<val> <DFSEL6>=<val> <SIP6>=<val> <DIP6>=<val>
      l3 tunnel_init show <INtf>=<val>
      l3 tunnel_init clear <INtf>=<val>
      l3 tunnel_term add <TYPE>=<val> <VRF>=<val> <DIP>=<val> <SIP>=<val>
           <DipMaskLen>=<val> <SipMaskLen>=<val> <DIP6>=<val> <SIP6>=<val>
           <L4DstPort>=<val> <L4SrcPort>=<val> <OuterTTL>=<bool>
           <OuterDSCP>=<val> <InnerDSCP>=<val> <TrustDSCP>=<val>
           <VLanid>=<val> <PBMP>=<val> <AllowIPv6Payload>=<bool>
           <AllowIPv4Payload>=<bool> <Replace>=<bool>
      l3 tunnel_term delete <TYPE>=<val> <VRF>=<val> <DIP>=<val> <SIP>=<val>
           <DipMaskLen>=<val> <SipMaskLen>=<val> <DIP6>=<val> <SIP6>=<val>
           <L4DstPort>=<val> <L4SrcPort>=<val>
      l3 tunnel_term show <TYPE>=<val> <VRF>=<val> <DIP>=<val> <SIP>=<val>
           <DipMaskLen>=<val> <SipMaskLen>=<val> <DIP6>=<val> <SIP6>=<val>
           <L4DstPort>=<val> <L4SrcPort>=<val>
      l3 dscp_map add
      l3 dscp_map destroy <DscpMapId>=<val>
      l3 dscp_map set <DscpMapId>=<val> <Priority>=<val> <Color>=<val>
                      <DSCPValue>=<val>
      l3 dscp_map show <DscpMapId>=<val>
      l3 dscp_map_port set <Port>=<val> <Priority>=<val> <Color>=<val>
                      <DSCPValue>=<val>
      l3 dscp_map_port show <Port>=<val>
      l3 prefix_map add <IP>=<val>
      l3 prefix_map delete <IP>=<val>
                      - Destroy ip6 prefix map to IPv4 entry
      l3 prefix_map clear
                      - Flush ip6 prefix map to IPv4 entry
      l3 prefix_map show [entry]
                      - Show ip6 prefix map to IPv4 entry
      l3 nat_ingress show
                      - Show all NAT Ingress Rules
      l3 nat_egress add <DNAT=0/1> <DIP=new_ip> <SIP=new_ip> <DIPMASK=mask>
           <SIPMASK=mask> <SRCPORT=new_port> <DSTPORT=new_port>
                      - Add a new Egress Translation Rule
      l3 nat_egress show
                      - Show all NAT Egress Translation Rules
      l3 route test perf
                      - Run IPv4/IPv6 route performance test

Further reading

bcmsh.md (published as https://kamelnetworks.github.io/sonic/bcmsh.html), SDK-6.5.20 HSDK Getting Started Guide (GS-SDK6520-R), plus SDK-6.5.20/21/22 Release Notes (RN-SDK65xx-R), SDK-6.5.20..6.5.27 Resolved-Issues-Improvements.xlsx (7 files), sonic-mgmt test suite files invoking bcmcmd (19 files), scripts/generate_dump (show techsupport collector), Cisco Nexus 9000 Series NX-OS Programmability Guide, Release 7.x – Broadcom Shell, Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 9.3(x) – Troubleshooting VLANs, Dell Networking OS10 Routes are Not Installed (KB 000322394), H3C S12500-F系列交换机 Probe命令参考(R100x系列)-6W100, H3C S5830V2[S5820V2] Probe Command Reference – 01-Probe Commands, FBOSS: BcmSwitch gatherSdkState/rcload, validated shell commands, diag shell stress test, vendor-diag-shell skill, debugging doc, BCM-SDK命令的使用.pdf, 交换芯片Helix4 L2 L3功能验证, MPLS_cint脚本实现 (MPLS via CINT and diag commands), Edgecore AS5610-52X reverse engineering: Cumulus 2.5 baseline captures.