Forwarding table sizes

Trident 3 has a pool of shared hash memory, the unified forwarding table (UFT), that is divided between the MAC (L2) table, the L3 host table, the exact-match table and ALPM route storage when the chip is initialised. The division is set in config.bcm and cannot change while the switch runs.

Shared table banks

Each table has a dedicated part and can take shared banks on top:

Variable	Dedicated	One shared bank adds
l2_mem_entries	32768	32768
l3_mem_entries	16384	32768
fpem_mem_entries	0	16384

The SDK rounds each request up to whole banks. A request for one exact-match bank gets two. At most 8 shared banks can be given out in total. With l3_alpm_enable set, the MAC, host and exact-match tables can use at most four shared banks; ALPM uses the rest. If the requests do not fit, chip initialisation fails and syncd does not start.

For example, l2_mem_entries=163840 (used by some SONiC Trident 3 HWSKUs) asks for 131072 entries beyond the dedicated 32768, which is four shared banks. That fits next to ALPM only if l3_mem_entries and fpem_mem_entries stay at their dedicated sizes.

L3 host entries and route prefixes are different resources: host entries (exact /32 and /128 neighbours) live in the host table, route prefixes in the LPM TCAM or, with ALPM, in the ALPM banks.

Example: a larger MAC table

To double the MAC table on a switch without ALPM:

-l2_mem_entries=32768
+l2_mem_entries=65536

This uses one shared bank. After a syncd restart, check the value the SDK read and the capacity SONiC sees:

Unvalidated

drivshell> config get l2_mem_entries
drivshell> echo $config_var_val

Unvalidated

admin@sonic:~$ crm show resources all

Route tables

Without ALPM, route prefixes are stored in the LPM TCAM.

ipv6_lpm_128b_enable

Allows IPv6 prefixes longer than /64. Every SONiC Trident 3 file sets it.

num_ipv6_lpm_128b_entries

How many of those long IPv6 prefixes to reserve room for (default 4096). Each takes the space of several IPv4 or short IPv6 prefixes.

lpm_scaling_enable

1 also allows 64-bit IPv6 prefixes in the unreserved paired TCAM. The Celestica Seastone2 files set it.

l3_alpm_enable

1 (parallel) or 2 (combined) moves prefixes to ALPM, which holds many more routes in shared UFT banks. The Dell S5232F-ON files use l3_alpm_enable=2 with lpm_scaling_enable=0.

The number of routes that fit depends on the prefix lengths and the mix of IPv4 and IPv6; there is no single “route count”. crm show resources all shows what SONiC can still add.

ECMP

l3_ecmp_levels

1 (default) or 2. Two levels (hierarchical ECMP) are needed for VXLAN overlay ECMP across underlay ECMP and split the ECMP tables between the levels.

l3_max_ecmp_mode

ECMP scaling mode; 1 in most SONiC Trident 3 files.

SAI_NUM_ECMP_MEMBERS in sai.profile

The largest ECMP group SONiC creates (64 in the lab HWSKUs). It is a SAI setting, not a configuration variable.

VXLAN and routing into tunnels

VXLAN in SONiC needs the flex-flow VXLAN engine, routing in and out of tunnels (RIOT) and SAI tunnel support. The Seastone2 VXLAN HWSKU sets:

sai_tunnel_support=1
use_all_splithorizon_groups=1
bcm_tunnel_term_compatible_mode=1
flow_init_mode=1
l3_ecmp_levels=2
riot_enable=1
riot_overlay_l3_intf_mem_size=8192
riot_overlay_l3_egress_mem_size=32768
riot_overlay_ecmp_resilient_hash_size=16384

The three riot_overlay_* sizes reserve router interfaces, next hops and resilient-hash entries for the overlay out of the shared L3 tables; the underlay gets less.

All table variables are in the table variable reference.