Configuration variables

The SDK and the Broadcom SAI are configured by configuration variables, also called SOC properties: name=value pairs such as l2_mem_entries=32768 or portmap_1=1:100. They decide things that cannot change while the switch runs: which SerDes lanes form which port, how the shared forwarding tables are divided, how the buffers are set up, which optional features are started.

On SONiC the variables come from the HWSKU’s config.bcm file. This chapter explains where that file is, how it is read, how variable names are looked up, and when a change takes effect. The pages below cover port mapping, table sizing and buffers in detail, and the variable reference lists every variable the Trident 3 SAI and SDK read.

Where the configuration comes from

Each SONiC platform directory has one directory per HWSKU (hardware SKU), /usr/share/sonic/device/<platform>/<hwsku>/ on the host. Inside the syncd container the same directory is /usr/share/sonic/hwsku/. It holds port_config.ini (SONiC’s port names and lanes), the buffer and QoS templates, sai.profile and the config.bcm file. sai.profile names the file:

SAI_INIT_CONFIG_FILE=/usr/share/sonic/hwsku/td3-seastone_2-32x100G-vxlan.config.bcm
SAI_NUM_ECMP_MEMBERS=64

The other sai.profile keys, such as SAI_NUM_ECMP_MEMBERS, are read by the SAI directly; they are not configuration variables and cannot be set in config.bcm.

When syncd starts, the SAI reads config.bcm line by line and stores every name=value pair in the SDK’s configuration table. It then adds a few values of its own before it initialises the chip:

  • l3_intf_vlan_split_egress=1, always, whatever the file says;

  • bcm_tunnel_term_compatible_mode=1, sram_scan_enable=1 and mem_cache_enable=1 if the file does not set them;

  • scache_filename, the warm-boot state file.

During initialisation each part of the SDK and the SAI looks up the variables it needs. A variable that nobody looks up has no effect, and there is no warning: a misspelt name or a variable meant for another chip is accepted and ignored. config dump shows which variables were looked up (see Inspecting the configuration).

Two more variables point to diag-shell scripts that the SAI runs during a cold start: sai_preinit_cmd_file before the chip’s init misc step, sai_postinit_cmd_file after initialisation. Dell S5232F-ON HWSKUs use the first to load the LED firmware with m0.

File format

  • One name=value per line. The name ends at the first =; spaces and tabs around the name and the value are removed.

  • A line whose first character is # is a comment. There are no end-of-line comments: in l2_mem_entries=32768  # default the value is 32768  # default. Numeric variables stop reading at the space, but string values keep the text.

  • Empty lines are ignored. Keep lines shorter than 256 characters.

  • If a name appears twice, the later line wins.

  • Names are case-sensitive and are not checked. PHY_AN_ALLOW_PLL_CHANGE=1, found in some SONiC files, is never read; the SDK looks up phy_an_allow_pll_change.

  • Numbers are C-style: 32768, 0x8000. Port bitmaps are hexadecimal numbers with one bit per logical port (bit 1 is logical port 1).

Names, scopes and precedence

Many variables apply to one port, lane, unit or index. The SDK looks them up by trying a list of names, from the most specific to the plain name, and takes the first one that exists. A variable that is not found anywhere gets a default chosen by the code that reads it.

Global variables

name.<unit> (l2_mem_entries.0), then name.<chip> (name.BCM56870_A0), name.<chip group> (name.BCM56870), and finally name. The unit and chip suffixes matter only when one configuration serves several chips; on SONiC there is one unit, 0. Some SONiC files write every variable with .0 (portmap_1.0=1:100); this is the same as writing it without.

Per-port variables

name_<port name> (serdes_preemphasis_ce0), name_<port type> (serdes_if_type_ce: all ce ports), name_<logical port> (serdes_preemphasis_1), and finally name for all ports. Each of these can also carry the unit or chip suffix. Port names depend on the port’s speed (see Ports and port bitmaps), so HWSKU files use logical port numbers.

Per-lane (physical port) variables

name{<physical>.<phy>.<phy port>.<lane>}, name{<physical>.<phy>}, name{<physical>}, then the per-port forms of the logical port on that lane. <phy> is 0 for the switch’s own SerDes and 1 and up for external PHYs, so SONiC files write phy_chain_tx_polarity_flip_physical{9.0}=1.

Indexed variables

name_<suffix><N> (ptp_bs_fref_0), then name_<suffix>, then name. Per-port indexed variables combine both: serdes_preemphasis_lane2_9 is lane 2 of logical port 9.

MMU buffer variables

buf.<object><N>.<attribute>, falling back to buf.<object>.<attribute>, each with the per-port forms. See Buffers and queues.

The SDK keeps a per-variable flag when a lookup hits, but it does not record which of the forms matched. If two forms are set, the more specific one wins: with phy_an_c73=1 and phy_an_c73_66=0 port 66 gets 0 and every other port 1.

When changes take effect

Almost every variable is read once, while syncd initialises the chip. Editing config.bcm changes nothing until syncd restarts:

  • sudo config reload restarts syncd together with the other SONiC services and reinitialises the chip. Traffic stops while it runs.

  • A cold sudo reboot does the same and is the safest choice after a port map or table-size change.

  • Do not change config.bcm before a warm or fast reboot. These restart syncd from saved state that was built with the old values.

Edit the file on the host, under /usr/share/sonic/device/, not inside the container. The change survives a reboot but not the installation of another SONiC image, which brings its own copy of the device directory; keep the modified HWSKU in your own configuration management.

A port map change must go together with port_config.ini (SONiC’s view of the same ports) and, for SONiC 202012 and later, the PORT table in config_db.json. Port mapping and SerDes has a complete example.

Changing variables from the shell

config changes the SDK’s in-memory copy of the variables. The file is not touched, and a later syncd restart reads the file again. Because the variables are read during initialisation, a change made with config usually only takes effect after init, which reinitialises the chip behind SONiC’s back and leaves the switch inconsistent until syncd is restarted. Changing the chip configuration live describes this for experiments.

Warning

On SONiC, config refresh does not reread config.bcm: it empties the in-memory configuration. Anything that reads a variable afterwards gets the built-in default. Do not use it on a running switch.

Inspecting the configuration

config show lists the variables, optionally only those whose name contains a string. config get puts one value in the shell variable config_var_val for use in scripts:

Hardware-validated

drivshell> config get l2_mem_entries
Environment variable (config_var_val) was set
drivshell> echo $config_var_val $config_var_not_set
32768 0

config dump shows every variable with a flag. 0x0004 means the SDK has looked the variable up since start-up; 0x0000 means nothing has read it so far. That is the quickest way to find a variable that has no effect. On a Celestica Seastone2 running SONiC, two variables from its config.bcm were never read:

Hardware-validated — Excerpt: variables read (0x0004) and not read (0x0000)

drivshell> config dump
...
hash:  41, line:   0, flag: 0x0004, portmap_9=9:10:1
...
hash: 291, line:   0, flag: 0x0000, serdes_lane_config_dfe=on
...
hash: 488, line:   0, flag: 0x0004, l2_mem_entries=32768
...
hash: 890, line:   0, flag: 0x0000, module_64ports=1
...

module_64ports and serdes_lane_config_dfe had no effect on that switch. 0x0000 is also normal for variables that are read only on demand: memlist_enable and reglist_enable are read when listmem or listreg runs.

To compare a switch with its file, capture the shell’s view and the file side by side (the shell output is indented by four spaces):

Unvalidated

admin@sonic:~$ bcmcmd "config show" | sed -n 's/^    //p' | sort > /tmp/live.txt
admin@sonic:~$ docker exec syncd sh -c 'cat /usr/share/sonic/hwsku/*.config.bcm' | grep -v '^#' | sort > /tmp/file.txt
admin@sonic:~$ diff /tmp/file.txt /tmp/live.txt