Redundant System Clock Synchronization

About this task

The phc2sys application can be configured to accept multiple source clock inputs. The quality of these sources are compared to user-defined priority values and the best available source is selected to set the system time.

The quality of the configured sources is continuously monitored by phc2sys application and will select a new best source if the current source degrades or if another source becomes higher quality.

The following section describes how to configure phc2sys with multiple clock sources, and manage and monitor the phc2sys application.

HA Phc2sys configuration

Procedure

1. Create a phc2sys instance with multiple source selection enabled.

   Note

   See the HA Phc2sys Configuration Parameters section for additional information on parameters required for configuration.

   # Set bash variables to simplify commands
   INSTANCE=phc2sys1
   TYPE=phc2sys
   IFNAME1=phc1-iface1
   IFNAME2=phc1-iface2
   HOST=controller-0
   

   # Create instance
   system ptp-instance-add $INSTANCE $TYPE
   
   # Create interface(s)
   system ptp-interface-add $IFNAME1 $INSTANCE
   system ptp-interface-add $IFNAME2 $INSTANCE
   
   # Assign host interfaces
   system host-if-ptp-assign $HOST data0 $IFNAME1
   system host-if-ptp-assign $HOST data1 $IFNAME2
   

2. Assign global instance parameters. The parameters described in HA Phc2sys Configuration Parameters are required for HA Phc2sys functionality. The domainNumber value can be included if all interfaces use the same ptp4l domain. If ptp4l instances use different domains, see the interface configuration options for each interface domain numbers.

   # Assign required global instance parameters
   system ptp-instance-parameter-add $INSTANCE \
   ha_enabled=1 \
   ha_phc2sys_com_socket=/var/run/$INSTANCE \
   domainNumber=24
   

3. Assign optional global instance parameters. These parameters determine what clock quality values are required by phc2sys to consider a source valid for selection. See HA Phc2sys Configuration Parameters for more details on the behaviour of these parameters.

   system ptp-instance-parameter-add $INSTANCE \
   ha_max_gm_clockClass=6
   

4. Assign CLI options. This parameter is required for HA Phc2sys operation.

   system ptp-instance-parameter-add $INSTANCE \
   cmdline_opts='-w'
   

5. Assign required interface parameters. The ha_domainNumber parameter can be ommited if a shared domain number is used and is configured under the global instance parameters.

   system ptp-interface-parameter-add $IFNAME1 \
   ha_priority=100 \
   ha_uds_address=/var/run/ptp4l-ptp1 \
   ha_domainNumber=24
   
   system ptp-interface-parameter-add $IFNAME2 \
   ha_priority=90 \
   ha_uds_address=/var/run/ptp4l-ptp2 \
   ha_domainNumber=25
   

6. Assign the instance to a host and apply the PTP configuration.

   system host-ptp-instance-assign $HOST $INSTANCE
   system ptp-instance-apply
   

HA Phc2sys Configuration Parameters

Note

The parameters listed here can be used in combination with the default phc2sys parameters. Use the phc2sys help page for additional parameters.

Global Instance Parameters

ha_enabled

The ha_enabled is a global setting to enable or disable the HA multiple ptp4l inputs support. The default is 0 (disabled). Set 1 to enable HA multiple input support.

ha_stability_timer

The ha_stability_timer option is a global setting used to configure the stability timer. The value is expressed in seconds, and the value 0 disables the timer. The default value is 0 (disabled).

When the timer is enabled, phc2sys waits a few seconds after a source becomes a candidate for selection before allowing it to be selected as the synchronization source. This can be useful to prevent frequent source switching if a given source is frequently changing quality. A value of 300 would cause phc2sys to ensure that a source has had stable quality for 300 seconds before it is eligible to be used as a synchronization source.

The stability timer only applies to sources that were in a degraded state and have recovered to meet the selection threshold. If the active source degrades and phc2sys determines that a new source must be selected, this happens without delay.

The ha_stability_timer is important when the clock has different priorities (ha_priority) to prevent multiple and consecutive clock switches. When the service starts with both clocks in a valid state, the higher priority source is selected, and when it degrades the lower priority is selected quickly. Then when the higher priority recovers it will only become active after the stability timer expires.

ha_phc2sys_com_socket

Provides a path for the communication socket used for monitoring and maintenance activities. This value is required for system alarming to check the phc2sys state. It is recommended to create the socket in the /var/run directory, for example, ha_phc2sys_com_socket=/var/run/<phc2sys instance name> See :ref: manage-phc2sys for managing interaction with phc2sys via this socket.

Source Selection Parameters

The parameters in this section are applied at the instance level. For example, system ptp-instance-parameter-add <phc2sys instance name> ha_max_gm_clockClass=7.

These parameters set the threshold for a source to be considered eligible for use in setting the system time. A source that does not meet one or more of the requirements will not be selected. Phc2sys then selects the best available source that meets the threshold. The interface parameter ha_priority is used as a tiebreaker for sources of equal quality, with the higher priority being selected.

The default value for each parameter is in effect even if the parameter is not explicitly defined in the configuration file.

Note

In most cases, users may use the “GM clock values” set of parameters to configure the selection threshold. The local clock values are available for additional fine tuning if required, but are generally not a good indicator of the quality of a clock source compared to the GM values.

Parameter	Default Value	Range	Description
GM clock values
ha_max_gm_clockClass	6	6-255	The ha_max_gm_clockClass option is a global setting used to configure the maximal GM clock class requirement. It ranges from 6 to 255 and its default is 6. A source with a higher clockClass will not be selected.
ha_max_gm_clockAccuracy	0xfe	0x00-0xff	A source must have a clockAccuracy value equal or lower than this parameter to be selected.
ha_max_gm_offsetScaledLogVar	0xffff	0x0000-0xffff	A source must have an offsetScaledLogVariance value equal or lower than this parameter to be selected.
ha_gm_timeTraceable	0	0/1	If this parameter is set to 1, then a source must have the timeTraceable flag set in order to be selected.
ha_gm_frequencyTraceable	0	0/1	If this parameter is set to 1, then a source must have the frequencyTraceable flag set in order to be selected.
Local clock values
ha_max_local_clockClass	255	6-255	Same as ha_max_gm_clockClass, but only considers the local clock class, not the class received from the parent time source.
ha_max_local_clockAccuracy	0xfe	0x00-0xff	Only considers the local clockAccuracy.
ha_max_local_offsetScaledLogVar	0xffff	0x0000-0xffff	Only considers the local offsetScaledLogVariance

Interface Parameters

ha_uds_address

The ha_uds_address is a required parameter for each source configured in phc2sys. This provides a path to the associated ptp4l instance on that source which allows phc2sys to determine the quality of the source.

This value should match the uds_address parameter of the associated ptp4l instance. For example, ha_uds_address=/var/run/ptp4l-ptp1.

ha_priority

The ha_priority option is an interface setting used to configure the clock priority. The lowest priority is 0, highest is 254, and the default value is 0.

If all clock quality parameters are equal between multiple sources, then the highest priority source is selected.

If multiple sources have the same clock quality and the same priority, then phc2sys will select the first interface in the configuration file.

An interface configured with ha_priority 254 will force phc2sys to select that source, regardless of its status.

If more then one interface is configured with ha_priority 254 the first interface in the configuration file with priority 254 is selected as clock source.

ha_domainNumber

The ha_domainNumber option is an interface setting to configure the domain number for a UDS socket. It ranges from 0 to 127. The ha_domainNumber has no default value, if the ha_domainNumber is not configured for a given interface, the global domainNumber setting is used.

This allows for configurations where multiple ptp4l instances providing clock source quality information may be in different domains.

If all ptp4l instances on the system are in the same domain, it is recommended to use the domainNumber parameter in the global section.

Example Configuration File

The following is an example configuration file for HA Phc2sys which shows how services can be configured with multiple interfaces tracking separate ptp4l instances for quality information.

[global]
##
## Default Data Set
##
domainNumber 24
ha_enabled 1
ha_gm_frequencyTraceable 1
ha_gm_timeTraceable 1
ha_max_gm_clockAccuracy 0x20
ha_max_gm_clockClass 6
ha_max_gm_offsetScaledLogVar 0x4e5d
ha_max_local_clockAccuracy 0x20
ha_max_local_clockClass 255
ha_max_local_offsetScaledLogVar 0xffff
ha_phc2sys_com_socket /var/run/phc1
ha_stability_timer 60
logging_level 7
message_tag phc1

[enp81s0f1]
##
## Associated interface: enp81s0f1
##
ha_priority 100
ha_uds_address /var/run/ptp4l-ptp11

[enp138s0f0]
##
## Associated interface: enp138s0f0
##
ha_priority 101
ha_uds_address /var/run/ptp4l-ptp12

Manage Phc2sys

The ha_phc2sys_com_socket displays information about the state of the source selection algorithm, as well as performs test and troubleshooting operations.

To perform these operations, locate the socket file for the phc2sys instance. This should be configured in the global instance parameters for the phc2sys instance. For example, /var/run/<instance name>.

These commands must be run on the specific host being managed, as access to the socket file is required. For example:

[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "<command>" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>

Informational Commands

status

The status command prints a list of configured sources along with the quality characteristics that have been read from the associated ptp4l instances.

The status command also indicates which source has been selected.

[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "status" | socat -t 1 - UNIX-CONNECT:/var/run/phc0

act   interface   priority   clockClass   clockAcc   offset   time   freq   gm.clockClass    gm.clockAcc   gm.offset

*     enp138s0f0   100          248       0xfe       0xffff    yes    yes        6              0x20         0x4e5d
      enp81s0f1    100          248       0xfe       0xffff    yes    yes        6              0x20         0x4e5d

forced lock

The forced lock command prints “False” if phc2sys has not been forced to select a specific interface via a management operation. It prints “True” if a force lock has been set via a management operation.

[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "forced lock" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>

valid sources

The valid sources command prints the interfaces that meet the quality threshold for selection. If none of the configured interfaces meet the quality threshold then the string “None” is printed.

[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "valid sources" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>

clock source

The clock source command returns the current source that is selected by phc2sys.

[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "clock source" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>

Management Operations

enable lock

The enable lock <interface> command forces phc2sys to use the given interface as the time source. This operation can be helpful for debugging operations that require the clock source to stay fixed.

Note

phc2sys can also be forced to select an interface by configuring it with ha_priority=254 which is preferred if the lock needs to persist between reboots and restarts of the phc2sys service.

Phc2sys can only be locked to a single interface at a time.

[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "enable lock enp81s0f0" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>

disable lock

The disable lock command returns phc2sys to normal operations where it selects the best quality clock source based on the configured thresholds.

[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "disable lock" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>

disable source <interface>

The disable source <interface> command removes a given interface from the list of sources considered by phc2sys for selection. This can be used for troubleshooting and testing by allowing sources to be removed and re-enabled without restarting the phc2sys service.

Phc2sys will not allow a source to be disabled if it is the only source currently configured and enabled.

[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "disable source enp81s0f0" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>

enable source <interface>

The enable source <interface> command returns a previously disabled interface to the selection pool, allowing phc2sys to consider it for selection based on its quality.

[sysadmin@controller-0 ~(keystone_admin)]$ echo -n "enable source enp81s0f0" | socat -t 1 - UNIX-CONNECT:<ha_phc2sys_com_socket>

HA Phc2sys Alarms

New alarms have been added to report the state of HA Phc2sys. To view system alarms on StarlingX, use the fm alarm-list command.

Phc2sys HA has no source clock

Example EID: host=controller-0.phc2sys=phc0.phc2sys=no-source-clock

The “no source clock” alarm is raised when phc2sys does not have any sources that meet the specified threshold in the user configuration. This is due to the quality of the available sources not meeting specified thresholds. The alarm clears when an available source meets the specified quality requirements.

The ha_phc2sys_com_socket “status” and “valid sources” commands can be used to assist with troubleshooting this alarm.

phc2sys HA source clock is not locked to a PRC

Example EID: host=controller-0.phc2sys=ptp01.interface=enp81s0f1.phc2sys=source-clock-no-prc-lock

The “source clock is not locked to a PRC” alarm is raised when one of the HA phc2sys interfaces is indicating a clockClass that is of lesser quality than the specified maximum clockClass in the user config. The alarm indicates which interface is degraded. The presence of this alarm should coincide with phc2sys selecting a different source if a better one is available. The alarm will clear when the clockClass for the source meets the selection threshold.

The ha_phc2sys_com_socket “status” and “valid sources” commands can be used to assist with troubleshooting this alarm.

phc2sys HA has selected a lower priority clock source

Example EID: host=controller-0.phc2sys=phc1. phc2sys=source-clock-low-priority

The “selected lower priority clock source” alarm is raised when phc2sys is configured with multiple sources and the selected one that is not the highest priority. This alarm indicates that there is degraded quality for the highest priority source and that phc2sys has taken action to switch. The alarm will be cleared when the highest priority source becomes available for selection.

phc2sys HA automatic source selection has been disabled

Example EID: host=controller-0.phc2sys=phc0.phc2sys=forced-clock-selection

The “automatic source selection disabled” alarm is raised when an operator has used the ha_phc2sys_com_socket enable lock <interface>`command to force ``phc2sys` to use a specific source. This alarm is informational and can be cleared by running the disable lock command via the ha_phc2sys_com_socket.

phc2sys HA source selection algorithm selected new active source

Example EID: host=controller-0.phc2sys=ptp02.interface=enp138s0f0.phc2sys=source-failover

The “selected new active source” event is raised when phc2sys switches between sources for any reason. This is not raised as a persistent alarm, but rather as an event, viewable via fm event-list. This event enables users to see when source selection events have occurred. There is no action required to clear this event.

Phc2sys Source Selection Behaviour

When phc2sys is configured with multiple sources and “ha_enabled”, a selection algorithm runs repeatedly to determine which source to sync the system time to. The source selection algorithm operates by determining a list of valid sources (i.e. sources that meet the configured selection requirements), and then choosing the source from that list with the highest priority.

When phc2sys detects that a source’s quality has changed, the list is re-calculated and a new source is selected if a better one has become available.

The source selection operates in the following way:

For each configured clock source:
    if local.clockClass > ha_max_local_clockClass -> disqualify source
    if local.clockAccuracy > ha_max_local_clockAccuracy -> disqualify source
    if local.offsetScaledLogVariance > ha_max_gm_offsetScaledLogVar -> disqualify source
    if source.timeTraceable = 0 AND ha_gm_timeTraceable = 1 -> disqualify source
    if source.frequencyTraceable = 0 and ha_gm_frequencyTraceable = 1 -> disqualify source
    if gm.clockClass > ha_max_gm_clockClass -> disqualify source
    if gm.clockAccuracy > ha_max_gm_clockAccuracy -> disqualify source
    if gm.offsetScaledLogVariance > ha_max_gm_offsetScaledLogVar -> disqualify source

From list of qualified sources, select the source with highest “ha_priority”. If qualified sources have the same “ha_priority”, select the first one as it appears in the phc2sys config file.

If there is no valid source, select the source with lowest “local.clockClass”. If “local.clockClass” is tied, select the first source in phc2sys config file from among the tied sources.