synce4l Monitoring and Alarms¶

This section describes how to monitor synce4l operation and the alarms raised by the platform for SyncE conditions.

Alarm 100.119 ¶

Alarm 100.119 covers all SyncE operational conditions. It is raised by the collectd PTP plugin based on the DPLL lock status reported by synce4l via its SMC socket.

Entity Instance ID	Severity	Description
`host=<hostname>.synce.interface=<interface>.synce=source-loss`	major, escalating to critical	The DPLL has lost its active source. Raised at major while in holdover; escalates to critical when the holdover timer expires and the DPLL enters freerun.
`host=<hostname>.synce.instance=<instance>.synce=not-running`	major	The `synce4l` service is enabled but not running.

The source-loss alarm is a single alarm whose severity changes (major in holdover, critical in freerun); it is not two separate alarms. The source-loss alarm is keyed on the interface, while the not-running alarm is keyed on the instance.

To view active SyncE alarms:

~(keystone_admin)]$ fm alarm-list --nowrap | grep 100.119

To view details for a specific alarm:

~(keystone_admin)]$ fm alarm-show <alarm-uuid>

Alarm Lifecycle ¶

Raised when:

not-running: The synce4l service is enabled but not running (stopped, failed, or killed). Raised at major severity.
source-loss (holdover): The active clock source is lost (e.g., GNSS signal interrupted, upstream SyncE link down). The DPLL maintains output frequency from stored calibration data. Raised at major severity.
source-loss (freerun): The DPLL has exhausted holdover (holdover timer expired) or has never locked to a source. The alarm severity escalates to critical.

Severity escalation:

When a source is lost, the alarm follows a two-stage progression:

The DPLL enters holdover and the alarm is raised at major severity.
If the source does not recover before the holdover timer expires, the DPLL transitions to freerun and the alarm escalates to critical severity.

The holdover timer is controlled by the holdover_seconds monitoring parameter. See Configure synce4l Service for details on monitoring parameters.

Cleared when:

The synce4l service is running and the DPLL transitions to locked state (active source with valid signal detected).

Note

The not-running alarm is the authoritative indicator that synce4l is stopped or has failed. The SyncE lock state notification is not a proxy for service health: it reflects the EEC DPLL state read from the kernel, which can remain Locked while synce4l is stopped if the DPLL still has a valid physical reference (for example, a GNSS-fed source). Use alarm 100.119 not-running for service health and the SyncE lock state for the clock’s synchronization state. See PTP Notification Status Conditions.

Holdover Behavior ¶

When the active source is lost:

The DPLL firmware enters holdover immediately. Output frequency is maintained from internal memory.
The collectd SyncE plugin detects the holdover state and drives the advertised quality level to holdover_ql (default 0x04, SSU-A) by sending SET_QL to synce4l, which transmits it on all ports via ESMC.
Alarm 100.119 (source-loss) is raised at major severity.
If a fallback source is available (another pin with valid signal and lower priority number), the firmware transitions to it after the holdover period.
Once locked to a new source, the advertised quality level is updated (pass-through for a SyncE source, or static_ql for a non-SyncE source) and the alarm is cleared.
If no source recovers before the holdover timer expires, the plugin drives freerun_ql (default 0x0F, DNU) and the alarm escalates to critical.

Note

Source failover timing (E825 / zl3073x). When the active source is lost, the DPLL firmware controls the holdover dwell and re-lock timing, not synce4l or the platform. Observed values on E825 are approximate:

GNSS to SyncE failover: approximately 100–120 seconds (firmware holdover dwell plus frequency acquisition on the new source).
SyncE to GNSS failback: approximately 15–30 seconds (revertive to the higher-priority source).

These durations are distinct from the holdover_seconds monitoring parameter, which controls when the platform escalates the alarm, and from the recover_time parameter (default 300 seconds), which governs only how long synce4l waits before trusting a recovered source on a port. Neither parameter changes the firmware timing above.

If failover takes longer than roughly 150 seconds or never completes, verify that the upstream device provides a valid Layer-1 frequency reference: ESMC frames alone are not sufficient, the PHY must output a recovered clock within the DPLL pull-in range.

Resolution Steps for Alarm 100.119 ¶

The following steps address each alarm condition raised by alarm 100.119. Follow the subsection that matches the synce field in the active alarm’s entity instance ID.

synce=not-running ¶

This condition indicates that the synce4l service is enabled but not running. Check the service state and logs to identify the failure cause.

Check the service status:

~(keystone_admin)]$ systemctl status synce4l@<instance_name>.service

Check the systemd service lifecycle log for start failures:
```
~(keystone_admin)]$ grep synce4l /var/log/daemon.log
```

Check the synce4l operational log for errors:

~(keystone_admin)]$ grep <instance_name> /var/log/user.log

Verify the instance exists, is assigned to the host, and has an interface with a port assigned:
```
~(keystone_admin)]$ system ptp-instance-show <instance_name>
~(keystone_admin)]$ system ptp-interface-list
```
The ptp-instance-show output lists the assigned hostnames and parameters. The ptp-interface-list output shows each interface with its parent instance.
If the previous steps required a configuration correction (for example, adding a missing interface or port), apply the change:
```
~(keystone_admin)]$ system ptp-instance-apply
```

source-loss (holdover)¶

This condition indicates that the active clock source has been lost and the DPLL is maintaining output frequency from internal memory. Identify the lost source and restore the physical reference before the holdover timer expires.

Check which source was previously active by examining the log:
```
~(keystone_admin)]$ grep <instance_name> /var/log/user.log
```
Verify physical connectivity on the source port (link status, cable):
```
~(keystone_admin)]$ ip link show <port_name>
```
For GNSS-based sources, verify GNSS signal quality using the gnss-monitor instance if configured.
On E825, if the DPLL does not lock to the SyncE recovered clock, verify that both synce_rclka=enabled and pin_state_synce=selectable were configured on the clock instance. See Configure synce4l Service.

Note

A pin in the SELECTABLE state is eligible for selection but is not necessarily locked. The DPLL only locks to a pin when a valid frequency reference is physically present on it. If a pin shows SELECTABLE but the DPLL stays in holdover, the upstream signal (recovered clock, GNSS, or external input) is missing or outside the DPLL pull-in range, even though the pin is eligible.

Verify the ESMC traffic is arriving on the upstream port (for T-BC configurations):

~(keystone_admin)]$ sudo tcpdump -enn "ether proto 0x8809 and ether[14] = 0x0a" -i <port_name> -c 5

source-loss (freerun)¶

Freerun indicates the DPLL has never locked or has been in holdover too long. Follow the same steps as holdover, and additionally:

Verify that at least one configured clock source has a valid signal.
Verify the external_source board label matches the actual pin label registered by the kernel driver.

Monitor ESMC Traffic ¶

Use tcpdump to verify ESMC frames are being sent and received on SyncE-configured ports:

~(keystone_admin)]$ sudo tcpdump -enx "ether proto 0x8809 and ether[14] = 0x0a" -i <port_name>

Expected output for a working SyncE port transmitting PRC quality (network_option=1, SSM 0x02):

12:57:02.525757 aa:bb:cc:dd:ee:ff > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 60: unknown (10), length 46
        0x0000:  0a00 19a7 0001 1000 0000 0100 0402 0000

Key fields in the ESMC frame:

Destination MAC 01:80:c2:00:00:02 — the Slow Protocols multicast address.
EtherType 0x8809 — Slow Protocols.
Subtype 0x0a — identifies the frame as ESMC.

Locating the quality level in the frame:

The -x option prints the frame payload in hex, starting at the ESMC subtype byte (0a) shown at the beginning of the 0x0000 line.
The base quality level is carried in the QL Type-Length-Value (TLV) field as the low four bits of the SSM code byte. In the dump above, the QL TLV is the sequence 01 00 04 02 (type 1, total TLV length 4, SSM code 0x02 = PRC for network_option=1). The SSM code is the byte immediately following 01 00 04.

When extended_tlv=1 is configured, the frame carries an additional Extended QL TLV (type 2) after the base QL TLV. In the payload dump it appears as the sequence 02 00 14 YY, where YY is the extended SSM code:

12:57:03.101204 aa:bb:cc:dd:ee:ff > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 60: unknown (10), length 46
        0x0000:  0a00 19a7 0001 1000 0000 0100 0402 0200
        0x0010:  1421 aabb ccff fedd eeff 0000 0000 0000

01 00 04 02 — base QL TLV: SSM code 0x02 (PRC).
02 00 14 21 — Extended QL TLV: type 2, length 0x14 (the total TLV length, 20 bytes), extended SSM code 0x21 (ePRTC for network_option=1). The 16 bytes following the extended SSM code carry the sender clock identity, a flag byte, and the cascaded clock counters.

To read the quality level quickly, the SSM byte can be located directly in the tcpdump hex as shown above. For full field-by-field decoding, save the capture with -w <file>.pcap, transfer it off the host, and open it in Wireshark or tshark, which label each ESMC field by name.

If no ESMC frames are observed:

Verify the synce4l service is running.
Check that the port is assigned to a synce4l interface.
Verify tx_heartbeat_msec is configured (default: 1000ms).

Check Service Logs ¶

synce4l logs its operational messages to /var/log/user.log. To view logs for a specific instance:

~(keystone_admin)]$ grep <instance_name> /var/log/user.log

The systemd service lifecycle (start, stop, restart, failures) is logged to /var/log/daemon.log:

~(keystone_admin)]$ grep synce4l /var/log/daemon.log

To enable debug logging temporarily (without restarting):

~(keystone_admin)]$ system ptp-instance-parameter-add synce1 logging_level=7
~(keystone_admin)]$ system ptp-instance-apply

Important

Remember to reset logging_level to 6 (info) after troubleshooting to avoid excessive log volume.

Key log messages to look for:

dpll device for '<name>' - init successful — DPLL netlink connection established and running.
new best source — source selection has changed.
set QL on — QL is being advertised on a port.
dpll device for '<name>' - reinit requested — indicates a pin resolution or initialization failure (check configuration).

Verify DPLL Lock Status ¶

To check the current DPLL lock status at the kernel level:

~(keystone_admin)]$ sudo python3 -c "import pynetlink; dpll=pynetlink.NetlinkDPLL(); [print(f'dev{d.dev_id}: {d.lock_status}') for d in dpll.get_all_devices()]"

Expected output for a locked DPLL:

dev0: LockStatus.LOCKED_AND_HOLDOVER

Common lock status values:

LOCKED_AND_HOLDOVER — Locked with holdover acquired (normal operation)
HOLDOVER — Active source lost, maintaining frequency from memory
UNLOCKED — Freerun, no reference

To check which pin is currently active:

~(keystone_admin)]$ sudo python3 -c "import pynetlink; dpll=pynetlink.NetlinkDPLL(); [print(f'pin{p.pin_id}: {p.pin_board_label} {p.pin_state}') for p in dpll.get_all_pins() if 'CONNECTED' in str(p.pin_state)]"