PTP Instance Troubleshooting

The most common error encountered using multi-instance PTP is a failure to start one or more instances after running the ptp-instance-apply command. This is often due to an invalid configuration or missing parameter.

This section provides some troubleshooting steps to assist with determining the error.

Example

After running the ptp-instance-apply command, the 250.001 alarm will appear if the PTP instances are not created or started properly. The system may also show the 200.011 alarm if, after an unlock, PTP instances were unable to start.

The following example shows these alarms as they would appear in tabular terminal output.

| 250.001  | controller-0 Configuration is out-of-date.          | host=controller-0     | major    | 2022-02-25T21: |
|          |                                                     |                       |
| 200.011  | controller-0 experienced a configuration failure.   | host=controller-0     | critical | 2022-02-25T20: |
|          |                                                     |                       |          | 47:59.561262   |

1. Determine if there was a PTP-instance failure by looking at the latest runtime puppet logs.

   sudo less /var/log/puppet/latest/puppet.log
   
   # Searching for "Error" in the log file shows this entry
   2022-02-28T17:26:49.529 ESC[1;31mError: 2022-02-28 17:26:49 +0000 Systemd start for ptp4l@ptp4l-legacy failed!
   

2. Once the instance has been identified, examine the config file for configuration errors.

   ~(keystone_admin)]$ cat /etc/linuxptp/ptpinstance/ptp4l-ptp4l-legacy.conf
   
   [global]
   ##
   ## Default Data Set
   ##
   boundary_clock_jbod 1
   clock_servo linreg
   delay_mechanism E2E
   domainNumber 0
   message_tag ptp4l-legacy
   network_transport L2
   summary_interval 6
   time_stamping hardware
   tx_timestamp_timeout 20
   uds_address /var/run/ptp4l-ptp4l-legacy
   

3. Start the service manually and check for errors.

   ~(keystone_admin)]$ ptp4l -f /etc/linuxptp/ptpinstance/ptp4l-ptp4l-legacy.conf
   

   In this example the ptp4l program indicates that there is no interface specified, which is confirmed by the contents of the config file above.

4. Check using the relevant system commands to see if there is an interface assigned to this instance and add one as required.

Additional Tools

PMC

PTP management client.

Used to interact with ptp4l and read or set various PTP parameters.

$ man pmc

# General command format:
sudo pmc -u -b 0 -f <path to ptp4l.conf for targeted instance> -s <path to uds socket for target instance> 'COMMAND GOES HERE'

For example:

pmc -u -b 0 -f /etc/linuxptp/ptpinstance/ptp4l-ptp1.conf -s /var/run/ptp4l-ptp1 ‘get PORT_DATA_SET’

PHC_CTL

Directly control PHC device clock.

Used to perform operations on the physical hardware clock (phc). PHC_CTL can be used to set the time on a NIC, check the delta between the NIC and the system clock, adjust the clock frequency.

$ man phc_ctl

# Example commands

phc_ctl <ptp_interface> get
phc_ctl <ptp_interface> cmp

# This syncs the NIC clock to the system clock
phc_ctl <ptp_interface> set

TCPDUMP

Check if PTP traffic is sending or receiving on a given interface.

You can capture L2 PTP traffic by filtering on proto 0x88F7

sudo tcpdump ether proto 0x88F7 -i <ptp_interface>

# Write it to file

sudo tcpdump ether proto 0x88F7 -i <ptp_interface> -w <output_file.pcap>

PTS Troubleshooting

The following tools and procedures are specific to PTS deployments using unicast PTP over IP networks.

Verify Network Connectivity to the Grandmaster

PTS uses UDP ports 319 and 320 for PTP event and general messages respectively. Before troubleshooting PTP synchronization, confirm that the host can reach the grandmaster on these ports.

# Test reachability - IPv4
nc -u -z -v <grandmaster-ipv4-address> 319
nc -u -z -v <grandmaster-ipv4-address> 320

# Test reachability - IPv6
nc -6 -u -z -v <grandmaster-ipv6-address> 319
nc -6 -u -z -v <grandmaster-ipv6-address> 320

If the ports are not reachable, check firewall rules and routing.

sudo iptables -L -n -v
sudo ip6tables -L -n -v
ip -6 route show

Capture PTP Traffic on Unicast Interfaces

To confirm that PTP packets are being exchanged with the grandmaster, capture traffic on the upstream-facing interface. Unlike L2 PTP, unicast PTP uses UDP ports 319 and 320 rather than EtherType 0x88F7.

# Capture on a specific interface
sudo tcpdump -i <interface> -n 'udp port 319 or udp port 320'

# Capture with timestamps
sudo tcpdump -i <interface> -n -tttt 'udp port 319 or udp port 320'

# Filter for a specific grandmaster address
sudo tcpdump -i <interface> -n \
    'host <grandmaster-address> and (udp port 319 or udp port 320)'

# Save capture to file for offline analysis
sudo tcpdump -i <interface> -n -w /tmp/ptp-capture.pcap \
    'udp port 319 or udp port 320'

Check PTP Port and Parent Clock State

Use the pmc tool to inspect the port state and parent clock information for a ptp4l instance. In a PTS deployment, the port state should be SLAVE when synchronized to the upstream grandmaster.

# Check port state - should show SLAVE when synchronized
sudo pmc -u -b 0 \
    -f /etc/linuxptp/ptpinstance/ptp4l-<instance-name>.conf \
    'GET PORT_DATA_SET'

# Check parent clock identity and grandmaster details
sudo pmc -u -b 0 \
    -f /etc/linuxptp/ptpinstance/ptp4l-<instance-name>.conf \
    'GET PARENT_DATA_SET'

# Check time properties (timeTraceable, frequencyTraceable)
sudo pmc -u -b 0 \
    -f /etc/linuxptp/ptpinstance/ptp4l-<instance-name>.conf \
    'GET TIME_PROPERTIES_DATA_SET'

A port state of LISTENING or UNCALIBRATED indicates that the instance has not yet synchronized to the grandmaster. Check network connectivity and verify that the grandmaster address in the unicast master table is correct.

Enable Debug Logging for ptp4l

To increase the verbosity of ptp4l log output for a PTS instance, add the logging_level parameter and re-apply.

~(keystone_admin)]$ system ptp-instance-parameter-add <ptp4l-instance> \
    logging_level=7
~(keystone_admin)]$ system ptp-instance-apply

Debug output is written to /var/log/user.log.

sudo tail -f /var/log/user.log | grep <instance-name>

To disable debug logging, remove the parameter and re-apply.

~(keystone_admin)]$ system ptp-instance-parameter-delete <ptp4l-instance> \
    logging_level=7
~(keystone_admin)]$ system ptp-instance-apply