PRECISION TIME PROTOCOL RED HAT ENTERPRISE LINUX 6.

4
INTRODUCTION The Precision Time Protocol (PTP) is a protocol used to synchronize clocks in a network. When used in conjunction with hardware support, PTP is capable of sub-microsecond accuracy, which is far better than is normally obtainable with NTP. The clocks synchronized by PTP are organized in a master-slave hierarchy. The slaves are synchronized to their masters which may be slaves to their own masters.
PTP Grandmaster

synchronized by Global Positioning System (GPS). By using a GPS-based time source, disparate networks can be synchronized with a high-degree of accuracy. One of the main advantages over the Network Time Protocol (NTP) is hardware support present in various network interface controllers (NIC) and network switches. This specialized hardware allows PTP to account for delays in message transfer, and greatly improves the accuracy of time synchronization. While it is possible to use non-PTP enabled hardware components within the network, this will often cause an increase in jitter or introduce an asymmetry in the delay resulting in synchronization inaccuracies, which add up with multiple non-PTP aware components used in the communication path. To achieve the best possible accuracy, it is recommended that all networking components between PTP clocks are PTP hardware enabled. Time synchronization in larger networks where not all of the networking hardware supports PTP might be better suited for NTP. With hardware PTP support, the NIC has its own on-board clock, which is used to time stamp the received and transmitted PTP messages. It is this on-board clock that is synchronized to the PTP master, and the computer's system clock is synchronized to the PTP hardware clock on the NIC. With software PTP support, the system clock is used to time stamp the PTP messages and it is synchronized to the PTP master directly. Hardware PTP support provides better accuracy since the NIC can time stamp the PTP packets at the exact moment they are sent and received while software PTP support requires additional processing of the PTP packets by the operating system. PTP support is divided between the kernel and user-space. The kernel includes support for PTP clocks, which are provided by network drivers. The actual implementation of the protocol is known as linuxptp, a PTPv2 implementation according to the IEEE standard 1588 for Linux.

GPS

Time Slave
S S

Boundary Clock
M

Time Slave
S

Time Slave
S S

Boundary Clock

Time Slave
S

M S

- Master port - Slave port

The hierarchy is created and updated automatically by the best master clock (BMC) algorithm, which runs on every clock. When a clock has only one port, it can be master or slave, such clock is called an ordinary clock (OC). A clock with multiple ports can be master on one port and slave on another, such clock is called a boundary clock (BC). The toplevel master is called the grandmaster clock, which can be

Copyright 2013 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks of Red Hat, Inc., registered in the U.S. and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

www.redhat.com

PREREQUISITES PTP support is provided as a Technology Preview in Red Hat Enterprise Linux 6.4.
1

SOF_TIMESTAMPING_TX_SOFTWARE SOF_TIMESTAMPING_RX_SOFTWARE

Before proceeding, please make sure the latest kernel update for Red Hat Enterprise Linux 6.4 has been installed. In order to use PTP, the kernel network driver for the intended interface has to support either software or hardware time stamping capabilities. For 6.4, the following network drivers support time stamping:
Driver igb ixgbe tg3 Description Intel 1G Intel 10G Broadcom 1G Software Time Stamping No Yes Yes Hardware Time Stamping Yes* Yes* No

For hardware time stamping support, the parameters list should include: SOF_TIMESTAMPING_RAW_HARDWARE SOF_TIMESTAMPING_TX_HARDWARE SOF_TIMESTAMPING_RX_HARDWARE

Finally, the linuxptp package must be installed on the system to provide PTP master/slave time synchronization functionality: # rpm -q linuxptp
linuxptp-0-0.6.20121114gite6bbbb.el6.x86_64

TUTORIAL The linuxptp package includes the ptp4l and phc2sys programs for clock synchronization. The ptp4l program implements the PTP boundary clock and ordinary clock. With hardware time stamping, it is used to synchronize the PTP hardware clock to the master clock, and with software time stamping it synchronizes the system clock to the master clock. The phc2sys program is needed only with hardware time stamping, for synchronizing the system clock to the PTP hardware clock on the NIC. Starting ptp4l The ptp4l program tries to use hardware time stamping by default. To use ptp4l with hardware time stamping capable drivers and NICs, you must provide the network interface to use with the -i option: # ptp4l -i eth3 -v By default, messages are sent to /var/log/messages. However, specifying the -v option enables logging to standard output (useful for debugging purposes.) To enable software time stamping, the -S option needs to be used: # ptp4l -i eth3 -v -S The command line options and other options, which can't be set on the command line, can be set in a configuration file.

This list will be expanded as additional network drivers are enabled to support time stamping capabilities in future releases of Red Hat Enterprise Linux 6. In addition to hardware time stamping support being present in the driver*, the NIC must also be capable of supporting this functionality in the physical hardware. The best way to verify the time stamping capabilities of a particular driver and NIC is to use the ethtool utility to query the interface: # ethtool -T eth3

Time stamping parameters for eth3: Capabilities: hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE) software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE) hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE) software-receive (SOF_TIMESTAMPING_RX_SOFTWARE) software-system-clock (SOF_TIMESTAMPING_SOFTWARE) hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE) PTP Hardware Clock: 0 Hardware Transmit Timestamp Modes: off (HWTSTAMP_TX_OFF) on (HWTSTAMP_TX_ON) Hardware Receive Filter Modes: none (HWTSTAMP_FILTER_NONE) some (HWTSTAMP_FILTER_SOME) ptpv1-l4-sync (HWTSTAMP_FILTER_PTP_V1_L4_SYNC) ptpv1-l4-delay-req (HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ) ptpv2-event (HWTSTAMP_FILTER_PTP_V2_EVENT) ptpv2-sync (HWTSTAMP_FILTER_PTP_V2_SYNC) ptpv2-delay-req (HWTSTAMP_FILTER_PTP_V2_DELAY_REQ)

For software time stamping support, the parameters list should include: SOF_TIMESTAMPING_SOFTWARE

Copyright 2013 Red Hat, Inc. Red Hat, Red Hat Linux, the Red Hat Shadowman logo, and the products listed are trademarks of Red Hat, Inc., registered in the U.S. and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

www.redhat.com

No configuration file is read by default, so it needs to be specified at runtime with the -f option. A configuration file equivalent to the -i eth3 -v -S options shown above would look like this: # cat /etc/ptp4l.conf
[global] verbose time_stamping [eth3] 1 software

The ptp4l program can be also started as a service by running: # service ptp4l start When running as a service, options are specified in the /etc/sysconfig/ptp4l file. More information on the different ptp4l options and the configuration file settings can be found in the ptp4l(8) man page. Using pmc The linuxptp package also includes a PTP management client, which can be used to obtain additional information from ptp4l. # echo GET CURRENT_DATA_SET | pmc -u -b 0
sending: GET CURRENT_DATA_SET 90e2ba.fffe.20c7f8-0 seq 0 RESPONSE MANAGMENT CURRENT_DATA_SET stepsRemoved 1 offsetFromMaster -142.0 meanPathDelay 9310.0

# ptp4l -f /etc/ptp4l.conf Below is example output from ptp4l when the PTP clock on the NIC is synchronized to a master: # ptp4l -i eth3 -v
selected /dev/ptp0 as PTP clock port 1: INITIALIZING to LISTENING on INITIALIZE port 0: INITIALIZING to LISTENING on INITIALIZE port 1: new foreign master 00a069.fffe.0b552d-1 selected best master clock 00a069.fffe.0b552d port 1: LISTENING to UNCALIBRATED on RS_SLAVE master offset -23947 s0 adj +0 path delay 11350 master offset -28867 s0 adj +0 path delay 11236 master offset -32801 s0 adj +0 path delay 10841 master offset -37203 s1 adj +0 path delay 10583 master offset -7275 s2 adj -30575 path delay 10583 port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED master offset -4552 s2 adj -30035 path delay 10385

# echo GET TIME_STATUS_NP | pmc -u -b 0

sending: GET TIME_STATUS_NP 90e2ba.fffe.20c7f8-0 seq 0 RESPONSE MANAGMENT TIME_STATUS_NP master_offset 310 ingress_time 1361545089345029441 cumulativeScaledRateOffset +1.000000000 scaledLastGmPhaseChange 0 gmTimeBaseIndicator 0 lastGmPhaseChange 0x0000'0000000000000000.0000 gmPresent true gmIdentity 00a069.fffe.0b552d

The master offset value is the measured offset from the master in nanoseconds. The s0, s1, s2 strings indicate the different clock servo states: s0 is unlocked, s1 is clock step and s2 is locked. Once the servo is in the locked state (s2), the clock will not be stepped (only slowly adjusted) unless the pi_offset_const option is set to a positive value in the configuration file (described in the ptp4l(8) man page). The adj value is the frequency adjustment of the clock in parts per billion (ppb). The path delay value is the estimated delay of the synchronization messages sent from the master in nanoseconds. Port 0 is a Unix domain socket used for local PTP management. Port 1 is the eth3 interface (based on the example above.) INITIALIZING, LISTENING, UNCALIBRATED and SLAVE are some of possible port states which change on the INITIALIZE, RS_SLAVE, MASTER_CLOCK_SELECTED events. In the last state change message, the port state changed from UNCALIBRATED to SLAVE indicating successful synchronization with a PTP master clock.

Setting the -b option to zero limits the boundary to the locally running ptp4l instance. A larger boundary value will retrieve the information also from PTP nodes further from the local clock. The retrievable information includes: stepsRemoved is the number of communication paths to the grandmaster clock offsetFromMaster and master_offset is the last measured offset of the clock from the master in nanoseconds meanPathDelay is the estimated delay of the synchronization messages sent from the master in nanoseconds

if gmPresent is true, the PTP clock is synchronized to a master, the local clock is not the grandmaster clock

gmIdentity is the grandmaster's identity

Starting phc2sys The phc2sys program is used to synchronize the system clock to the PTP hardware clock on the NIC: # phc2sys -i eth3
phc phc phc phc phc phc -2059 -38454 -74878 -111312 -65 11 s1 s2 s3 s4 s4 s4

while true; do echo 'GET TIME_STATUS_NP' | \ /usr/sbin/pmc -u -b 0 2> /dev/null | \ grep -q 'gmPresent.*true' && break sleep 1 done Verifying time synchronization When PTP time synchronization is working properly, new messages with offsets and frequency adjustments will be printed periodically to the ptp4l and phc2sys (if hardware time stamping is used) outputs. These values will eventually converge after a short period of time.
master master master master master master master master master master master master master master master master master master master master master master master master offset -111684 s2 adj offset -99129 s2 adj offset -62940 s2 adj offset -36080 s2 adj offset -15833 s2 adj offset -5035 s2 adj offset -1574 s2 adj offset 632 s2 adj offset 581 s2 adj offset 293 s2 adj offset -138 s2 adj offset -471 s2 adj offset -275 s2 adj offset 528 s2 adj offset 3770 s2 adj offset -1028 s2 adj offset -1522 s2 adj offset -904 s2 adj offset -207 s2 adj offset -193 s2 adj offset 164 s2 adj offset -282 s2 adj offset -89 s2 adj offset 133 s2 adj -40746 -61697 -55246 -47268 -37845 -31797 -29847 -28113 -27974 -28088 -28431 -28806 -28751 -28030 -24630 -28297 -29099 -28938 -28512 -28560 -28261 -28658 -28550 -28354 path path path path path path path path path path path path path path path path path path path path path path path path delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay delay 19461 19461 16458 16458 14862 13412 13412 12443 12443 12149 12176 12176 12188 12188 8122 9024 9451 9404 9404 9442 9421 9415 9401 9371

1361543543.393018747 1361543544.393117219 1361543545.393201158 1361543546.393285328 1361543547.393512720 1361543548.393605988

drift drift drift drift drift drift

0.00 0.00 0.00 -36414.43 -36433.93 -36430.63

Normally, PTP operates in the International Atomic Time (TAI) timescale2, while the system clock is kept in Coordinated Universal Time (UTC)3. The current offset between the TAI and UTC timescales is 35 seconds. The offset changes when leap seconds are inserted or deleted, which typically happens every few years. This information can be obtained from ptp4l via the PTP management protocol, but the phc2sys program available in Red Hat Enterprise Linux 6.4 doesn't implement this. The -O option needs to be used to set this offset manually: # phc2sys -i eth3 -O -35 Once the phc2sys servo is in a locked state, the clock will not be stepped. This means that the phc2sys program should be started after the ptp4l program has synchronized the PTP hardware clock. The phc2sys program can be also started as a service by running: # service phc2sys start When running as a service, options are specified in the /etc/sysconfig/phc2sys file. More information on the different phc2sys options can be found in the phc2sys(8) man page. To modify the phc2sys service to wait for the PTP clock to be synchronized to a master, the following code can be added to the start function in the /etc/init.d/phc2sys file:

Serving PTP time with NTP The ntpd daemon can be configured to distribute the time from the system clock synchronized by ptp4l or phc2sys by using the LOCAL reference clock driver. To prevent ntpd from adjusting the system clock, the ntp.conf file must not specify any NTP servers. The following is a minimal example of ntp.conf: # cat /etc/ntp.conf
server fudge

127.127.1.0 127.127.1.0 stratum 0

Serving NTP time with PTP NTP to PTP synchronization in the opposite direction is also possible. When ntpd is used to synchronize the system

clock, ptp4l can be configured with the priority1 option (or other clock options included in the best master clock algorithm) to be the grandmaster clock and distribute the time from the system clock via PTP: # cat /etc/ptp4l.conf
[global] priority1 127 [eth3]

and thus improve the PTP synchronization accuracy (at the cost of increased power consumption.) The kernel tickless mode can be disabled by adding nohz=off to the kernel boot option parameters.

REFERENCES The Linux PTP project http://linuxptp.sourceforge.net/ IEEE 1588 Standard http://www.nist.gov/el/isd/ieee/ieee1588.cfm

# ptp4l -f /etc/ptp4l.conf With hardware time stamping, phc2sys needs to be used to synchronize the PTP hardware clock to the system clock: # phc2sys -c /dev/ptp0 -s CLOCK_REALTIME -O 35 To prevent quick changes in the PTP clock's frequency, the synchronization to the system clock can be loosened by using smaller P (proportional) and I (integral) constants of the PI servo: # phc2sys -c /dev/ptp0 -s CLOCK_REALTIME -O 35 -P 0.01 -I 0.0001 Improving accuracy Test results indicate that disabling the tickless kernel capability can significantly improve the stability of the system clock,

[1] For more information on "Technology Preview" feature status, please refer to: https://access.redhat.com/knowledge/solutions/21101. [2] TAI, from the French name Temps Atomique International, is the International Atomic Time scale based on a continuous counting of the SI second. TAI is currently ahead of UTC by 35 seconds. TAI is always ahead of GPS by 19 seconds. [3] UTC, Coordinated Universal Time, popularly known as GMT (Greenwich Mean Time), or Zulu time. Local time differs from UTC by the number of hours of your timezone.