CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time

In Grid versions 12.2.0.1 and later, although the time is the same in all nodes, the following error is seen in the CRS alert log file.

CRS-2412: The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time.

Time on the nodes can be checked as follows.

[root@exadbadm01 ~]# dcli -g dbs_group -l root date
exadbadm01: Fri Dec 18 18:00:31 +03 2020
exadbadm02: Fri Dec 18 18:00:35 +03 2020
exadbadm03: Fri Dec 18 18:00:31 +03 2020
exadbadm04: Fri Dec 18 18:00:31 +03 2020
exadbadm05: Fri Dec 18 18:00:31 +03 2020
exadbadm06: Fri Dec 18 18:00:31 +03 2020
exadbadm07: Fri Dec 18 18:00:31 +03 2020
exadbadm08: Fri Dec 18 18:00:31 +03 2020

This error is caused by bug number 30296374   .

The following steps can be taken to solve the problem. Commands should be run with the root user.

1. Learn the ctssd log level on all nodes with the following command.
   /u01/app/12.2.0.1/grid/bin/crsctl get log ctss CTSS
2. Update ctssd log level to 4 on all nodes.
   /u01/app/12.2.0.1/grid/bin/crsctl set log ctss CTSS=4
3. Shut down the ctssd daemon on all nodes.
   /u01/app/12.2.0.1/grid/bin/crsctl stop res ora.ctssd -init
4. Start the ctssd daemon on all nodes.
   /u01/app/12.2.0.1/grid/bin/crsctl start res ora.ctssd -init
5. Reset the ctssd log level detected in step 1 on all nodes.
   /u01/app/12.2.0.1/grid/bin/crsctl set log ctss CTSS=<value received in step 1>

Afterwards, no errors will be seen in the crs alert log file.

With dcli, you can run commands on all nodes via a server as follows.

dcli -g dbs_group -l root /u01/app/12.2.0.1/grid/bin/crsctl get log ctss CTSS
dcli -g dbs_group -l root /u01/app/12.2.0.1/grid/bin/crsctl set log ctss CTSS=4
dcli -g dbs_group -l root /u01/app/12.2.0.1/grid/bin/crsctl stop res ora.ctssd -init
dcli -g dbs_group -l root /u01/app/12.2.0.1/grid/bin/crsctl start res ora.ctssd -init
dcli -g dbs_group -l root /u01/app/12.2.0.1/grid/bin/crsctl set log ctss CTSS=<value received in step 1>