Problem Description

Scenario 1: Single board goes offline:

The line board SLQ4 on slot 5 of the current network OSN3500 suddenly goes offline, open the network element panel on the T2000 network management, it shows that the problematic veneer is grayed out, but there is no alarm reported, at the same time, with the command line :cfg-get-phybd; it has not been able to query the physical veneer on slot 5, confirm the business impact with the customer and find that the business is normal.

Scenario 2: COMMUN_FAIL alarm:

A newly installed OSN3500 device, the single boards in slots 5, 7 and 8 report COMMU_FAIL alarms at the same time, and two COMMUN_FAIL alarms are reported on each single board at the same time. By checking the alarm parameter, it is found that the alarm parameter indicates that the 485_A serial bus and the 485_B serial bus of the single board report alarms at the same time, which may be the case that the two buses are bad. It is possible that both buses are bad.

Alarm information

Scenario 1: The single board is offline:

There is no alarm, just when you check the device panel on the T2000 network management, you see that the veneer becomes gray, and at the same time, you use the command line :cfg-get-phybd; to query the physical veneer on slot 5 already, and confirm the business impact with the customer, and find that the business is normal.

Scenario 2: COMMUN_FAIL alarm:

The single boards in slots 5, 7 and 8 report COMMU_FAIL alarms at the same time, and two COMMUN_FAIL alarms are reported on each single board at the same time. By checking the alarm parameters, it is found that the alarm parameters indicate that the 485_A serial bus and the 485_B serial bus of the single board have reported alarms at the same time.

Processing

Scenario 1: The single board is offline:

Since the business is normal, you cannot do a single board reset or unplug the single board again, and you can only try to check it with the command line:

:optp:9,0,96,1,30,06,5,1,8,0,0,50,0,f; //Test the 485_A communication from the business board at board position 5 to the crossover board at board position 9

Test result 485_A communication is normal;

:optp:9,0,96,1,30,06,5,2,8,0,0,50,0,f; // Test 485_B communication from 5 board position business board to 9 board position cross board

Test result 485_B communication is normal;

:cfg-get-phybd.

Query result there is no physical board in the slot where the faulty single board is located;

:cfg-get-bdverinfo:5

Query can not find any information, suggesting that the single board is not in position;

Determine that the 485_A,485_B bus is normal and the LAN SWITCH bus is faulty;

Scenario 2: COMMUN_FAIL alarm:

Unplug the veneer boards with alarms one by one, and check whether the motherboards are backward pinned;

After unplugging one single board, find that the alarms of other single boards disappear at the same time;

Plug the other boards back in, and no more alarms are reported;

Determine the faulty veneer;

Plug the faulty single board into other slots, and find that the COMMUN_FAIL alarm is reported, and after a period of time, other slots report this alarm;

Command line query 485 bus situation:

:optp:9,0,96,1,30,06,5,1,8,0,0,50,0,f; //Test the 485_A communication from the faulty board to the cross-board

:optp:9,0,96,1,30,06,5,2,8,0,0,50,0,f; // Test the 485_B communication from the faulty board to the cross board

Test result: the faulty board 485_A,485_B is not available;

485_A,485_B communication is normal on the other boards that are affected by the alarm; it is judged to be a false alarm;

Finally determined to be a single board on the 485 bus damage.

Root cause

Scenario 1: The single board is offline:

Analyzing the operation of the equipment, it is presumed that the service bus connecting the cross board is normal, so the service is not interrupted, and the single board is offline, indicating that the communication between the master control and the single board has been interrupted, but then there is no report of the BD_STATUS alarm? It is inferred that only the LAN SWITCH communication is interrupted, and the other communications are normal, and the single board operation status is not stable.

Try to find out the reason through the following operations:

1. Check whether the physical board is online ( cfg-get-phybd to check the physical board position)

2、Check the business impact situation (check the customer business impact situation)

3、Command line query inter-board communication status (query 485 bus communication and LAN SWITCH communication)

Scenario 2: COMMUN_FAIL Alarm:

At the same time there are multiple single boards reported at the same time, it is impossible for multiple single boards to be damaged at the same time, so it is suspected that a single board is damaged affecting other single boards, or the mother board has problems such as reverse pinning. Check the alarm parameters and identify the channels affected by the alarm. 485_A and 485_B in the problem have reported COMMUN_FAIL alarms.

Try to identify the cause by doing the following:

1、Query whether the physical board is online ( cfg-get-phybd query physical board position)

2、Check the business impact situation (check the customer business impact situation)

3、Command line query inter-board communication status (query 485 bus communication and LAN SWITCH communication)

4, check whether the motherboard pinback (pull out the faulty single board and cross board, check the motherboard pin condition)

5, the problem of single board replacement slot test (the single board with alarms replaced to other slots, to observe the alarm situation)

Suggestions and Summary

When there are alarms in batch, check the actual status of the veneer through the command line to determine whether there are false alarms;

Locate faults without resetting, unplugging and unplugging single boards, without affecting the business;

Familiarize yourself with the principles of inter-board communication.

Keywords:

Inter-board communication: There are two ways to communicate between OSN 7500/3500/2500/1500 boards:

1 ) 2HDLC channels between single boards, following the HDLC protocol, with the interface electrical specification RS485,

2 ) 1LAN SWITCH channel;

485 bus: mainly realizing functions related to multiplexing segment, SNCP, TPS, with a rate of 4Mbits/s;

LAN SWITCH bus: Ethernet communication between the motherboard (or CXL board) and the veneer and users;

Motherboard Pin Down: Misplaced insertion of the veneer board causes the motherboard pins to tip over or break off, potentially causing a short circuit;

OPTP command: send test packets from the source board bit to the host board bit, the host board bit receives the test packets and sends the test response packets, and then the source board bit receives the test response packets, after checksum, to determine whether the test passes;

COMMUN_FAIL alarm: According to the principle that 485 bus can be self-sending and self-receiving, each single board sends a test packet to itself every 20S (and the test packet sent by each single board can only be received by the board), and if the test packet is not received for 7 consecutive frames, then the communication failure alarm is reported;

Board not in position: BD_STATUS is the Board not in position alarm, which is the main alarm.