Description of the problem

After upgrading an OptiX OSN 3500 device of an M operator at the representative office, the SLD64 status of two line boards is abnormal, and the service of about XXX wireless GSM sites is interrupted, affecting about X users. After the problem appeared, an attempt was made to use the network element database backed up before the upgrade to recover, but the recovery failed and the service was interrupted.

Processing

1. After upgrading, the SLD64 boards of slot8 and slot11 are found to be offline, and the related services are interrupted;

2、Customer maintenance personnel found that the N4SLD64 physical board of slot8 and slot11 changed from the N1SLD64 logical board before the upgrade to the N2SL64 logical board, and modified the logical board from N2SL64 to N1SLD64.

3. After modifying the logical board type, the veneer goes online but the service is not restored;

4. Huawei notified the local Chinese side and TAC of the problem, and Huawei's Chinese side began to intervene to deal with the problem;

5. After communicating with the organization's GTAC, the decision was made to perform a rollback;

6. Remove the two R10 version GSCCs that have been upgraded, insert the R8 version GSCCs, and start to use the database that was backed up before the upgrade to recover;

7, recovery timeout, recovery failure, business is still interrupted (DC password error, resulting in timeout);

8, give up the recovery from the database, GTAC according to the database backed up before the upgrade to create a cross-configuration script, the local command line to send;

9、Downloaded successfully, but the business is still all interrupted, found that the database backed up before upgrading has problems, can not be used;

10、Start looking for valid data from recent backups, use the business configuration collected from the November inspection to make cross-configuration scripts;

11, the implementation of the script, business recovery.

Root cause

The synchronization switch between DRDB and FDB inside the master single board GSCC was abnormally turned off, resulting in the data stored in FDB being historically configured and not matching the currently running service. This leads to two problems:

1. After the upgrade, the master control resets, and after the reset, the master control re-issues the data, and the data issued is the old data in the FDB, which is inconsistent with the current configuration of the logical board position and the business configuration, so the upgrade is reported successfully but the business is interrupted;

2、Network data backup is backed up from the FDB, so the data backed up by the DC before the upgrade is also problematic, and cannot be recovered based on this database, leading to rollback failure, resulting in business interruption.

The database backup function is enabled by default, and the faulty network element was found to be disabled by inspection before the faulty network element was not repaired.

#9-48:szhw [48-***_OSN-4 ][][2014-12-12 14:31:50+06:00]>

:dbms-get-autobackup

DBMS-AUTO-BACKUP : disable

#9-48:szhw [48-***_OSN-4 ][][2014-12-12 14:31:52+06:00]>

:dbms-get-cyclebackup

CYCLE-BACKUP-SWITCH : disable

#9-48:szhw [48-***_OSN-4 ][][2014-12-12 14:31:53+06:00]>

:mml-get-evtflag

CMD EVT FLAG: disable

Solution

Find the most recent valid network element database backup file before the upgrade, or the network management backup script before the upgrade, and create a business recovery script accordingly to re-delete and reconfigure the full network crossover.

Recommendations and Summary

1, regular inspection, for inspection found in the abnormal timely tracking and resolution, even if the customer does not take the initiative to request a solution, but also to take the initiative to promote the solution;

2、Upgrade action strictly refer to the upgrade guide, especially for the steps that do not understand, but also to strictly implement, because it is likely to be the key filtering measures for a potential pitfalls;

3, strengthen the understanding of product knowledge, distinguish the meaning of the concepts of network element data backup (backup from FDB), network management data upload (upload from DRDB), network management script backup (DRDB dynamic valid data), network management on the synchronization of the entrance (backup of network element data to the SCC), the failure of the timely find valid data.

4, upgrade strict compliance with the process to do a variety of backups, in case of emergency.

This chapter of related technical information and SDH equipment troubleshooting process provided by Shenzhen Optical Transmission Network Technology Co., Ltd (www.opticaltrans.com), reproduced please retain! Huawei SDH Optical Transmission Equipment,SDH Transmission Equipment Sales