Basic KPI Troubleshooting

Basic Key Performance Indicators

RRC Success Rate - This is to evaluate Accessibility of the network.

ConnectionEstablishmentSuccess/ConnectionEstablishmentAttempt

Connection Drop Rate (CDR) This is evaluate disconnections/drops for all kind of services (QCI1 to QCI9)
E-RAB Release whilst Active/(Initial ERAB Setup Success+ Additional ERAB Setup Success)

Voice over LTE Drop Rate (VDR) - This is to evaluate call drops during Volte call.
E-RAB Release whilst Active*/(Initial ERAB Setup Success*+ Additional ERAB Setup Success*)
*Use values from only QCI1

Handover Success Rate (HOSR) This is to evaluate mobility performance of the network.
Inter eNB handover Success/Inter eNB handover Attempt

Throughput DL/UL - Cell level Data speed in DL and UL

DL Throughput_HSI =(('DlMacTrafficBytes.9'[eCell,Time]*((8.0*0.743)*1000.0))/(('DlActTti.9'[eCell,Time]*1024.0))
UL Throughput_HSI = ((('UlMacTrafficBytes.9'[eCell,Time]*(8.0*0.2*1000))/(('UlActTti.9'[eCell,Time])*1024.0)

Copyright 2016 AIRSPAN NETWORKS INC.

RRC Success Rate Troubleshooting

1. PCI & RSI Planning Validation- Refer Basic Parameter Planning Guideline *
1. *If Auto PCI and Auto RSI is enabled (Check SON Profile from Netspan) then ignore this step.
2. Clock Issue: Refer Alarm Document for clock related issues.
1. If No clock alarms but RRC Success Rate < 50% then swap L2 Switch & monitor the performance
2. If multiple cells are having poor RRC Success Rate then try CSS reboot.
3. Congestion : If Max. Connected Users > 30 then
1. Check if neighbouring cells are OOS - resolve it and monitor the performance.
2. Check # connected users at NBR cells, if possible increase the power of NBR cell & monitor the performance.
3. If SMCs are not planned tightly at affected floor then add small cell for traffic offload and monitor the performance.
4. If SMCs are tightly planned then do re-design the floor with additional capacity and get validated from NPE team .
4. Planning and design validation
1. Check if eNBs placed in close proximity is causing SMC to SMC interference.
2. Physical optimization if installed locations are different than plan
3. If above two conditions are met then re-design.
5. Walk Test: Do CVT/Walk to validate coverage gap
1. Increasing in eNB power can help to improve RRC success rate.
2. If power is at maximum, plan additional small cells.
3. if the outdoor signal strength is high and causing interference then increase SMC powe rif not improved, coordinate with Samsung
Optimization team to do physical optimization
Copyright 2016 AIRSPAN NETWORKS INC.

RRC Success Rate Flow Chart

Copyright 2016 AIRSPAN NETWORKS INC.

HOSR Troubleshooting
1. PCI & RSI Planning Validation- Refer Basic Parameter Guideline *
1. *If Auto PCI and Auto RSI is enabled (Check SON Profile from Netspan) then ignore this step.
2. Clock Issue: Refer Alarm Document for clock related issues
1. If No clock alarms but RRC Success Rate < 50% then swap L2 Switch & monitor the performance
2. If multiple cells are having poor RRC Success Rate then try CSS reboot.
3. Audit NRT table
1. Validate SMC installation plan along with surrounding Macro locations.
2. Check if all possible neighbours are reported in NRT and their X2 links are setup to form required NBR relationship.
3. Check stats to find out specific NBR relationship causing handover failures.
4. Neighbour cell Availability- Check if neighbouring cells are OOS and causing handover failure because of coverage gaps or poor
coverage areas generated.
5. Neighbour cell Accessibility Check if the neighbouring cells are having Accessibility issues, Refer to RRC Success Rate
troubleshooting module.
6. Walk Test
A. Pingpong HO
1. To identify Ping-Pong Handover, do walk test and check if Handovers are happening back n forth at same locations.
2. If both SMCs are strong (RSRP) then 1. Reduce power or 2. Physical optimization required.
3. If Macro interference causing this then increase power to make SMC dominant.
Copyright 2016 AIRSPAN NETWORKS INC.

HOSR Troubleshooting
4. else coordinate with Samsung Optimization to optimize the macro coverage
B. Coverage Gaps
1. Do walk test to identify if Handover failures are happening in poor coverage and bad SINR
2. Increase eNB power to coverage and check HOSR.
3. If power is at maximum, additional small cell planning is required.

Copyright 2016 AIRSPAN NETWORKS INC.

HOSR Flow Chart

Copyright 2016 AIRSPAN NETWORKS INC.

CDR & VDR Troubleshooting

1. PCI & RSI Planning Validation- Refer Basic Parameter Guideline *
1. *If Auto PCI and Auto RSI is enabled (Check SON Profile from Netspan) then ignore this step.
2. Check HO Performance - Need to check if HO failures are increased.
1. Refer HO Success Rate module
3. Planning and design validation
1. Check if eNBs placed in close proximity is causing SMC to SMC interference.
2. Physical optimization if installed locations are different than plan
3. Else do redesign and NPE will validate.
4. Walk Test: Do CVT/Walk to validate coverage gap
1. Increasing in eNB power can help to improve CDR & VDR.
2. If power is at maximum, plan additional small cells. Raise to NPE for redesign

Copyright 2016 AIRSPAN NETWORKS INC.

CDR & VDR Flow Chart

Copyright 2016 AIRSPAN NETWORKS INC.

Throughput DL/UL Troubleshooting

1. PCI Planning Validation- Refer Basic Parameter Planning Guideline *
1. *If Auto PCI is enabled (Check SON Profile from Netspan) then ignore this step.
2. Congestion : If Max. Connected Users > 30 or DL PRB % > 75% & UL PRB % > 75% then
1. Check if neighbouring cells are OOS - resolve it and monitor the performance.
2. Check # connected users at NBR cells, if possible increase the power of NBR cell & monitor the performance.
3. If SMCs are not planned tightly at affected floor then add small cell for traffic offload and monitor the performance.
4. If SMCs are tightly planned then raise to NPE/NOC team for re-design the floor with additional capacity.
3. Backhaul Testing: Run Iperf test with maximum data load.
1. If throughput lower than data sent then raise issue to Backhaul team
4. Planning and Design validation
1. Check if eNBs placed in close proximity is causeing SMC to SMC interference.
2. Physical optimization if installed locations are different than plan
3. Else raise to NPE/NOC team for re-design.
5. Walk Test: Do CVT/Walk to validate coverage gap
1. Increasing in eNB power can help to improve Throughput DL/UL.
2. If power is at maximum, plan additional small cells.

Copyright 2016 AIRSPAN NETWORKS INC.

10

Throughput Flow Chart

Copyright 2016 AIRSPAN NETWORKS INC.

11

Alarm & Troubleshooting

Small Cell Alarm List

Sr. No Alarm ID

100

Alarm Description

Node Lost Comms

Severity

Critical

Probable Cause

Action

Power Failure / Backhaul Issue

1. Check Reachability through CSS

2. Check power availability on site
3. Check physical backhaul issue
1. Check clock status
2. Make sure MME connectivity is OK
3. Check for any PCI Allocation Failure alarm

557

Channel Out of Service Critical

Clock Issue / MME not reachable

556

MME connectivity lost Major

Loss of the S1AP SCTP connection to 1. Verify the extent of the alarms on the nearby nodes
a specific MME
2. Contact NOC for further advise

580

SGW connectivity lost Major

SGW is not reachable

1. Verify the extent of the alarms on the nearby nodes

2. Contact NOC for further advise

565

Auto PCI Allocation fail Major

No PCI resource available in the PCI

pool

Report to NOC for further advise

587

Wi-Fi Board
Connectivity

Major

Wi-Fi board is not active, Wi-Fi

software version is not updated

Report to NOC for further advise

579

eSON Server
connection Lost

Major

Server unreachable

Report to NOC

Copyright 2016 AIRSPAN NETWORKS INC.

13

Small Cell Alarm List PTP / GPS

Sr. Alarm
No
ID

Alarm Description

572 Node Loss of Sync

570

Severity
Critical

Node PTP Grand Master

Major
Lost

Probable Cause
The node does not have a valid sync
source and remain in the OOS state

1. For ISC, check the GM connectivity

2. For OSC, check GPS connectivity

GM unreachable

1. Check status in Netspan connection to GM

2. Report to IP team to check connectivity to the GM

PTP Grand Master Lost

PTP Degraded Stream

1. Verify additional alarms involving GPS or GM.

2. Rectify physical issues for GPS alarms
3. Report to IP team for GM-related alarms

571 Node in Holdover

Major

599 PTP Degraded Stream

Warning High PDV

213 GPS SNR Low

224

GPS Signal
Unacceptable

Minor

Minor

Action

1. Check additional alarms and events in Netspan

2. Report to NOC.

Skyview partially obstructed

GPS antenna faulty
GPS antenna cable faulty

1. Check onsite if GPS location has clear skyview

2. Check connectors, they should free from moist
3. Check gps cable on both ends make sure properly tighten
4. Try to swap new gps antenna

No satellite detected

1. Check onsite if GPS location has clear skyview

2. Check connectors, they should free from moist
3. Check gps cable on both ends make sure properly tighten
4. Try to swap new gps antenna

Copyright 2016 AIRSPAN NETWORKS INC.

14

Node Lost Comms

Description The node is not reachable from Netspan

Yes

Probable Cause : Power / IP / Backhaul issue

Impact : Services will be suspended , No Ping possible.

Yes

Actions to be taken :

1. Management VLAN (IP) Issue

a.

Request IP team to check reachability from CSS to Netspan

b.

If not reachable check Netspan route in CSS

c.

If Netspan is reachable from CSS, check reachability from CSS to eNB

d.

Check reachability of L2 SW from CSS

Check power on the site

b.

LED should be ON in the node

No

Check
eNB
Reachabi
lity

Check
Routes in
CSS

No
Check
Power

No

Report to
concern
team

Yes
Check
Physical
connecti
vity

2. Power Failure:
a.

From CSS
Check
Netspan
Reachabil
ity

Yes

Take
necessary
actions

No

3. Backhaul Issue

Report to
NOC team

a.

Check port status on L2 (Up / Down)

b.

Check the physical connections between CSS to L2 Switch and L2 Switch to SMC e-NB. (Loose
connection check )

c.

Clean SFPs used at L2 Switch and SMC e-NB. (Dust /Moisture check )

d.

Check fiber/CPRI cable (if damaged, bent)

Copyright 2016 AIRSPAN NETWORKS INC.

15

Channel OOS
Description The node is manageable but RF is not radiating
Probable Cause : Clock Issue, MME connectivity, PCI allocation failure
Impact : Services will be suspended

No

Actions to be taken :
1 . Clock Issue

a. In Netspan, clock status should be connected & Locked

Check
MME
Alarm

Yes
Check
PTP
Alarms

Report to IP
Team

Yes

b. If it is not Locked, check Grand Master (ISC) or GPS (OSC) connectivity or

refer to PTP & GPS alarm chart

No

2. MME Connectivity Issue

a. Check MME Alarm (Details in MME Connectivity Lost Alarm)

Check PCI failure

Events

3. PCI Allocation Failure

a. Check PCI alarms / events (Details in PCI Allocation failure)
b. If alarm is present, possible no PCI resources available then report to SON
team

Copyright 2016 AIRSPAN NETWORKS INC.

Report to
NOC team

16

MME Connectivity Lost

Description Loss of the S1AP SCTP connection to a specific MME
Probable Cause : MME is not reachable
Impact : If alarm present on all defined MMEs then service affecting (Channel OOS)

Actions to be taken :
1. Verify other nodes at the same site with the same MME connection if it has the same alarm
a. If it has same alarm report to IP team and check MME status
b. Otherwise, report to IP team to check:

i.

Check 602 vlan tagging on L2 switch

ii.

Check 602 gateway reachability with source as eNb on CSS

2. If everything is OK contact NOC team

Copyright 2016 AIRSPAN NETWORKS INC.

17

SGW Connectivity Loss

Description Loss of S1 connection to the SGW
Probable Cause : SGW is not reachable
Impact : If alarm present on all defined SGWs then service affecting

Actions to be taken :
1. Verify other nodes with the same SGW connection if it has the same alarm
a. If it has same alarm report to IP team and check SGW status
b. Otherwise, report to IP team to check :

i.

Check 601 vlan tagging on L2 switch

ii.

Check 601 gateway reachability with source as eNb on CSS

2. If everything is OK contact NOC team

Copyright 2016 AIRSPAN NETWORKS INC.

18

Auto PCI Allocation Fail Alarm

Description Automatic allocation of the PCI failed
Probable Cause : No PCI resource available in the PCI pool ( Channel OOS)
Impact : Services will be affected
Actions to be taken :
1. Report to NOC for further advise

Copyright 2016 AIRSPAN NETWORKS INC.

19

Wi-Fi Board Connectivity Alarm

Description The unit is not able to communicate with the Wi-Fi board (applies to AirVelocity with Wi-Fi only)
Probable Cause : Wi-Fi board is not active, Wi-Fi software version is not updated
Impact : Wi-Fi services will be affected

Actions to be taken :
1. Report to NOC for further advise

Copyright 2016 AIRSPAN NETWORKS INC.

20

eSON Server Connection Lost

Description Connection with eSON server is lost
Probable Cause : eSON server unreachable
Impact : Not service affecting but some SON performance will be affected

Actions to be taken :
1. Check Netspan status
2. Report to IP team to check connection to eSON server

Copyright 2016 AIRSPAN NETWORKS INC.

21

PTP and GPS Alarm Chart

GPS SNR LOW

PTP Degraded
stream
High PDV
(Event)

2.5 Min

Check Sky,
antenna no
obstruction

GPS Signal
Unacceptable

Node in Holdover

Node in Holdover

Node PTP Grand

Master Lost

8 Hrs

Node Loss Sync

Node Loss Sync

Channel OOS

Copyright 2016 AIRSPAN NETWORKS INC.

22

Node Loss of Sync

Description The maximum holdover time has expired, node has no valid sync source
Probable Cause : GM is not reachable or GPS connectivity issue
Impact : Services will be suspended (Channel OOS)
Actions to be taken :
1 . For ISC, check the GM connectivity
a.

In Netspan, clock status should be connected

b.

If it is not Locked, check Grand Master or refer to PTP & GPS alarm chart

c.

Report to IP team

2. For OSC, check GPS connectivity

a.

In Netspan, clock status should be locked

b.

Ensure GPS is well installed having sky clearance

c.

Check physical connectivity between eNB and GPS antenna (Moist, connector)

d.

If problem persist then try to replace GPS antenna

Copyright 2016 AIRSPAN NETWORKS INC.

23

Node PTP Grand Master Lost

Description Node has no valid 1588 clock source
Probable Cause : GM not reachable
Impact : Service will be suspended (Channel OOS)
Actions to be taken :
1. Check status in Netspan connection to GM

2. Report to IP team to check connectivity to the GM.

Copyright 2016 AIRSPAN NETWORKS INC.

24

Node in Holdover
Description Node has no valid clock source
Probable Cause : GM not reachable, High PDV
Impact : May lead to Holdover expire and probable service-affecting

Actions to be taken :
1. Verify additional alarms in Netspan like GPS SNR Low and Node PTP Grand Master Lost and events like
High PDV
2. Report to IP team to check connectivity to the GM.
3. If GPS problem, refer to the GPS SNR Low action item.

Copyright 2016 AIRSPAN NETWORKS INC.

25

PTP Degraded Stream

Description PDV conditions in the 1588 PTP stream have changed
Probable Cause : PDV reaches the threshold level
Impact: May lead to Holdover and probable service-affecting

Actions to be taken :
1. Check additional alarms and events in Netspan
2. Report to NOC.

Copyright 2016 AIRSPAN NETWORKS INC.

26

GPS SNR Low

Description Low SNR of the GPS signal
Probable Cause : Sky view partially obstructed ,GPS antenna faulty , GPS antenna cable faulty
Impact : May lead to GPS lock loss
Actions to be taken :
1. Check onsite if GPS location has clear skyview
2. Check connectors, they should free from moist
3. Check gps cable on both ends make sure properly tighten

4. Try to swap new gps antenna

Copyright 2016 AIRSPAN NETWORKS INC.

27

GPS Signal Unacceptable

Description Satellite not detected
Probable Cause : Sky view partially obstructed ,GPS antenna faulty , GPS antenna cable faulty
Impact : May lead to GPS lock loss
Actions to be taken :
1. Check onsite if GPS location has clear sky view
2. Check connectors, they should free from moist
3. Check GPS cable on both ends make sure properly terminated

4. Try to swap new gps antenna

Copyright 2016 AIRSPAN NETWORKS INC.

28

THANK YOU