Load Balancing and High Availability for BMC Remedy Action Request System

Kelly Deaver, BMC Technical Marketing

February 1, 2008

Agenda

Why use server group and load balancing How server group enhances BMC Business Service Management Products covered Examples of possible environment Licensing Installing What to do and what not to do Configuring Managing BMC Remedy Knowledge Management and load balancing

Copyright February 1, 2008 BMC Software, Inc.

Why Use Server Groups with Load Balancing

Symptoms
Screens taking too long to build/refresh Users not able to login (server busy or down)

Server group as the solution

Increased performance
Administrative tasks can be assigned to a specific AR System Server User interactions are distributed between servers

Redundancy for fail over

If one AR System Server or AR System Mid-tier fails, others pick up the load and continue

Copyright February 1, 2008 BMC Software, Inc.

BSM and AR System

Powered Powered By By AR System AR System

Copyright February 1, 2008 BMC Software, Inc.

Products Covered in this Presentation

BMC Remedy Action Request System BMC Atrium CMDB BMC Remedy FTS (Full Text Search) BMC Remedy Flashboards BMC Remedy Asset Management BMC Remedy Change Management BMC Remedy Service Desk BMC Service Level Management BMC Knowledge Management

Copyright February 1, 2008 BMC Software, Inc.

Configuration - Basic
AR System Servers
Administrative Operations Escalation, Reconciliation, etc.

Clients

Load Balancer

Database

Top server is primary for Administration and Escalation operations. Only one at a time can do this. The other two are for back up of Admin operations. All three are used for User interaction.

Copyright February 1, 2008 BMC Software, Inc.

Configuration With Firewall and Load Balancer

AR System Servers
BMC1 (real server) 11.40.35.47:2020

BMCRemedy (virtual server) 11.40.34.94:3030

BMC2 11.40.35.49:2020

Clients

Firewall

Load Balancer

Database

Here the Client comes in through a firewall to port 3030. The load balance then acts as in the earlier example but the port for each server is 2020

BMC3 11.40.35.51:2020

Copyright February 1, 2008 BMC Software, Inc.

Configuration Firewall, Web Servers & 1 Load Balancer

As in the previous slide the client requests passes through the firewall and into the load balancer but this time it is a web client. The load balancer distributes the load to the 3 web servers. Each web server is paired with an AR System server by making an IP address entry in the hosts file of the web server BMCRemedy (virtual server) 11.40.34.94:3030

Web Servers AR System Servers

Web 1

Web 2

Clients

Firewall

Load Balancer
Web 3
BMC3 11.40.35.51:2020

Database

Create a hosts file entry for each AR System Server on the companion web server -

BMCRemedy 11.40.35.51

Copyright February 1, 2008 BMC Software, Inc.

Configuration Firewall, Web Servers & 2 Load Balancers

After Initial connection is made between the mid-tier and the AR Server it remains a connection until either server fails. When one server fails another pairing is made. Even if the other server comes back up, doesnt rebalance.

Web Servers Only For Failover

AR System Servers

Clients

Firewall

Load Balancer

Database

Once a connection has been made through the 2nd load balancer, session affinity must be maintained.
9 Copyright February 1, 2008 BMC Software, Inc.

Session Affinity

Why
User instance information is cached on the AR System server Large data transfers may involve chunking. All blocks of data need to follow the same path. If encryption is used, encryption is set between the two servers.

How
Determine the sticky affinity timeout period
AR System default user session timeout is 60 minutes The Sticky Timeout must be longer than the user session timeout

In AR.cfg / AR.conf
Internal-User-Instance-Timeout (the session timeout)
The default setting is 1 hour, the maximum is 2 hours, and the minimum is 30 minutes.

On Your Load Balancer

Set the sticky affinity timeout
10 Copyright February 1, 2008 BMC Software, Inc.

Configuration DMZ, Firewall, Web Servers & 2 Load Balancers (Higher Security, WANs, etc) Q? A
Clients Firewall
12 5 16 4 1 13 14 3 2 15 Web Servers Load Balancer

DMZ

AR System Servers 6 11 7 10 Load Balancer 8 9 Database

LAN

11

Copyright February 1, 2008 BMC Software, Inc.

Configuration DMZ, 2 Firewalls, Web Servers & 2 Load Balancers (Better Performance)
Load Balancer

DMZ

Clients

Firewall

2 Network Pipes To Reduce Bottlenecks

Web Servers

Firewall
AR System Servers

LAN

Load Balancer

Database

12

Copyright February 1, 2008 BMC Software, Inc.

Configuration - NOT
Load balancing is for spreading the load, not condensing it. Plus you need to have a sticky connection between the db and the AR Server.

AR System Servers

Clients

Load Balancer

Load Balancer

Database For Failover Use A Disk Array Instead

13

Copyright February 1, 2008 BMC Software, Inc.

Purchasing and Licensing

Purchases required per environment (dev, test, production)

1 copy of AR System per server running it 1 copy of each application (all servers in the group use the same application)

License 6.x or 7.0.x on each server

You must specify server group when ordering license keys
You have to order through BMC Support-Licensing, not online key generation

License is stored in license file on each server

1 AR Server Application (server group name must be consistent) Fixed User (*SITE*) Floating User (server group must be consistent)

Licensing 7.1 is easier

For in-depth info see the session BMC Remedy Customer Enforced Licensing Simplification Thursday 2PM
14 Copyright February 1, 2008 BMC Software, Inc.

Licensing 7.1 Servers

License is stored in shared database
Create an entry for each AR System Server (with required key matching each host) Create one shared entry for each application Create one shared entry for Fixed Users (system or application) Create one shared entry for Floating Users (system or application)

15

Copyright February 1, 2008 BMC Software, Inc.

Install Order
Install on the initial server
AR System, Approval Server, Assignment Engine, Email Engine Mid-Tier Server (same or separate physical server) Flashboards, CMDB Applications For subsequent AR System Servers All install paths must be consistent on each server in the group Install AR System Server
Choose the same database when installing the servers in a server group Select the Share database option

License the servers appropriately Install CMDB

Specify Share Server Group only executables and config entries are created

Install Flashboards
Forms are only installed on the first install/primary server by default

Applications

Install subsequent Mid-Tier Servers

16

Copy the application\bin sub-directories from the initial server Configure ar.cfg/.conf and armonitor.cfg/.conf

Copyright February 1, 2008 BMC Software, Inc.

Configure Each Server

Server group name (shared)
Name shared by all servers in the group (BMCRemedy) Must match name of group applied with licenses Check Interval Dont configure Dont configure How often other servers check if it is online through the through the Full Text Search indexing load balancer load balancer Put index directory on a shared file system and point all servers there. Email Engine Port needs to match on all servers BMC Remedy Flashboards Port needs to match on all servers BMC Service Level Management Import the public key of the Data Collection Point security certificate from the initial server BMC Remedy Distributed Server Option (DSO) Specify any server in the group as the source Must specify explicitly a server for distributed returns, deletes and updates

17

Copyright February 1, 2008 BMC Software, Inc.

Configure Each Mid-Tier Server

Host file on each web server

<Load Balancer Virtual Server Name> <Actual Server IP> BMCRemedy 11.40.35.51

18

Copyright February 1, 2008 BMC Software, Inc.

Configure Each Server AR.cfg

Server Name (unique)

by default, the fully qualified name of the server
bmc.mycompany.com

You can specify a name in the ar.cfg file

Server-Connect-Name : bmc1

Used by the other servers to talk to the server

Copy from ar.cfg on initial server to all others

Plugin <full path>omfobjiefilapi.dll" Plugin <full path>arfslasetup.dll" Plugin <full path>arfcbdata.dll" Plugin-Path <concatenation of paths> Alternate-Approval-Reg: T

19

Copyright February 1, 2008 BMC Software, Inc.

Configure Each Server ARMonitor.cfg

Copy from armonitor.cfg on initial server to all others

<full path>/brsvcbrie.exe" -d <AR System path> -m <full path>/arrecond.exe" -i <AR System path> -m <full path>/arcmdbd.exe" -i <AR System path> -m

20

Copyright February 1, 2008 BMC Software, Inc.

Configure Operation Ranking

Create 1 entry for each server/operation combination. Business Rule Engine, Approval Server and CMDB must share ranking with Administration. They have to run on the same server.

Servers are used based on ranking, lowest to highest. Null excludes the server from the operation The number of times the server does not respond before the next server for the operation takes over

21

Copyright February 1, 2008 BMC Software, Inc.

Managing : Log - arsrvgrp

Tracks
starting/stopping operations evaluation of other servers recache requests Event time

22

Copyright February 1, 2008 BMC Software, Inc.

Managing : Server Status Signaling

You can review the status of your server group from the servgrp_board table in your database.
23 Copyright February 1, 2008 BMC Software, Inc.

Configuring Knowledge Management

Top server is primary for Administration and Escalation operations. Only one at a time can do this. The other two are for back up of Admin operations and primary for User interaction.

KM Servers

AR System Servers

Application Server
Hummingbird Search Server

Clients

Load Balancer

Data Server
Knowledge Solutions

24

Copyright February 1, 2008 BMC Software, Inc.

Configuring Knowledge Management Servers

You must copy and edit files on all servers Details can be found in the White Paper
Configuring BMC Remedy Knowledge Management in a Load Balanced Environment

25

Copyright February 1, 2008 BMC Software, Inc.