Redbooks Wiki: Optimizing Lotus Domino Administration

Table of Contents
Part 1. Getting Started ............................................................................................................... 8 1.1. Documentation ............................................................................................................... 8 1.1.1. Content to document ............................................................................................. 8 1.1.2. Server Details ........................................................................................................ 9 1.1.3. Architecture Overview ........................................................................................... 9 1.1.4. Security Standards .............................................................................................. 14 1.1.5. Backup Standards ............................................................................................... 14 1.1.6. Naming Standards............................................................................................... 15 1.1.7. Instructions for End Users ................................................................................... 17 1.1.8. Conclusion ........................................................................................................... 18 1.2. Daily, Weekly, and Monthly Checklist .......................................................................... 19 1.2.1. General recommendations .................................................................................. 19 1.2.2. Actions ................................................................................................................. 20 1.2.3. Conclusion ........................................................................................................... 23 1.3. Security Checklist......................................................................................................... 24 1.3.1. Additional References ......................................................................................... 25 1.4. Domino Authentication Options.................................................................................... 26 1.4.1. Choosing the Domino Authentication Options..................................................... 26 1.4.2. SmartCards (1) .................................................................................................... 27 1.4.3. Lotus Notes and HTTP Password Synchronization (3)....................................... 27 1.4.4. LDAP (4 and 5).................................................................................................... 28 1.4.5. SPNEGO (6)........................................................................................................ 28 1.4.6. Lotus Notes and Operating System Single-Sign On (7)...................................... 29 1.4.7. Tivoli Directory Integrator (9) ............................................................................... 30 1.4.8. Additional Resources........................................................................................... 30 1.5. Agents and the Domino Administrator ......................................................................... 31 1.5.1. Agent Triggers ..................................................................................................... 31 1.5.2. Determining Which Agents Are Scheduled to Run on the Server....................... 32 1.5.3. Agent Manager Settings That Affect Agent Execution ........................................ 32 1.5.4. Full Text Searches and Agents ........................................................................... 33 1.5.5. Performance Impact of New Agents and Applications ........................................ 34 1.5.6. Troubleshooting Problems with the Agent Manager Task .................................. 34 1.5.7. Writing an Agent .................................................................................................. 34 1.6. Expanding the Domino Domain ................................................................................... 35 1.6.1. Registering and Securing a New Server ............................................................. 35 1.6.2. Replicating Critical Databases............................................................................. 36 1.6.3. Mail Routing......................................................................................................... 39 1.6.4. Monitoring and Managing Multiple Servers ......................................................... 40 1.6.5. Clustering............................................................................................................. 40 1.7. Design, Replication, and Mixed Releases: Avoiding Design Ping-Pong ..................... 41 1.7.1. Description of the Problem .................................................................................. 41 1.7.2. Preventing the Problem ....................................................................................... 41 1.7.3. Working with Mixed Clusters ............................................................................... 43 1.8. Routine Maintenance Best Practices ........................................................................... 44 1.8.1. Fixup and Transaction Logging ........................................................................... 44 1.8.2. Regular Maintenance .......................................................................................... 44 1.8.3. Database Corruption ........................................................................................... 45 1.8.4. Hard Disk Fragmentation..................................................................................... 45 1.9. Mobile Access .............................................................................................................. 46 1.9.1. Traveler................................................................................................................ 46 1.9.2. BlackBerry ........................................................................................................... 48 1.9.3. Lotus iNotes Ultra-Lite ......................................................................................... 50 1.9.4. IMAP/POP ........................................................................................................... 52

Part 2. Managing Users and Clients ........................................................................................ 54 2.1. Optimizing Lotus Notes Client Administration Tips ...................................................... 54 2.1.1. General Client/User Management Tips ............................................................... 54 2.1.2. Recent Contacts .................................................................................................. 55 2.1.3. Local Replicas ..................................................................................................... 55 2.1.4. Smart Upgrade .................................................................................................... 56 2.2. Managing a User's Inbox.............................................................................................. 57 2.2.1. Using Inbox maintenance to manage mail file size ............................................. 57 2.2.2. Quota Enforcement Options ................................................................................ 58 2.2.3. Quotas with DAOS Enabled ................................................................................ 58 2.2.4. Enforcing Quotas on Local Replicas ................................................................... 59 2.3. Policies ......................................................................................................................... 59 2.3.1. Introduction to Policies ........................................................................................ 59 2.3.2. Using Policies to Standardize Secure and Simplify Your Environment .............. 60 Part 3. Effective Server Administration .................................................................................... 65 3.1. Monitoring..................................................................................................................... 65 3.1.1. Monitoring Options .............................................................................................. 65 3.1.2. What should be monitored? ................................................................................ 66 3.1.3. Monitoring Profiles for Domino ............................................................................ 66 3.1.4. Domino Event Monitoring .................................................................................... 75 3.1.5. Further Reading................................................................................................... 79 3.2. Mail Routing ................................................................................................................. 80 3.2.1. Managing Spam .................................................................................................. 80 3.2.2. Mail routing and multiple directories.................................................................... 81 3.2.3. Journaling ............................................................................................................ 82 3.2.4. Out of Office Notification...................................................................................... 85 3.2.5. Mail routing in a clustered environment............................................................... 85 3.3. Mass Mailings............................................................................................................... 87 3.3.1. The Mass Mailing Problem .................................................................................. 87 3.3.2. The Mass Mailing Solution .................................................................................. 88 3.3.3. Conclusion ........................................................................................................... 94 3.4. Multiple Directories....................................................................................................... 94 3.4.1. Condensed Directory Catalog, Extended Directory Catalog or Directory Assistance 94 3.4.2. Hints and Tips...................................................................................................... 95 3.5. Server Clustering Options ............................................................................................ 96 3.5.1. Keep It Simple ..................................................................................................... 96 3.5.2. Redundant Domino Parts .................................................................................... 97 3.5.3. Domino Cluster.................................................................................................... 98 3.5.4. OS Cluster ......................................................................................................... 100 3.5.5. Internet Cluster Manager (ICM)......................................................................... 102 3.5.6. iNotes High Availability Configuration ............................................................... 103 3.5.7. IMAP failover (Domino 8.5 new feature) ........................................................... 103 3.5.8. Lotus Traveler Server High Availability ............................................................. 103 3.5.9. Load Balancer ................................................................................................... 104 3.5.10. Software proxy (IBM HTTP, nGinx, etc) ............................................................ 105 3.5.11. Sametime and QuickR High Availability ............................................................ 105 3.5.12. Disaster Recovery Plan ..................................................................................... 106 3.6. Transaction Logging................................................................................................... 106 3.6.1. General Transaction Logging Recommendations for 8.5.x Servers ................. 106 3.6.2. Transaction Logging Tips .................................................................................. 107 3.6.3. NOTES.INI Recommendations for Domino 8.5.x Servers ................................ 107 3.6.4. Domino 8.5.x and ODS 51 Updates .................................................................. 108 3.7. Domino Attachment Object Service (DAOS) ............................................................. 109 3.8. Managing Domino Indexing ....................................................................................... 111 3.8.1. View Indexes ..................................................................................................... 111

3.8.2. Full Text Indexes ............................................................................................... 112 3.8.3. Domain Indexes................................................................................................. 116 3.9. Backup a Domino Environment.................................................................................. 116 3.9.1. Backup Basics ................................................................................................... 118 3.9.2. Backup Strategy ................................................................................................ 118 3.9.3. Backup Software ............................................................................................... 122 3.9.4. What to back up................................................................................................. 123 3.9.5. Backup procedures............................................................................................ 124 3.9.6. Backup Scripts................................................................................................... 126 3.9.7. Recommendations............................................................................................. 127 3.9.8. Summary ........................................................................................................... 128 3.9.9. Reference Reading............................................................................................ 128 3.10. Restore .................................................................................................................. 128 3.10.1. Disaster Recovery ............................................................................................. 129 3.10.2. Static File Recovery........................................................................................... 130 3.10.3. Domino Data File Recovery............................................................................... 130 3.11. Procedure to Restore Deleted Documents on IBM i.............................................. 136 3.11.1. Operating Type Save Restore Procedure ......................................................... 137 3.11.2. BRMS Full Save Restore Procedure................................................................. 138 3.11.3. BRMS Incremental Save Restore Procedure.................................................... 140 3.11.4. Additional Resources......................................................................................... 141 3.12. The Domino HTTP Server ..................................................................................... 141 3.12.1. General Server Configuration............................................................................ 141 3.12.2. iNotes................................................................................................................. 141 3.12.3. Troubleshooting and Tuning.............................................................................. 142 3.12.4. Additional References ....................................................................................... 142 3.12.5. Some Tips.......................................................................................................... 142 3.13. Domino HTTP Server Security .............................................................................. 143 3.13.1. Server Access ................................................................................................... 143 3.13.2. User Security and Authentication ...................................................................... 145 3.13.3. Database Security ............................................................................................. 146 3.13.4. File System Security.......................................................................................... 150 3.14. Setting up a Redirection Application for Lotus iNotes users.................................. 151 3.14.1. Create the iNotes Redirect Application ............................................................. 151 3.14.2. Configure the iNotes Redirect Application......................................................... 152 3.14.3. Configuring the iNotes Redirect Application as the Default Home Page .......... 156 3.15. Securing Lotus iNotes............................................................................................ 158 3.15.1. iNotes and the Notes ID files ............................................................................. 158 3.15.2. Active X Controls ............................................................................................... 158 3.15.3. Browser Cache Management............................................................................ 159 3.15.4. Encrypting Offline Databases............................................................................ 160 3.15.5. S/MIME .............................................................................................................. 161 Part 4. Tuning the Environment ............................................................................................. 162 4.1. Health Check.............................................................................................................. 162 4.1.1. High Level Checklist for Health Check .............................................................. 162 4.1.2. Performing the Health Check ............................................................................ 163 4.2. Document Configuration Tuner (DCT) ....................................................................... 172 4.3. Establishing a Performance Baseline ........................................................................ 173 4.3.1. Recommended Metrics...................................................................................... 174 4.3.2. Collecting Domino Statistics .............................................................................. 175 4.3.3. Reporting Database........................................................................................... 175 4.4. Domino Tuning Tips (all platforms) ............................................................................ 176 4.4.1. View Index Updates........................................................................................... 176 4.4.2. Disable Transaction Logging For Certain Databases ....................................... 177 4.4.3. Replication ......................................................................................................... 177 4.4.4. Internal Caches ................................................................................................. 179

4.4.5. Multiple Mail Boxes............................................................................................ 181 4.4.6. Tips for Server Based Mail Rules...................................................................... 181 4.4.7. Tuning User Sessions ....................................................................................... 181 4.4.8. Domino Configuration Tuner (DCT) .................................................................. 182 4.5. Tuning for Virtualized Environments .......................................................................... 182 4.5.1. The Pros and Cons of Virtualization.................................................................. 182 4.5.2. Static Resources ............................................................................................... 183 4.5.3. Best Practices for Guest VM ............................................................................. 183 4.6. Domino on Windows Tips .......................................................................................... 185 4.6.1. System Page Pool ............................................................................................. 185 4.6.2. Other Tuning Tips for Windows Servers ........................................................... 186 4.6.3. Additional references......................................................................................... 186 4.7. Domino on Linux Tips ................................................................................................ 186 4.7.1. Monitoring Server Resources ............................................................................ 186 4.7.2. Operating System Limits ................................................................................... 186 4.7.3. Linux Services ................................................................................................... 187 4.7.4. TuneKrnl ............................................................................................................ 187 4.7.5. Troubleshooting and Debug Tips ...................................................................... 187 4.7.6. Disabling concurrent I/O and direct I/O on Domino servers on AIX .................. 188 4.7.7. Tuning Java for Domino on AIX ........................................................................ 188 4.7.8. Perfpmr for AIX.................................................................................................. 188 4.8. Domino on IBM i Tips................................................................................................. 188 4.8.1. Overview............................................................................................................ 188 4.8.2. Performance ...................................................................................................... 189 4.9. Tuning Tips for the Domino HTTP Server.................................................................. 192 4.9.1. HTTP Server Threads ....................................................................................... 192 4.9.2. HTTP Requests ................................................................................................. 193 4.9.3. JVM Heap .......................................................................................................... 193 4.9.4. Database Performance...................................................................................... 194

0.1 Preface
Note: This PDF document is the original text from the Optimizing Lotus Domino Administration wiki found in the URL in which this document originated. Always refer to the wiki version for the latest updates. This IBM Redbooks wiki provides you with information on how to optimize Lotus Domino administration. The focus is to provide Lotus Domino administrators with information on how to get most of their valuable time. Optimization of a Lotus Domino environment is not only a matter of how to set specific configuration parameters on a server or on a client; it is more a conceptual approach on how to address specific needs of the environment. In this Redbooks wiki, we share our experiences and industry best practices about how an optimized and smart Lotus Domino environment should look like and the checklists and steps you should perform to ensure a smooth and optimized Domino environment. Ideas and concepts presented here are meant to be an introduction, and are not meant to be a complete list. If there are existing wiki articles, technotes, or whitepapers available that have detailed discussion on the topics being presented we provide the reference links.

Using this wiki

Each article in this Redbooks wiki is intended to be used stand-alone. However, you can navigate back to the TOC and access all of the articles in the series by clicking the "Table of Contents" link at the top of each article.

0.2 About the Authors

This book is produced by a team of specialists from around the world. Paul Band is a certified IT specialist with IBM Software Services for Lotus in the UK. He has been working with Lotus software within IBM for 10 years, having started with Domino Development in R4.6. Paul has worked on countless customer engagements ranging from troubleshooting customer issues to designing global enterprise collaboration environments. In his spare time, Paul tries to improve his golf swing.

Thomas Hampel is an IT Architect at IBM Germany. His key areas of focus are migration projects, performance and delivery service architecture for customers who outsourced to IBM. He is working with Lotus Domino since version 3 and is an IBM Certified Advanced Lotus Developer as well as IBM Certified Advanced Lotus Administrator in various versions up to 8.5. He is also an IBM Certified Advanced Security Professional.

Amy Hoerle is an Advisory Software Engineer in the Lotus Support Center. She has been focusing on Lotus products running on the IBM i Operating system for over 10 years. She is also a frequent presenter at the COMMON, A Users Group, annual meeting. When not working, Amy spends her time caring for her children, volunteering at their school, reading or working in her garden. Gladstone Lang is a IT Specialist at ITD/SSO, Hortolandia/SP/Brazil. He is one of the account focals for the Email and Collaboration Service Line working with Lotus products. Before joining IBM in 2004, Gladstone used to work at an IBM Client and Training Center.

Vladislav Tatarincev is the Technical Director and co-owner of CYONE. www.cyone.eu. He has a Master of Computer Science from Latvian University. He has been working with Domino from release 4.5, for more than 10 years. He is also an IBM Certified Security Professional. Vladislav is the author of many freeware tools for Domino. His key areas of focus for Lotus Domino are: Performance, Traveler, Security. His hobbies include: diving, shark diving, wreck diving, underwater archeology, and motorbikes. Wei-Dong Zhu (Jackie) is an Enterprise Content Management Project Leader with International Technical Support Organization. She has more than 10 years of software development experience in accounting, image workflow processing, and digital media distribution using C, C++, Java, and Lotus Notes scripts. Jackie holds a Master of Science degree in Computer Science from the University of the Southern California. She is a Certified Solution Designer for IBM Content Manager and has managed and lead the production of many ECM redbooks and Lotus Domino redbooks wiki projects.

Part 1. Getting Started 1.1. Documentation

How many times have you come across a project or new client, where you have to migrate hundreds of databases, application, and servers? You assume that everything is documented and has always been well documented. As you begin to work on the project, you realize that the existing documentation does not contain what you needed or expected. This article describes how to properly document your Lotus Domino environment and what kind of documentation you should have based on the size of your corporation. The article serves as a valuable resource as you begin to document your environment or assist you in evaluating your current documentation. Before you start, here are some questions to consider regarding your existing documentation: Where is the documentation stored in? Is there a defined place for all documentation? Have you ever inherited an environment from someone else? If so, is the document about this environment up to date as you were told?

Documentation helps the current and the next administrator or technical project manager to quickly understand the environment. Properly and up-to-date documentation helps the continuation and growth of your environment. Lets look at what documents should be created and what information should be documented.

1.1.1. Content to document

What makes a good documentation? Just listing the number of servers and host names is certainly not enough in all cases. A well documented environment helps to identify problems more efficiently, resolve tickets faster, and helps to find bottlenecks. The following table lists the kind of information that should be part of your documentation and what implementation size benefits most from this type of documentation.
What Content to Document Server details Architectural overview Naming standards Instructions for end users Small + + (+) (+) Medium +++ ++ ++ ++ Large +++ +++ +++ +++

1.1.2. Server Details

The very first and basic documentation is to have the server configuration described. Often, the Domino Directory is misunderstood to be online documentation of the server configuration. However, there is much more to be detailed for example, to allow someone who is new to the environment to understand the building blocks. The server configuration is essential for upgrade projects because it can uncover pitfalls such as incompatibilities before they would cause a problem. For each logical Lotus Domino server in your environment, the following (but not limited to) configuration elements should be detailed: Server names and IP addresses Platform details such as hardware, operating system Lotus Domino version with fix packs or hot fixes installed (if any) Additional Lotus products with name, version and patch level 3rd party software installed with information on licensing details, install instructions and describing how to access the installation package

For this part of the documentation, it is advisable to use the server name as the primary key, e.g. have one document per server where you keep track of the current server configuration details. The resulting document must be able to be referenced in other parts of the documentation. That's why a Notes application is a good choice for this kind of information. Document links can be used to cross reference information.

1.1.3. Architecture Overview

An architectural overview diagram is a graphical overview of your servers, clusters and their inter-connections. The main focus is set on Lotus Domino servers and how their logical communication paths like mail and replication or SMTP connections. The main purpose of the architecture overview diagram is to communicate a simple, brief, clear and understandable overview of the environment that the administrators are working with. Most likely, this is a drawing, much larger than one sheet of paper, which is the building of your environment. This diagram should include: Lotus Domino Servers with their name and primary usage type and cluster membership (if applicable) Mail routing connections Replication paths For one way connections like "Pull only", use arrows to point out the direction of the data flow Inbound and outbound SMTP connections Incoming and outgoing 3rd party connections like ODBC or other type of connections Major non-Domino systems which Domino is communicating with

Some recommendations for creating an architectural overview diagram are: Do no try to put too much information into the same drawing. It is important to get the concept rather than all the tiny little details at this point in time. Focus on the Domino level, ignore details such as hardware and operating system, patch level, etc. Do not use abbreviations without explaining them in (e.g.) a legend. Make sure when you build this overview, others can understand it easily. Especially for complex environments, it is a key element to a successful documentation because even the person who created this overview might not remember all the details later. Work with your application development team to get an understanding of the 3rd party connections. There is a high chance that administrators do not know what developers have done in the past. It is essential to get a full understanding of these interconnections to avoid problems when applying changes to the infrastructure. Make sure to describe the type of connection, e.g. by using different colors for each connection type. Include a legend within the drawing for more details. An example for this legend is shown below.

Keep this overview up to date by making sure changes in the infrastructure are reflected in the drawing as soon as possible. In best case, the documentation is updated as part of the change implementation.

Tools and Method to Create an Architectural Overview Diagram

To create a drawing, any vector oriented graphic software is acceptable. Even CAD software or simple bitmap oriented graphic software will work. A broad number of software products are available on the market, from commercial software products to freeware applications. The product choice is yours. Just make sure to be familiar with its usage and be comfortable with the licensing details. Some examples for applications which are free of charge are listed below.
Dia http://projects.gnome.org/dia/ yED http://www.yworks.com/en/products_yed_about.html

After you have selected the software of your choice, create a drawing in the following way: 1. Create an overview of the Lotus Domino Domains and outline how they are connected. For this first step, the Domain can be represented by a cloud icon or similar. Review the replication topology within the Domain by looking into the Replication Topology of each Domain. This information can be retrieved from the Lotus Domino Administrator client within the Replication \ Replication Topology tab as shown on the picture below.

10

Note: The Domino Administrator client will retrieve this information from the Domino server which runs the maps task. This task is not automatically started on a Domino server. For details on how to enable the maps task, refer to the Lotus Domino Administrator Help http://www.ibm.com/developerworks/lotus/documentation/domino/ Add the administration server of each Domain and walk along the replication path. Ensure that in the end of this process, all servers of a Domain were added.

Example

Here is an example of how a simple architectural overview diagram can look like:

11

Standards Especially in large organizations, it is important to describe standards that apply to the entire corporation. These standards can apply to every single detail of the environment, sometimes predefined by other people in your company e.g. your operating system standard or similar. Even if there are no regulations, it is advisable to define simple but effective software standards, giving you and your peers the opportunity to work towards a common target.

Hardware Standards
Start with documenting the hardware type and size, and used for what purpose. From the Domino point of view you start by Defining server classes, this is where you defer the server usability according to registered users or user access, location size, server main task, etc. For example, in Company A, the architect has defined that: Small Servers should not exceed 150 users or host application for small locations (up to 150 users); Intermediate Servers can work as administration/application hub and should not exceed 1000 registered users or host application where the concurrent users should not exceed 1000 users; Large Servers should not exceed 3000 registered users or host application where the concurrent users should not exceed 3000 users. They may also be used for high performance infrastructure servers (e.g. central SMTP gateways).

In a second step, for each of the server classes above, define: Server Parameters: This is where you define based in the Server Class how your servers configuration are going to be; type and amount of CPUs; how much memory they should have, etc. Hard Disk Layout: In here, you identify how your servers hard disk is or should be configured, where would the operating system be installed; where the binaries are going to be; and where is your data configured. Also in here, you should determine which type of disk array you are going to be using or used according to each different Server Class.

Be aware that vendors change their server hardware models quite often. New and more powerful servers are being offered while the older ones might not be available anymore. This is why you should consider the hardware standards as a rough guidance for new Domino administrators or people who are not familiar with Domino itself. Keep them updated on a yearly basis and do not hesitate to brainstorm smaller adjustments, e.g. to use a more powerful CPU if it becomes available. Note: Domino is a very I/O intensive application. When you choose a server model, choose the system with best I/O thruput for best performance!

12

Software Standards
An important part of the environment documentation is the software standard. You determine what will be the server operating system according to each Server Class. Again, we can see that when you determine the Server Class correctly, it facilitates everything that comes after. Below is the suggested software standards that help you in your documentation: Operating System (OS): You determine what should be the OS version and which service pack needs to be installed according to the products requirements. Depending in your Server Class, you can determine a specific OS. Anti-virus Software: For anti-virus software, you have to differentiate between antivirus software for the OS and anti-virus software for Lotus Domino: Anti-virus software at the OS level: You document the software version, patch level applied, how the virus pattern files are being updated (how often they should be updated), and also the files to be ignored by the OS anti-virus. Anti-virus software at the Lotus Domino level: You document the software version, patch level applied, and how the virus pattern files are being updated (how often they should be updated). Lotus Domino server: Document what is the Lotus Domino Server version and Fix Packs applied according to each sub-software requirements. We all know that for every product, there is a recommended Lotus Domino Server version with a specified Fix Pack, which is very important to follow. Also, you document what should be the folder naming convention for your Lotus Domino Server binary folder and data folder Backup software: This is a tricky area. In many circumstances in a large company, the Domino administrators do not work with the backup administrators. It is very critical for every environment to have a good backup standard and policies very well defined: Domino server backup tool and policies: Document what are covered and how your Domino server backups occur (daily, weekly, monthly) and what type (Incremental, Full, the use or not of Transaction Logging Archiving). OS backup tool and policies: Document what are covered and how your OS backup occur (daily, weekly, monthly and what type for each Incremental and/or Full). Special backups: If your company follows some sort of rule that request a longer retention period or any special tasks that need to be done with the Lotus Domino Server backup, this is the place where you should document it. Environment monitoring: You document how your environment is being monitored (which tool, and what if being monitored at the OS level and at the Lotus Domino Server level). Server reporting: You have to differentiate between reporting of operating system specific data and reporting of Domino specific data. This document only covers the Domino related reporting; OS settings: Document the mandatory OS settings for all Domino servers. These mandatory settings are necessary to support a stable and secure environment and to minimize the support efforts.

Document what and how your OS should be set (e.g. drive letter, volume name, if Windows update should be automatic or not, network card naming convention, registry keys for OS tuning, etc): Drive Indexing: If option is turned off as per Lotus recommendation (for each drive) or if the service is turned off in the control panel (which covers all disk drive).

13

System Page file: Document how your systems page file is set. Time settings: To assure a smooth mail routing and replication between all Domino servers, it is important that the servers have been set to the correct time. Time zone: The operating systems time zone has to be set to the correct value according to the physical location of the server. In addition, the setting "Automatically adjust clock to daylight saving changes" should be enabled and Domino servers should be set to "use OS time zone settings". Regional settings: Document how your servers Regional Settings are set, when working on Windows OS environment;

Not all of the information must be described for a small environment. In large and growing environments with multiple administrators and teams working in different time zones, it is clearly a benefit to have common settings lay out.

1.1.4. Security Standards

One of the key components in a documentation is a description of the security standards in your environment. This information is not only essential in case of a disaster, its also supposed to clearly describe whats allowed and whats not. Describe the different roles and the limitations for each role. E.g. User Administrator, Database Administratror, Application developer Defined server access level, e.g. how the server security tab is supposed to be set up. Part of this is already described in the naming standards, so the focus should be on the concept rather than the naming as such. Limitations and permissions within the environment defined in form of a corporate policy Think about who is allowed to run agents, who is allowed to sign applications, script libraries, etc. Have you got a specific role defined who will perform this work? Handling of user id's and passwords, where are they stored, who (which role) Handling of access deny groups Where backup copies of key system (cert.id, server.id's, administrator id's.) files are located and who got access to them in case of an emergency

Please note that the list above wont be a full list of items to be considered - additoinal elements may apply to your environment, so please ensure to update this chapter of your documentation on a regular basis.

1.1.5. Backup Standards

Documentation should also include a description of your environment is backed up and what users can expect from a restore. This should include: Backup policy, descrbing your corporate rules for how long a backup is retained "What" explaining what kind of information is backed up how often. Here the focus should be on Domino data and not so much on other elements such as the operating system "When", backup jobs are starting and when your backup window ends. "Where", describing which servers do perform which role within your backup concept, e.g. if one server is backing up data for all servers, etc.

14

"How", describing what backup software is used, where its installed Details about the restore process and an estimate of how long it will take to have the restore available

For more details about backup concepts for Domino please refer to the chapter "Backup a Lotus Domino Environment".

1.1.6. Naming Standards

Naming standards allow people who are new to the environment to understand how to maintain consistency while extending or changing elements within. The following list represents configuration details of a Lotus Domino environment where naming standards could be applied: Server names e.g. Define when to use what kind of server name. As the previous example, Company B uses CCNNNNXXX where CC stands for ISO country code, NNNN server Type (MAIL, HUB, APPL, etc), and XXX stands for the server sequential number according to its Type. Domain names. Cluster names. Domino named Networks. Access control lists. Directory names. e.g. Describe the directory structure of a Domino server and where to put which type of applications. Note: You should document your company standards for servers directories (folders) names for system databases, users Mail database, mail-in databases, and application databases. File names. e.g. Define how to get a unique file naming in place. Document the naming convention used by your company for file naming convention at the Domino side. You can cover the naming convention for: users mail database, mail-in database, common Domino application database, Domino application based in the default templates; etc. Group names. Document the naming convention used by your company for groups used in the Domino environment. Make sure to define groups following Domino standards: multipurpose, access control list only, mail only, server only, and deny list only. Also, clearly define which characters are allowed and which are not. Rooms and resources. For information, see http://www.ibm.com/support/docview.wss?rs=899&uid=swg21251010 User names. Describe the naming convention used by your company for users names naming convention. Cover all the details that are in use in your environment, such as: First Name, Last Name, Full Name, Short name, Internet Address and any other important field related to user that is in-place at your company. e.g. Which characters are allowed or how and when to use middle initials. Email addresses. e.g. How an email address is computed if users can choose any address of their choice or if there are restrictions within your organization.

15

Agent signer names If you use specific ID files to sign or run your agents, then describe what syntax to be used for them.

Note: There are limitations defined by IBM Lotus for each of the elements listed. For more detail, refer to IBM Technote 1091216.

Access Control List

In this section of the documentation, include information on the database access control lists (ACLs). They provide a flexible mechanism for the database owner to determine the rights that individuals and groups have with respect to a Notes database. The ACLs for a database are server specific, in that each server enforces the ACL as specified in its version (replica) of the database. For ACL documentation, include how your companys default template ACLs are set and how the ACL of the system databases are set. With this kind of information, you can improve your level of standardization and fine tune your monitored environment. To document ACL settings, we provide the following suggestions as how this can be done: Define basic ACL rules, e.g. Define ACL entries which must be present in all applications or must be present in all mail files. This can be all the groups and default IDs that should be in all databases ACL. Normally, these are groups that are related to IT Administrator and Administrative Server groups such as Domino Administration Process server, Server Backup, and so on. Databases that are under investigation. In most companies, the legal department can request access to any database that may be part of some sort of litigation process. Create and document the standards for this scenario. Define how the ACL for Domino System databases should look like in your environment. Include every ACL entry with its access rights and roles from the following system databases: names.nsf, admin4.nsf, catalog.nsf, log.nsf, certlog.nsf, events4.nsf, ddm.nsf, domlog.nsf, domcfg.nsf, busytime.nsf / clubusy.nsf, report.nsf, statrep.nsf, mail.box / mailX.box (where X stands for the number from 1 to n depending on the amount of mail.box set in the configuration document), statmail.nsf, mtstore.nsf, and da-CC.nsf (where CC stands for the naming convention used for your companys Country code or Site code).

Format to be used to document naming standards

It does not matter what format you use to document naming standards; however, it is important that this part of the documentation is available to everyone. It must be simple and easy to access and should be versioned. There are multiple options to achieve this. Some examples are: Stored in a Notes application, e.g. A discussion database where all users have READER access. Create a file based document (e.g. PDF) and store it in (e.g.) Lotus QuickR or any other existing document management system.

16

An internal web page, where users can access the information by using their web browser.

There are other methods. Use your favorite ones. Again, make sure the result is accessible for all your users who need the documentation. The following recommendations should be kept in mind when detailing naming standards: In small implementations of Domino, naming standards most likely do not need to be defined because the number of servers and domains are relatively consistent. Nevertheless, it might be useful to define some naming standard to set the scene for future growth. Within larger Domino environments, corporate naming standards are defined or need to be defined to define precise rules and limits for a team of administrators. This is why it should include all the elements we mentioned in their naming standards. Especially in large environments, allow users and IT people to improve these naming standards by offering a discussion forum where people can ask why a certain standard is defined in this way. Keep answers accessible for others and be openminded for changes suggested by your users.

1.1.7. Instructions for End Users

To reduce the number of help desk tickets and service requests, end users need to be provided with a completely different kind of documentation. Most likely, end users do not want to know about the detailed server configuration or their replication paths. Instead, they look for guidance on how to manage standard requests in an efficient way. Descriptions of common processes such as How to reset request new User ID or how to reset a password is what end users are usually looking for. The list of processes can be endless. Always aim to reduce the time support staff needed to spend for answering the same question over and over again. Consider this part of the documentation to be one of the first places for people who are new to your organization. Its purpose is to be used for self assistance platform or a form of guidance for your end users: Process descriptions What-if scenarios Background information and cross reference (e.g. to naming standards) Help desk phone number(s)

Additionally, keep the following recommendations in mind: Include information for 1st level help desk, e.g. where to route tickets or how to reach self service portals. Do not mix end user instructions with training material. If people are new to Lotus Notes, there are better existing resources to refer them to. You do not have to create your own training material. For more information please take a look into: http://www-01.ibm.com/software/lotus/training/ http://www-01.ibm.com/software/lotus/training/multimedialibrary.html

17

Define the language which is to be used. Depending on your corporation and regional distribution, not all users might understand English. The language is defined by what users understand best, not by what administrators would like to use. Extend this part of the documentation as needed. Whenever you experience a growing amount of questions or requests in a certain area, add one more instruction and cross reference them where needed. Do not send mass mails to your users communicating how a new process looks like. Instead, put the process description into this part of your documentation and refer users to it by sending a link to the respective entry.

Format to be used for documenting end user instructions

Although there are many different ways to provide information, not all method provide the required functionality: Simple to access, e.g. via web browser or Lotus Notes client Distributable to a broader population Accessible even when having a problem e.g. Stored locally on the users desktop

One advisable method is to set up a new Lotus Domino application based on the Help template. IBM Technote 1164526 describe how to do this. Keep in mind that process descriptions are likely to change. Especially in larger organizations, they are not easy to categorize because the processes differ e.g. between countries, regions, business units or even between departments.

1.1.8. Conclusion
With a well documented environment, any infrastructure changes and any emergency situation can be faced with more efficiency and more professionalism. It is important to understand that documentation is not a static document. It is a living document with your environment and needs to be updated on a regular basis to keep its value.

18

1.2. Daily, Weekly, and Monthly Checklist

Lotus products typically run smoothly. To ensure that they continue to run smoothly, administrators should be equipped with the right set of tools for analyzing problems when they occur. This article provides considerations and recommendations on what administrators can do to improve efficiency or optimizing their system.

1.2.1. General recommendations

As a general recommendation, administrators should always be equipped with the latest knowledge in the Domino area, understand how to analyze system performance, and be able to troubleshoot server crashes.

Knowledge
One of the very basic elements is to know about capabilities of an environment, know about bugs or problems, and practical methods to improve system functionality. Optimizing an environment requires continuous improvement and fresh ideas. However, fresh ideas are not always written down in a book yet. So you should always keep track of the up-to-date information. We recommend few hints to help you or an administrator to do this: Sign up for My Notifications within the IBM Support portal. With My Notifications you can receive daily or weekly announcements through e-mail, custom Web pages and RSS feeds. These customizable communications can contain important news, new or updated support content, such as publications, hints and tips, technical notes, product flashes (alerts) and downloads and drivers. The tool allows you to customize and categorize the products you want to monitor and any of the available delivery methods to suit your support needs. http://www01.ibm.com/software/support/einfo.html

Since the early beginning Lotus Domino administrators have been collaborating with each other to exchange information, best practices, hints, etc. One of the key elements is to share knowledge with the community. This is why the biggest source of information is the community itself. Make use of this valuable information by: Signing up for the IBM Developerworks Lotus Community to read about Lotus software products and strategy from those who develop it. The Lotus Blogs are brought to you from IBMers who focus on Lotus software. http://www.ibm.com/developerworks/lotus/community/ Actively participate in the product discussion forums. Do not hesitate to ask your questions and feel free to answer questions from other people. ttp://www.ibm.com/developerworks/lotus/community/ Reading the blog posts of the Domino community. This following site provides a good start because it consolidates Blog posts from

19

various Lotus community blogs. You may want to use the RSS Feed Reader embedded in your Lotus Notes client in order to stay up to date. www.planetlotus.org Getting in touch with other Domino Administrators in your region by attending community meetings or round tables. Setting up your own Blog and post about your experience with Lotus products.Encourage the community to provide feedback by registering your Blog at: www.planetlotus.org

Performance
Troubleshooting performance related issues is not a simple task. Usually, these kind of problems do not show up over night. Most likely, it is a phenomenon which develops over time and this is also the reason why a performance analysis must take into account that the root cause may be located outside of the Lotus Domino ecosystem. Different tools and techniques apply to different platforms or areas that you want to focus. IBM already provided a detailed article about performance best practices. To get an idea of which tools you need for your platform, take a look at IBM Technote #7008849 - Notes Domino Best Practice: Performance http://www-01.ibm.com/support/docview.wss?uid=swg27008849

Server Crash Analysis

For troubleshooting server crashes, the following tools are valuable to have. Be familiar with their capabilities, functionality and usage. Analyzing server and client problems sometimes requires taking a closer look into NSD files created by Lotus while a crash occurred. By default, these files are stored within the Data directory of the client or server. Administrators can set up an Automatic Diagnostics database to collect these NSD files in a central database which will help to quickly get access to these files. For more information, see: IBM Technote 1085850 describes how to set up this Automatic Diagnostic Database http://www-01.ibm.com/support/docview.wss?uid=swg21085850 IBM Technote 1272213 provides the latest version of the utility. http://www-01.ibm.com/support/docview.wss?uid=swg21272213 IBM Support Assistant is a desktop software which helps to find answers or solve problems for various different IBM software products. It supports multiple IBM products, not just Domino alone. http://www-01.ibm.com/software/support/isa/ Lotus Notes Diagnostic is a Lotus Notes application which can analyze NSD files, search for known problems and collect suggestions for a possible resolution of the problem. http://www-01.ibm.com/support/docview.wss?uid=swg24019151

1.2.2. Actions
To help administrators in maintaining smooth Domino operations, optimizing server stability, performance, and security, we provide recommendations for daily/weekly/monthly tasks that are to be carried out. Actions outlined in the sections below represent general best practice, but do not include maintenance activities such as compact or fixup tasks which typically run scheduled. Focus is set on what administrators are required to do to keep the Domino environment healthy.

20

This is not an all-in-one perfect list as individual actions vary based on your environment. If any of the described actions is performed, but does not result as expected, administrators need to investigate further as there may be undiscovered problems. On the other hand, not every action needs escalations or emergency actions. So even if you are not investigating further, it is highly recommended to at least document an abnormality with date and time of when the event occurred followed by comments or clarifications. Actions can be automated or may even be part of an existing monitoring solution. In this case, the task shall be understood as task to verification of functionality. To get an understanding of efficiency, where valuable time is spent, we recommend documenting how much time administrators spend for which reoccurring action and activity. After a certain period of time, the reported hours should be reviewed to identify where (e.g.) small tasks take up a lot of the administrators time. In return, a conclusion would be to evaluate if such a task or action can be automated in some way to optimize your environment.

Daily Actions
The following list represents the daily actions an administrator should carry out for optimizing server runtime, performance and security: Check and resolve problems reported in Domino Domain Monitoring database (DDM). The type and number of issues shown are depending on your DDM configuration. Check if there were any servers crashing. If there is a problem, find the root cause by analyzing NSD files using tools referenced in the general recommendations section earlier. Check available free disk space. This daily check can be automated by creating an event in a properly configured Domino Domain Monitoring database (DDM). Verify daily backup jobs ran successfully. How to do this depends on the backup software used in the environment. All backup software vendors provide log files which administrators can check. Some can report success and failures through mail or other notification methods. Check if anti-virus software is running properly and patterns are up to date. Keep in mind to include the operating system anti-virus software in this check because it is as important as the anti-virus software on Domino servers. Check for replication problems by reviewing the replication log of your server(s). The amount of time spent for this action can be minimized by setting up DDM replication monitors which only reports failures to DDM. Monitor mail routing by checking mail routing queues. Check key statistic values on your servers and compare them with values from past days. For the number of peak transactions per minute, a fixed limit can not be provided because the capabilities depend on the underlying hardware. Over time, you will get an idea of when workloads become an issue and you will get to know how to balance workload.

For the daily check, focus on these statistics: Server.trans.PerMinute.peak Replica.cluster.WorkQueueDepth.avg Replica.cluster.WorkQueueDepth.Max

21

Note: This list does not claim to be complete. Depending on your environment, additional or fewer actions may apply e.g. caused by third party tools which either requires administrators to manage them separately or which help them to perform some of the tasks mentioned above automatically.

Weekly Actions
In addition to the daily actions, administrators should perform the following actions on a weekly basis, preferable in the beginning of a week to review the previous weekend: Monitor Administration Requests database (Admin4.nsf), Check the views Errors by Date and Individual approval required and take appropriate action. Review Domino server statistics and statistic trends. Especially search for workload peaks and document them. Take actions if you can see an unexplainable peak by reviewing log files further and explore options to balance workload within your environment. For more details, refer to Notes/Domino Best Practice Statistic and Events http://www-01.ibm.com/support/docview.wss?uid=swg27009310 Clean up your server and remove restored NSF files which are no longer required on your server. Typically, a restore is only required to be kept for a certain period of time, e.g. 7 days after this time, the restored file can be removed from disk.

Once again, additional actions may apply to your environment.

Monthly Actions
In addition to the weekly actions, administrators should perform the following actions on a monthly basis: Monitor Domino server memory consumption and take actions to provide sufficient memory for the Domino server to avoid performance problems. The total amount of memory required depends on more than just the Domino server alone. Third party tools on the operating system level that consume additional memory must also be taken into account. Check for new releases or patches affecting your environment. Sign up to mailing lists as described in General Recommendations above. If any changes are applied to the infrastructure, update the documentation to reflect the current environment. Run Lotus Domino Configuration tuner against all servers in the environment and review recommendations made by DCT. http://www-10.lotus.com/ldd/dominowiki.nsf/dx/domino-configuration-tuner Check Certlog.nsf for expiring certificates and ID files, If required initiate the recertification process as described here http://www01.ibm.com/support/docview.wss?uid=swg21326765 Check server disk fragmentation, because fragmented disks may result in slow server performance. Note: You can run defragmentation tools on a server, but Lotus Domino must be down to avoid damage to your data! For more details, see IBM Technote #1229817 http://www-01.ibm.com/support/docview.wss?uid=swg21229817

22

Review problems (if any) that have been reported in the previous month. Dig into problems and try to find and fix the root cause. For Domino Web servers, monitor web server requests. The following IBM technotes can be helpful when working with Domino HTTP logs:
#1382231 - How can I gather and sort data from HTTP access? http://www-01.ibm.com/support/docview.wss?uid=swg21382231 1161104 How to reduce the size of Domino Web Server Log (domlog.nsf) http://www-01.ibm.com/support/docview.wss?uid=swg21161104

As mentioned previously, this list does not claim to be complete. Depending on your infrastructure, the focus may vary and additional actions may apply.

Yearly Actions
In addition to the monthly actions, administrators should perform the following actions on quarterly or yearly basis: Perform tape backup restoration tests to ensure valid recovery data. Just checking backup logs and reacting on errors alone is not everything. There is nothing worse than a restore which cannot be done because (e.g.) backup media is broken or it is empty for whatever reason. Once in a while, such as on a quarterly or yearly basis, backups should be verified by restoring to a full server. The restore can be done on an isolated server to avoid effect on your production environment. Update your documentation. We recommend updating documentation whenever a change is applied to the infrastructure or at least with only a small delay. A quarterly or yearly update is recommended to verify essential parts are correct. The larger an environment is, the more frequent administrators should schedule this task. Perform a server health check of your environment. For more information on this topic, refer to the IBM OpenMic Call Health Check for Lotus Domino servers in IBM Technote #1432995 http://www-01.ibm.com/support/docview.wss?uid=swg21432995

1.2.3. Conclusion
In this article, we recommended a set of actions you should perform on daily, weekly, monthly, quarterly, and yearly basis. These are not complete lists of actions. Depending on your specific environment, come up with your own lists to perform on regularly basis, to ensure smooth Domino operation and optimized system performance.

23

1.3. Security Checklist

According to some estimates, 80-90% of threats come from current or ex-employees. This article describes activities which administrators should perform on a regular basis to keep their environments secure. Server security consists of different areas, such as server physical security (server room), operating system (OS) access by OS administrators, and services that run on this machine. The security level of a server is defined by the weakest node in the chain. The following checklist guides you on what needs to be checked regularly to keep your servers in good shape from a security perspective: Check for unneeded account by enabling the License Tracking in Configuration document. For small and medium environments, check every six months. For large environments, check quarterly. According to statistics, about 10% of the accounts in large environments are not used or not closed properly. For more information, see License Tracking Run ACL analysis on all databases. Check to make sure that there is no high Default access, such as Default=Manager or Designer. Also, if your server is accessible from the Internet make sure that all databases have an Anonymous entry. Note: Catalog.nsf may not have all databases. It lists existing entries. It does not show that Anonymous entry is missing. You can write a simple LotusScript agent to check the ACL or use the Domino Administrator client to perform a mass ACL update. Consider enabling LOG_USERSESSION=2 server notes.ini parameter as this will log the IP address of the PC from which the user accesses the Domino server. Ensure that the Domino web server is logging requests. You may filter some resources from being logged. For more information, see License Tracking Check that the Domino web server logs for attacks and scan your web site for brute force password attempts. If you have HTTP access to your server, consider deploying SSL for authorization so that passwords are not transmitted in plain text over the network. Check DDM database daily for new tickets. Consider enabling Security Probes in Lotus Domain Monitoring. For more information, see Security Probe Make sure that no person records have attached USER.ID files. You can run a scheduled script to check this. In the case that the attached USER.ID is found, the script should notify the administrator. Many organizations have default start password. If password checking is not enabled, old copies of USER.ID may be used for the wrong purposes. Check if deny access group is updated and is populated in server documents. Check for unneeded tasks running on server. For example LDAP running on public server with Anonymous access allowed. Check that only needed ports are open. Ask firewall administrators which ports are allowed from outside (Internet). Make sure only needed ports are open. If a port is no longer needed, close it on Domino and at the firewall level. Every opened port is a potential way to get inside your system. If you plan to do vulnerability scanning with third-party software, do this outside of working hours. Notify administrators who are responsible for this system when you plan to perform the test.

24

Check against information theft. There are third-party solutions that allow you to check if anyone is accessing unauthorized data. There are Data Leakage Prevention systems that can protect you against information theft. Ensure that the Domino server has Internet password locking feature enabled. If somebody does a brute force attack on a server, you can see this in the internet lockout database. For more information, see Securing an IBM Lotus Domino Web server:
Using the new Internet lockout feature

Consider implementing stronger and more complex passwords. Do this step by step. If a users password does not comply with policy, the user will be asked to change the password. If the user cancels the password change procedure, Lotus Notes will notify the user that the current password is not complying with policy and the client will close. Review the Security tab of your servers. Check who can enable Full Access Administration mode, who can sign scripts that has server operating system access, etc. Enable notification for enabling Full Access Administration to others, or a special mailbox. For more information, see Technote # 1197579

Keep in mind, security can impact system performance and user experience. The more secure the environment, the harder for users to access data to perform their work. You should find a balance between the needed security level required by the business holders and user comfort.

1.3.1. Additional References

For additional references and reading, see: Lotus Security Handbook http://www.redbooks.ibm.com/abstracts/sg247017.html Security Considerations in Notes and Domino 7: Making Great Security Easier to Implement http://www.redbooks.ibm.com/abstracts/sg247256.html

25

1.4. Domino Authentication Options

This article describes the Domino authentication options to help you determine which option best suits your environment.

1.4.1. Choosing the Domino Authentication Options

Traditionally, Lotus Notes use USER.ID file for authentication. Release 8.5.x added some new options for authenticating as well as a great feature that significantly reduces time for ID management ID VAULT. Authentication using SmartCards is another option. The following flowchart helps you understand which Domino authentication option is best suited for your environment.

Each authentication option listed and numbered in the flowchart is described in the following sections.

26

1.4.2. SmartCards (1)

SmartCards provide high security for Lotus Notes access. Regular Lotus Notes USER.ID files can be stored on SmartCards and be protected with a PIN number. Every time you log into Lotus Notes, you need to enter the PIN number, so the SmartCards can unlock the protected USER.ID. You can also protect SERVER.ID files with SmartCards as well. In that case, every time a server is started you need to enter the PIN number to unlock the SERVER.ID. If you decide to use SmartCards with Lotus Notes ID, you need to know how to configure Lotus Notes/Domino for SmartCards usage. Keep in mind that SmartCards users may require a separate policy. This is because periodic password changes and some other options are not applicable for SmartCards protected USER.ID files. For more information, see the following articles:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/enabling-smartcards-for-notes-login/ http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.notes85.help. doc/sec_smartcard_t.html

==Single-Sign On (2)== If you have many Domino Web servers, then Single-Sign-On (SSO) based on LTPAToken can be used. When a user connects and authenticates to the first server, the browser receives a secret ticket (token) that is stored in the browser. If you need to authenticate to a new server, the browser will pass this ticket to the server (limited to servers inside your Domain) and you will be authenticated without an additional password prompt. For example, you have 5 servers (3 mail servers in a cluster, and two applications servers, one of which is an internal application server and the other one is an external application server). In this case, if you do not use (SSO), you have to enter your password several times on each server. If SSO using a LTPAToken is used, you log in only once. Be cautious when you configure your server. If you use Internet Sites, then you use one LTPAToken definition. If do not use Internet Sites, then you may use another LTPAToken definition, which is stored in another document. It is recommended that you check the ($WebSSOConfigs) view for duplicated documents and that on all servers you use or not use Internet Sites documents. This may save you time while deploying SSO. You may also later use LTPA SSO during deployment of an Instant Messaging (Sametime Entry) or Sametime Meeting Server. For example, you can configure the Lotus Notes embedded Sametime client to log into Sametime using the SSO token. Thus, you can eliminate the need for Lotus Notes users to enter an HTTP password to log into Lotus Sametime. For more information, see the following articles: Can Sametime work with Internet Sites enabled? http://www-01.ibm.com/support/docview.wss?uid=swg21157740 Configuring Single Sign-On between Lotus Quickr and Lotus Sametime http://www-10.lotus.com/ldd/lqwiki.nsf/dx/configuring-single-sign-on-between-lotusquickr-and-lotus-sametime

1.4.3. Lotus Notes and HTTP Password Synchronization (3)

27

Lotus Notes provides an option for users to set the HTTP password same as the Lotus Notes password. The advantage of setting the same password is that the user has one less password to remember. If the user uses the same password for both systems (Lotus Notes and HTTP access), there is no need to spend time to set the HTTP password. It happens automatically if it is set in Security Settings and added to Domino policy. When needed, user can submit request to change the user's HTTP password. Lotus Notes and HTTP password synchronization can be the first step to make authentication easier for the users. This also helps reduce the number of help desk calls. Lotus Notes and HTTP synchronization is available in Release 6.x, 6.5, 7.x, 8.x, 8.5. For more information, refer to Security Setting help for enabling Lotus Notes and HTTP
password sync

Note, password synchronization does not work with Shared Login.

1.4.4. LDAP (4 and 5)

Lotus Domino users are registered in NAMES.NSF - the Domino directory. If you deploy one more system, you may need to maintain multiple user accounts for single user in your environment. User and password management can be a costly task in each system. You can use Lotus Domino server for authenticating other users with the help of LDAP protocol. Other systems may use Domino users for authentication with LDAP. The benefit of this approach is if something is changed, e.g. password is changed or a new user is added, all systems that use Domino for authentication automatically reflect the changes and and see changes. You do not need to register or remove the user from different systems. For more information, see IBM Infocenter, Planning LDAP Features

1.4.5. SPNEGO (6)

Lotus Domino 8.5.1 introduced a new way for user authentication: Users can authenticate in Domino web server with their Windows credentials. The benefit of the SPNEGO authentication is that users are not asked to enter passwords. According to research, password management in systems is rather expensive. Reducing the number of passwords users need to have helps decrease the number of help desk calls. If you currently use pre 8.5.x Release of Lotus Notes and plan to upgrade, consider migrating from Single-Sign-On service to 8.5 so you can use Shared Login because this solution does not synchronize password, thus it is easier to administer. SPNEGO means Simple and Protected GSSAPI Negotiation Mechanism. The clients browser and server can negotiate and the server can get information from Windows Active Directory regarding which user is trying to access the system. In this way the server will map the Windows user to the Domino users. If you consider adding SPNEGO authentication to products such as Microsoft Quickr, Sametime, consult IBM Consultants. Check requirements for Windows Active Directory and the Domino version before you decide to deploy this.

28

For more information, see: Deploying Windows single sign-on for Web clients (SPNEGO) in an existing Domino environment http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Deploying_SPNEGO Who supports SPNEGO authentication in a Lotus Quickr Domino 8.1 or 8.2 environment? http://www-01.ibm.com/support/docview.wss?uid=swg21422957

1.4.6. Lotus Notes and Operating System Single-Sign On (7)

Lotus Notes prior to 8.5.x offered an option that enabled users to not enter their Lotus Notes password. Lotus Notes Single-Sign-On service was synchronizing the password between Lotus Notes and the Windows operating system. If you currently use pre 8.5.x Release of Lotus Notes and plan to upgrade, consider migrating from Single-Sign-On service to Shared Login because this solution does not synchronize password, thus it is easier to administer. For more information, see
Does Notes support Single Logon with Novell?

==Shared Login (8)== Notes Shared Login (NSL) is a feature introduced in Release 8.5. It allows you to unlock your Lotus Notes ID with your Windows credentials. If the person is logged into the Windows operating system, a special Windows service is responsible for unlocking Lotus Notes USER.ID and the user can log into Lotus Notes without a password prompt. If the user forgot his/her password, you need to reset his/her Windows Domain password. Lotus Notes policy security settings in Release 8.5 has options on how to notify and enable Shared Login for users. If you have enabled Lotus Notes and HTTP password synchronization and you later enable Shared Login, users will now have to manage their HTTP password separately. If needed, for some users, you can enable Shared Login in the security preferences of the Lotus Notes client (grayed out by default).

29

Tips: Do not mix Shared Login with Single-Sign-On from Release 6.x-7.x. Single Sign-On from old versions of Lotus Notes was synchronizing the Lotus Notes and Windows passwords, Shared Login in 8.5 does not have to synchronize the passwords. A special Windows Service UNLOCKs the USER.ID of the user. If you are upgrading from an environment which used operating system Single-Sign-On, it is recommended to move to the Shared Login feature. Operating system Single-Sign-On is maintained for backward compatibility. For more information, see Using Notes shared log in to eliminate Notes password prompts
http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.notes85.help.doc/ sec_nsl_desc_t.html

1.4.7. Tivoli Directory Integrator (9)

Users who have purchased Lotus Domino 8.5.x are entitled to use Tivoli Directory Integrator to synchronize data with Domino. For more information, see TDI Entitlement If Tivoli Directory Integrator (TDI) is used to synchronize data with Domino, you do not need to buy additional licenses. If TDI is used to synchronize non-Domino data, you need to buy additional licenses. TDI allows integration of different systems and synchronization of data. For example, it enables you to integrate Lotus Domino directory and Windows Active directory as well as export or import users to and from textfiles or CSV files. TDI is a very powerful tool. You will need to take some time to learn its architecture before getting started with it.

1.4.8. Additional Resources

For more information, see:
Robust Data Synchronization with IBM Tivoli Directory Integrator Home page of TDI Users community materials. Lotus Domino integration page Tivoli Directory Integrator - integration with Lotus Domino You can find here online education and tutorial

30

1.5. Agents and the Domino Administrator

As a Domino Administrator, you may consider agents are for developers to write and handle. However; there are a number of things every Domino Administrator should know relating to agents. This article provides the list of these agent-related topics including how can an agent trigger, how to determine which agnets are schedule to run on the server, full text searches and agents, performance impact on agents, and how to write an agent.

1.5.1. Agent Triggers

Agents can be triggered in a number of ways and can also run under a number of server tasks including agent manager, server and http. When an agent is created, the developer can choose how that agent can run. An agent can be triggered on an event such as being called from a menu, a mail message being delivered or a document being created. It is important to understand which task an agent will be running under in order to be able to recognize the impact of an agent. If you see a sudden increase in CPU usage on your server, you may want to be able to easily identify or eliminate an agent as the potential cause. The list of triggers is as follows: On Schedule: This type of agent will run on the server and on schedule that was defined by the developer. The agent may run multiple times per day, weekly or monthly. Action menu selection: For this type of agent to run, a user must selects the agent from the Actions menu in the Lotus Notes client. When an agent runs in this manner, the code will run either on the client or in the server task depending on how the agent was coded. Domino administrator does not get a log or see a list of agents triggered in this way. Agent list selection: This type of agent trigger is reserved for agents that will be called by another agent or only ran from the designer client. These agents will run in the designer client or within the server task that called the agent. Before new mail arrives: This type of agent is triggered by the router task immediately prior to the message being written to the Inbox of a mail or application database. This type of agent will run under the router task. After new mail has arrived: This type of agent is triggered by a new mail message being written to the Inbox of a mail or application database. Once the message is delivered, the agent is scheduled to run in 2 minutes. This type of agent is controlled by the agent manager task; this ensures that if multiple mail messages are written into the database within a short period of time, the server will be able to process those new mail messages during one execution rather than running multiple times within seconds. After documents are created or modified: When a new document is created or a document is updated, this type of agent will run. The agent will be set to run as a scheduled agent under the agent manager task once a document has been created or modified, but not more than every 30 minutes.

31

1.5.2. Determining Which Agents Are Scheduled to Run on the Server

The Domino Administrator client provides an easy way to determine which agents are scheduled to run on the server. In figure 1 you can see that there are 5 agents scheduled to run. For illustration purposes, the agents are named with their trigger type. As you can see, only the agents that will be run by the agent manager task are listed. The next scheduled run time can be easily seen under the Next column. If you place your cursor over the schedule, the server will then display information on the schedule for that agent. "A Scheduled Agent on a Server" is set to run every 60 minutes. If you were to hover over "An After documents are created" or "An After New Mail Arrives" agent, you will see a listing of "Agent will not run" to indicate to you that the agent will be scheduled based on another trigger.

1.5.3. Agent Manager Settings That Affect Agent Execution

There are several settings that determine how many agents can run on your server at any time and how long they may run. You find these settings in the server document, Server Tasks... Agent Manager tab as shown below in figure 2. For example, by default, only 1 agent may run at a time during the day and 2 agents may run concurrently during the night. Depending on the number of agents that need to be processed in you environment, you can raise this number. Why would you need to increase the number of agents to run concurrently? In the section above, you saw how to determine which agents are scheduled to run. If you saw many of those agents scheduled to run in the past, and they are not agents triggered by a document update or mail message, then the agent manager task is not keeping up and you may want to consider increasing the Max concurrent agents parameter for daytime, nighttime or both. Another parameter to consider is the Max LotusScript/Java execution time. If you have agents that take over 10 minutes to process, you must change this parameter or the agent will be stopped by the agent manager task before it is finished. This setting is a safety precaution against infinite loops and to ensure one agent does not prevent all other agents from running.

32

1.5.4. Full Text Searches and Agents

An agent may often times need to search for information and then take action. Full text searches are resource-intensive operations when a full text index is not maintained. For more information, refer to 3.8 Managing Domino Indexing. To determine if any agent is performing full text searches on an application without a full text index, you can search your server logs for the following message: Warning: Agent is performing full text operations on database 'application/application1.nsf' which is not full text indexed. This is extremely inefficient. You can easily do this using the log analysis tool. To access the log analysis tool, go to the Domino Administration client, Server... Analysis tab. In the right panel select Tools Analyze Log... as shown in figure 3.

At this point, you should be presented with a Log Analysis window. From here, you can click on the Words tab to enter a string to search. In this case, the word inefficient is enough as shown in figure 4. You can then click OK to begin the search.

33

Once the search as finished, you can review the Log Analysis Results. Below in figure 5 is the desired result of the search No matching entries found.

1.5.5. Performance Impact of New Agents and Applications

It is a good idea to have a performance baseline before adding a new application or agent to your environment. This helps you to determine the impact of that application or agent in your environment. For more information on this, refer to 4.3 Establishing a Performance Baseline.

1.5.6. Troubleshooting Problems with the Agent Manager Task

For troubleshooting, see technote 7002851 - Troubleshooting Script for Notes Scheduled Agents (and Agent Manager).

1.5.7. Writing an Agent

After you become more comfortable with Domino administration, you may want to move into automating processes or writing an application. To get started with LotusScript, you can refer to the Domino Designer help which includes a Lotus Script tutorial or LotusScript Self-Help Training Modules. Additional resources can be found in the Lotus Notes and Domino Application development wiki.

34

1.6. Expanding the Domino Domain

At some point, you will most likely be in the position to expand your Domino Domain. Whether this is to add a Sametime or Traveler server into your domain or add an additional mail server; when you make your first transition from a single server to a multiple server environment, there are some things you need to keep your environment running smoothly. This article walks you through the process of expanding your Domain.

1.6.1. Registering and Securing a New Server

The first step when adding a new server to your Domino domain is to register the new server. Similar to registering a user, the registration process will create a server document and server ID file for the new server. The registration process will also automatically add the server name to the LocalDomainServers group if it exists. At this point, you can modify the server document to set the proper authorities for the new server. In the example shown in figure 1, you can see that the server administrators have been defined, the administrators have also been allowed the programmability restriction of Sign or run unrestricted methods and operations. In the server access section, database and replica creation has been restricted to administrators only and the Terminations group has been denied access. You are now ready to configure the new server using the configuration wizard appropriate for the platform you are using for this new server.

35

1.6.2. Replicating Critical Databases

In a multiple server environment, there are 2 databases that must be replicated among your servers names.nsf and admin4.nsf. This is critical in order for the administration process to function correctly. In small companies with 10 Domino servers or less, replication is typically a push/pull replication from the administration server as shown in figure 2. Replication is scheduled using a connection document. In the example you can see that connection is from Server1/Company A to Server2/Company A. On the Replication/Routing tab, the replication type is Pull Push which means changes made in those databases will flow to and from each server. The Files/Directory paths to replicate has been defined as names.nsf and admin4.nsf so only those databases will replicate. Finally, on the Schedule tab, the replication is configured to run every 20 minutes every day.

36

For more information the administration process including example replication topologies for larger Domino deployments and adminp basics, refer to the Notes/Domino Best Practices: Administration Process article. Be aware that to prevent replication conflicts and to ensure proper operation of the adminp task, the domino directory (names.nsf) must have the same server specified as the administration server for ALL domino servers in the domain. This can actually be expanded to any application. You will not have one server in your domain that is the administration for every database in your domain; however, each application replica should use the same administration server. While only names.nsf and admin4.nsf must be replicated, you can choose to replicate all databases in common between the servers. In this case, be aware that nearly all templates will replicate between the servers. For more information, refer to the Domino Template Replication and Design article on the Domino wiki. In most cases, this is desired; however if you need to co-exist in a multiple release environment and do not want the design to be updated to the latest release, see 7.5_Design_replication_and_mixed_releases__avoiding_design_ping-pong information. for more

In figure 2 above, you saw an example of probably the most common and simple replication configuration. There are a number ways you can improve on that schedule when working in enterprise size deployments of Domino: Set specific times in the schedule rather than an interval. Set a replication time limit. Avoid replicating databases during scheduled maintenance times. Consider time zones. Avoid pushing massive updates to the Domino Directory during prime business hours, especially important for large enterprises or large directories containing a hundred thousand users or more.

37

What is the difference between using the times and setting a time range and repeat interval? The difference is that with the repeat interval the replication will not always occur at the same time. That is because the repeat interval starts when the replication completes. So if the first replication starts 12:00 and takes 15 minutes, the next replication will occur at 12:35, not 12:20. When setting a specific start time, it can be easier to troubleshoot replication if you know exactly what time it will occur. Set a replication time limit to avoid one replication event from starting before the previous replication is completed. Keep in mind that if you want very frequent replication on some databases, it will be much simpler to use a repeat interval. To consider these keys points, imagine that Company A has recently expanded their enterprise globally and added a group of servers in Rio de Janeiro. The company now has two groups of servers Server Group US and Server Group Brazil. They have decided they want to replicate names.nsf, admin4.nsf and their mail files every 60 minutes. They run their backups at 12:00 a.m., the design task at 1 a.m. and updall task at 2 a.m. They run weekly compacts on all databases on Saturday from 4 a.m to 7 a.m. How can you create a replication topology that satisfies the business requirements and avoids the maintenance windows assuming these servers are located in the EST and BRST time zones?
EST Time 12:00 a.m. 3:00 a.m. Daily 4:00 a.m. 7:00 a.m. Saturday 9:00 p.m. 12:00 a.m. Daily 1:00 a.m. 4:00 a.m. Saturday BRST Time 3:00 a.m. 6:00 a.m. Daily 7:00 a.m. 10:00 a.m. Saturday 12:00 a.m. 3:00 a.m. Daily 4:00 a.m. 7:00 am. Saturday Comments Nightly maintenance window for US servers Compact for US servers Nightly maintenance window for Brazil servers Compact for Brazil servers

You can now easily see what times you should avoid scheduling large replication events depending on which server you initiate the replication. Assuming Server 1 is located in the EST time zone, you can create a connection document as shown in the following figure.

38

By creating the connection document in this way, you can accommodate for the maintenance windows and time zones for both servers for every day but Saturday. You can then create another document for Saturday. The following figure shows the example for Saturday from the server 2 located in Brazil to Server Group US.

1.6.3. Mail Routing

Typically you want all of the servers in your Domino domain to be able to exchange mail as needed. If all of your Domino servers are in the same Notes Named Network (NNN), then Domino automatically know how to route mail between your servers. If they are not in the same NNN, then you need to have two connection documents: server1 to server2 and server2 to server1. What is the Notes Named Network? The NNN is a way to logically group your servers based on your physical network. It is defined in the server document in the Notes Network parameter which is found on the Ports Notes Network Ports tab in the server document as shown in figure 6. By default, the first server in a Domino domain is created into a network called Network1 and additional servers default to TCPIP Network. Thus you should choose a value for your servers or set the network name to match if appropriate based on your network topology.

39

In order to send mail between two Domino servers, the server must know where to send the mail and must be able to make a connection to the remote location. If you have properly configured your NNN or connection documents. you have satisfied the first requirement. The second requirement requires a close working relationship with your network administrator. The Domino servers uses the Notes remote procedure call (NRPC) protocol over the port 1352 when sending mail or replicating. If you have a firewall between your Domino servers, you must ensure that port 1352 is open in both directions in order for mail to be successfully exchanged.

1.6.4. Monitoring and Managing Multiple Servers

Monitoring the log or Domino console manually is a rather straight forward task in a single server environment. As you add more servers to your environment, this task becomes increasingly more difficult. Domino provides multiple tools to simply this task so it is important to remember to use these tools and add your new server to this environment. Information on configuring monitoring and alerts can be found in 3.1 Monitoring. Here is a simple checklist for you: Add the new server to DDM Hierarchy. Configure Diagnostic Collection for the new server if you do not use a global configuration document for this. Configure statistics collection document for new the server.

1.6.5. Clustering
Another common reason for expanding the Domino domain is to be able to provide high availability to your users. To date, there is no hardware or software that will or can guarantee the system will be up from now until the end of time. Therefore your Domino server will be down at some point. Whether the server is down for routine maintenance, upgrades or a natural disaster; Domino application clustering is simple to configure and use to provide high availability for your users. Key clustering concepts for the new Domino Administrators are: Domino clustering is application level clustering. Failover is limited to the Lotus Notes client. A browser accessing iNotes will not automatically failover to a cluster server without additional infrastructure. Mail can be routed to another server using Domino clustering. For more information refer to 3.2 Mail Routing. Another applications such as the resource reservation database can be clustered using the natively available Domino clustering. A Traveler server cannot be clustered.

For detailed information on clustering refer to the article Understanding IBM Lotus Domino server clustering and 3.5 Server Clustering Options.

40

1.7. Design, Replication, and Mixed Releases: Avoiding Design Ping-Pong

Do you have multiple Domino servers in your environment? Do you upgrade your Domino servers over multiple days, weeks or months? If so, you should be aware of the options available to prevent a problem where the design task and replication are changing the design of your databases daily. This article explains to you how to avoid design pingpong scenario.

1.7.1. Description of the Problem

There are several ways you may notice that your servers are doing "design ping-pong". You notice in your log that each night when the design task runs, all of your databases are updated. Your users noticed that one day, their mail database was upgraded and the following day their mail file is not upgraded any more. You noticed the replica task is replicating more data than normal. What can cause this behavior? It is the combination of replication and the design task. By default, the design task runs at 1 a.m. on your server. By default, the line ServerTasksAt1=catalog, design is in the NOTES.INI of every server. When the design task runs, it compares the date and time of the design elements in the template with the design elements in the database. If there is a mismatch, then the design task replaces the existing design element with the one from the template. This is important so that when you upgrade your Domino server, your databases get the latest updates and fixes. Now imagine you have a Domino server you just upgraded to 8.5.2 that replicates with an 8.5.1 server. The following sequence of events may occur: Day 1: Domino 8.5.2 server upgrades the design of all mail files and system application databases with the latest 8.5.2 updates. During the next replication event between the servers, the 8.5.2 design replicates to any replica databases present on the 8.5.1 server. Day 2: The Domino 8.5.1 server updates the design of all databases with the copy of design elements it has stored in the copy of its templates. During the next replication event between the servers, the 8.5.1 replicates design to any replica databases present on the 8.5.2 server. Day 3: The cycle begins again; the Domino 8.5.2 server upgrades the design...

1.7.2. Preventing the Problem

There are a number of ways you can prevent design ping-pong from occurring: Replicating templates between servers Limiting the design task Removing templates from some servers Selective replication

41

Replicating Templates Between Servers

Most templates have the same replica ID between releases. If you allow templates to replicate in your environment, then the latest design will get replicated to any templates that are in common between the servers. For more information, refer to the Domino Template Replication and Design Domino wiki article. In this scenario, since all servers have the same design elements when the design task runs, it will recognize that the database has already been updated and thus not update the database again.

Limiting the Design Task

By default, all Domino servers run the design task at 1 a.m. If you need to run in a mixed release environment for a period of time and do not replicate templates, you can choose to modify your server configuration and not run the design task on one or more servers in your domain. The design task is started at 1 a.m. due to the ServerTasksAt1=catalog, design entry in the server NOTES.INI file. You can remove the design task by entering the following command at the Domino console: set config ServerTasksAt1=catalog. Once you have upgraded all of your servers you can easily re-enable the design task using the set config command again: set config ServerTasksAt1=catalog,design.

Removing Templates from Some Servers

The design task cannot update the design of databases if the template does not exist on the server.

In the scenario described above, the administrator can remove the templates from the 8.5.1 server. Later, as part of the server upgrade, the templates are copied back to the Domino server data directory. Be aware that removing the templates can cause a performance problem on the server if you have clients or servers configured to replicate these files. For details on the potential performance impact of removing the templates, review technotes 1299812 and 1426125.

42

Selective Replication
When replicating Domino databases and applications, you control which data can replicate. For example, if you only replicate a small subset of databases, you may want to temporarily disable replication of design elements. You can do this in the replication settings of a database as shown in figure 1.

1.7.3. Working with Mixed Clusters

The approaches discussed in this article work great for replica servers, but they may not work for clustered servers. That is because the cluster replicator always replicates design elements. The easiest way to prevent the design from going to the old server is to not run the design task on either server in the cluster. However; without the new design, you cannot use the new features. Also system databases such as names.nsf and admin4.nsf are backwards compatible so the new design will work fine on your older servers. Because of the complexities here, it is best to upgrade your clustered servers together or within a short period of time.

43

1.8. Routine Maintenance Best Practices

Having a weekly maintenance schedule is a crucial part of maintaining good health of your Domino server. Domino provides a simple way to schedule maintenance tasks using Program Documents in the Domino Directory. This article provides best practices for routine maintenance.

1.8.1. Fixup and Transaction Logging

One of the benefits of transaction logging is that if a server is restarted following a fault (server failure, power failure, hardware failure etc.), logged databases do not require a consistency check before users can access it. After you set up transaction logging, Fixup is not needed or used to bring databases back to a consistent state. Therefore for transaction logged servers, the Fixup task should not be scheduled to run regularly.

1.8.2. Regular Maintenance

Regular Maintenance of the Domino Directory is essential to maintain the health of a server and optimize performance. Administrators can use the following tips as a starting point to create a maintenance schedule tailored to their environment. IBM technote 1248830 provides a short summary.

Weekly Reboot of Windows Servers

Many customers reboot their Windows servers on a monthly schedule to prevent memory leaks.

Recover Database White space

When documents and attachments are deleted from a database, Domino tries to reuse the unused space, rather than immediately reduce the file size. Sometimes Domino cannot reuse the space or, because of fragmentation, cannot reuse the space effectively until you compact the database. The following command can be scheduled to run at the weekend using a Program Document: compact -b -s 10 (Archive Transaction Logging enabled) compact -B -s 10 (Circular or ArchivedTransaction Logging disabled)

With the above command, any database that has more than 10% white space will be compacted. The -b, or -B means that the server will perform an in-place compaction. The b (lowercase) switch should be used with transaction logs so that it does not assign new DBIIDs to the databases.

44

Note: New with 8.5.2, the compact -ODS switch performs a copy-style compact only if the current ODS is less than desired default ODS. This is useful if you need to perform a compact to implement new features (such as document data compression).

Compact Administration Requests database

The following command can be scheduled to run at the weekend using a Program Document: compact admin4.nsf -c -t

With the above command, a copy style compact is performed. This is useful because it can solve database corruption (a new DBIID will be assigned). The -t switch will disable transaction logging for that system database. Note: If a new DBIID is assigned to a database, a full backup of the database should be performed as soon as possible. This is because old transaction logs can no longer be restored to that database. For more information about the database DBIID, read IBM technote 7003909.

1.8.3. Database Corruption

There is no need to run fixup or updall as part of the weekly maintenance schedule. Fixup should be run only if you suspect corruption in a database and updall is run every night by default. You only need to run updall with switches if you suspect view corruption or other such problems within the views. If a database corruption is detected, run compact -c first. If the a view index has become corrupted, run updall -r -t . The database corruption troubleshooting guide has additional suggestions to minimize corruption within a database.

1.8.4. Hard Disk Fragmentation

SAN or RAID-5 based Domino data volumes can be affected by fragmentation. The mechanical hard-disk is the slowest component of a server. Maintaining file fragmentation can help maintain performance. The Hard Disk fragmentation Notes/Domino wiki article explains in-depth hard-disk fragmentation and its effect on Domino servers. Note: File fragmentation also affects workstations. Regular o/s de-fragmentation helps maintain disk performance on users' client machines too.

45

1.9. Mobile Access

For mobile access, there are different options available for a variety of device types. This article discusses various options and solutions available for mobile access. The following table summarizes the supported device types and the available features for each.
Device Type Blackberry iPhone/iPad/iPod Android Windows Mobile X X X Traveler BES Server X X X X POP/IMAP iNotes Ultra Lite

1.9.1. Traveler
Lotus Traveler is a Push-mail solution included in Lotus Notes licence. It is available at no additional charge if you have purchased Lotus Notes 8.x or Lotus Notes.8.5.x. It is recommended to run Traveler on a separate server. Traveler must installed on top of the Domino server and it is called an Advanced Product (a product/component that is installed on top of Domino). The version of Lotus Traveler you may install depends on Lotus Domino version you server is currently running. Review the software requirements to check which version is supported for this Traveler version you desire (for example, Lotus Traveler 8.5.2 can be installed only on Lotus Domino 8.5.2). Additional information on Traveler can be found in the following link: http://www-01.ibm.com/software/lotus/products/notes/traveler.html Lotus Traveler supports Windows Mobile, Nokia Symbian, Apple iPod/iPhone/iPad. Android phone support is added in fixpack 1 of 8.5.2. To deploy Lotus Traveler, you need: 1. Install Lotus Traveler on top of Lotus Domino. 2. Distribute Traveler clients on mobile phone (they can be downloaded from Traveler homepage, or distributed remotely).

Installing Traveler Server

Lotus Traveler server is supported on Windows 32, Windows 64 and Linux OS. Linux OS support was added in release 8.5.2. Lotus Notes Traveler system requirements can be found in the following link: http://www-01.ibm.com/support/docview.wss?rs=475&uid=swg27007909 Things to check before you start installation: The Domino Servlet Manager is enabled. To do this, open the Server Document of the Traveler server, Internet Protocols -> Domino WEB Engine tab. Set the Java Servlets to Domino servlet manager.

46

Verify that the Lotus Traveler server is included in LocalDomainServers group so it has Manager access to all mail files. If your Traveler server is located in a DMZ, enable Configuration Directory. When using a configuration only directory the Domino Directory will contain only configuration files as all other information will be requested from other servers that has a full copy of names.nsf. It is recommended to have a SSL certificate on a server. This improves the security of your server. Using a HTTPS connection ensures that no one can intercept (listen to the network traffic) when a password is sent from the phone to the server. With a standard HTTP connection the password is transmitted in plain text. Traveler versions prior to 8.5.2 use two ports, one for AutoSync and the second for data transmission. In 8.5.2, there is only communications on HTTP or HTTPS port.

For complete and updated information on things to check prior to installing Lotus Notes Traveler, review the information from the following link: http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Before_you_install_LNT8521 Installing a Traveler server on both Windows and Linux is quite easy. On Windows, the LotusTraveler setup file is launched, and you choose what you need to install, see options below. On Linux, the Traveler installation can be performed in two ways, via graphical mode or silent install. The Silent install is preferred method since it does not require graphical mode which maybe missing on some systems. Just configure a silent install answer file and run installation with silent install switch: Clients (installs only images for Nokia/Windows mobile/Apple) Server only (install only server part without clients Both (recommended option) - this will install both option, server and client

After you complete the Traveler installation wizard, pay attention to last screen of installation wizard. Check to make sure the installation is completed successfully. Lotus Traveler stores information about registered devices in lotustraveler.nsf database. Do not delete device documents from this application, Lotus Traveler can be managed only with console commands.

Troubleshooting
If you have issue with phone, try to understand if this is the problem for one user or for all users. This helps you understand, where to look for solution on device or on a server. The most popular problem is autosync is not working. In most cases restarting the phone solves this problem. If only one user has problems, try to login to Traveler server using a web browser with the user's credentials. You may see additional valuable information that may help you as you investigate the problem. You may add additional logging for this device/user with Lotus Traveler console commands. When the problem is solved, do not forget to disable logging, as this generates debug XML files on server, and they fill disk space.

47

In case the entire server is not functioning, search the console for JAVA errors and search for a solution from IBM support. Reinstalling Traveler is also an option. Reinstalling does not require much time and it will fix problems with missing components of Traveler. For more troubleshooting information, refer to the following references: Traveler 8.5.2.1 Server Troubleshooting: http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Server_troubleshooting_LNT8521 Traveler 8.5.2.1 Client/Device Troubleshooting http://www10.lotus.com/ldd/dominowiki.nsf/dx/Troubleshooting_known_limitations_and_restricti ons_LNT8521 Traveler Frequently Asked Questions (FAQ) and Common Issues http://www-01.ibm.com/support/docview.wss?uid=swg21450615

For Lotus Traveler console commands information, refer to the following link: http://www.lotus.com/ldd/dominowiki.nsf/dx/Console_commands_LNT8521

1.9.2. BlackBerry
If you are looking for information about Domino and you BES, consult this table as it shows the Domino Release/Server OS and BES: http://na.blackberry.com/eng/support/software/server_domino_ver_march_05_10.pdf

Domino Server Installation on a Windows Server Before Blackberry Enterprise Server Install
A Domino Server installation is recommended on the server where the Blackberry Enterprise Server will be installed. The installation makes the administration easier. The installation should follow a usual Domino installation as shown below: 1. Check if the server document for the new server has been created and exists in the Domino Directory. 2. Prepare for the Domino server build, placing the following files on the server in a temporary location: Server ID file , Domino server executable file, and Domino system files. 3. Install the Domino server code by running the self-extracting installation executable, answering the prompt questions. 4. Copy or move the files from the temporary location to the permanent place. 5. Execute a compact in the system files by opening a command prompt. Use ncompact D filename.nsf (for more info about compact). 6. Configure the Domino server by double clicking on the server program icon, answering the prompt questions. 7. Click on Domino server program icon to start the server for the first time. Server is ready for installation of BES.

48

Preparing the Environment

To have the environment prepared, perform these steps (which help to prevent problems in the future): Check if the Domino Server is a member of the LocalDomainServer group in the Domino Directory and if Adminp is running. Try opening admin4.nsf, Check if the server can access the other servers in the environment. Check if the setup application for the BlackBerry Enterprise Server can access the Domino environment variables in the NOTES.INI file. Verify that the remote DB2 or SQL Server is accessible. This step must be performed if you choose to have Blackberry Manager to use authentication in a SQL or DB2 Environment.

To verify, type CMD and then type telnet 1433. If you have connection, the cursor will be blinking. If that happens, close the Command Prompt because you can connect to server.

Post Blackberry Enterprise Server Installation

After you complete the BES installation and you want to be sure Domino is properly running, check the console for messages such as Message sent to handheld, OTAFM or OTAC. These messages indicate that messages are flowing from the server to the environment. Start the Blackberry Controller after the messages are being delivered that will prevent a silent crash causing nbes and domino to restart on the server again. If dispatcher is not starting up, it is necessary to check SQL or DB2 environment. In most of the cases, the connection between the BES and the server is causing the problem. There are some cases where a recycle of the SQL machine that holds the SQL/DB2 needs to be performed. NBES is related to the Domino task that starts the BES for Domino. To start it using load nbes command at Domino Console and to finish it use tell nbes quit command at Domino Console. Blackberry Controller can be stopped while server is initializing users. It must be started once all the users were initialized and mails are flowing properly. We can Stop/Start anytime it to prevent a silent crash (unexpected reboot).

Locating a Problematic Blackberry State Database

Many times if a Blackberry server crashes it is due to a problematic end user Blackberry state database. To find out where the problem is, you can do the following: From the console log file, locate problem thread ID [1114:000D-1384] Thread=[1114:000D-1384] Stack base=0x07110090, Stack size = 12068 bytes PANIC: LookupHandle: null handle

49

Do a search in NSD on "FATAL" to find the problem thread id to confirm. If there are multiple crashes, several NSDs can also be checked to ensure it is the same file causing the problem. FATAL THREAD with PARAMETER DATA 12/143 [ nserver: 1114: 1384]

Search NSD on "Open Databases" and look for file with problem thread ID. [Domino Install directory]\data\BES\state\123456789.nsf Version = 43.0 SizeLimit = 0, WarningThreshold = 0 ReplicaID = 0x87256f00:0x004b5bba bContQueue = NSFPool [ 000f6545] Offline = No DeleteInProgress = No FDGHandle = 0xf0240409, RefCnt = 1, Dirty = N DB Sem = (FRWSEM:0x0244) state=-1, waiters=0, refcnt=1, nlrdrs=0 Writer=[ nserver: 1114: 1384] SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1, refcnt=1 Writer=[ nserver: 1114: 1384], n=0, wcnt=0, Users=-1, Owner=[ nserver: 1114: 1384] By: [ nserver: 1114: 000d] DBH= 154, User=CN=COMPANY/NEWORG

From the information we get from the NSD, we see that the 123456789.nsf file caused the server to crash.

Fixing a problematic Blackberry state database

You can fix a problematic database by doing one of the following options (you must decide which one can be applied to your environment): Deleting or renaming the .NSF file with the problem and recreating it. Executing maintenance (fixup, compact or an updall) on the DB using the Domino console.

For more information, tips and tricks from RIM Website: http://na.blackberry.com/eng/support/blackberry101/tips/

1.9.3. Lotus iNotes Ultra-Lite

Lotus iNotes Ulta-Lite is for mobile devices and can also be with a traditional PC browser. For reference, go to http://www01.ibm.com/support/docview.wss?rs=899&uid=swg21315871 Lotus iNotes runs on the following client operating systems: Microsoft Windows XP Professional and Microsoft Windows Vista Business and Enterprise Editions using the following browsers: Internet Explorer or Mozilla Firefox

50

Novell SUSE Linux Enterprise Desktop 10 using the following browsers: Mozilla Firefox RedHat Enterprise Linux Desktop 5.2 using the following browsers: Mozilla Firefox Macintosh OS X 10.5 using the following browsers: Mozilla Firefox Safari 3.1.x Apple iPhone and iPod Touch firmware version 1.1.4 or later (for the ultra-light mode)

There is increase in mobility request to have mail configured or available on devices using a web browser. When enabling iNotes, confirm the following settings are enabled/configured: End user mail file: ACL to Anonymous=No Access ACL Advanced tab = Maximum Internet access to Editor. Server Address Book: Must be Domino 851 or greater Forms85.nsf File Must be the only Formsxx.nsf located on server iNotes directory. If not, it can cause an end user issue when opening in the browser. Be sure to remove any old Formsxx.nsf listed from old installations.

Remember that any change that you do with FormsXX (like removing old files) requires you to reload HTTP on server (restart task http). If you do not execute this command, it will give you a Read Only error when trying to open a mail file via Internet Browser. Directory Assistance File This should be on the lastest template available. SSL Configuration Install the SSL certificate by copying the necessary files to Domino Data directory (.kyr & .sth files). Open your the server document for your server and go to the Configuration -> Servers -> All Servers -> your server -> Ports -> Internet Ports tab and enter the SSL Key file name. You should also verify the SSL port is enabled. HTTP Config Define the hostname and/or IP addresses to be used by this server. To do this, open your Address book and go to Configuration -> Servers -> All Servers -> your server -> Internet Protocols -> HTTP. Set the Bind to host name field to enabled and set the Hostname(s) field to the proper host names and/or IP addresses for your server.

51

Set a Home URL for the server, for more information see 3.14 Setting Up a Redirection Application for Lotus iNotes Users. For example: https://yourservername/homepage.nsf?Open

Ensure the operating system is not running any other http task that could interfere with the Domino server. If it is, disable it before enabling Domino's http server. If you do not do that, Domino will not be able to use port 80/443.

Verify HTTP and SSL is working properly: Via the Domino Administrator client review the active tasks. Verify that the HTTP task is running. To verify a SSL connection is in use, access the server using this syntax -> http://yourserveraddress and confirm that you see the little lock on the bottom of your browser.

For more information, go to http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.help.do mino.admin85.doc/H_LOTUS_INOTES_ULTRALITE_MODE_STEPS.html

1.9.4. IMAP/POP
What is IMAP? You can view just the heading and the sender of the message and then decide whether to download the mail. You can also create and manipulate folders or mailboxes on the server, delete messages, or search for certain portions or an entire message. Domino IMAP users can: Replicate messages from the server that runs the IMAP service and store them on end user local replica. Access messages directly from the server (different from POP3 users who download the messages first).

What is POP3? Post Office Protocol 3 (POP3) is old and less sophisticated e-mail protocol. When you read your mail, all of it is immediately downloaded to your computer and it is no longer kept on the server. This can be a problem if you want to access your mailbox on the server or different computers due to possible hardware problems, virus, or preference .

52

The IMAP protocol is preferred over the POP3 protocol because if we enable both, we can downgrade the performance of the server running Domino. Other considerations: When using POP3, we will force the end user computer to connect to server, start a push/pull to bring messages to the end user computer. When using IMAP, the end user will synchronize the local replica with the server only for new messages to avoid server to be in a sync mode for a long time which consumes CPU of the server. Other item that must be considered is that the messages should be stored in MIME in order to prevent CPU issues.

For more information about performance, go to:

http://www-01.ibm.com/support/docview.wss?uid=swg21240413 http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.help.domino.a dmin85.doc/H_SETTING_UP_POP3_USERS_STEPS.html http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.help.domino.a dmin85.doc/H_ENABLING_A_MAIL_FILE_FOR_IMAP_ACCESS_STEPS.html

53

Part 2.

Managing Users and Clients

2.1. Optimizing Lotus Notes Client Administration Tips

As a Domino Administrator, you know that in order for your users to be productive and happy, the Lotus Notes client must be working properly. Keeping the client configured for optimal performance and up to date is important. There are a number of tools available to help you accomplish those goals. This article introduces you to these features and provides you the access to the best documentation on these topics.

2.1.1. General Client/User Management Tips

Understand the installation options available to you. Clients may be installed manually or deployed automatically. For more information refer to the Domino wiki article Notes Client Deployment or the IBM Redpaper Distributing Notes Clients Automatically. You can also customize the Lotus Notes installation, refer to Roadmap: Customizing your Notes client installation. Lotus Notes is extremely powerful software; however, it seems that users are often not aware of how to use many of the features or realize they exist. End-user training can resolve this issue. There are many available options using third-party vendors, free videos from sources such as YouTube or the IBM Multimedia Library. There are also a number of videos related to learning Lotus Notes available in the Domino wiki. Starting at version 8.0, the concept of a "basic" client was introduced. The Standard client is Eclipse based and has a heavier footprint from the previous Lotus Notes client version. For more information, refer to technote 1264877. Trying to understand what client type to use and which functions are available can be confusing. The Domino wiki provides a feature comparison for the Lotus Notes standard, Lotus Notes Basic, Lotus iNotes and Traveler clients to simplify this task for you. Use the Domino Smart Upgrade feature to simplify client updates. You can use smart upgrade to install maintenance releases, fix packs and/or other patches. For more information refer to the Smart Upgrade topic below. Know your clients. Do you know what client versions are still being used at your company? Not sure? You can use license tracking to help you answer this question. Ensure users only have editor access to their mail files. This is important to prevent users from changing their ACL permissions or granting other access as well as forbidding them to accidentally delete their own mail file. Refer to the InfoCenter for information regarding the access levels in the ACL. Note that users may still use the delegation functionality within the mail and calendaring features with editor access. Also, giving users editor access to the mail file on the server prevents the user from creating a full text index on their mail file. This is desired to ensure all full text indexes

54

in your environment are created in a way to have the least amount of impact on your server. Did you know that you can automate application roll outs? If you are still sending application links to users or manually creating local replicas for your remote users, there is an easier way! Using policies you can add bookmarks or have databases automatically replicated to clients. For more information, please refer to the 2.3 Policies article. Managing unread marks can be confusing. For example, do you know unread marks are stored in a table? Do you know unread marks are unique to a user and may not replicate? To understand how unread marks work and options available, refer to technote# 1140018. Use a local replica and a mobile directory catalog to allow your users to send mail and perform directory look-ups off-line. For information, refer to the information below on local replicas and the wiki article 3.4 Multiple Directories. Be aware of client issues affecting your users. Configure automatic diagnostic collection (ADC) to work for you. With ADC, if the Lotus Notes client crashes, the nsd and other relevant diagnostic data will be sent to the fault recovery database on your Domino server for your review. Crashes can be easily monitored and categorized. For more information refer to What is the Automatic Diagnostic Data Collection tool. The maintenance tasks (fixup, updall, and compact) can be run in the Lotus Notes client. The tasks are located in the client executable directory and the name is preceded with an "n", for example nfixup.exe, ncompact.exe and nupdall.exe. The switches are the same as on the server.

2.1.2. Recent Contacts

Starting with the Lotus Notes client version 8, the concept of Recent Contacts was introduced. With Recent Contacts, anyone who you send/exchange mail with is automatically added to your "recent contacts" list. In addition to Recent Contacts, the type-ahead feature was enhanced. When addressing a message, type-ahead provides results in order where the contact you interact more often with is at the top of the list. This can be a good and be a very time-saving feature, it can also have some negatives. For example, if someone you exchange mail with gets a new e-mail address, a new entry will be added to your recent contacts list and the old address will remain. This can be frustrating with the type ahead feature. You should delete the old address to avoid sending emails to the old address. For more information on the recent contacts feature including information on disabling this feature, refer to recent contacts FAQs.

2.1.3. Local Replicas

Using a local replica to access mail has become a more popular option over the years. It is easy to understand why when you consider a properly configured client with a local replica and directory catalog will provide users access to their mail from anywhere connected or disconnected. Also, users working from a local replica require less server memory and CPU to service compared with a user directly accessing their mail on the server. This means you may be able to add more users to a server or postpone an upgrade when moving from server based mail files to local mail files. When you hear this

55

statement do you immediately think "I don't want to go to each client?" or "My users are going to complain because they won't see new mail immediately" or "If the mail is replicated locally it is a security risk as anyone could read that data if the PC or laptop is stolen". Fear not, Domino 8.5 takes away these concerns. Using desktop policies, you can automatically deploy local replicas, configure a default replication schedule and ensure the data is encrypted on the client. Even better, Domino 8.5.2 implements an enhancement to local replicas with a new feature called managed mail replicas. With managed mail replicas: Users are automatically redirected to the server if the replica is unavailable for any reason. Not all mail has to be replicated to the client or mail can be retrieved from the server "on demand." Replication may be triggered immediately upon receipt of a message.

Simply put a managed mail replica eliminates the negatives of using a local replica. For more information regarding using a server based mail file versus a local replica or managed replica refer to the following article: http://www10.lotus.com/ldd/dominowiki.nsf/dx/IBM_Lotus_Notes_and_Domino_8.x_local_mail_repli cas_Advantages_considerations_and_best_practices For more information regarding Managed mail replicas refer to: Configuring managed replicas using the Desktop Settings document Managed Mail Replica: Use Mail free of network delays and server outages

2.1.4. Smart Upgrade

Smart Upgrade is the process to notify the user to update his/her client to a later release and perform the installation. As you learn about using Smart Upgrade, there is one key point that is often times overlooked. Policies, while recommended, are not required to use smart upgrade. Keep this in mind as you learn about and implement Smart Upgrade to prevent issues with users being upgraded before you are ready. The prerequisites for using Smart upgrade are: A Lotus Notes Client (version 6 or higher) already installed Connectivity to a Domino server Smart Upgrade database Smart Upgrade database defined in a Server configuration document.

For more information refer to:

Understanding Lotus Notes Smart Upgrade Notes Client Deployment Steps to create a Smart Update Database How to deploy non-versioned patches via SmartUpgrade

56

2.2. Managing a User's Inbox

A higher number of documents contained within the mail file Inbox creates extra work for Domino mail servers. The Update task must perform more processing to maintain the sort order in the View Index when new mail is added, or old ones deleted. When one considers the cumulative effect of many hundreds or thousands of mail files in large organizations, Inbox management is a key factor in maintaining performance and stability of the environment. This article provides Administrators with some tips on how to keep control of an ever growing estate of mail files.

2.2.1. Using Inbox maintenance to manage mail file size

The Inbox maintenance feature is designed to improve server performance by reducing the size of users' Inboxes in mail files. This has the effect of reducing the View Index size for $Inbox and also reduces the amount of work the Update task must perform. When you enable the Inbox maintenance feature, the Administration Process periodically runs the Inbox Maintenance agent on each users' home mail server. The Inbox Maintenance agent resides in the mail template, MAIL8x.NTF. The agent removes documents from the Inbox based on settings you define in the Server document or the mail policy settings document. These documents are still stored in the mail file and can be accessed via the All Document view. There are two methods to enable the Inbox maintenance feature; both in the Domino Directory (names.nsf). The first, is in the Server Document; Server Tasks -> Administration Process tab. The second is in the Mail Policy document; Mail -> Basics tab. The settings that are specified in the Server document override the Inbox Maintenance settings in the mail policy settings document. Administrators should therefore choose one method only, based on the following strategy: Goal: Implement Inbox maintenance globally on the mail server (Use Server document). Goal: Implement Inbox maintenance selectively to a subset of users or groups (Use Policy Document).

For further information, read Using Inbox Maintenance to manage mail file size, in the IBM Lotus Domino Administrator Information Center.
Frequently Asked Questions

57

2.2.2. Quota Enforcement Options

For a beginner's guide, read Understanding Quotas for IBM Lotus Domino Mail Databases. Administrators have a number of options available to enforce (or not) mail file quotas. The options available differ in strength of enforcement and help Administrators to implement organization policy. Administrators can set two thresholds; warning level and maximum level, and whether the maximum level is enforced. Notifications can be sent to user's Inbox to give early warning that they are close to exceeding their mail file size quota. Administrators have three options of quota enforcement. These options are outlined in figure 1 and instruct the mail router how to treat mail addressed to an over-quota user.
Over Quota Enforcement Delivery Anyway Non Deliver to Originator Hold mail and retry Description Use this if quotas will not be enforced by the mail router. This is the default setting. New messages are not delivered and returned to the sender. They are informed that the message could not be delivered because the recipient's mail file was full. New messages are held in the MAIL.BOX. Further options must be set to refine behavior. Administrators need to be aware that using this setting can lead to very large MAIL.BOX sizes if large messages are held for long periods.

For further information, read Setting Quota controls for the router, in the IBM Lotus Domino Administrator Information Center.

2.2.3. Quotas with DAOS Enabled

When calculating the size of a mail file to determine whether it conforms to configured mail quota or warning threshold limits, the server treats attachments stored using the DAOS as though each user owned the entirety of the attachment file. Therefore the full size of the attachment in every message delivered to a mail file counts towards the mail file quota. Likewise, when a user deletes a message, the full size of the message is removed from the mail file quota. The actual file size of the mail database that uses attachment consolidation therefore does not necessarily reflect its logical size. For example, a user's mail file might exceed its quota limit of 100MB even though the physical size of the file is only 65MB. IBM Technote #1405456 discusses how DAOS functions with Mail Quotas. Administrators familiar with earlier versions of Domino may have viewed the Space Used column in the Files tab of the Domino Administrator. The Space Used column gave an indication of the actual amount of data stored within the NSF file, as opposed to blank space. Note that Domino 8.5 saw the last of the Space Used Column. For further information, read the DAOS and The Space Used Column wiki article.

58

2.2.4. Enforcing Quotas on Local Replicas

If a quota is set on a server-based replica, by default the quota is not enforced in a local replica and neither are quota warning notifications displayed. Administrators can configure their environments so a quota warning is displayed when users attempt to create a new mail message or calendar invitation on a local replica. IBM Technote #1247798 provides a comprehensive guide to the procedure for implementing this feature.

2.3. Policies
Have you looked into Domino policies yet? If not, then you have not yet seen how powerful policies can be. Since there are many reasons to use a policy and a number of policy types, you may wonder how you should get started with policies.

2.3.1. Introduction to Policies

The purpose of a policy document is to assign settings to users. There are a number of available setting types in Domino 8.5.x. including Registration, Setup, Desktop, Security and more. For a list and definition of each setting type refer to the Domino Infocenter Policies topic. There are two types of policies: Explicit Organizational

An organizational policy is a policy that will be applied to the entire organization or organizational unit. For example, if you work for Fictional Software Company A, your user name may be User One/IT/Company A. In this example you could create an organizational policies for */IT/Company A or */Company A.

59

An explicit policy is a policy that is explicitly applied to a user. This could mean that the policy is defined in the users person document or is assigned to the user within the policy document itself. When assigning a policy to a user or group within the policy document itself, the policy is then considered to be a dynamic policy. For more information on dynamic policies, you can refer to the Domino wiki article How Dynamic Policies can reduce your administration workload. For more information on assigning policies, refer to the Domino Administrator help topic Planning and assigning policies.

2.3.2. Using Policies to Standardize Secure and Simplify Your Environment

Policies can be used to simply resolve a number of common problems.
Problem You Need to Solve Enforce corporate security standards such as Changing passwords every x days Password minimum length Security risk from having Notes.id files stored on a network drive with their default password Enable favorite calendaring features by default for users for example: Displaying new and unprocessed calendar and to documents in the calendar Automatically showing cancelled meetings as cancelled in the calendar Automatically checking for conflicts when adding appointments to the calendar Configure server side archiving for all users Ensure all administrators use consistent settings when registering new users The Policy Answer Organizational security policy

Organization security policy with an assigned ID Vault for secure ID file storage Organizational mail policy

Organizational archiving policy Organizational registration policy

60

Get the new sales application replicated locally to your sales team. Configure default settings for all new Lotus Notes Install Define default settings for mobile users

A dynamic desktop policy assigned to the sales team. Organizational setup policy A dynamic Traveler policy assigned to users as they purchase a new mobile device supported by Traveler Organizational desktop policy Explicit desktop policy

Automatically upgrade mail file design when a client is upgraded. Set a NOTES.INI parameter at the client for a remote user

The list could go on and on. Policies can be a Domino administrators best friend or biggest headache. Here are some definitions, hints and tips that will help you succeed with your policy roll out: Test first! Before rolling out a new organizational policy, create an explicit policy and assign it to a small set of users. If it goes well, then create your organizational or organizational unit policies for the rest of the employees. Create policies based on your organizational structure. A policy implementation of managers, employees and contractors will be too vague to work in most cases. Assign policies via groups or individually within the policy document, policy assignment tab rather than explicitly in the person document. Assigning policies in the person document is time consuming and difficult to manage. Create exception policies where needed for executives. Be sure to include a detailed description to help you identify the policy. It is also recommended to hide the group document from the general user population using a readers list. For more information on hiding group documents, refer to Limiting access to group documents section of the Mass mailings article. When learning policies for the first time, the language used can be confusing. For example, there is no option to create an organizational security policy. What does that really mean? A policy of type organizational where the policy name matches your organization, for example */organization, which has a security settings document applied as shown in figure 3.

61

A desktop policy applies to all location documents. If you only want to apply a specific setting to a single location you should still make this change manually. There is no undo with policies. Once a policy has been applied, you can change the policy to push out a new value, but there is no way to choose put it back to the original value. When using policies to push replica databases or bookmarks to clients, never delete the database on the server without removing the database entry from the policy. As a safety net, always create database redirects when you delete a database or template to prevent a problem with a policy pointing to a non-existent database as shown in Figure 4.

62

Know your environment. Policies will behave differently depending on the client version being used. This would be expected as many new functions have become available with the later client versions. Be sure your clients can support the function you want to roll out. A policy may be implemented by a number of processes. Archive policies are used when running compact a or compact A. Mail policy values are rolled out by the administration process (adminp) every 12 hours or immediately with the tell adminp process mailpolicy command. A setup policy is read and used during initial client setup. The remaining policy setting types are implemented by the Lotus Notes clients dynamic client configuration (dcc). You can be certain dcc is running if you see the entry Notes configuration settings have been refreshed in the status bar of the client. Policies are stored within the users Contacts database (names.nsf). This database should be using the latest design to prevent problems when using policies. When using a desktop policy to control settings within iNotes you must also have a mail policy. This is due to the fact that it is the job of the adminp task to update the iNotes profile document from the desktop policy and adminp will only runs when that process when a mail policy is present. When using a How to apply value of Set initial value in a settings document, the value you are setting will be pushed to the client when the policy is first saved and any time the policy is modified or saved. Policy signatures are important. Policies will only be applied and used when signed by an administrator with proper authorities and a valid key. Thus, it is recommended that you sign your policies with a generic administration account or resign policies before the administration id can expire or when they administrator leaves the company. To see who is the signer of a policy and settings documents using the Policies and Settings views in the Domino Administrator client as seen in Figure 5. By placing the policy or settings document in edit mode and saving it the signature will be updated with your signature.

63

For a detailed description and an example of the different policy settings, refer to the policies self training modules or review the Domino Policy Blog.

64

Part 3.

Effective Server Administration

3.1. Monitoring
Monitoring a Domino environment means a repeating systematic collection and supervision of the environment and its process and individual tasks within. The main functionality of monitoring is to identify if certain parameters of a system or environment exceed their defined boundaries and react in a defined way, for example, by alerting. Due to the highly configurable nature of Domino and a variety of tasks it can perform, the aim of this article is to define a monitoring strategy that covers the most common components of a Domino environment. This monitoring strategy should be treated as a base line that requires further customization to accommodate your specific Domino environment.

3.1.1. Monitoring Options

We do not recommend a single solution or a single tool for all customers. Certainly large implementations of Domino or those with high usage have different needs than smaller Domino environments. IBM offers the following monitoring options: Server Monitor This is a basic monitoring option built into the Lotus Domino Administrator client. It is great for small environments or as an additional monitoring for one of the following solutions. Domino Domain Monitoring (DDM) This is a server feature built into Lotus Domino and enabled by default. DDM is great for detecting, understanding and acting on run time issues. DDM probes log events. The administrators need to check the events that have been logged. Event generators and event handlers together with statistics collection can be used to monitor a Domino environment. For information on how they are handled, see: http://www.ibm.com/developerworks/lotus/tutorials/lsdom6stats/index.html IBM Tivoli ITCAM for Applications This is part of an enterprise class monitoring solution, which is extremely scalable and capable to monitor much more than Lotus Domino alone. Its functionality can be deployed agent-based or agentless. It leverages best-practice models that focus on performance monitoring of key Lotus Domino components including servers, mail routing, replication, calendar, database, and clusters. Deeper Domino administrative capabilities are available with IntelliWatch http://www-01.ibm.com/software/tivoli/products/composite-application-mgrapplications/index.html IBM Tivoli Intelliwatch Pinnacle for Distributed Systems This is an automated problem detection and correction, system-wide product configuration options, custom reporting capabilities, fault recovery and more. For more information, see: http://www-01.ibm.com/software/tivoli/products/intelliwatch/

65

Note, there are also 3rd party solutions on the market you can use which are not listed here.

3.1.2. What should be monitored?

A common mistake is to limit monitoring to the application Lotus Domino itself. A number of other components should also be monitored to ensure their work and to avoid spending time on analyzing issues which are caused by a completely different area. This list provides a brief overview of which elements should be monitored: Network (LAN / WAN) Platform (Hardware, Operating system) Storage & Backup environment Application with its components

Within this article, we focus on the last part Application which in our case is Lotus Domino.

3.1.3. Monitoring Profiles for Domino

For ease of reference, different monitoring profiles should be defined within an environment. By grouping monitors in this way, it is possible to create profiles which are applicable to specific server configurations or functions in the Domino environment. The following server roles shall be understood as an example: Generic Domino Servers Applied for all domino servers. Mail Servers Servers hosting end user mailboxes. Web Servers Servers hosting Web sites (HTTP services). Cluster Servers Servers providing cluster service. Special Application Servers - Servers providing additional services.

Additional profiles can be defined based on your environment needs. Make sure to document additional server profiles and include a definition when to use which monitoring profile.

Action
Monitoring by itself is useless unless you take actions in case of an event or problem. These actions can be defined for each response level and also for each event in detail. Which action is the most important or convenient depends on your corporate environment. In small implementations of Lotus Domino, it might be enough to mail the administrator to take action some time later. In large environments, there might need to have a solution which supports 24x7 monitoring and alerting. In this scenario, it is often required to integrate Lotus Domino monitoring results into an enterprise-wide monitoring system or help desk system.

66

Actions depend to different factors like the size of the environment and the availability of systems for alerting or ticket management. Lotus Domino supports a number of notification actions which can be used further on to build custom integrations to 3rd party systems, for example, to automatically open a help desk ticket in your custom help desk application. Figure 1 shows event handler methods.

If a Tivoli Enterprise Console is already available, then forwarding events to this console is recommended. This is most likely the case for medium and large Lotus Domino installations.

Hint: Cell Phone Alert

In order to receive cell phone alerts, there is a special cell phone configuration which some providers support. Providers can be requested to forward email messages to a phone as text messages (SMS). This allows notifying administrators via SMS when a critical event occurs. If properly configured, a cell phone can receive email messages in SMS form. The email must be addressed to a specific email address and domain name which is defined by your provider. In most cases, it is your cell phone number followed by the providers gateway domain name. for example, 0123456789@.. Consult your cell phone carrier for details about how to enable this configuration. Be aware that this may add extra cost to your phone bill. To notify multiple people about the same event, create a group in your Domino directory (e.g. AdminAlert-HTTPServers) which contains a list of these special email addresses.

Mapping Response Level to Severity Level

For further understanding the configuration details later on in this article, we map the response level to severity levels which are widely used in help desk systems.
Response Level N/A Severity Level Sev 1 Description Highest level of attention required, serious impact to business expected.

67

Fatal Critical

Sev 2 Sev 3

High attention required, system is functioning but may lead to service disruption if no action is taken Requires attention of a Domino administrator, if not handled in a timely manner this may lead to further problems Should be brought to administrators attention, but doesnt require immediate attention Previous severity now stabilized

Warning Reset

Sev 4 N/A

Profile: Generic
A default monitoring profile should be applied to every Domino server, regardless of it is designated role. In general, where a monitor is considered important and critical enough that it will impact server function, the monitor interval can be set to 5 or 10 minutes. Otherwise an interval of hourly is predominant. The Generic Domino Server Profile should include the following monitors:
Monitor Name Mail Probe Response Level Warning (high) Trigger On time out Details and Interval Mail Delivery Monitoring probe Send Interval: 10 minutes Time out threshold: 10 minutes Server availability Fatal (nonclustered servers) Critical (clustered servers) Reset Task adminp Fatal Reset Becomes Available Task event Fatal Reset Becomes Available Alternative : Every 10 minutes Becomes Unavailable Alternative : Hourly Task Status Monitor Becomes Unavailable Task Status Monitor is unavailable is available TCP Event Monitor Every 5 min

68

Task amgr

Fatal Reset

Becomes Unavailable Becomes Available

Task Status Monitor Alternative : Every 10 minutes Task Status Monitor Alternative : Every 10 minutes Task Status Monitor Alternative : Every 10 minutes Task Status Monitor Alternative : Every 10 minutes Task Status Monitor Alternative : Every 10 minutes Task Status Monitor Alternative : Every 10 minutes

Task stats

Failure Reset

Becomes Unavailable Becomes Available

Task update

Fatal Reset

Becomes Unavailable Becomes Available

Task router

Fatal Reset

Becomes Unavailable Becomes Available

Task replica

Fatal Reset

Becomes Unavailable Becomes Available

Task DAOSMgr

Fatal

Becomes Unavailable

Task MTC

Fatal Reset

Becomes Unavailable Becomes Available

Task Status Monitor Alternative : Every 10 minutes Statistic Event Generator Alternative : Hourly Statistic Event Generator Alternative :

Domino Statistic Replica.Failed

Warning Failure

Increase of 10 Increase of 10

Domino Statistic Server.Sessions.Dropp ed

Warning Failure

Increase of 50 Increase of

69

100 Domino Statistic Server.Users Warning Failure Increases above Y Increases above X

Hourly Statistic Event Generator (X and Y depend on size of server) Alternative : Hourly

Domino Statistic Agent.Hourly.Unsucces sfulRuns

Warning

Increase Above 0

Statistic Event Generator Alternative: Hourly For details, see IBM Technote 1232603

Domino Statistic Agent.Daily.Unsuccessf ulRuns

Warning

Increase Above 0

Statistic Event Generator Alternative: Daily For details, see IBM Technote 1232603

ACL Change (names.nsf)

Warning (high)

On ACL change.

Database Event Generator Monitor ACL Change File name: names.nsf Servers: all Domino servers in scope

ACL Change (Admin4.nsf)

Warning (high)

On ACL change.

Database Event Generator Monitor ACL Change File name: admin4.nsf Servers: all Domino servers in scope

70

Profile: Mail Server

The following monitors will be applied to all Domino servers designated as mail servers: Generic Monitoring Profile.

In addition, the following monitors are recommended:

Monitor Name Task calconn Response Level Fatal Reset Becomes Available Task sched Fatal Reset Becomes Available Domino Statistic Mail.Dead Warning Failure Reset Increases above Y Decreases below X Domino Statistic Mail.Waiting Warning Fatal Increases above Y Alternative : Every 10 minutes X and Y depend on size of environment Domino Statistic Mail.Trans..Failures Warning Fatal Reset Increases above 500 Decreases below X Alternative : Hourly X and Y depend on size of environment Increases above 100 Statistic Event Generator Increases above X Alternative : Hourly X and Y depend on size of environment Statistic Event Generator Increases above X Alternative : Every 10 minutes Statistic Event Generator Becomes Unavailable Alternative : Every 10 minutes Task Status Monitor Trigger Becomes Unavailable Details & Interval Task Status Monitor

Profile: Web Server

The following monitors will be applied to all Domino servers designated as Web servers: Generic Monitoring Profile. Domino Mail Server Monitors Profile (if needed).

71

In addition, the following monitor profile should be added:

Monitor Name Task http Response Level Fatal Reset Trigger Becomes Unavaila ble Becomes Available Domino Statistic HTTP.PeakConnecti ons Warning Failure Increases above Y Increases above X Statistic Event Generator (X and Y depend on size of server) Alternative: Every 60 minutes For details see IBM Technote 1232603 Domino Statistic Domino.Threads.Act ive.Peak Warning Failure Increases above Y Increases above X Statistic Event Generator (X and Y depend on size of server) Alternative: Every 60 minutes For details see IBM Technote 1232603 Details & Interval Task Status Monitor Alternative: Every 10 minutes

Profile: Domino Cluster

Any Domino Servers configured as a Domino cluster should have the following Domino Cluster Server monitoring profile applied in addition to the basic profiles: Generic Monitoring Profile. Domino Mail Server Monitors Profile (if needed)

In addition, the following monitor profile should be added:

Monitor Name Respo nse Level Fatal Reset Trigg er Details & Interva l Task Status Monitor Alternat ive: Every 10 minutes

Task clrepl

Beco mes Unav ailabl e Beco mes Availa ble

72

Task cldbdir

Fatal Reset

Beco mes Unav ailabl e Beco mes Availa ble

Task Status Monitor Alternat ive: Every 10 minutes Statistic Event Genera tor Alternat ive: Every 60 minutes For details see IBM Techno te 123260 3

Domino Statistic Replica.Cluster.Failed

Critical 1 Reset <1

Domino Statistic Server.Cluster.OpenRedirects.LoadBalanceB yPath.Unsuccessful

Warnin g Critical Fatal Reset

1 5 10 <1

Statistic Event Genera tor Alternat ive: Every 60 minutes For details see IBM Techno te 123260 3

Domino Statistic Server.Cluster.OpenRedirects.LoadBalance. Unsuccessful

Warnin g Critical Fatal Reset

1 5 10 <1

Statistic Event Genera tor (X and Y depend on size of

73

server) Alternat ive: Every 60 minutes For details see IBM Techno te 123260 3 Domino Statistic Server.Cluster.OpenRedirects.FailoverByPat h.Unsuccessful Warnin g Critical Fatal Reset 1 5 10 <1 Statistic Event Genera tor Alternat ive: Every 60 minutes For details see IBM Techno te 123260 3 Domino Statistic Server.Cluster.OpenRedirects.Failover.Unsu ccessful Warnin g Critical Fatal Reset 1 5 10 <1 Statistic Event Genera tor Alternat ive: Every 60 minutes For details see IBM Techno te 123260 3 Replica.Cluster.WorkQueueDepth.Avg Warnin g 500 Statistic Event

74

Genera tor Alternat ive: Every 60 minutes For details see IBM Techno te 123260 3

Example of documenting Monitoring Profiles

Server Name Generic Mail Hub Web Cluster Custom App.

ServerA/ITSO ServerB/ITSO SametimeA/ITSO

Yes Yes Yes

Yes

Yes Yes

3.1.4. Domino Event Monitoring

Although the profiles above can be implemented in different monitoring systems, it is possible to monitor Lotus Domino event from the Domino Monitoring Configuration (events4.nsf) database. To prevent too much information from being shown, administrators should monitor all Domino events defined as Fatal, Failure or Warning (high), as defined in the table below. Each event type sub classifies each message with a severity level. These severity levels are defined, in the Lotus Domino server, as:
Severity Level Fatal Failure Warning (high) Warning (low) Normal All Response Level Fatal Critical Warning N/A N/A N/A Meaning Imminent system crash Severe failure that does not cause a system crash. Loss of function requiring intervention. Performance degradation. Status messages. All of the above messages.

75

Severities

For best results you may wish to change the following default settings: Remember to document changed defaults, so you can reapply them after an upgrade of Lotus Domino to a higher version.
Value Text Old event severit y Warning (Low) New event severit y Normal Reason

0x02CC

Database is being Compacted; Compact must finish before use. Recovery Manager: Assigning new DBIID for (need new backup for media recovery). Recovery Manager: Restart Recovery complete. (/ databases needed full/partial recovery) Database is currently being indexed by another process Full Text Error (FTG): Exceeded max configured index size while indexing document NT in database index Recipient user name not unique. Several matches

Compact task runs against (e.g.) a system database which is in use.

0x0EA2

Warning (Low)

Normal

Backup software is requested to take a new full backup of this application.

0x0EA8

Warning (Low)

Normal

This only indicates that the server has been restarted completely.

0x0F13

Warning (Low)

Normal

This is only informational.

0x0F3B

Warning (High)

Normal

We do not want to FT large attachments - so this error is normal.

0x1104

Failure

Normal

We cannot do anything about, because the recipient is chosen by the sender, and

76

found in Domino Directory. 0x1105 User not listed in Domino Directory Error registering mail rule for database Warning (Low) Normal

when sent offline or to email address not validated by Client. Failure occurs every time a user writes wrong name in SendTo field. Rules is controlled by users - we can not fix this every time - and it has no consequence for the server. This is only informational. This is only informational. Many users may try to access Admin server or servers with limited access, e.g. because they have had access before. Normal (ex. Users try to see calendar details and does not have any public access or higher). Information about an user has been redirected to cluster-server. Information about an user has been redirected to cluster-server. Information that a database was not able to failover to cluster-server Normal under compact

0x1149

Warning (High)

Normal

0x131B 0x131C 0x1321

Database created by Database deleted by ATTEMPT TO ACCESS SERVER by was denied

Warning (Low) Warning (Low) Warning (Low)

Normal Normal Normal

0x1323

ATTEMPT TO ACCESS DATABASE by was denied Failing over from for replica id , directing open to Failing over from , directing open to Unable to redirect failover from Operation cannot be performed at the current time database compaction

Warning (High)

Normal

0x1357

Warning (Low)

Normal

0x135C

Warning (Low)

Normal

0x135E

Warning (Low)

Normal

0x138C

Warning (Low)

Normal

77

in progress. 0x1519 A DDM report document (NoteID 0x) could not be opened. Replicator was unable to initialize (from ): Your account is locked out; see your system administrator to reset it documents ( bytes) indexed in LDAP Server: Warning: Invalid credentials specified on Bind request, DN is Database was marked for delete and has been deleted Admin Process: does not appear in the ACLs of any databases designating as their Administratio n Server does not appear in the Readers or Authors fields of any databases designating as their Administratio n Server Warning (High) Normal If a DDM report has been manually deleted, and then another instance of the error is logged, then this error is coming. Failure occurs every time a replica stub is made. Many users forget to change their password in time; we consider this to be fixed by the user himself. Indexing is normal.

0x1614

Failure

Normal

0x19FC

Warning (Low)

Normal

0x330A

Warning (Low) Warning (Low)

Normal

0x9AC0

Normal

Normal behavior, see IBM Technote 1219847.

0x331A

Warning (Low)

Normal

This is only informational.

0x3320

Warning (Low)

Normal

AdminP process is normal.

0x3327

Warning (Low)

Normal

AdminP process is normal.

78

0x3346

The database is transactionall y logged. A full backup of it needs to be performed on for media recovery. Router: Message contains no recipients does not appear in the unread lists of the databases on . Admin Process: does not appear in design elements of any databases designating as their Administratio n Server Not all specified languages were found in design template

Failure

Normal

Backup software is requested to take a new full backup of this application.

0x336D

Warning (High

Normal

Information on missing recipients in a message. AdminP process is normal.

0x33C4

Warning (Low)

Normal

0x33E3

Warning (High)

Normal

AdminP process is normal.

0x3032

Normal

Warning (High)

This error has to be handled, otherwise refresh design of the database fails.

3.1.5. Further Reading

For more information about how to use and configure DDM, refer to the following IBM Technote and the IBM Redpaper:
http://www-01.ibm.com/support/docview.wss?rs=463&uid=swg27009312 http://www.redbooks.ibm.com/abstracts/redp4089.html

79

3.2. Mail Routing

If your company is like most companies using Lotus Domino, you are using your server to route mail. This article provides a brief introduction to Domino mail routing concepts and helps you to learn more about how to manage a Domino mail environment. Since there are many great resources already available, this article helps guide you to these resources. For a description of the tasks, databases and templates involved in mail routing, refer to Mail routing with the Domino mail server.

3.2.1. Managing Spam

As a Domino administrator, one of the most common concerns is how to deal with spam. There are a number of resources and options available. To understand how to use the options available within the Domino server configuration, refer to the articles Controlling spam: Advanced SMTP settings in Lotus Domino and Understanding SMTP authentication and securing your IBM Lotus Domino 8 server from spam. In addition to the SMTP settings, you may also want to create a server-based mail rule to filter mail. For information on creating mail rules refer to the Domino Infocenter topic Filtering new mail using rules. In additional to the security capabilities in Domino, many companies use a spam filtering service or software. One example would be Lotus Protector for Mail Security. If you are using a 3rd party spam filtering service, you may want to configure your server that only mail which is being sent from your vendor will be accepted by your server. You can easily do this by defining SMTP Inbound Connection Controls. In order to do this, you will need to obtain a list of hosts, IP addresses or IP ranges used by your vendor. You can then define those entries as well as your internal hosts in the field Allow connections only from the following SMTP internet hostnames/IP addresses. You will find this setting in the configuration document on the Router/SMTP Restrictions and Controls SMTP Inbound Controls tab. In the example shown in figure 1, you can see an IP range, IP address, generic host name and the internal private network range allowed for inbound SMTP mail connections. The host name must only match the end of the name so in this case both server1.YourVendorDomain.com and server1.mail.YourVendorDomain.com would be allowed to connect to the Domino server. It is important to consider internal inbound connections if you have any other applications or hardware such as a printer or copy machine that may route mail through your Domino server.

80

In some companies checking outgoing messages is equally important as checking inbound. If you are in the position that you want all outbound mail to be scanned for possible spam by your vendor, you can easily do this by configuring your vendor as an outbound relay server. You will find the Relay host for messages leaving the local internet domain field in the configuration document for your server, Router/SMTP Basics tab. In figure 2 the relay host is set to Server.YourVendorDomain.com. Note that if you specify an IP address, it must be enclosed in square brackets. Also, only one value is allowed in this field, so use caution when configuring a relay server as a relay server failure will prevent all outbound mail from routing.

3.2.2. Mail routing and multiple directories

When working specifically with mail routing, the topic of multiple directories is frequently brought up. Maybe you need to have secondary directories with customer or vendor contacts for mail addressing. Perhaps your users have been requesting a way to perform address look-ups when disconnected from the server. Possibly you need to prevent the server from searching secondary directories for mail addresses. Domino provides ways to accomplish all of those goals using directory assistance, extended directory catalogs and/or mobile directory catalogs. For more information refer to 3.4 Multiple Directories.

81

3.2.3. Journaling
Mail journaling allows administrators to keep a copy of specific messages or all messages as they route through the Domino server. This can be important for security or required for companies with pending litigation. Mail journaling is configured with a combination of settings specified in the configuration document and a mail rule. To access mail journaling settings open the configuration document and access the Router/SMTP Advanced Journaling tab as shown in figure 3.

Some of the fields are rather self-explanatory while other settings will determine access and usability of the journal. There are two methods available for journaling: Copy to local database Send to mail-in database

The default method is to Copy to local database. When this option is selected, data will be automatically encrypted for the user selected in the Encrypt on behalf of user field except for those fields listed in the Field encryption exclusion list field. The second method is to choose Send to mail-in database. Why might you choose one versus the other? The advantage of using a mail-in database is that multiple servers can journal to the same database. The disadvantage is that as administrator you must manage the encryption and database rollover as this will no longer be done for you. Unless you have a tool to manage the mail-in database, using the default option of Copy to local database is recommended. The database management options are rather straight forward. You can choose to create a new journal based on size or date. By default a new journal will be created each day. At approximately midnight the current mail journal will be renamed to mjmmddyyyy.nsf, for example mj11302010.nsf. The last field in the Basics section of the journaling tab is Journal Recipients. Whether or not you enable this setting you will be able to see the original values chosen in the TO, CC and BCC fields for the message. In some cases,

82

this may be a group. By default, you will just see the group name listed in the journal. Who were members of the group? This could change. To ensure you see all of the actual recipients of the message, you will want to set Journal Recipients to Enable. Assuming you have chosen to copy the journaled message to a local database that the server will manage for you, determining if the field encryption exclusion list should be modified and which user should be used for encryption is slightly more complicated. The values you choose will effect what data will be seen by users accessing the journal when not using the ID listed in the Encrypt on behalf of user field and what data can be included in a full text index. A full text index is built by the Domino server, so only data that can be read by the Domino server can be included in the full text index. The following table will help you match your company requirements with the proper journaling configuration.
Requirements: Data must be secured with full data access restricted to one or two users. Message subject, body and recipients must be encrypted. Full text searching of message subject and bodies is not required. Configuration Details: This is the default configuration. To satisfy these requirements, register a new user and specify that user in the field. For example, At fictional Software Company A , they created a user named Mail Journaling/Administration/Company A. This user should then added to the ACL of the mail journal and we suggest that the person document have a readers list to hide this person from the general user population. The id file and password are then shared and accessed only by the users designated at the company with this authority. Based on the default field encryption exclusion list, anyone with reader authority to the journal will only be able to read the date the message was sent and who was the original sender as shown in Figure 4. When the message is opened with the Mail Journaling/Administration/Company A id, the entire message is visible.

Depending on your security requirements you may want to further secure the id used by creating the id in a private organizational unit and only provide password reset authority to those administrators who are authorized to access the mail journal. For more information on using an ID vault refer to ID Vault All users with reader access or above to the mail journal application should be able to view all messages in the journal. Users who access the journal must .

In order to satisfy these requirements, the Encrypt on behalf of user field must be blank which will disable encryption. The entire message can now by seen and data access is controlled by the ACL of the mail journal as shown in Figure 5.

83

be able to perform complete full text searches of subject and body of all e-mails.

All user with reader access or above to the mail journal application should be able to view the date, sender, recipients and subject of all messages in the journal. The message body should remain hidden unless accessed by the appropriate id file. A full text search must be able to used to search for recipients, senders and subjects. Mail journaling has been running with encryption. A new requirement or lawsuit requires that certain mail journals be sent for

In order to satisfy these requirements, register a new user and specify that user in the field. For example, At fictional Software Company A, they created a user named Mail Journaling/Administration/Company A. This user should then added to the ACL of the mail journal and we suggest that the person document have a readers list to hide this person from the general user population. The id file and password are then shared and accessed only by the users designated at the company with this authority. The field encryption exclusion list should be modified to include the SendTo, CopyTo, BlindCopyTo and Subject fields. Once done, anyone with reader authority to the journal will only be able to read those new fields as shown in figure 6. When the message is opened with the Mail Journaling/Administration/Company A id, the entire message is visible as shown in figure 6 .

An agent can be used to decrypt the documents in the journal if needed. For details refer to technote 1089495.

84

review. Data encryption must now be removed from some journals.

Journaling notes.ini parameters:

JournalLimitForwardToAdmins JournalLimitForwardToMailAddress JournalLimitForwardToDomain JournalLimitForwardToSendCopyTo

3.2.4. Out of Office Notification

Starting with Domino version 8.0, there are two methods of out of office notification within Domino. The first is controlled by an agent and the second it controlled by the router task and has many benefits over the agent. For information on this topic refer to The IBM Lotus Notes and Domino Out of Office service: Best practices.

3.2.5. Mail routing in a clustered environment

In a clustered environment, when the primary server is down you mostly likely want any mail destined for users on the downed server to be delivered on its cluster mate. There are several configuration pieces that must be in place in order for this to work. First, a replica of each mail file must exist on another server in the cluster. Second, cluster mail failover must be enabled in the configuration document for all servers in the cluster as well as any Domino server that will be attempting to send mail directly to the server. You can find the cluster failover setting in the Router/SMTP Advanced Controls tab. The Cluster failover setting should be set to Enabled for last hop only as shown in figure 7. The setting of Enabled for all transfers in this domain may also be used; however use caution to avoid mail routing loops with this setting. What is the difference? When failover is allowed for all transfers if a hub server is down, mail can be redirected to another hub server. This setting should only be used in enterprise size deployments and as the administrator you should ensure that both servers are able to send the mail to the desired destination to avoid problems with the mail getting stuck on an alternate hub.

85

The cluster failover parameter was configured via a notes.ini parameter in order releases of Domino. If you have inherited an environment you should ensure that MailClusterFailover is not specified in your notes.ini or is set to 1 to prevent problems with mismatched settings. To see if the setting is currently in use on your server you can use the console command show config mail* and review the output. MailClusterFailover should not be included in the output just like the example shown in figure 8. If so, you know that the notes.ini setting is being defined manually in the notes.ini or in a configuration document and should be removed.

> show config mail* MAILSERVER=CN=server1/O=Company A MAILTYPE=0

Figure 8: Example of show config mail*

The next piece that must be in place for cluster failover to work is that the cluster.ncf must be populated. The cluster.ncf file is a text file with a list of all known clusters and cluster members. It is populated automatically when it connects with a server that is a member of a cluster. In enterprise size environments the default cache size may be too small and prevent failover, For more information refer to technote 1102957. The final configuration piece that is required in order for mail cluster failover to work is a fully populated cluster database directory (cldbdir.nsf). The cluster database directory contains a list of all replica databases in the cluster. As an administrator, you can choose to disable cluster replication for a database. If a database has been marked out of service in the cluster database directory, then cluster failover will not occur. In figure 9 below you can see the mail\utwo.nsf has been disabled on server1.

Lastly, if you cannot determine why the router task is not properly delivery mail to a cluster server, you can enable additional debug logging by setting RouterDebugClusterFailover=1. This setting is dynamic and can be enabled or disabled

86

using the set config command, for example set config routerdebugclusterfailover=1. For example, here is an example of cluster failover working normally: Error connecting to server Server2/Company A: The server is not responding. The server may be down or you may be experiencing network problems. Contact your system administrator if this problem persists. Router: Cluster failover starting for server server2/Company A in domain COMPANY A; mail file mail\uthree Router: Cluster failover found server server2/Company A in cluster MailCluster Router: Cluster failover found server SERVER1/COMPANY A is a cluster mate of server server2/Company A in cluster MailCluster Router: Cluster failover found local failover replica mail\uthree.nsf Router: Failing over mail transfer from server2/Company A to [$LocalDelivery]; mail file mail\uthree.nsf Router: Message 005758B8 delivered to User Three/Company A

When no replica exists or the replica has been marked out of service then the following message will be posted: Router: Cluster failover could not find server by Rep ID for Server2/Company A mail/uthree; Unable to find path to server. Check that your network connection is working. If you have a working connection, go to Preferences - Notes Ports and click Trace to debug

3.3. Mass Mailings

Mass mailings are mail messages addressed to a large number of users. It could be your quarterly employee newsletter sent out to all employees or a sales flyer to all customers. In any case, it is a message with a large recipient list that can affect normal mail delivery on your server.

3.3.1. The Mass Mailing Problem

When the Domino server routes mail, for the most part, it works in a first in first out basis. It is also a multi-threaded process that is able to send multiple mail messages at any given time. When a large mail message is received, the router task of the Domino server will use as many threads as it can to deliver the message. This can be a good thing in that Domino will get the message sent to all of the recipients as quickly as possible; but in the process, it may appear as the router is hung. In this case, you see the router task taking CPU and memory, and all new mail will stay in the mail.box file(s) while the router completes its work on the message. If the message has a large number of recipients, users may see a slow down when accessing the server. This happens while the router is parsing the list of recipients and determining which messages should be delivered locally. This step is commonly referred to as the name lookup step and involves briefly locking the $users view in the Domino directory (names.nsf). Since the $users view is also required for user authentication to the server, these lookups can cause a brief delay for the end user.

87

3.3.2. The Mass Mailing Solution

Optimizing the performance of mass mailings on your server can be achieved in a number of ways. That is because there are many factors that determine the impact of sending the message. This includes the group size, how the recipients are specified, how the message is sent, who can access the group and router configuration.

Using Sub Groups

One simple way is to break up very large group into multiple subgroubs. For example, at Fictional Software Company A, they send monthly technical tips to all of their customers. They have created a group called Product B Customers. In this group they have multiple subgroups listed Product B Customers A-K, Product B Customers L-Q and Product B Customers R-Z. Sending a message to the group containing multiple subgroups will be more efficient than sending a message to one large group. This is because of the way the router must parse each message as it prepares to send the message to each user. Even better would be to send a message to each subgroup, but most users will reject this idea as it requires more work on their part.

Properly Using Mail Addressing

Are your mass mailings sent to a group listed in the to, cc or bcc field? The router task must process a message differently if a large group is specified in the to or cc field versus the bcc field. When processing a bcc list, the router makes a unique copy of the message for each recipient. That is very intensive work. You can reduce this overhead by using the to or cc field when appropriate or set Disable_BCC_group_expansion=1. By setting Disable_BCC_group_expansion, you are preventing this unique copy process from occurring. However, as a result, the users will no longer see their name or e-mail address listed in the to, cc, or bcc field when they receive the message. For additional information on the Disable_BCC_group_expansion notes.ini parameter refer to technote 1089346.

Using Low Priority

Another option when sending mass mailings is to avoid using your production mail server or avoid sending mass mailings during primary business hours. If you use a specific account for your large distributions, selecting your administration server as the mail server for that account would be a simple solution to avoid causing problems for your users. However; if you only have one server or many users that must send mass mailings, then an alternate solution would be to force users to send mass mails as low priority messages and therefore to be delivered off hours.

88

How do you set a message to be sent as low priority? This is done when composing the message and selecting Delivery Options. From there the user can set the Delivery priority to Low as shown in figure 1.

To force your mass mails to deliver during off-hours, you need to take two actions (1) Define times for low priority mail routing and (2) configure a rule to only allow mass mailings to be accepted for delivery if they are sent as low priority by the user. By default the Low priority mail routing time range is from 12:00 AM 6:00 AM. You can change this range to be the best times for your environment in the configuration document, Router/SMTP Restrictions and Controls Transfer Controls tab as shown in figure 2.

89

Be sure that if you take down your server each evening for backups that it does not conflict with your routing time. You should also be sure that you have not disabled the message priority functionality on your server. The Ignore message priority setting is found in the configuration document, Router/SMTP Advanced Controls tab as seen in figure 3.

90

The next step is to configure a server based mail rule to define what should be considered a mass mailing and thus be sent as low priority. A server based mail rule will affect all mail going in and out of the Domino server so be aware that if you set the value to low you could reject inbound mail directed to a large number of users. In the example seen in figure 4, any message addressed to more than 50 recipients that is not a low priority message will be rejected. The user will receive a Delivery Failure Report stating that the message was rejected for policy reasons.

Limiting Access to Group Documents

Limiting access to group documents can also be an effective way of limiting which users in your organization can send mass mailings. A readers list can be placed on any document stored in a Domino database, including group documents. To create a readers list on a document, right click on the document and select Document Properties. On the security tab, remove the check next to All readers and above. Now only the users or groups with a check next to their name in the list as shown in figure 5 will be able to access the document. It is critical that when you use a readers list that you check the LocalDomainServers group in order to allow the group and group changes to replicate throughout your Domain. It is also recommended to also include the LocalDomainAdmins group. There is no undo when it comes to a readers list. If you lock the document down and the only person able to access the document leaves the company, there will be no

91

way to unlock or see the document. Thus, it is important to protect yourself by selecting the LocalDomainAdmins or other group as appropriate in the readers list.

In the case of a group document, if a user is not part of the readers list, then the user will not be allowed to send a message to the group from the Lotus Notes or iNotes clients. If an unauthorized user attempts to send a message using a POP or IMAP client, they will receive a non-delivery report with the error Not authorized to send mail to this user or group.

Server Configuration
There are a number of NOTES.INI parameters that can be used to further tune your Domino server and prevent server problems with mass mailings. This includes RouterMaxConcurrentDeliverySize, RouterMaxEffectiveSize and RouterMaxEffectiveSizeIncAttach. RouterMaxConcurrentDeliverySize allows the router to open only one copy of a message at a time if it is greater than the specified size in bytes. This setting provides a performance improvement for messages with large attachments sent to multiple users. Users may notice a slight delay for a message with a large attachment to reach all recipients, but message will be delivered. The proper size for this parameter varies depending on your environment. Many customers find a starting value of 1048576 (1 megabyte) to be helpful. RouterMaxEffectiveSize sends a delivery failure notification to a user if the message they are trying to send is greater than the effective message size, specified in kilobytes. The Effective size of the message is calculated by taking the size of the message times the total number of recipients (effective size = message size in kilobytes * number of recipients). Note that the message size used in this calculation does not include attachments unless RouterMaxEffectiveSizeIncAttach=1 is set. How you set this value depends on whether or not you want to include attachments in your calculations. RouterMaxEffectiveSize is similar to the Maximum message size parameter that can be configured in a configuration settings document (figure 6) or a rule configured based on size (figure 7). The difference is that when using the Maximum message size or a

92

server based mail rule the recipient count is not considered.

93

Transferring the Work Out of the Router Task

One final possibility for dealing with mass mailings is to find a way to transfer the work away from the router task and into another task. You could do this by developing an agent to sent the distribution. The agent could break up the message sending to groups of small users, queue the message for delivery at a later time or use any of the other strategies discussed in this article. There are also a number of 3rd party tools that will manage and create large mailings without impacting server performance.

3.3.3. Conclusion
In summary you have seen a number of ways you can manage mass mailings in your Domino environment. Some options involve user training and others are transparent to your users. Now that you understand the many options available to you, you can determine which set of options are the right choice for your business.

3.4. Multiple Directories

As a Domino administrator, you may be asked to integrate multiple directories in your Domino environment. This could be anything from creating a simple address book to track customers or vendors, integrating two Domino environments or something as complex as configuring single sign on (SSO) in your environment. For more information regarding authentication options for Domino, refer to 1.4 Domino Authentication Options. Alternately, you could be trying to find a way for users that exist in another directory to access your server via the web (web authentication). This article provides an overview and introduction to configuring secondary directories in your environment. For a list of features available, refer to the help document Comparison of directory catalogs and directory assistance. If you are creating your first secondary directory, you should be aware that the directory should be created with the Domino Directory (pubnames.ntf) template as personal address books are not recommended for use with directory assistance or directory catalogs. Once your directory has been created and populated, you can then begin to configure access to the directory for your users. If you have multiple Domino servers in your environment, each server that needs to access the directory should have a local replica for use.

3.4.1. Condensed Directory Catalog, Extended Directory Catalog or Directory Assistance

To determine whether or not you should be looking into directory assistance or directory catalog and for an overview of the steps required to implement, refer to figure 1 below. You can click on each entry in the diagram for information on accomplishing that step.

94

3.4.2. Hints and Tips

Some other things to keep in mind if you are creating your first secondary directory or catalog: Directories must be created using the Domino directory (pubnames.ntf) template. Condensed directory catalogs should not be used on the server and cannot be used for group authorizations. For example, a group listed in a condensed catalog cannot be used to grant access to an application in an ACL, but an extended directory catalog or a secondary address book referenced via directory assistance can be used for group authorization.

95

To determine if any secondary directories are currently in use by your Domino server, enter the Domino console command show xdir. A server should always have access to a local replica copy of any secondary directory. Use an * instead of the actual server name to have the server check for the file name locally as shown in figure 2.

3.5. Server Clustering Options

This article addresses improving Domino uptime. Domino uptime can be improved in many ways. One way is by making all components redundant and the second way is making the infrastructuremore simple. Yes, simple. If you have a simple and clear infrastructure, it runs better. In this article, we discuss Domino components only. Redundant power, cooling, network, HDD and other components that lay below Domino need to be redundant as well.

3.5.1. Keep It Simple

The rule of thumb is not to mix configurations. If you have a mail cluster, let it provide mail to users. Do not put other things on this server, including all applications, web server, and products, such as Lotus Quickr, Traveler, and Sametime. For these types of products, you should use a separate server. On top of Domino, you can have only one such add-on product. So even if you have an additional server for such products, do not put them all on top of each other. IBM Domino license states that if you need to install Traveler, you do not need to buy additional licenses (for additional Domino server). An advantage of putting Traveler on a separate server is that it improves security, because Traveler will be placed in the DMZ zone. Even if somebody does hack it, it will be an empty server. Nothing else will be compromised because all databases and applications will be located on other servers that are not available from the internet. If you use Sametime Entry to enable users chatting with each other, or you use Sametime for audio and video conferencing, then again, you should place it on a separate server. This will make your servers and services less dependent from each other. Tips: Not every such product will work with any patch level of Domino. For example, a fix for your mail environment, might be not compatible for the Microsoft Quickr server

96

running on the same machine. Before you upgrade to production environment, make a clone and test to make sure it works fine first.

3.5.2. Redundant Domino Parts

There are different approaches on how to improve availability of the service to users and systems that access Domino. Depending on how users and applications access Domino, there are different ways on how to achieve this goal. Is it possible to measure availability of a server? Yes. There are two factors in this calculation: PLANNED downtime and UNPLANNED downtime. Planned downtime is unavailability of a server which was forecasted and planned beforehand and it does not impact users. For example, you plan to put a new version of Domino or patch operating system during off hours. No one will notice that you brought the server down. After you complete your task you bring the server back online. On the other hand, if there is a power failure during the normal working hours or the operating system or Domino crashes, this is considered UNPLANNED downtime. During this time, users cannot send or receive mails or access databases. They will call the help desk and register support calls. This unplanned downtime impacts the user's work activity. The following table lists different approaches you can use to reduce unplanned Domino downtime depending on the size of your Domino environment (small, medium, or large)
Approach Domino Cluster Small + Medium +++ Large +++++ Notes Seamless failover for Lotus Notes users. Note #1 Note #2 OS cluster ICM-Internet cluster manager Hardware LoadBalancer POP3/IMAP/HTTP proxy +++ ++ + + ++ ++ ++ + + +++ +++ + Note #3 Note #4

97

The following flowchart provides guidance on the approach best suites to your environment to reduce system downtime.

Notes: #1. Requires Enterprise or Utility license for each server. #2. Requires Database adaptation for failover. All data (as determined by the Domino Administrator) is duplicated on the servers in the cluster. If data is corrupted on one server and deleted during a consistency check (Fixup), it will be automatically restored (replicated) from other. If data is deleted on one server, it is deleted from the other servers as well. #3. You can use Domino Express licenses if your organization is less than 1000 users. #4. Servers uses one disk to store data. So if one server goes down, the other server is up using the same data. If data is corrupted, both servers will operate on corrupted data. Servers use the same data. We explain how you can reduce the downtime of your Domino server using various approaches for the remaining sections of this article.

3.5.3. Domino Cluster

Small: Medium: Large: Choose this approach if: Organization size is Medium, Large or Domino downtime in unacceptable.

98

A Domino cluster is a group of servers of two or more servers that provides failover for Domino applications. Failover is a process in which the Lotus Notes client is redirected from one server to another when the primary server is not responding or is overloaded. The advantage of failover is that users will continue to have access to critical resources. In the most current versions of Domino, user will not receive an error messages and the failover happens transparently for the user. In case of a Domino cluster, database replicas of clustered databases are located on different servers. Clustering provides not only failover, but also load balancing. Overloaded servers can pass users to other servers that are not so busy. In general, there are two types of operating system clusters, Active-Active, and ActivePassive. For Active-Active cluster, all servers are available and serve users' requests at the same time. For Active-Passive cluster, when one server (the Active server) serves users' requests, the other server (the Passive server) waits and does not serve users. When the Active server goes down, the Passive server notices that and it will become the Active server and will start providing service to users. Domino clustering requires a license for every server included in the cluster. It needs to be purchased from IBM or an IBM Business Partner. Thus, there are additional license costs for this solution. Some databases automatically support clustering and failover. Other databases, developed in house, do not support clustering by default and need to be programmed to support clustering. Some databases provided by IBM already have clustering support, for example Mail Database. Designer Help lists LotusScript functions that help to make databases work in a cluster. For instance, Database.Open is a regular database open function and Database.OpenWithFailover is a cluster version of it. Database.OpenWithFailover will try to open database. If the database is not available, it will automatically failover to another server. Enabling a database to support cluster mode is not about substituting one function with another. There are different challenges that need to be solved by developers. For instance, two documents are modified at the same time on different

99

servers. You need to think about document locking to prevent Save Conflicts. Agents are another issue to consider in a clustered environment. Domino supports "After new mail has arrived" agents failover, but it does not support scheduled agent failover. The following technique can be used to solve scheduled agent to work in a clustered environment: There is one master agent that triggers all other agents (slave agents). This master agent is scheduled on Any server. It is executed every X minutes on all nodes of cluster. To avoid this agent to actually running on all servers at once, we need somehow try to run the agent first on one server. If it is down, then we run the agent on other nodes. You can create a profile document in a database where this Master Agent lives. You can put two fields in this profile: PrimaryServer and TimeStamp. PrimaryServer is the name of server on which agent should run and trigger all other agents. When the master agent runs successfully on one server, it updates profile document with a timestamp and this profile document is replicated to other servers via cluster replication. Master agents on other servers checks if there is an up-to-date timestamp. If yes, then they quit execution as they know that the Master agent on the primary server is already at work. If there is a big gap, between NOW() and the last timestamp, then the Master Agent on the other server understands that the Primary server is dow, and it backs up the Master agent. It then triggers all of the other agents. Nodes of a Domino cluster can be located in different buildings, cities or countries. Domino provides the Geo-Clustering option. In case of fire, or if it is impossible to work in the building, all information is available from the alternative location.

Summary
Domino Clustering is the best way to provide high-availability to Lotus Notes users. Failover occurs and is seamless in the more recent versions of Domino, otherwise a prompt to redirect to a different server is displayed. You may build a Domino cluster on top of different operation systems. If you cannot provide cluster awareness of your applications, you can choose other solutions such as an operating system cluster.

Additional Resources on this.

For more information, see
Understanding IBM Lotus Domino server clustering http://www.ibm.com/developerworks/lotus/documentation/d-ls-dominoclustering/index.html Lotus Domino R5 Clustering with IBM eServer xSeries and Netfinity Servers http://www.redbooks.ibm.com/abstracts/sg245141.html

3.5.4. OS Cluster
Small: Medium: Large:

100

One type of clusters is the operating system (OS) cluster. There are two or more servers and Domino process is running on one server. When there is a need to switch server or there is a hardware failure, the other server starts the Domino process on the other server. In both cases, server will run under one and the same SERVER.ID and same IP address. In case of Domino cluster, they have different names. If ServerA is running, and we need to do some maintenance, we give command to the other server to take control. Then the same ServerA is started on the other node. Users may experience a small delay when the first server goes down and the second server starts up. Almost every operating system provides an option to build an OS cluster. An additional licence may be required for this. For example, on Linux -Heartbeat daemon provides a solution to build cluster on Linux OS basis. In configuration of this daemon, you define primary node and processes that need to be monitored. If one server goes down, the second node takes control. It will map shared disk, where Domino data is located and will start Domino server on another machine. The OS clustered Domino server appears to end users under the same name and same IP address. If there are systems that access Domino by host name, such as POP3/IMAP/LDAP/HTTP, they will successfully reach the server. If you have less than 1000 users in your company, you may use the Domino Express licence for the OS cluster. Since a limitation of the Express licence is that you have less than 1000 users and that Domino clustering is not used. From that point of view, you can have Domino high availability at relatively low price.

101

If you use an OS cluster, then you will NOT use the Lotus Notes client failover feature, from Release 8.x and 8.5.x. Domino supports failover for opened databases. in case of OS cluster, users may need to re-open databases. You MAY use an OS cluster in conjunction with Domino cluster. This is a supported configuration.

3.5.5. Internet Cluster Manager (ICM)

In the earlier sections, we discussed high availability of the entire Domino server. Starting from Release 5.x, there is an option for high availability Internet Cluster Manager, also referred as ICM. ICM provides a failover for WEB clients who access your iNotes server or intranet and company homepage. ICM is an additional task that is loaded on a Domino server. It is quite easy to setup ICM if you already have a working Domino cluster. ICM is an addition to Domino cluster, and it works only with Domino cluster. In the Domino configuration you define which Domino cluster ICM should look for, if you have many clusters in your Domino environment. ICM plays a re-director role, like a dispatcher who (re)directs landing planes. When a new HTTP request comes in, ICM knows which servers are available and redirects users to one of them. When one server goes down, ICM notices that and the subsequent new requests are directed to another available server. After some time laps, ICM sends Domino a ping command to check which servers in the clusters are available. When new requests come in, ICM knows which servers are up and which ones are down and sends the new requests to the running servers.

102

A working scenario of ICM

We have two mail servers, mail1.company.com, and mail2.company.com. ICM listens for requests on webmail.company.com. When users type the webmail.company.com address, this request goes to ICM. ICM already knows which servers are up and it will send the user redirection requests. The URL is changed to mail1.company.com or mail2.company.com in users browser and the user is asked to authenticate. It is advised to run ICM on a separate server than the Domino servers. Otherwise, if ICM runs on an overloaded server, there is a probability that this server may become unavailable and ICM will be not reachable for clients. If that occurs, ICM will not redirect requests. If the ICM runs on a separate machine it just sends back redirect information to clients. The ICM system should have low unplanned downtime.

What will happen if ICM is down

The cluster will be up, but users will not be redirected. If you want to improve availability of ICM you can have several ICM servers and assign multiple IP addresses to one host. Then, the failover of ICMs will happen at the DNS/OS level. With this approach you can have a high level of ICM availability. Deploy Single-Sign-On for web servers will give your users additional benefits. If mail servers share one common LTPA Token, then if users change the URL from mail1.company.com to mail2.company.com, there will be no additional login screens. If one mail server goes down while the users read mail, users have to go back to webmail.company.com host, and ICM will redirect users to another available server. With Single-Sign-On set up, users will get to their mails without additional login forms.

3.5.6. iNotes High Availability Configuration

For information on iNotes high availability configuration, see http://www-10.lotus.com/ldd/dominowiki.nsf/dx/iNotes_High_Availability_Configuration

3.5.7. IMAP failover (Domino 8.5 new feature)

Release 8.5.2 of Domino added new functionality to IMAP users. Now IMAP users can failover to different servers, and the IMAP client will not be confused. If you have many IMAP users, you may benefit from this feature. There are some additional steps needed to configure IMAP support on Domino Clustered servers so refer to Technote #1429885 or the 8.5.2 Administrator help. In conjunction with multiple IP addresses assigned to one host, or software proxy, IMAP users will be redirected between available servers. For additional information, see Failover and the IMAP server http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg21429885

3.5.8. Lotus Traveler Server High Availability

At the moment Lotus Traveler does not provide an option for native clustering. An enhancement for Lotus Traveler support is registered. You can still build many Traveler servers. If a user manually changes the IP address of the server, the Traveler client will do a Full Sync of Data, which is quite time/traffic consuming task. This is because of the design of Traveler server.

103

One of the best solutions to make Traveler available in cluster is to put it on OS cluster or use Proxy server which works in Active-Passive mode. Related Topics: Traveler Clustering and failover http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/index.jsp?topic=/com.ibm.help.lnt8 51.doc/Clustering_and_failover.html

3.5.9. Load Balancer

One more option for improving availability of the servers is load balancer. Load balancer is a hardware device or software that can check if target servers are available. When new request comes in, it will redirect request to one of the available servers. Hardware balanced can be used to cluster POP3/IMAP/SMTP users between servers. In addition to POP3/IMAP/LDAP/SMTP protocol failover, you can use load balancer to switch Lotus users between Domino servers. If you want to use IP-Sprayer (load balancer) with Lotus Notes, you should have additional parameter described in Technote 1233210. You can deploy this parameter with the help of policy/desktop settings. Notes client fail to connect to Domino servers behind a network sprayer http://www-01.ibm.com/support/docview.wss?uid=swg21233210

104

3.5.10.

Software proxy (IBM HTTP, nGinx, etc)

There are software programs that work like proxy servers, and they can do automatic failover of servers from which they request data. Some solutions like IBM HTTP can provide failover (reverse proxy) of HTTP/HTTPS protocols. Some others can serve SMTP, POP3, IMAP, HTTP for example NGINX. There are different vendors and every proxy functionality is different from the protocol prospective. Depending on your need, if only HTTP failover is needed or additional protocols like SMTP, POP3, you can select which solution to be used.

3.5.11.

Sametime and QuickR High Availability

Sametime and QuickR server may be also clustered. If these resources are critical for you, deploy clustering which will provide high availability for Lotus Sametime or Lotus QuickR users. Follow the links below to be guided how to deploy cluster for Lotus Sametime and Lotus QuickR. QuickPlace clustering guide: configuration and managing places http://www-01.ibm.com/support/docview.wss?uid=swg21248809

105

3.5.12.

Disaster Recovery Plan

This section deals with Domino recovery if you have OS / hardware failure. Recovery plan is a document that defines a sequence of actions and responsibilities during server restore. Test recovery is a procedure that needs to be done after you deploy a new backup solution. In addition to this, a test recovery needs to be repeated every year to be sure changes done in the environment are reflected in the backup policy. Test recovery shows that everything is fine with backup procedure. Test recovery should be ordered without the backup team, so they will not prepare additional (full) backup copies. The purpose of this to understand what problems you have in the test case, and eliminate problems in future. When you put a new server in production, be sure that this server is included in daily backup procedure. Nobody knows when recovery will be needed or what data will be needed. It can be one text or configuration file, stand-alone mail file, database that is part of application or the entire server. It is vital to have a recovery plan for your Domino environment. You should write it like you are going on vacation, you know that there will be problems, and you do not want to have calls to your cell phone. Your colleagues should be able to do the recovery according to your documentation. Describe what need to be restored, how to restore them and in what sequence, installation locations, IP addresses and phone numbers. This document should be kept up to date. It should be printed and stored in an available place. Do not keep this only in electronic format. If the system is down you will not be able to access it. It is advised to do the test recovery of the entire server. You can do this on a separate machine, a test server. Be sure to restore it on an IP isolated machine so when you bring the server up it does not replicate other production servers. If you do the full recovery once, you will be able to do this again smoother and faster in a real life. Do spend time describing the steps you performed in the document. In a real life, you will do this at least several times faster than the first time. Test the recovery. Find and highlight the things you may have documented wrong in your current backup plan. For example, you backup .nsf files by a Domino specific backup solution and backup everything else with an OS backup solution, except the Domino DATA folder. In that case some important files, such as cert.id, server.id, notes.ini may be excluded from backup. Test recovery is ensuring that everything is fine with your backup solution and approach. In your recovery plan describe sequence of the restore. How should the entire server be restored, one mail file, or one document (alternative location, then copy paste).

3.6. Transaction Logging

Transaction logging is a real time log of transaction occurring on your Domino server. If you are starting with transaction logging for the first time refer to the article Notes/Domino Best Practices: Transaction Logging.

3.6.1. General Transaction Logging Recommendations for 8.5.x Servers

When working with transaction logging at Domino version 8.5.x, your first decision must be the type of transaction logging. If you are using a backup utility that will manage the transaction logs for you or need point in time recovery, then you will want to use archived

106

style logging. Be aware that archived logs are not cleaned up by Domino automatically. When using archived style logging, you must use a backup utility to clean up the old logs to avoid filling all available disk with old logs. If you are not going to be a backup utility to manage the archived logs or performing point in time recovery, then you should choose to use the circular logging style. The maximum size for circular logs is 4GB and that is the recommended value for all but the smallest implementations of Domino. One source of confusion over many years is where to place the log directory. It may be easier to state where you should not place your log directory. It should not be placed on the same physical disk as your Domino data directory. That means you may place it within the data directory when using a RAID disk array (like IBM i). For more information refer to Transaction Logging Best Practices hardware recommendations.

3.6.2. Transaction Logging Tips

There are some things that every Domino administrator should know in order to prevent problems with transaction logging: Domino must always be able to obtain write access to the transaction log files, thus you should not run anti-virus on the transaction log directory. The transaction log tracks database changes through the DBIID. Thus, you should never have multiple .nsf files with the same DBIID. The Lotus Notes client and Domino server will ensure this never happens. However; if you are moving, when copying files or restoring files at the OS level, be sure that you do not accidentally duplicate a DBIID. If you need to make copy or restore a file at the OS level, you can easily protect yourself by running a fixup or copy-style compact on the current database to change the DBIID before you make the copy or restore. If you have a problem with the transaction logs, always keep a copy of the log files, console.log and any logasio*.txt files you may have in the IBM_TECHNICAL_SUPPORT directory. Without that information, Lotus Support may not be able to assist you in determining the cause of your problem. To determine if a system database should be have transaction logging enabled, refer to the table in the DAOS Best Practices article.

3.6.3. NOTES.INI Recommendations for Domino 8.5.x Servers

To help optimize your transaction logging configuration, there are some NOTES.INI parameters that you should be aware of and may want to set on your Domino server. NSF_DONT_LOG_MAILBOX_BODY=1: This setting will prevent the message body from being logged for messages in the mail.box file(s). Since the mail.box files are very busy, this can improve the performance and throughput of your mail.box file(s). If your mail files are transaction logged, the message will then be logged at the point the message is delivered rather than at the mail.box files. RM_NO_LOG_LARGE_OBJECTS=1: No attachments larger than 1 MB placed into the server mail.box file(s) will be transaction logged. This can improve the efficiency of mail delivery for large messages.

107

Schedule_DisableTXNLogging=1: This setting will disable transaction logging for the scheduling databases (busytime.nsf or clubusy.nsf).

3.6.4. Domino 8.5.x and ODS 51 Updates

Starting at Domino 8.5 and ODS level 51, log compression is not enabled. Transaction Logging will store the data in the same format as the database. If the data is already compressed, it will be compressed in the transaction logs. If the data is not compressed, the data will not be compressed in the transaction logs. What data is being referred to here? It could be data documents, design documents or attachments. There are some cases where, for performance reasons (most often on system z and IBM i systems), you may want to compress your transaction logs, but not compress your data. In this case there are two NOTES.INI settings that can accomplish this for you: NSF_COMPRESS_TXN_LOGS - Compress both design and data documents within the transaction logs. NSF_ENABLE_LZ1_TXN_LOGS Compress attachments stored within the transaction logs using the LZ1 algorithm whether or not they have been stored compressed within the Domino database.

Enabling LZ1 compression for attachments

To enable LZ1 compression in your environment, open the Lotus Domino Administrator client, connect to your server, and select the Files tab. On the right panel, select Advanced Properties. Select the option to enable Use LZ1 Compression and select OK to apply as shown in figure 1 below. After this option is set, the LZ1 compression will work from this period on all new attachments, everything before this activation will remain the same way (without compression). To covert the existing attachments to LZ1 compression, you can run the compact task with the -ZU parameters.

Enabling design document compression

Open the Lotus Domino Administrator client, connect to your server, and select the Files tab. On the right panel, select Advanced Properties. Select the option to enable Compress database design and select OK to apply as shown in figure 1 below.. In order to compress the design,a copy-style compact must now run on the database.

Enabling document data compression

Open the Lotus Domino Administrator client, connect to your server, and select the Files tab. On the right panel, select Advanced Properties. Select the option to enable Compress document data and select OK to apply as shown in figure 1 below.. In order to compress the existing documents a copy-style compact must run on the database.

108

3.7. Domino Attachment Object Service (DAOS)

The Domino Attachment and Object Service (DAOS) is a new feature introduced in Domino 8.5. DAOS is a mechanism which shares identical attachments between databases on the same Domino server. When configured, separate and complete attachments are no longer stored within a database, rather a single copy of an attachment is stored on the file system with documents referring to them on the disk. DAOS is implemented such that the creation and retrieval of attachments is transparent to all transactions, including all user functions. The benefit of DAOS is file size reduction. Any Domino database is eligible for participation in DAOS. This article discusses when to deploy it in your environment. For information on DAOS architecture, see: IBM Lotus Domino going green: The new Lotus Domino attachment and object service

109

The following flowchart provides the guideline on when to deploy DAOS.

There are various technotes and articles available about DAOS including how to configure it and how to estimate its impact. How can you be sure if every server needs this? Should it be enabled on Traveler? No! Should it be enabled on company cluster servers hosting several hundreds of gigabytes of data? Definitely Yes! When DAOS is used on a server, additional tasks are running on a server. DAOS also requires changes in your backup procedure; because in addition to NSF files the .NLO files need to be backed up.

110

DAOS is introduced in 8.5.0, but to be on a safe side, before implementing DAOS, you should make sure you are on the recommended version of Domino for optimal DAOS operation. You also need to disable "Shared Mail" if it is used. Use this above flowchart as a guideline. Do not run DAOS Estimator during working hours. Do this after working hours. You can also use DDM(Domino Domain Monitoring) to monitor DAOS. The following references illustrate the configuration and administration requirements for DAOS:
Download the Domino Attachment and Object Service Estimator Tool version 1.5 IBM recommendation for conversion to DAOS enable a database IBM Lotus Domino going green: The new Lotus Domino attachment and object service Achieving ultimate storage and server cost savings with DAOS in IBM Lotus Notes and Lotus Domino 8.5 IBM recommendation for conversion to DAOS enable a database DAOS Backup and restore

3.8. Managing Domino Indexing

Without an index, finding the data you are looking for would be extremely difficult. There are 3 different types of indexes used when using Lotus Domino -- view indexes, database full text indexes, and Domain indexes. There are also multiple tasks involved with Domino indexing including update, updall, chronos, kvoop and domidx. If you are new to Domino administration, this can be overwhelming. This article gives you the information you need to get started with and manage Domino indexing.

3.8.1. View Indexes

What is a view index? Anytime you open a Domino database, you are seeing information presented based on a view index. You could be looking at the Inbox of a mail file, a personal folder or a view in a custom application, but how that data is presented to you and whether or not you are seeing the latest update depends on the view index. Since view indexes are stored within a Domino database, you may wonder how to see if a view index exists, if it is up to date or the size of the index. The Domino Administrator provides a tool to easily see this information. To access the tool you will go to the Files tab in the Domino Administrator client and right-click on the database you want to see. You should then select Manage Views as seen in figure 1.

111

Within the Manage the views of this database window, you can see the name of each view index, the index size, the creator of the view, refresh interval, discard interval and the internal note id representing the view. In figure 2 you can see the views for a typical mail file.

Now that you understand how view indexes are used and where they are stored, you may want more information on how the view indexes are updated and the tasks responsible. For this information refer to the Domino administration help topics Indexer tasks: Update and Updall and Keyboard shortcuts that update or rebuild views.

3.8.2. Full Text Indexes

A full text index is an index designed to be used for searching within a Domino database. The full text index may only contain words found within a document or may optionally include text within numerous supported attachment types. A full text index is created and stored outside of the application in a .ft folder. A user can easily see if their mail database has a full text index by the icon displayed above the search bar as shown in figure 3.

112

When creating a full text index, there are several choices that need to be made including the update frequency and whether or not attachments should be read and included in the index. Be default, attachment indexing is turned off, but the default update frequency is different depending on the method used to create the index as shown in figure 4. If a user selects to create the index from within their mail file by clicking Not indexed as seen in figure 3 the index will be created with an immediate update frequency. This is typically not recommended for most applications. When creating the index from the Domino Administrator, the default is daily updates, which is recommended for most applications. You will also notice that the wording for the attachment indexing is also different. The most complete attachment indexing is binary indexing performed by the kvoop task which implements KeyView technology for reading the attachments. This type of indexing is referred to as with file filter in the Domino Administrator client and called conversion filters on the database properties Create Full-Text Index panel. There are additional options which can include indexing encrypted fields, sentence and paragraph breaks and case-sensitive searches. In general it is best to create the index with the fewest index options needed based on the application to minimize the size of the full text index.

Having a full text index on a database has some pros and some cons. Creating and maintaining a full text index requires additional resources on your system. However, if a database is not full text indexed and a user needs to perform a search, a temporary index is created and then destroyed. If this is done many times, it would be more efficient to have a permanent full text index. If an agent performs full text searches and a full text index is not created for that database, the server will advise you to create a full text index by logging the message Warning: Agent is performing full text operations on database 'mail/database.NSF' which is not full text indexed. This is extremely inefficient. Because of the cost involved with building and maintaining temporary and full text indexes, there are numerous options available to Domino Administrators to control who can create an index, how often and what can be full text indexed. See the table below for the options.

113

Scenario All users allowed to create full text indexes at their discretion with any refresh interval they choose.

Details In order to create an index, the user must have a minimum of designer access to the database. The user can create the index within the Application properties, the Full Text tab.

Pros/Cons Pro: Does not require administrator intervention Con: This is not best practice. IBM recommends a maximum access setting of editor in the ACL of the mail file. Con: Users can create indexes with an automatic or immediate update frequency on large mail files which can impact the performance of the entire server.

No full text indexing is allowed except if created by an administrator using the Domino Administrator client.

As an administrator you can prevent users from creating their own full text indexes by setting UPDATE_NO_FULLTEXT=1 into the NOTES.INI on your server. With this setting the following will occur:

Pro: Administrators can fully control when a full text index is created and the update frequency used. Pro: Prevent performance problems created by having mail files with full text indexes set for automatic or immediate updates. Con: Users may be upset if they have been able to create full text indexes in the past.

Current full text indexes will continue to be updated. Administrators can create new full text indexes from the Domino Administrator client. Users cannot create a full text index.

Available disk space on the system is low and you want to conserve space by removing unused indexes.

By default, view indexes are purged if they have not been accessed in 45 days. You can reduce this by settingDEFAULT_INDEX_LIFETIME_DA YS=<# of days> in the NOTES.INI of your server.

Pro: Disk space used by indexes that are not being used can be reclaimed and reused. Con: If you set the value too low, views may need to be rebuilt more frequently than before and thus it will have a negative performance impact. Pro: Conserve server resources by maintaining full text indexes rather than creating and deleting a temporary index each time the user performs a full text search.

No temporary indexes are allowed on the server. If a database/applica tion must be searched, then a full text index must be created.

As an administrator, you can prevent all temporary indexes from being created and deleted by setting FT_FLY_INDEX_OFF=1

114

Con: If you do need to search a database only one time, you will need to manually create and delete the full text index. No temporary indexes are allowed to be created on the specific application. At ODS version 48 or higher you can set a database property called Dont allow simple search. When this property is selected, the database does not contain a full text index and a user attempts a full text search a message stating Application must be full text indexed before search is allowed. Pro: You can avoid the server overhead of creating a temporary full text index, but not prevent it for all databases. Con: May lead to help desk calls for users that want to be able to perform full text searches. Pro: As an administrator you can set a standard for all attachments on the server. Pro: You can easily disable attachment indexing which will lead to smaller full text index sizes as well as fewer system resources needed when building and maintaining indexes. Con: Unable to accommodate situations where applications have different requirements which determine whether or not attachment indexing should be used. Need to disable binary attachment indexing (kvoop process). As an administrator you can prevent the binary attachment indexing process, also known as the keyview filter, from indexing attachments by setting FT_BINARY_FILTER_OFF=1 Pro: You can easily prevent the kvoop process from running on your server. Con: This does not work in Domino version 8.5.0 and 8.5.1 (SPR JSTN825PAV) You search an application and when reviewing the resulting documents you observe that When opening a document that is a result of a search, Domino will highlight all of the search words found within the search. Domino will also search the attachment for the search string to determine whether or not the attachment Pro: Opening documents after a search is faster for users and require less server resources.

It is desired to force or prevent the full text indexing of attachments or attachment types.

There are several ways that an administrator can control attachment full text indexing. To disable all attachment indexing set FT_Index_attachments=2 To force all full text indexes to include attachments set FT_Index_attachments=1 To exclude specific attachments types from being indexed set FT_INDEX_IGNORE_ATTCHMENT_TY PES=<list of file types separated by commas

115

when opening a document containing an attachment it is slower than opening a document without an attachment. It is desired that temporary files created and used by the indexing process are created outside of the data directory.

should be highlighted. You can prevent this search and thus prevent the attachment from being highlighted by setting FT_LIMIT_HIGHLIGHT_FILTER=1 in the server notes.ini file.

Con: Attachment will not be highlighted so user may not realize the result of their server is contained in the attachment.

During the indexing process the server will need to create and manipulate temporary files. You can specify the path where you would like these temporary files stored by setting view_rebuild_dir=<complete path to desired directory into the server notes.ini file.

Pro: Fewer files in the data directory will make the server more efficient. Con: Administrator must be aware of where the files are located to ensure the desired storage location is available for the server. Pro: Occasionally may be necessary when troubleshooting or can be used to compare system performance with and without chronos. Con: Chronos is responsible for updating view and full text indexes set for hourly updates. With chronos disabled, those views and indexes will not get updated until the nightly updall process runs.

Need to temporarily disable the chronos task.

Chronos can be disabled by setting Debug_Disable_Chronos=1.

3.8.3. Domain Indexes

A domain index is a full-text index over multiple Domino applications or databases. The domidx task is responsible for building and maintaining a domain index. A domain index can be especially helpful in legal situation where auditors may need to be able to search your entire organization for specific data. For information on planning and creating a domain index refer to an article that is still as relevant in Domino 8.5.x as it was in Domino 5 entitled Domino R5: Domain Search or the Domino Administrator help topic Planning the Domain index or the wiki article Key points to consider when creating a domain index.

3.9. Backup a Domino Environment

116

Lotus Domino is a database system which differs from a traditional file server. On file servers users are mainly accessing files during office hours so administrators can run a backup during non office hours. Lotus Domino servers instead are actively accessing their databases at all times, even when no user is accessing the mail file or application. Because an NSF file is being accessed at all time (in use), most operating system backup software products will skip them. Although this is acceptable for file servers, it is not for Domino servers. This article describes different technologies and strategies which can be used for efficiently backing up a Lotus Domino server and avoiding headaches when you need to restore this data. We provide information on how to define a backup plan which can provide confidence that you can restore your Lotus Domino functions and recover critical data as fast as possible. To set expectations correctly, the main purpose of a backup in the context of this article is to restore a server and its data in case of a disaster. Do not mix up backup with archiving because they are completely different topics.

117

3.9.1. Backup Basics

As mentioned earlier, a Lotus Domino server backup can not be handled like a file server because Domino is keeping a large number of files opened and is performing background actions against those open files even if there is no user accessing them. Relying on normal backup software which is not aware of how to handle open files can be dangerous, as this software might claim a file as locked which Domino is trying to access. This will cause the error This database is currently being used by someone else... in Domino. In short, the problems are: Backup software and Domino are claiming accessing the same files. Without control, data will be corrupted. Downtime.

Generally there are several accepted strategies, tools and features can be used to efficiently back up Lotus Domino. The backup solution for Domino only covers the Domino data (including attachments) and program files of the Domino servers to be backed up. It is beyond the scope of this article to describe a backup and restore solution for the operating system, which is supposed to be covered by the operating system level backup.

3.9.2. Backup Strategy

This article describes different backup strategies for Lotus Domino where clearly not every concept will fit all environments. We assume that with a growing size of an environment, the need for reducing downtime and increasing functional capabilities will change. The following table provides a suggestion as which strategy shall be applied for which size of an environment.
Strategy Offline Backup Clustering / Shadow Backup Open File Backup Domino Application Level Backup Replication Small +++ ++ + + (~) Medium + ++ ++ +++ +++ Large

Offline Backup
As mentioned earlier, Lotus Domino databases can not be backed up by simple file backup software because Domino is claiming access to the NSF files.

118

Offline backup describes a method where the Lotus Domino server is being shut down before the backup starts and will be restarted when backup has finished. This action ensures that the database files are not in use. Of course this method will cause downtime for the server and therefore is not a recommended option for every environment. In small environments where downtime is acceptable, this strategy can very well be a considerable option because data can be backed up without using specific backup software. Instead, simple backup software like the one used for backing up your operating system is enough. For automating a Lotus Domino server shutdown, administrators should use a script at the operating system level which is scheduled to start before the backup starts. Some backup software can also execute operating system commands as part of the backup job itself, typically called pre- and post processing. For a Domino server running on Microsoft Windows use the following commands:
Before Backup starts After backup has finished net stop Lotus Domino server net start Lotus Domino server

Use this method when Server downtime of several hours is acceptable for your business. Certified backup software for Domino is not available.

Note: Do not use offline backup method if you want a point-in-time restore. Point-in-time restore is only possible with application level backup in combination with archived transaction logging which is a feature of Lotus Domino. Domino server will be down as long as backup is in progress. If the backup job hangs, Lotus Domino will not start up without manual intervention. Never store backup sets on the same machine that you back up. In case of a disaster, you might not be able to recover any data!

Open File Backup

If downtime is not acceptable, a good (but expensive) method is to use a specific enterprise software package with a feature to back up open files on Domino servers. This type of software allows centralized backup of all the Domino servers in your network without taking them offline. This method does not require specific Domino certified software as the backup software is using different technologies to back up open files. Use this option when Domino servers do not use transaction logging. Downtime is not acceptable, but using a Domino certified backup software is not possible. You do not need to restore Domino data at a specific point in time. Otherwise, you need an application level backup which includes transaction logs.

119

Domino Application Level Backup

This method probably describes the most common method of backing up Lotus Domino servers, where operating system and Domino data are being backed up separately by software which is certified to use the Lotus Domino Backup API. An application level backup does not include any operating system files, so you need to implement two backup systems on the same server: One for backing up the operating system and its data. An application level backup which is the a Lotus Domino using certified backup software.

In a standard Notes database (NSF), the attachments are stored inside the NSF file itself, and the database is self-contained. In order to back up a standard Notes database, only the NSF file itself needs to be backed up. Starting with Lotus Domino 8.5, a new feature called DAOS is introduced where the NSF files that participate in DAOS no longer contain all data. Instead they contain only references to attachment content which is stored separately in files of type *.NLO As a result, backing up the NSF alone is no longer enough. The NLO data needs to be backed up as well. For more details, refer to IBM Technote 1358548 - http://www01.ibm.com/support/docview.wss?uid=swg21358548 Note: Do not forget that important data is stored within the Lotus Domino program directory. For example, the NOTES.INI and the Server.ID should be part of a regular back up!

Transaction Log Backup

Transaction log backup is actually not a strategy itself. It is an extension to the Domino application level backup, because in order to take a benefit of transaction log backups, administrators must perform the restore with a software which is certified to work with Lotus Domino. Transaction Logging is a process which captures database changes and writes them sequentially to a so transaction log, which should be on a separate disk drive (not just a partition!). Domino builds transaction log files on this drive and does not delete them until the 3rd-party backup software backs them up using API. Transaction logging improves performance and ensures data integrity of databases by committing to the transaction log first before writing to the Notes database. Transaction logs in Domino can be configured in two modes:, circular and archived mode. Archive mode - The transaction logs (*.txn) must be backed up with certified Domino backup software, for example IBM Tivoli Storage Manager for Mail, Data Protection for Lotus Domino. If no backup is taken, then the log files will grow further and further until the transaction log disk drive is full and your server will crash. Ensure that you have enough disk space available to support a time frame of at least 24 hours without any backup.

120

Circular mode - Use a configurable fixed amount of disk space to improve the server performance. It does not offer to restore a database at any point-in-time. In this mode, you can only restore the database as it was when you took the backup.

More information about transaction logging can be found in the following documents: IBM Technote 7003543 Transaction Logging on Domino Servers http://www-01.ibm.com/support/docview.wss?uid=swg27003543 IBM Infocenter Article Setting up a server for Transaction Logging IBM Technote 7009309 Best Practices Transaction Logging http://www-01.ibm.com/support/docview.wss?uid=swg27009309

Replication
Probably the least efficient method for backing up data is replication where Administrators configure Lotus Domino to replicate important databases across different servers to ensure that a live backup of the information is available somewhere else. This method requires much planning because: Deleted documents can and will replicate. You cannot restore a document which was just deleted by accident. You can configure the database to not replicate deletions. This adds a significant management overhead. In addition, it might not acceptable for all databases. Design elements can and will replicate. This causes design and data corruption which in the end might make the backup useless. Critical files which are local to the server itself are not replicated. For example, NOTES.INI and Server ID files are not replicated thus backed up. If a servers data drive fails and you do not have a backup of these files, then the system is in trouble.

Never rely only on replication as your method of database backup. A damaged or accidentally changed database may replicate, and then your only recourse is to recover the database from a server backup tape. This method must be combined with another backup concept in order to provide any value.

Cluster, Shadow Copy, and Shadow Backup

This backup strategy is a mixture between Replication and Offline backups. Assumption: Both servers are hosting the same data and replicate on a regular basis, or even are part of the same Domino cluster (as shown in the example below). During office hours, both servers are active and users can work on both instances. During the non office hours, Server B is taken offline and is backed up using the offline backup method described earlier. While backup is in progress, ServerB is down. During this time, users can still access Server A because it is part of the same cluster. Users can continue to do their work and changes will replicate back to Server B once Server B is started up again.

121

The challenge in this strategy is to keep both servers synchronized because not every database created on Server A automatically has a replica on Server B. Furthermore, all data must successfully replicate from one server to the other, which for example, is not the case if applications use reader name fields where servers are not a member of. Additional and more advanced options are offered by Enterprise Storage Systems where similar strategies are used. First, the online data is being mirrored to a dedicated storage pool. To perform a backup, the flash copy process is interrupted as long as the backup is in progress. Describing this technology and the different vendor options is beyond the scope of this article. Consult your storage specialist for more details. Note: Lotus Domino currently does not support Microsoft Volume Shadow copying. For further details, refer to:http://www01.ibm.com/support/docview.wss?uid=swg21196479 To understand how Domino clustering works, refer to this old (but valid) IBM Redbooks publication: http://www.redbooks.ibm.com/abstracts/sg245141.html

3.9.3. Backup Software

The backup software you use for your Domino servers depends on the backup infrastructure back-end in your environment. Although there are a variety of certified backup products available for Lotus Domino on the market, we focus on the following IBM products within this article: IBM Tivoli Storage Manager http://www-01.ibm.com/software/tivoli/products/storage-mgr/ IBM Tivoli Storage Manager for Mail, Data Protection for Lotus Domino (further described as TDP) http://www-01.ibm.com/software/tivoli/products/storage-mgrmail/ Which is certified for backing up Lotus Domino data incl. transaction logs

To perform an application level backup of Lotus Domino, the following components are required: IBM Tivoli Storage Manager Server (TSM Server) IBM Tivoli Storage Manager Agent (TSM Client) for static data backup and restore including DAOS TSM for Mail Data Protection for Lotus Domino (TDP) 5.5.2.1. See description below.

122

Tivoli Storage Manager for Mail - Data Protection for Lotus Domino (TDP)
TDP provides the function to do online backups (this means without the need to stop the Domino server) of Domino databases and transaction log files. TDP only acts on databases (*.ns*), templates (*.ntf), and transaction log files (*.txn). TDP also handles the restore of data. TDP communicates with the TSM server via the TSM application program interface - the TSM client. For the communication with the Domino server, TDP uses Lotus Domino API. There are several ways to operate TDP: command line interface, GUI, and integration within Tivoli Manager for Domino. For more information about system requirements of TDP, refer to IBM Technote 1297052. Note: TDP offers no functions for disaster recovery of a Domino server because only the data is backed up and not the program files. To support disaster recovery, TDP and the TSM client must be used in combination.

TDP and the Interdependences with Domino Server Tasks

On a Domino server with transaction logging enabled, Domino assigns a Database Instance Identifier (DBIID) to each Domino database. When Domino records a transaction in the transaction log file, it includes the DBIID. During restore, Domino uses the DBIID to match transactions to databases. There is no relation between the Replica ID and the DBIID. There are two database maintenance activities: COMPACT (all switches beside "-b") FIXUP, which causes the Domino server to assign a new DBIID to a database

From that point forward, all new transactions recorded in the transaction log file use the new DBIID. However, any old transactions still have the old DBIID and no longer match the database's new DBIID. As a result, Domino cannot restore the old transactions to the database. In this scenario, TDP must be used to make a new backup of this database. That is done by the incremental type of TDP backup.

3.9.4. What to back up

The table below shows which data must be included in the Domino backup. The operating system files are not included in this list as they vary between different operating systems.
# 1 Data Program code Description Backup of the Domino program code \*.* 2 Static data Backup of the Domino data directory with all subdirectories and directory links. Backup Utility File backup e.g. TSM client File backup

123

\*.* The following files and directories must be excluded:

e.g. TSM client

o o o o o o o

.ntf (templates) .ns* (databases) .box (mailboxes, SMTP mailboxes) .txn (transaction log files) .txn.dad (transaction log files) .ft\ (full-text index directories) .tmp (temporary files within the Domino Data directory)

Incremental backup

Backup of new Domino databases, databases excluded from transaction logging, and databases changed through a FIXUP or COMPACT program document.

Certified Domino Backup utility, e.g. TDP using incrementa l option Certified Domino Backup utility, e.g. TDP using selective (full backup) Certified Domino Backup utility, e.g. TDP using archive log option File backup e.g. TSM client

Domino database backup

Backup of system databases, templates, mail files and application databases

Transaction log backup

Backup of transaction log files Daily multiple (every 2 hours) for Transaction log files from type archive. No backup for transaction logs from type circular

DAOS data

Backup of DAOS data provided on DAOS drive. According to weekly and monthly full backup

3.9.5. Backup procedures

There are different kinds of backup that are handled by Domino certified backup software: Full backup Incremental backup Selective backup

124

Archive log

Apart from the application level backup, one lower level type of backup must be mentioned: DAOS Backup

We describe these backups in the following sections.

Full Backup
A full backup is a backup that includes all files that can be backed up using Domino certified backup software. Unlike a file based full backup, a Domino certified backup software can backup Domino files while Domino is online.

Incremental Backup
Online backup of complete databases via TDP where at least one of the following conditions have to be fulfilled: The database is not excluded from the backup by rules (include-exclude-lists). The database is enabled to use transaction logging and the DBIID is changed. All database transactions use the DBIID as key which is unique for a database on a server. The DBIID of the transactions in the log must match the DBIID of the database so that the recovery of the transaction logs is possible. The run of the fixup or the compact task changes the DBIID. Only if compact is started with the "-b" option (in-place compaction), the DBIID stays unchanged. The database is not included in the transaction logging and is modified since the last backup. Changes in the database itself or in the access control list are used to determine the backup of that database. The database is new or is just added to the backup list.

Important: The incremental backup function of TDP should not be mixed up with the rules valid for the TSM client's incremental backup, like backing up file if the modification date and time of the file is changed. TDP backs up all databases which fit one of the conditions mentioned above. Both, the incremental backup and the selective backup of TDP are full backups. This means that both backup operations store complete databases.

Selective Backup
The selective backup selects that data out of the data pool of Domino that will be backed up. Use this feature to backup all files that can be backed up using TDP. Leave out the cluster member mail files. This helps saving space.

Archive Log
The archive log with TDP stores filled transaction log files on the TSM server so that space allocated to these files can be reused by the Domino server. The archive log command is available if archival transaction logging is enabled on the Domino server. Filled transaction log files must be archived frequently enough to ensure the transaction log storage space is never allocated completely and stops the Domino server. Transaction log files stored on the TSM server are automatically restored as needed for a

125

database recovery. Archived transaction log files are retained on the TSM server as long as a database backup exists that needs these log files for a complete recovery. All the backup procedures mentioned above can be executed automatically using command line scripts.

DAOS Backup
DAOS backup is done according to the schedules of TDP full backups (weekly and monthly basis). The data should be written to the same management classes that are being used for Domino backup and therefore has the same retention periods. DAOS backup should be done using the TSM Client so it backs up only the files that have been changed (therefore it is incremental). For more information on DAOS backup backup and recovery, refer to this Tivoli Field Guide: http://www-01.ibm.com/support/docview.wss?uid=swg27015114

3.9.6. Backup Scripts

This section represents an example backup schedule for an environment where Domino Data needs to be restorable for 12 months. The following table contains steps to back up various files types, with their data retention settings, the script names, and the generic schedule settings for the scripts on the Domino servers:
# Data Program code Pattern Domino program code *.* Static Domino data files, Domino data directory with subdirectories and directory links. 2 Static data \*.* Exclude: 3 days Included in the daily incremental TSM backup of the OS Retention Script / Schedule Included in the daily incremental TSM backup of the OS

3 days

o o o o o

.ntf .ns* .box .txn .txn.dad

126

o
Incremental backup

.ft\

Domino data files

30 days

incremental.cmd /daily dombackupWK.cmd /

Domino database backup

Domino data files

6 weeks

Weekly (with the exception of the last weekend of each month) dombackupMT.cmd /

Domino database backup

Domino data files

12 months

weekly (the last week-end of each month) archivelog.cmd /

Transaction log backup

Transaction log files

30 days

daily or multiple times per day (every 2 hours) dombackupWK.cmd / dombackupMT.cmd /

DAOS backup weekly

DAOS Files

6 weeks

DAOS backup weekly

DAOS Files

12 months

weekly (the last week-end of each month) inactivatelogs.cmd / weekly included in dombackupWK.cmd / weekly

No backup of files. This command inactivates *.txn on the TSM server.

Note, the steps 6 through 9 are valid only for Domino servers with archival transaction logging enabled.

3.9.7. Recommendations
Here are some tips which can help in any environment: Publish your backup standards so that end users know when and how often a backup is taken. Perform backup restoration tests to ensure valid recovery data. Define what data you store for how long, and carefully think about what happened afterwards. The oldest backup is what you are able to restore. Evaluate if it makes

127

sense to take a yearly backup (acting as an archive version) which is stored in a safe place. Ensure your backup environment is properly monitored and follow up any errors as soon as possible. Document when an error caused a restore to be not available. Work to institute charge-back to end-users for backup resources consumed. This way, you can have your users understand the cost of a good backup infrastructure. Understand the broader backup environment which includes the operating system below of Domino as well as the backup back-end, for example, tape robots, and their schedules etc. Collaborate with other teams. share your knowledge with other people who are not familiar with the Domino platform and how backup is done within.

3.9.8. Summary
Performing Lotus Domino backup need to fulfill requirements similar to other application systems like an SQL database. The most efficient backup concept depends on the size of your environment and the provided backup infrastructure (if any).

3.9.9. Reference Reading

Information provided within this article is based on the following documents: Lotus Knowledge base Technote #7009311"Notes/Domino Best Practices: Backup & Recovery" http://www.ibm.com/support/docview.wss?uid=swg27009311 Lotus knowledge base Technote 1358548 DAOS Backup and Restore http://www-01.ibm.com/support/docview.wss?uid=swg21358548 IBM InfoCenter http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/com.ibm.help.domino.ad min85.doc/H_ABOUT_BACKING_UP_THE_NOTES_SERVER.html Lotus Knowledge base Technote #1196479 Is Microsoft Volume Shadow Copy supported for Domino backups? http://www-01.ibm.com/support/docview.wss?uid=swg21196479 IBM Knowledge Base Technote #1095238 Some considerations on the use of TSM/TDP as a backup solution for Notes/Domino http://www-01.ibm.com/support/docview.wss?rs=0&uid=swg21095238 IBM Redpaper Choosing the right backup strategy for Domino 6 for iSeries http://www.redbooks.ibm.com/abstracts/TIPS0377.html?Open IBM InfoCenter IBM Tivoli Storage Manager for Mail Data Protection for Lotus Domino http://publib.boulder.ibm.com/infocenter/tivihelp/v1r1/topic/com.ibm.itsmfm.doc/dpdo mw.htm

3.10.Restore
Within a Lotus Domino environment, there are different restore scenarios: A full system restore - Which is required to recover from a disaster like a complete hardware or site failure. A restore of one or more database - Replacing the existing one or more databases. Restore of one or more documents within a database.

128

In general, the data restore technology depends on the scenarios. Also, the server type and the backups available are of interest. The following table lists different type of data and the tools involved in the restore process:
Data Type Program files Static Domino data files Domino data files Utility File backup software, e.g.TSM client File backup software, e.g.TSM client Certified Domino Backup utility, e.g. TDP

In addition to the table above, Domino servers where transaction logging is enabled requires:
Data Type Transaction log files DAOS data Utility Your certified Domino Backup utility, e.g. TDP Your file backup software, e.g.TSM client

DAOS enabled transaction logs require a different tool to backup the DAOS storage directory. DAOS stores attachments separately from the database files, and are not backed up with the traditional backup and restore products. Transaction log files use a product external to Domino for backup and restores. Without transaction logging, DAOS and NSF may both use flat file recovery. Three parts of a restore process can be distinguished: Disaster recovery Static file recovery Domino data file recovery

3.10.1. Disaster Recovery

This process is to restore a whole Domino server by recover all files and directories. A prerequisite for this process is to have a fully recovered operating system with all registry and configuration settings. One disaster recovery scenario is to perform a bare-minimum-restore, which means to recover all software to a blank piece of hardware. Here the main challenge is to restore the operating system so that it is fully functional as before. How to do this depends on your servers operating system and your configuration. Consult the operating system vendor as well as your backup software vendor for more detailed instructions. If you can not bring the original server back online, for example, because of an hardware failure or similar problems, consider restoring Lotus Domino to a different hardware. Domino itself is not forced to use the same operating system or the same hardware. Lotus Domino can be moved if you take some items into consideration. IBM Technote 1087009 http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg21087009 describes how to move a Lotus Domino server from one hardware to anther. The same

129

concept can be used in case of a disaster recovery to rebuild the server on a different hardware than before. As soon as operating system and installed operating system applications are recovered, it is time to restore Lotus Domino. For this to be done, tools and processes used for disaster recovery are described in the next two sections.

3.10.2. Static File Recovery

Static files are Domino program files and files in the Domino data without Domino databases and templates. In our example, the TSM client will be used to restore these files. It is possible to restore the most recent backup or a file from a specific date, given that it is still retained within the backup. This restore process is not different than restoring a file from a file server. Because Domino is constantly using some of the files within the program directory, when restoring Static Domino files, consider shutting down Domino before restoring the static files.

3.10.3. Domino Data File Recovery

Individual restores might be requested from users, for example, to restore a particular database (.nsf file) for a specific point in time from the backup environment. Users typically expect to get access to the data in the same way as before. To accomplish different data restore, the following strategies are available: Stand-alone - Provides the restore in the form of a stand-alone application. It is accessible for users. The user needs to find and restore individual Notes documents. Merge - Restore data back into production by merging current data with the restored data. Replace - Replace the current application completely with the .nsf file obtained from the backup.

Each of these options is described in the following sections.

Stand-alone Restore
For a stand-alone restore, Domino Administrators provide the restored data so that the user can get access without impacting the data in production. Although it sounds simple, there are small challenges which administrators need to be aware of for the restored databases: Restored databases are not any different from production databases when it comes to their functionality. Restored databases can and will run scheduled background agents (if not explicitly disabled). Have the same ACL and will therefore immediately allow users to access this file (if not configured otherwise).

130

Restored database can replicate (if not configured otherwise).

For restoring Domino data files, TDP/TSM is used in this example. On Lotus Domino servers without transaction logging enabled in archive mode, TDP recovers one or more designated files from the TSM server. In general, replication will be disabled for this restored database by the restore process (TDP/TSM) automatically, so restoring back to the server where files have been backed up is the preferred method, however there are cases where this is not possible e.g. because an old server has been switched off. The Domino data file recovery is a three-stage process: 1. Restore one or more data files. It is possible to restore the files under a different name, in a different directory or to a different server. (See figure 1 below). Domino data file restore can cover the most recent versions or a specific date for the files to recover. 2. Activate the Domino data files. This function brings restored databases online for use by the Domino server. It is optional to apply transactions from the transaction log to update the database. Transactions can be applied up to a specific point in time or up through the most recent changes recorded in the transaction log. If archival logging is in effect, TDP for Lotus Domino automatically restores archived transaction log files as needed. 3. After transaction log restore is completed and Domino database restore is therefore complete, TDP provides a list of DAOS Files that must be restored to complete the restore. Restore of the DAOS files is done using the TSM Client. TDP restores Domino data files at the database level. To restore single documents, the entire database must be restored. Afterwards the documents can be copied to the original database using the Notes client.

Perform Single Database Restore

To prevent the most common mistakes, figure 1 illustrates a flowchart that provides assistance for defining the most efficient restore method and defining your restore target. Note: The flowchart should be used as a reference only. It should not be mapped onto your environment as-is because every environment is unique and there are many things need to take into account that we might not mentioned here.

131

132

Number 1 2

Comments and considerations Is the database you are trying to restore encrypted? For details see the database properties and encryption settings. On the server where the backup was taken, is (or was) DAOS enabled on this server? Check the server document DAOS tab for details On the server where the backup was taken, is (or was) DAOS encryption enabled? Note: DAOS Encryption is enabled by default unless you have explicitly configured your server to not encrypt DAOS objects (*.NLO files). This can be beneficial if you want to be able to restore back to server other than the restore was taken from. In this situation, you must restore the database back to the server where the backup was taken. The server ID must be the same because the database itself or its DAOS objects are encrypted with this server ID. Restoring to a different server is not possible. Although it is more efficient to restore a database back to its original server, there are environments where corporate policies request the usage of a dedicated restore server. In this scenario, you can if you want - restore this database to a different server. Restore the Domino NSF file(s) using your backup and restore software. Make sure all of the following items are checked before you continue:

Change the replica ID. Some backup software can do that as part of the restore process. If not, you have to do this manually! Disable replication. Some backup software can do that as part of the restore process. If not, refer to IBM Technote 1094568 Disable scheduled agents, by enabling this check box in the database properties of the restored file.

133

(Optional) change the ACL of the restored file so that only the requestor can get access.

Restore DAOS Objects. For more detailed actions, refer to this DominoWiki article http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daosbackup-and-restore#Restoring+DAOS+objects Open the restored database (or request users to do that) and restore the documents requested. If the restored database(s) are no longer required, delete them from the server.

8 9

Merge Restore with Production

Once a database has been restored using the method described above, you might want to merge data restored with the one currently available in production. Clearly, merging a restore with production is NOT a recommended method because documents which have been deleted in the production replica will leave a deletion stub. So replicating a restored version of the same database will NOT restore the deleted document. Although those deletion stubs can be removed, there might be local replicas of the same database for example, on the users desktop which also contain those deletion stubs. Removing deletion stubs as described in IBM Technote 1095683 will not remove them in other replicas. Result: Deletions will be performed again once you clear the replication history.

Replace Production Database with Restore

If you need to replace a production mail file or application database with a restored version of the same file, you can of course delete the production database and store the .nsf file which was restored as a single database restore (see above) to the same folder and file name. Note this will not work if: You are attempting to restore a Domino system database; these files are in use by Domino and often require the ReplicaID to be consistent within a Domain. For technical details, refer to IBM Technote 1099635. As part of the restore process, you have changed the ReplicaID, but some (badly developed) custom applications require a specific ReplicaID to be used. Before changing the replicaID as you want it to have, think twice about the consequences! Any user who got a replica of this file will replicate old content back to the server, including modifications you might not want to have.

When replacing a production database with a restore, evaluate using a different replica for the restored version. This prevents from old or modified content to replicate back into the restore.

134

Optimizing the Restore process

A restore process which is optimized for providing server availability and user satisfaction would run as an half automated process which can look like this: Restore the ,nsf file to a place outside of the Domino Data directory. This will ensure the file would not replicate with other servers. A server side scheduled agent would open the database from this path outside the Domino Data directory and set the DB property Disable background agents in this file." The source code for such an agent can be found in IBM Technote 1380020. As an alternative, disable all scheduled agents as described in the end of IBM Technote 1201461 To put back the database online and change the replica ID at the same time, use the Domino server command Cluster Copy , for example > CL COPY C:\Restore\Filename.nsf RESTORE\filename.nsf This command creates a non-replicating copy of the database C:\Restore\Filename.nsf in the new location Restore\filename.nsf below of the Domino Data directory. For more details about this command and its usage, refer to this DominoWiki article Note: for this command to be available, make sure to set the NOTES.INI variable CLUSTER_ADMIN_ON is set to 1 (If required) change the ACL as needed.

Recommendations
In general, the following recommendations should be kept in mind when performing restores: If you want to use a dedicated server for your restores, consider to disable DAOS Encryption as described in this DominoWiki article http://www10.lotus.com/ldd/dominowiki.nsf/dx/DAOS_Deployment_Guide#Deciding+on+encrypt ion+for+NLO+files Define AND test your emergency restore procedures on a regular basis. When restoring custom developed Domino applications, consult the application developer for specific details on how to restore individual documents without impacting the applications logic. Consider using specific file names which clearly indicate the point in time of a restore. For example, . Restore\2010-11-02\filename2009-12-01.nsf indicating the file filename.nsf was restored on Nov. 2nd 2010 as it was on 1st of December 2009. Feel free to use your preferred method of naming files and folders, but make sure to use a consistent approach as this naming can be used for a cleanup script later on. Delete restored files when they are no longer required. This can be automated with a small script if you define that any restore will be removed (e.g.) 7 days after it was provided. Configure your Domino servers to enable Cluster Administration command line options. For more details, refer to this DominoWiki article.

135

3.11.Procedure to Restore Deleted Documents on IBM i

This article assists you when trying to recover documents that were accidentally deleted within a Domino application or database. It is assumed that you will be restoring a copy of the database prior to the document deletions. The restored database will be placed on the server and the user will then access the restored copy of the application to copy and paste the documents back into the existing database. There are a number of considerations that should be made when performing this type of restore: DBIID: You must ensure that you will not have 2 databases on the server with the same DBIID because multiple databases with the same DBIID could damage your transaction logs. Replication: You need to prevent any replication attempts to the restored database. To accomplish this you must prevent access to the database or ensure the database replica ID is changed during the restore process. Scheduled agents: If the application being restored has agents that normally would run on this server, you want to ensure these agents are disabled as part of the restore process to prevent the agent from running on this temporary file.

Since there are different restore requirements and/or limitations depending on the type of save you performed, figure 1 below and the subsequent explanations walk you through the restore process with these factors already considered for you. The complete steps and commands can be found in the appropriate sections below the diagram.

136

3.11.1. Operating Type Save Restore Procedure

1. Gather the tape(s) from last save of the database/ application. 2. Restore database to a folder outside of the data directory: If you do not have a restore directory already created, create one now. For example: CRTDIR '/Restore' Restore the database to the new folder. For example: RST DEV('/qsys.lib/tap01.devd') OBJ(('/notes/data/mail/mailfile.nsf' *INCLUDE '/restore/mailfile.nsf')) Set permissions so the QNOTES user profile can access the directory and object. For example: CHGOWN OBJ('/Restore') NEWOWN(QNOTES) SUBTREE(*ALL) 3. (Optional) Use an agent to disable scheduled agents from running on the database. A server side scheduled agent would open the database from this path outside the Domino Data directory and set the DB property Disable background agents in this file. The

137

source code for such an agent can be found in IBM Technote 1380020 or as an alternative simply disable all scheduled agents as described in the end of IBM Technote 1201461 4. Create a restore folder in the server data directory. In the Domino Administrator go to the Files tab. In the right hand panel select Folder New.... Give the folder the desired name such as Restore. 5. Use the CL Copy command to copy the database to the Restore folder within the data directory. Using the CL copy command will ensure that the database will have a new DBIID and replica id when placed on the server. The Domino console commands to issue are as follows: set config CLUSTER_ADMIN_ON=1 this will enable the use of the CL Copy command. CL COPY /restore/mailfile.nsf restore/mailfile.nsf Note that the initial entry is the source database with the complete path and the second entry is the destination with a path relative to the data directory. 6. Verify integrity of restored database and DAOS links by running fixup. For example: load fixup -j restore/mailfile.nsf Note: The fixup can be skipped if running 8.5.1 FP2, 8.5.2 or higher and restoring to the same server (refer to SPR DROO7YXTC3). The fixup process is critical when restoring to a different server as the fixup process is needed to update the .nlo file hints stored within the documents of the database. 7. Copy and paste the needed documents into the existing database. If you get an error regarding a missing .nlo file you can get the file name from the error message posted in ddm.nsf or generate a list of the .nlo files needed to be restored using the "listnlo" command. For example: tell daosmgr listnlo -o missing_nlo_files.txt MISSING restore/mailfile.nsf 8. Review output from the "listnlo" command above and restore any files listed. Note that the same file may be listed multiple times in the output, one for each reference to the missing attachment. The syntax for the restore is the same as step #2. For example: RST DEV('/qsys.lib/tap01.devd') OBJ(('/notes/DAOS/0001/AAC48E7FC07B6CC71ADD186896DC4F48509F165801A8 702F.nlo')) 9. Cleanup by remove the restored database. Remove the copy within the data directory using the Delete Database right click menu option in the Domino Administrator Files tab. To delete the file outside of the data directory use the following command: RMVLNK OBJLNK('/restore/mailfile.nsf') The restore directories may be retained for reuse or removed at your discretion.

3.11.2. BRMS Full Save Restore Procedure

1. Gather the tape(s) from last save of the Domino server. 2. Create a restore folder in the server data directory. In the Domino Administrator go to the Files tab. In the right hand panel select Folder New.... Give the folder the desired name such as Restore. 3. Set a directory ACL to prevent all users and servers from accessing the directory. This will prevent replication and thus prevent propagating deletions to this newly restored database. To enable this feature you must set enable_acl_files=1 in your NOTES.INI. To do this, enter the Domino console command set config enable_acl_files=1. You can then return to the Domino Administrator Files tab, right click on the folder and choose Manage Directory ACL. From there choose an administrator to be able to access the directory, but no other users or servers as

138

shown in figure 2.

4. Use RSTBRM, WRKLNKBRM, WRKMEDIBRM, or System i Navigator to restore the database to your restore folder. Here is an example of RSTBRM: RSTBRM DEV(*MEDCLS) OBJ(('/server1/data/mail/mailfile.nsf' *INCLUDE '/server1/data/restore/mailfile.nsf')) 5. Change the Replica ID of the restored database using CL Copy, an agent or 3rd party utility such as Antrid. For information on changing the replica id refer to technote 1094568. Here is an example of using CL Copy: set config CLUSTER_ADMIN_ON=1 this will enable the use of the CL Copy command. CL COPY restore/mailfile.nsf restore/mailfile2.nsf Note that the initial entry is the source database with the complete path and the second entry is the destination with a path relative to the data directory. Remove the copy that the database with the original replica id using the Delete Database right click menu option in the Domino Administrator Files tab. 6. (Optional) Disable any scheduled agents in the restored database manually or via an agent. A server side scheduled agent would open the database from and set the DB property Disable background agents in this file. The source code for such an agent can be found in IBM Technote 1380020 or as an alternative simply disable all scheduled agents as described in the end of IBM Technote 1201461. 7. Remove the directory ACL. To do this return to the Domino Administrator Files tab, right click on the folder and choose Manage Directory ACL. From there remove all entries from the Who should be able to access this directory list. 8. Verify integrity of restored database and DAOS links by running fixup. For example: load fixup -j restore/mailfile2.nsf Note: The fixup can be skipped if running 8.5.1 FP2, 8.5.2 or higher and restoring to the same server (refer to SPR DROO7YXTC3). The fixup process is critical when restoring to a different server as the fixup process is needed to update the .nlo file hints stored within the documents of the database.

139

9. Copy and paste the needed documents into the existing database. If you get an error regarding a missing .nlo file you can get the file name from the error message posted in ddm.nsf or generate a list of the .nlo files needed to be restored using the "listnlo" command. For example: tell daosmgr listnlo -o missing_nlo_files.txt MISSING restore/mailfile2.nsf 10. Review output from the "listnlo" command above and restore any files listed. Note that the same file may be listed multiple times in the output, one for each reference to the missing attachment. The syntax for the restore is the same as step #2. For example: RST DEV('/qsys.lib/tap01.devd') OBJ(('/notes/DAOS/0001/AAC48E7FC07B6CC71ADD186896DC4F48509F165801 A8702F.nlo')) 11. Cleanup by removing the restored database. Remove the copy within the data directory using the Delete Database right click menu option in the Domino Administrator Files tab.

3.11.3. BRMS Incremental Save Restore Procedure

1. Gather tape from the server saves starting with last full save prior to the desired restore date along with each incremental from the full save to the desired restore date of the Domino server. 2. If you need to recover to a point in time between now and last save, run an incremental (transaction logs only) save now. 3. Create a restore folder in the server data directory. In the Domino Administrator go to the Files tab. In the right hand panel select Folder New.... Give the folder the desired name such as Restore. 4. Create the required data area to force the replica ID to be modified during the restore with the following command: CRTDTAARA DTAARA(QUSRNOTES/QNNIDIFFID) TYPE(*CHAR) LEN(1) The data area needs to be owned by QNOTES. Set the owner with the following command: CHGOBJOWN QUSRNOTES/QNNIDIFFID *DTAARA NEWOWN(QNOTES) 5. Use RSTBRM, WRKLNKBRM, WRKMEDIBRM, or System i Navigator to restore the database to your restore folder. Here is an example of RSTBRM: STBRM DEV(*MEDCLS) OBJ(('/server1/data/mail/mailfile.nsf' *INCLUDE '/server1/data/restore/mailfile.nsf')) 6. (Optional) Disable any scheduled agents in the restored database manually or via an agent. A server side scheduled agent would open the database from this path and set the DB property Disable background agents in this file. The source code for such an agent can be found in IBM Technote 1380020 or as an alternative simply disable all scheduled agents as described in the end of IBM Technote 1201461. 7. Verify integrity of restored database and DAOS links by running fixup. For example: load fixup -j restore/mailfile.nsf Note: The fixup can be skipped if running 8.5.1 FP2, 8.5.2 or higher and restoring to the same server (refer to SPR DROO7YXTC3). The fixup process is critical when restoring to a different server as the fixup process is needed to update the .nlo file hints stored within the documents of the database. 8. Copy and paste the needed documents into the existing database. If you get an error regarding a missing .nlo file you can get the file name from the error message posted in ddm.nsf or generate a list of the .nlo files needed to be restored using the "listnlo"

140

command. For example: tell daosmgr listnlo -o missing_nlo_files.txt MISSING restore/mailfile.nsf 9. Review output from the "listnlo" command above and restore any files listed. Note that the same file may be listed multiple times in the output, one for each reference to the missing attachment. The syntax for the restore is the same as step #2. For example: RST DEV('/qsys.lib/tap01.devd') OBJ(('/notes/DAOS/0001/AAC48E7FC07B6CC71ADD186896DC4F48509F165801A 8702F.nlo')) 10. Cleanup by remove the restored database. Remove the copy within the data directory using the Delete Database right click menu option in the Domino Administrator Files tab.

3.11.4. Additional Resources

How to move a Domino server on IBM i from one system to another Summary of Backup and Recovery recommendations when using DAOS on IBM i Lotus Server Backups documentation for Backup Recovery and Media Services (BRMS for IBM i)

3.12.The Domino HTTP Server

Domino is more than a mail server. It can also be a powerful web server. With no additional work or configuration, any document on a Domino server may be viewed by the Lotus Notes client or a browser as the server automatically converts the document to HTML. As a Domino administrator, you want to know how to effectively manage your Domino server on the web. This article provides references to the existing information on Domino web server configuration.

3.12.1. General Server Configuration

Before you start working with the Domino server or if you take over the administration responsibilities for an existing Domino domain, you should be aware of some fundamental concepts.
Domino HTTP Server Security Building Web applications in Domino 6: A tutorial on Web site addressing Building Web applications in Domino 6: Accessing and protecting the file system Building Web applications in Domino 6: Web site rules

3.12.2. iNotes
Lotus iNotes is the method of accessing Domino mail via a web browser and is extremely powerful. There are many resources available for you as the Domino Administrator to assist with configuring and managing iNotes users.

Getting started with Lotus iNotes

Setting Up a Redirection Application for Lotus iNotes Users Securing Lotus iNotes

141

How to rename an iNotes (DWA) user with a Notes ID Notes.ini settings for iNotes Customizing iNotes Knowledge Collection: Directory Assistance and Lotus iNotes/Domino Web Access(DWA)

Lotus iNotes Administration

Adding widgets in iNotes Type-Ahead is on my default starting with iNotes at 8.5.1

iNotes and Sametime

Setting up Lotus iNotes with Sametime iNotes_WA_MacIM=1 required for Sametime integration from iNotes on a Mac (OSX operating systems)

3.12.3. Troubleshooting and Tuning

Tuning Tips for the Domino HTTP Server Troubleshooting Lotus Domino hangs and crashes How to enable HTTP Request logging in Domino Collecting data for HTTP crash on a Lotus Domino server Collecting data for HTTP memory crash on Lotus Domino server

3.12.4. Additional References

As you become more familiar with using Domino as a web server, you may want to start using it for more or new applications. Here are some references to get you started:
XPages Building Domino Web Applications

3.12.5. Some Tips

Internet site: Use IP address and host name. This is not supported with Sametime until version 8.5.1. SSO: Do not change name of LTPAToken.

142

3.13.Domino HTTP Server Security

HideTable

of Contents

If you have secured your Domino data for use with the Lotus Notes client, then your data is also secured when accessed from a browser. However; there are additional considerations after you enabled the Domino web server (http task). This article assists new Domino administrators with basic security recommendations and concepts when granting internet access to your Domino server. Items included are how to enforce server access settings and controlling anonymous access as well as links to great resources for Internet password lockout, SSL and more.

3.13.1. Server Access

The HTTP server honors the database access control list as well as document access such as readers and authors fields. When authenticating via HTTP, a Notes certificate is not required. In order to be able to access Domino information via a browser, the user must have a valid person document with an internet password or token if using single sign on (SSO). The user must also be allowed access to the server. By default, anyone listed in the Domino directory can access the Domino server via HTTP, but this can be controlled via the server access settings. Specifically in the server document, security tab, you can define who can access the Domino server as shown in figure 1.

143

144

By default, these settings are not honored by HTTP. To force HTTP to honor those settings, set Enforce server access settings to Yes in the Ports Internet Ports Web tab of the server document as shown in figure 2. You can also choose whether or not to allow Anonymous access to the server.

3.13.2. User Security and Authentication

To protect the integrity of your users and their passwords, Domino provides you with an "Internet Password Lockout" function. This allows the system to "lock out" a user after a certain number of failed log in attempts. For more information on Internet password lockout refer to the article Securing an IBM Lotus Domino Web server: Using the new Internet lockout feature.

145

The Domino server caches user names and passwords for 2 days. For that reason, you may observe that when the internet password is changed there is a period of time when both the old and new password will be accepted. If this is unacceptable in your organization, you can control the cache by using the NOTES.INI setting HTTP_Pwd_Change_Cache_Hours=<# of hours>. You should be aware that restarting the HTTP task will rebuild the cache and thus cause the server to no longer accept the old password no matter how many hours are specified by HTTP_Pwd_Change_Cache_Hours. If you have multiple web servers, you must also consider your replication topology as it may take some time for the new password to replicate throughout your environment. Many times, you may want users that are not defined directly in your directory (names.nsf) to be able to access data on the web. Alternately, you may have a user directory already configured that is used throughout your enterprise. Domino provides this functionality through directory assistance. For information on directory assistance refer to How to set up Directory Assistance in Domino or 3.4 Multiple Directories. Domino has multiple authentication types. You can choose to enable session authentication to minimize the number of log-in prompts presented to the user at both a single server and multi-server level. Here are some resources related to authentication and single sign on (SSO):
Name-and-password authentication for Web clients Preventing multiple password prompts in Lotus iNotes Webserver Authentication Troubleshooting Session-based authentication (single sign-on) How the Domino HTTP session authentication configuration affects which login prompt is sent to Web browsers Deploying Windows single sign-on for Web clients (SPNEGO) in an existing Domino environment DWA with Sametime integration Hints and Tips for Troubleshooting Single Sign-on and Authentication Issues with Domino and WebSphere Troubleshooting WebSphere Portal, Domino Extended Products, and Domino SSO issues

3.13.3. Database Security

When looking into Database security, there are typically several questions Domino administrators ask. Can anonymous users access the data? Is there a way to force SSL access to this data? How do I control what page is displayed when the database is opened? Can I prevent certain databases from being viewed with a browser? You will now learn how to answer these questions.

Anonymous Access to Domino data

For additional protection, Domino has an additional level of security within the Access Control List (ACL) of databases when accessing using a web based protocol such as HTTP, DIIOP, POP3 or IMAP. Any Domino database located on a server running the HTTP task should have an anonymous entry in the ACL. While you can disable anonymous access at the server level as seen above in figure 2, it is best practice to add an anonymous entry to the ACL of each database on the server. If anonymous is not present in the ACL, then the Default access will be granted to anonymous users. Add

146

the anonymous entry is especially important on mail databases to prevent anonymous users from accessing public calendar documents as many users allow or delegate read access to their calendar to everyone. You can also set a maximum authority value when accessing data from the internet. For example, if you have an application you want visible from the web, but do not want anyone to edit the data from the web, you could set the Maximum Internet name and password to reader. This means that when anyone access the Domino database using one of the web based protocols, they will only be granted reader access, even if they are specifically listed in the ACL of the database with greater access. For mail files, the recommended Maximum Internet name and password setting is editor. The Domino Administrator client makes it easy to modify the ACL of many databases at once. For example, to modify the ACL for all files in the mail directory, you can right click on the folder and select Access Control Manage. The Manage Multiple ACLs window will display. You can then see that at the top of the screen how many databases you will be modifying. From there you can use the Add button and enter the value of anonymous with an access level of No Access. Once added you should see anonymous listed in the Apply these changes to all X databases as seen in figure 3.

147

You can then select the Advanced tab to modify the Maximum Internet name and password setting to Editor as shown in figure 4.

Once you click OK, the client will then connection and modify the ACL for any selected database that you as the administrator have authority to modify or all databases if using full access administration authority. When finished, the client will tell you if the process completed successfully or if there were any errors as seen in figure 5.

It is important to note that any database that already contains an entry for anonymous will be listed in error. You can review the log or the status bar to see why an error occurred. See figure 6.

If this happens to you, you can run the tool again this type changing the anonymous

148

entry. This way you can be certain that all databases are set to no access without verifying each database.

SSL Access to Domino data

As an administrator you can force SSL access at the server level. For information on configuring SSL refer to the technote How to set up SSL using a third-party Certificate Authority (CA) or the Redbooks publication Domino Certification Authority and SSL Certificates. However; most companies have public data that does not need to be secured with SSL and private data that must be secured. In order to satisfy both requirements you must force SSL connections at the database level. If you do want to force SSL at the server level, you can do that by simply setting the TCP/IP port status to Redirect to SSL as shown in figure 7.

To force SSL at the database level you need to set the database property Require SSL connection. This is found on the basics tab as shown in figure 8. With this property enabled, if a user attempts to access the database without SSL, they will be automatically redirected to a secure connection. If you are planning to set this field on your mail files, it is best to reference the users to the mail file with a secure connection using the iNotes redirect application. Otherwise, the mail may not properly load for you due to the different connection type between the mail file and the forms85.nsf. Forms85.nsf access is needed in order for iNotes to display properly.

149

For more information about SSL refer to technote Frequently Asked Questions: Using Secure Socket Layer (SSL) with Notes/Domino.

Additional Database Properties

There are other database properties that affect how your database can be accessed from the web including:
Use JavaScript when generating pages Don't allow URL open Enable enhanced HTML generation When opened in a browser / Database launch properties

3.13.4. File System Security

Do you have sensitive data or pieces of your web application within the data directory of your Domino server? If so, the Domino server can access it and thus so can a user with a browser. To be sure your files are safe, review the article Building Web applications in Domino 6: Accessing and protecting the file system which is still technically accurate for Domino 8.5.x environments.

150

3.14.Setting up a Redirection Application for Lotus iNotes users

Do you currently use Lotus iNotes or are you considering doing so? If so, the topic of a redirect application has probably come up. Domino provides a redirect application to allow users to be routed to their mail by simply entering their user id and password. This article describes the steps required to create an iNotes redirect database and make it the default home page for the web site. To help you understand the process of creating and implementing this application, you will see a simple case study and Fictional Software Company A. The management team at Company A has decided to roll out iNotes for all of their users. They have decided that they would like users to be able to enter a URL of mail.companya.com and then the user should be able to enter their user id and password and get directly into their mail. Here are the steps the Domino administrator has taken to implement this request.

3.14.1. Create the iNotes Redirect Application

The first step is to create the iNotes redirect application. You can do this from the Lotus Notes or Domino Administrator client (File Application New). When creating the redirect application, you will first enter your desired Title and File name. You then will then select to create the application based on the IBM Lotus iNotes Redirect template as shown in figure 1.

151

3.14.2. Configure the iNotes Redirect Application

Once the redirect application has been created, you will be prompted with a Setup button which you can click to get started with the configuration. From there you will be presented with four options as shown in figure 2.

Server Settings
To get started with the configuration, click on Server Settings. The first choice to be made is the Redirection Type. The value to choose here is dependent on your environment. Use the table below to help you choose appropriate values for the Redirection type and associated settings. In the example for Company A, they will be using a Redirection type of Mail Server with TCP/IP domain of companya.com. Since Company A does not use a proxy server or a specific folder for their mail files, those settings will remain blank.
Scenario Multiple Servers running iNotes and users should be redirected to mail file located on their home mail server. Settings

Redirection Type of Mail Server Proper TCP/IP domain for the mail server should be specified, for example companya.com

Multiple Servers running iNotes, but all mail files have been replicated on one server located in a DMZ. All users should be redirected to the copy of the mail file on the DMZ server located in a folder called iNotes.

Redirection Type of Fixed Server Name to use should be set to the hostname of the DMZ server, for example dmz.companya.com Force path should be set to iNotes

Multiple mail servers and replica copies of the mail files. Users should be redirected to the replica copy of the mail file located on the server that they are currently connected to. Single mail server environment. Users should be redirected to their mail file on this server.

Redirection Type of Dynamic

Any redirection type can be used in this environment.

Redirect database exists on an application server, but users should be redirected to the appropriate mail server

Redirection Type of Mail Server Proper domain for the mail server should be specified, for example companya.com

The next decision to be made is SSL. At company A, the decision has been made to force SSL during sign-in, but not use an SSL connection when accessing mail. Thus they

152

have set "Do you wish to force SSL for the entire session?" to No, but set "Do you wish to force SSL only on authentication" to Yes. In the example provided here, the default SSL port of 443 is being used so that setting has not been changed. Figure 3 shows all of the server settings chosen by Company A. If you need to be sure SSL is used for authentication and for iNotes access you can change the "Do you wish to force SSL for the entire session" parameter to yes.

153

UI Settings
The next group of settings to be configured are the UI Settings. For the most part the UI settings are self-explanatory. In the example shown in figure 4, Company A decided to display their company logo on the standard redirect page. The redirect screen will be shown for 4 seconds to allow the user to change the log in options. You will see an example of the Enable Personal Options and Enable Login Options below.

154

When the user accesses the redirect database using their browser, they will see the screen like the one seen in figure 5 for 4 seconds. Since Company A has chosen to display their logo rather than the Lotus iNotes logo you can see that in the example.

If you choose Yes for the Enable Personal Options setting the Personal Optionslink will be displayed as shown above in figure 5. Once the user selects this option, they will see the screen shown in figure 6. Note that in order to allow the user change their log in options, you need to set Yes to Enable Login Options, you must also have Yes for the Enable Personal Options. In the Personal Options screen, there are two parameters Alternative Mail File Display Name and Default View. The alternative mail file display name can be used to enter another user name. In figure 6 you can see that User One has logged in and selected to change the personal options. If User One has access to User Two's mail, then User One could enter User Two in the Alternative Mail File Display Name field and then User Two's mail file would be displayed rather than User One's mail file after the redirection is complete. The Default View parameter can be used to switch between the different iNotes formats or home pages. In this case User One is choosing to use the Full mode. Options selected here are remembered and used the next time the user logs in.

155

Ultra-light/Mobile Settings
The Ultra-light/Mobile Settings configuration within the iNotes redirect database is very simple. You can determine whether or not your users can select the ultra-light mode and determine which mobile devices should be automatically redirected to the ultra-light interface. Within Company A they have rolled out iNotes ultra-light and thus it has been enabled as shown in figure 7.

Application Setup
After you have successfully chosen your settings the last step is to access the Application Setup option in the iNotes redirect application. Click on Click to Auto Set ACL Settings to properly configure the ACL for your redirect application (reference figure 8).

Once the ACL has been properly configured you will get a message confirming the changes as shown in figure 9.

You are now ready to begin using the iNotes redirect application.

3.14.3. Configuring the iNotes Redirect Application as the Default Home Page
Once you have configured, you may wish to have the iNotes redirect application be the default home page for your mail server. To do this, you will need to modify the default home page setting in the server document for the server or in the internet site document (if you use internet sites in your environment).

156

Setting the Default Home Page in a Server Document

If you are not using internet sites, you will need to set the default home page in the server document. To access the server document in the Document Administrator client, go to the Configuration tab. From this tab select Server All Server Documents. You can then open the document for the appropriate server. To do this, you will access the Internet Protocols... HTTP tab. Set the Home URL: to /<name of your redirect application ?Open. For company A, the correct value would be /iNotesredirect.nsf?Open as shown in figure 10.

Setting the Default Home Page in an Internet Sites Document

When using internet site documents, each site has its own home URL. Thus, the home URL must be defined within the internet site document. To access the internet site document in the Document Administrator client go to the Configuration tab. From this tab select Web Internet Sites. Within the site document, access the Configuration tab and define the Home URL. For example, set the home URL to /iNotesRedirect.nsf?Open

You have now seen how easy it is to implement the iNotes redirect application and set it as the default home page for your server.

157

3.15.Securing Lotus iNotes

There are a number of security topics that are specific to iNotes, including Notes ID files, browser cache management, encrypting offline databases, and encrypted mails (S/MIME). This article addresses these topics and covers the key points for these topics.

3.15.1. iNotes and the Notes ID files

For authentication purposes an ID file is not required when using Lotus iNotes. However; there are some functions that require an ID file including offline access to iNotes, recalling messages, and sending and receiving encrypted mail (S/MIME). When you register a user, you have the ability to have the ID file automatically stored in the user's mail file. Starting at 8.5.1, the password in the stored ID file can be synchronized using the ID Vault functionality of Domino. You can easily see if an ID file is stored within the mail file by accessing the security preferences within iNotes as shown in figure 1.

For additional information on synchronizing passwords or enforcing a custom password policy, refer to:
Can I synchronize a Notes ID password change with an imported Notes ID in a DWA mail file? How to implement a Custom Password Policy for DWA Users

3.15.2. Active X Controls

ActiveX controls provide a way to create distributed applications that work over the internet through the Internet Explorer browser. In Domino 8.5.x, there are several iNotes functions that are implemented via an ActiveX control including: ability to drag and drop attachments, ability to select multiple files when uploading and downloading files and browser cache cleanup. In order for an ActiveX control to install, the user must have Administrator or Power User authority on their workstation. They must also allow the installation of ActiveX controls within their browser. As an administrator you can control whether or not these controls will be used by enabling or disabling them at the server level in the configuration document as seen in figure 2 or for specific users in a mail policy.

158

3.15.3. Browser Cache Management

Browser cache manage is a way to define which temporary files stored on the PC during iNotes access should be cleaned up after the user exits iNotes. For more information refer to What is Browser Cache Management and how is it installed? Frequently Asked Questions regarding Browser Cache Management: Question: What happens when you log out of iNotes without the browser cache management feature enabled? Answer: The browser cache will be cleared when clicking logout, regardless if the Browser Cache Management feature is enabled. The benefit of Browser Cache Management is that the browser cache will be cleared without having to click logout. Just closing the browser, for example, will clear the browser cache. Question: Is there any way to force the user to restart their browser after automatically installing browser cache management? Answer: iNotes is a web based application and therefore must comply with the limitations of the browser. iNotes does not have the ability to force the browser to close.

159

Question: What does it mean by clear history when the browser window is closed? Answer: This refers to clearing the temporary files available in your web browser. The files are deleted from the appropriate directory. You can set the cache scrubbing level to remove all cache entries or only those related to the user's mail file. Since iNotes is a web application, it has limitations on what it can and cannot do. When it clears the browser cache, it deletes these files from the directory, the same as if you browsed to the directory and deleted them via Windows explorer. It is important to note that other than attachments, user's data is not stored on the hard drive, only design data to improve iNotes performance. Question: How does the attachment gets handle when you just read it and detach it? How does it remove from hard-drive? Answer: The same procedure applies: The file is deleted the same as if you browsed to the location and deleted it manually.

3.15.4. Encrypting Offline Databases

One thing great about iNotes is that if your server is accessible from the internet, then you can easily access your mail anywhere. Lotus iNotes also allow users to access their mail offline (requires DOLS). When a user configures an offline subscription, a local copy of their mail file is created. This can be a concern knowing that users could use a shared computer. One way to minimize this concern is to force a user to encrypt their mail file when going offline. This can be easily accomplished in the configuration document for your server. In the example shown in figure 3, the Domino Administrator defined that Medium encryption should be used and the user cannot change this setting.

160

3.15.5. S/MIME
Lotus iNotes fully supports sending and receiving encrypted messages using secure MIME (S/MIME). There are 2 requirements for this functionality. First, the user must access their mail via a secure connection (SSL). Secondly, a Lotus Notes id file must be stored within the mail file. Sending a secure message within iNotes is very simple. If the requirements have been met, the user can simply check the Encrypt option before sending the message.

If the recipient of a secure message does not have an ID file stored in their mail file, a warning message is displayed as shown in figure 5.

As you can see in the example, the error states that the body of the message is encrypted. This is an important thing for you and your users to understand. The subject of the message is not encrypted so all confidential information should be kept in the body of the message.

161

Part 4.

Tuning the Environment

4.1. Health Check

The purpose of a Health Check is to perform a reasonably deep analysis of the Domino server configuration and underlying services it requires (such as hardware resource, disk and network). A thorough health check aims to identify where best practices could be implemented. It should also uncover any vulnerabilities that may compromise integrity, availability or confidentiality of the information served by Domino. This article provides Administrators with a checklist to help them get started with performing a Health Check. Additionally, certain points are explorer in greater depth. A health check compromises three key elements: 1. Inputs (Gathered by reviewing hardware and software configurations) 2. Analysis (A review of the inputs, in the context of your Domino environment) 3. Recommendations (Actions identified from the analysis phase)

4.1.1. High Level Checklist for Health Check

If you cover all these points, you will have performed a comprehensive health check: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. Inspect Server Hardware Inspect Server Configuration Inspect Domino Mail Routing Inspect SMTP Mail Routing Inspect Domino Replication Inspect Directory Services Inspect Domino Clustering Inspect Domino Security of Core Databases (names, admin4, events4) Inspect Domino Server Topology Inspect NOTES.INI files Inspect Server Logs Inspect Server Statistics Inspect Server Documents Inspect Connection Documents Inspect Server Configuration Documents Inspect Program Documents Inspect Active Server Tasks Report on Disaster Recovery Procedures Report on Backup Procedures Report on Number of Databases and Size Report on Database Activity Report on Agents and Runtime

162

4.1.2. Performing the Health Check

A health check is more than just looking at your environment. A good health check requires that you document your findings to be reviewed with your team or management. It is also a way to track your environment health. Throughout the rest of this document you will see not just the steps to perform the check, but guidelines on what data your report should include.

Environment Diagrams
In order to provide some context to the health check it is useful to include high-level architecture diagrams. This will assist the reader to understand the key components that make up the Domino environment. Diagrams should succinctly describe the following information to the reader: The number and physical location of Domino servers IP addresses/host names Domino clusters The client types accessing the environment The network environment External access (from the internet) Mail routing topology Replication topology

For additional information refer to LINK to Doc topic

Current Client Environment

Use this section to get a picture of the user base that the environment serves. Describe the approximate number of users and their access methods (Notes client, browser, mobile device etc.). Person documents inside the Domino Directory often contain a list of Notes client versions used to access the server if you are unsure which client types are in use. This information is stored in the Administration tab. You can also enable license tracking to give you an easier way of reviewing this information.

Domino Server Builds

This section of your health check is similar to an inventory. It lists the Domino servers in the environment together with their role, host operating system and version (including any hot fixes). Take note of mixed version environments, especially within across members. Best practice suggests that cluster members should all be at the same Domino version (e.g. 8.5.2).

Hardware
Your health check should include an overview of the current hardware being used. Monitor key system resources over an extended period of time. You should aim to monitor the systems for weeks or months. This will help to iron out any irregular variation. Reporting on resource utilization has the following two benefits: Ascertain a steady-state baseline of resource utilization

163

Help identify any pattern of spikes in resource utilization that can have an adverse effect on the service. Help plan when additional hardware or a server migration may be required

Operating tools such are useful. As a minimum, the following attributes should be measured: Total CPU utilization Total Memory (RAM) utilization Average Disk Queue Length Network utilization

The tools used and the exact statistics will vary slightly between platforms. Below is an example using Windows Performance Monitor (Perfmon.exe). Figure 1 depicts a healthy server in terms of the resources outlined above. CPU utilization is typically low with spikes not exceeding about 50% during busy periods. There is plenty of free memory throughout the monitoring period. The average disk queue length is typically under 2, except occasional spikes.

Figure 1: Healthy server

164

Figure 2 depicts a server that reaches its CPU capacity and has most of its available memory utilized (up to 70%).

Figure 2: CPU capacity reaching peak

Figure 3 depicts a server where the average disk queue is very high, peaking at 80.

Figure 3: Average Disk Queue Length too high.

165

Disk Subsystem
The disk subsystem is a key component in the Domino server configuration. Various factors such as the location of files, the RAID level and free space should not be overlooked.

Disk Layout
Best practice suggests you should have separate disk volumes (separate spindles) for the following components: 1. 2. 3. 4. 5. 6. 7. Operating System Domino binaries (Program Code) Domino data directory DAOS Repository Transaction Log drive View Rebuild drive Swap Drive

Note: On IBM i the operating system automatically manages your drives through its single level storage capabilities. The operating system and all Domino components can reside in the primary auxiliary storage pool (asp) with no negative performance side affects. Using the primary asp is the default and recommended configuration for this platform.

Free Space
As a rule of thumb, maintain at least 20% free disk space to reduce the amount of file fragmentation. Performance usually degrades as a system runs out of disk space. If there is no available space remaining this can lead to a server crashing or panic.

RAID Level
RAID at the OS/Software level is not recommended. Hardware-based RAID controllers should be used in production Domino servers. There is always a balance to achieve between the disk IO performance, redundancy and usable disk space gained from each RAID level. Figure 4 lists the most appropriate RAID level for each Domino component. Note: On IBM i/5, RAID level 5 or 6 is suitable for all volumes.
Domino Component Operating System Domino Binaries Domino Data DAOS Repository Transaction Logs View Rebuild Temporary Files Windows/Linux/AIX RAID 1 RAID 1 RAID 10 or RAID 5 RAID 5 RAID 1 or RAID 10 RAID 1

166

O/S Swap drive Figure 4: RAID Levels

RAID 1

Furthermore it is best practice to use a separate RAID controller dedicated to the transaction log drive.

View Rebuild Drive

Dedicate a RAID 1 array to store temporary files used to perform view index rebuilds. This can improve performance especially during busy times when many users open their Inbox for the first time. IBM Technote #1090462 documents the procedure in more detail. Practice suggests that the rebuild drive is configured to use a dedicated RAID 1 array composed of two dedicated solid-state disk drives or physical disk drives. Note: For IBM i/5, the view rebuild directory may be in the primary ASP, but can be located outside of the server's data directory for increased performance.

Transaction Logging
During a health check a review of transaction logging configuration should be done. In most cases, Transaction Logging can safely be enabled in order to reap the benefits. An excellent guide to Transaction Logging best practices is available in IBM Technote #7009309. Transaction Logging configuration should differ according to several factors. These include: Available Server Disk Layout Available disk space Domino Server Usage (Sametime server or Mail server etc.) Type of backup strategy and backup tool availability

This topic is discussed further in the Transaction Logging section of this wiki.

Server NOTES.INI
To determine the health of your notes.ini consider the following: The server NOTES.INI configuration file contains settings that can modify default behavior and be used to tune the server. Care should be taken when modifying the NOTES.INI. Always take a backup and document any changes in your change management process. Directly modifying the NOTES.INI file can lead to mistakes. An accidental or incorrect change may cause Domino or Notes to run unpredictably. Therefore setting NOTES.INI parameters in the server configuration document or using the set config console command is safer. A text comparison tool is useful to highlight differences in NOTES.INI files. Look for consistency across server roles (e.g. mail, hub or application servers), and especially across cluster members.

167

The IBM Lotus Notes and Domino wiki is a good place to get familiar with current NOTES.INI parameters and their uses. The NOTES.INI file tends to contain more obsolete parameters on servers that have undergone upgrades from earlier Domino releases. NOTES.INI parameters typically become obsolete because its function is superseded by a UI setting (in the server configuration document, or server document itself).

An example is MAILCLUSTERFAILOVER=1. This parameter was superseded by a setting in the Router/SMTP tab of the server Configuration document in Domino R5. While the NOTES.INI parameter will not overwrite the setting in the Configuration document, it can lead Administrators to think its set when its really not. For NOTES.INI parameters obsolete in Domino 7, read IBM technote 1207338. For NOTES.INI parameters obsolete in Domino 8, read IBM technote 1327806.

Redundant Tasks
The ServerTasks parameter specifies tasks that begin automatically when the Domino server starts. These tasks consume memory and CPU so it is important to ensure only tasks necessary to the server's role are included. An example of a redundant tasks is the Rooms & Resource Manager (RNRMGR) running on a Sametime community server or SMTP running on an administration server. As part of a good health check you should ensure that all running tasks are still needed in the current environment.

Domino Configuration Tuner

In a nutshell, the DCT examines server configurations and compares them with an extensive set of best practice rules. It allows administrators to quickly and easily analyze an entire Domino domain and identify any parameters that are known to cause issues. The DCT is explored in more depth in 4.2 Document Configuration Tuner (DCT) . As part of any health check you should use the DCT tool and document the results and recommended actions.

Database Size
The convenience and versatility of electronic mail makes it somewhat like an ever growing file cabinet (or attic). Large mail files, especially ones that contain a large number of documents, have a negative impact in several ways. For example: They rapidly consume server disk space, especially when they are replicated to multiple servers. Database view indexes consume more disk space, consume more CPU time and take longer to update. The Inbox (and other folders) require more time to update and open. Full-text indexes are larger and take more server resources to maintain. Backups and restores take more time to complete. Retaining old documents may violate document retention policies. Having many large files open simultaneously can exhaust server resources, especially on Windows.

For a discussion of large Domino mail files, see the paper entitled How Large Databases Uniquely Affect IBM Lotus Domino Server Performance.

168

There are a variety of options available to Domino Administrators that help control the size of user's mail files. Firstly there are database size quotas, and the ability to withhold mail delivery from any mail file that is over its quota. Additionally, the size of individual messages that can be delivered by the router can be limited. Other advanced database properties discussed elsewhere (such as compression) can be effective at reducing the overall database size. These topics are covered in more detail in 2.2 Managing a User's Inbox. For best performance, keep as few documents in the Inbox as possible. As a rule of thumb, over 1000 documents is excessive. The Inbox Maintenance feature can be enabled can be run periodically to move documents out of users' inbox and into another folder.

Ports and Network configuration

As part of a health check you should review your network configuration including the ports defined on your server. Here are a list of tips to consider and document as part of your health check: The Domino server Ports configuration is defined by NOTES.INI parameters. The Ports parameter should contain all enabled ports defined on the server. The order is also important as the first port listed denotes which should be used for cluster communication. Port compression allows network traffic to be compressed before being sent to its destination. This may be either another Domino server, or a Lotus Notes client. Use the Administration client (Server -> Status tab, Tools -> Ports -> Setup) to enable or disable port compression. Do not enable port compression if your network switches and routers already compress traffic automatically as you there is a CPU overhead involved in compressing and decompressing the traffic. Best practice suggests a dedicated NIC and dedicated high speed network for Domino clustering to provide the fastest throughput. By using a dedicated network for cluster replication you offload the cluster's probe and replication network traffic from the user-access LAN, leaving more bandwidth for client communication with the cluster servers. This also provides a level of redundancy in the event of a network failure. For further information, read Configuring a Private LAN for a Domino Cluster.

Figure 6 is an extract from the server NOTES.INI of a cluster member that has the Server_Cluster_Default_Port parameter set. With this configuration, the cluster replication (clrepl) task only uses this defined port. If this port fails, the clrepl task does not fail over to any other port.
Ports=TCPIP_CLU,TCPIP TCPIP=TCP, 0, 15, 0,,32 SERVER_CLUSTER_DEFAULT_PORT=TCPIP_CLU TCPIP_CLU_TCPIPADDRESS=0,10.172.99.134:1352 TCPIP_TCPIPADDRESS=0,10.172.99.100:1352 Figure 6: Cluster configuration extract from NOTES.INI

169

The Server_Cluster_Auxiliary_Ports NOTES.INI parameter will allow cluster replication to fail over to the other available ports, even when using the Server_Cluster_Default_Port= parameter. Its use is documented in IBM technote#1259288. In this case, add the parameter like so: Server_Cluster_Auxiliary_Ports=TCPIP

Verify that all servers are configured to use the same line speed and duplex options as the switch to which they are attached to avoid potential network performance issues.

Replication Topology
As part of your health check your replication topology should be reviewed and analyzed. Consider the following: If you are using a hub & spoke topology, best practices suggests initiating replication from the hub. This is to maximize the amount of resource available on the spoke servers to server client requests. Best practice suggests the number of replicator tasks should equal to the number of spoke servers with which the hub replicates. However, you should not exceed 20 replicators to avoid putting too much load on the server. If the server you intended to replicate is not a hub server, the recommended number of replicators should equal the number of processors on the server. How often are you currently replicating? Is one replication event completing before the next event begins? How long does it take changes to be completely propogated in your environment. Is this timing still acceptable or have business conditions or requirements changed?

Mail Routing
As part of your health check your mail routing topology should be reviewed and analyzed. Consider the following: Verify the number of mail boxes on each Domino server. This setting is found in the Configuration document, Router/SMTP Basics tab. Follow the guidance in Determining how many mail.box databases to place on a server to verify the optimum number of mail boxes. Is SPAM currently under control in your environment? Do you need to make changes to your SMTP inbound controls or consider using a 3rd party service or product to better manage SPAM in your environment. Is dead mail accumulating in your mail boxes?

170

Server Security
In a complete health check security should also be considered. Here are a few tips. For a complete list refer to 1.3 Security checklist. The Security tab of the Server Document contains fields that control server access and permissions. Organisations should have controls in place to ensure people are given the minimum amount of access necessary to perform a task. For instance, Database Administrators do not need Full Administrator Access to the Domino Domain. User and Server IDs can be attached to respective documents in the Domino Directory, however best practice is to securely store them outside of the NAMES.NSF and use features such as Certificate Authority and ID Vault. Rather than grant default ACL access to the Domino Directory, Administrators can set Default to No Access, then grant just the appropriate certifiers. For example, */ORG to Reader.

Directory Assistance
As part of a health check you should review your directory architecture. For information on using multiple directories, see 3.4 Multiple Directories. Directory assistance is a feature that enables a server to look up information in a directory other than a local primary Domino Directory (NAMES.NSF). You can configure directory assistance to use either a remote LDAP or another Domino Directory. Secondary Domino Directory referrals should be configured to use a local replica to the server for best performance. Administrators should create replicas of additional Domino Directories referred, on all servers where Directory Assistance is enabled. Also consider the following: Are additional secondary directories required? Are all directories currently being used still needed? Should a mobile directory catalog or extended directory catalog be created? Is a mobile directory catalog being used on a server instead of the recommended extended directory catalog architecture?

Person Documents
As part of your health check, a review of the person documents is recommended. For example, field validation for the Person document does not specify that the Internet Address field be complete. Mail routing from external senders can still occur if an internet address is stored in the User Name field. To help prevent against incorrect mail routing, all members with a Domino mail box should contain a valid external email address in the internet address field.

Policies
Policies are a powerful tool that enables Administrators to control many user's client settings. These can be used to simply set client options (such as spell-check before sending e-mails). They can also be used to enforce company policy (such as e-mail message disclaimers).

171

As part of your health check review which policies are in force. Also check policy rules for any contradictory policies that might be applied to users. It is good practice to have at least a Desktop, Security and Registration policy: Desktop Policy to help standardize the Notes client. Benefits include a reduction in support calls and easier troubleshooting. Security Policy to enforce password for compliance in-line with corporate instructions. Registration Policy to standardize creation of new users and their accounts. For example, ensure all new users have Editor access to their mail file (rather than Manager).

The topic of Policies is discussed in more detail in this wiki. See 2.3 Policies.

Backup and Restore Procedures

As part of your health check you should review and document, if it has not been done already, your backup and restore procedures. A clearly defined backup procedure is vital for any business. The backup schedule for Domino environments will depend on whether Archived Transaction Logging is enabled. If the Transaction Logging type is circular logging, or is disabled, then daily full backups of the Domino data (NSF / NTF) is most likely required. If the transaction logs are backed up with a third-party backup tool, then the schedule must accommodate this. Transaction logs should be backed up frequently. Each evening an incremental backup captures the Domino data. A full backup should still be taken once a week. Equally important as the backup procedure is the restore procedure. Whereas the backups tend to be automated, a restore is usually performed by the technical operations group or Domino Administrators. It is therefore useful to perform periodic dry-runs, to ensure databases are restored as expected and the procedure can be performed smoothly when the business demands it.

4.2. Document Configuration Tuner (DCT)

The Domino Configuration Tuner (DCT) is just one of many tools that are included with the Domino 8.5.x server and Administration client. It can also be downloaded for free from this site: http://www-01.ibm.com/support/docview.wss?rs=463&uid=swg24019358 DCT provides easy-to-use self-service configuration analysis by Domino Administrators. Its goals are to identify server document and NOTES.INI configuration parameters that could be modified for more robust and better performance of Domino servers. DCT will analyze your configuration against a set of best practice rules identified by IBM. Furthermore, it warns of worst practice configurations if discovered. Configuration settings are flagged when their values are know to cause problems based on Technotes, Redbooks/Redpapers, internal knowledge from IBM development and support and prior customer experience. Out-of-range and unexpected values are reported so that undefined behavior can be prevented. Links to supporting to documentation are also provided.

172

DCT rules are tuned for Lotus Domino server 7.0 and later (although it can analyze any version of Domino) and requires the Lotus Notes client, standard or basic, version 8 and later. Inside the DCT tool is a button that installs the latest updates to the DCT template and downloads the latest rules. Rules are typically updated each month and up-to-date details of the current additions can be found on the Tuner Blog. No configuration changes to the production deployment are required to perform a DCT evaluation. A full analysis exhibits a negligible load on the environment, even in business hours. Therefore analysis can be performed at any time. Most of the rules that are evaluated have a wiki post in the Notes/Domino wiki and DCT will point to that wiki posting in your report. It is recommended for you to take advantage of DCT to feedback your results, especially if your experience does not match the recommendation in the findings. The Notes/Domino wiki entries provide a good place to do so. For further reading, visit the Domino Configuration Tuner entry in the Notes/Domino wiki.

4.3. Establishing a Performance Baseline

Having a set of historical baseline data gives Administrators a reference point to measure against when changes are made to the environment. Establishing the baseline also gives Administrators an opportunity to identify obvious performance bottlenecks or trends. For example, the CPU may reach peak levels every Monday morning or disk queue lengths may exceed normal operating boundaries during the same periods each week. To help capture accurate baseline historical data, follow this set of best practices: Capture a complete view of the infrastructures performance: Kernel statistics (such as CPU, memory and disk utilization).

173

Periodically capture the information to get long term trend analysis. Do not rely just on a quick snapshot of a system before a tuning event take place. It should be representative of normal system loads. Do not record or take inconsideration of data taken during extra-ordinary events (periods of unusually quiet activity, such as the New Year holiday period). Do not include data from periods with unusual events (such as system failures or high usage volumes). Record at a set period of time (For example, at least once a week, and preferably at least once a month). Include descriptions of what components or services were running on individual servers (ServerTasks, Backup or Anti-virus facilities) at the time of capturing the data.

After every major tuning changes or system upgrades (when the system is stabilized), take a new set of baseline statistics.

4.3.1. Recommended Metrics

The following metrics provide a basis for establishing baseline statistics. With experience, you will find other metrics useful or more applicable for the type of tuning performed.

Perfmon
Memory.Available MBytes Processor.% Processor Time LogicalDisk Avg. Disk Seconds Read LogicalDisk Avg. Disk Seconds Write LogicalDisk.Avg. Queue Length.

- o/s drive - Notesdata - tx log - DAOS - view rebuild temp. drive - swap

Events4 / Statrep
Server.AvailabilityIndex Server.Users Replica.Cluster.SecondsOnQueue.Avg Replica.Cluster.WorkQueueDepth.Avg

Note. The Replica.Cluster statistics are only relevant if the server being monitored is a member of a Domino cluster. They give an indication of the ability of the cluster to maintain synchronized replicas. For a comprehensive guide to Domino statistics and their definition, read Lotus Domino Statistics.

174

4.3.2. Collecting Domino Statistics

Administrators can use the COLLECT and EVENT server tasks to monitor and collect server statistics in entire Domino domains. The procedure is well documented in the Statistics and the Domino System, IBM Lotus Domino and Notes InfoCenter article. The sample interval should be no lower than 15 minutes, although Administrators may find 1 hour suitable. Statistic collection can be performed on any Domino server. A best practice is to configure statistic collection on an Administration server and remotely collect statistics from other servers. The topic of Centralized Vs Stand-alone statistics collection is discussed in IBM technote 1092952.

4.3.3. Reporting Database

Figure 1 shows an example of a customized Monitoring Reporting database (STATREP.NSF). It contains a document for each sample according to the frequency set in the events4.nsf, statistics profile. Add or remove columns, and choose from all available statistics. These statistics can easily be exported as a comma separate volume file, so graphs can be produced.

Figure 1: Monitoring results in statrep.nsf

175

4.4. Domino Tuning Tips (all platforms)

The following tips will help Administrators to optimize their Notes/Domino environment. It is important for Administrators to understand that any configuration change should be closely monitored for its effects. Make single changes at a time to ensure adverse effects are easy to spot and changes can be backed-out.

4.4.1. View Index Updates

It is possible to tune the Domino update task to take advantage of the additional CPU resource of multi-core hardware. There are a several NOTES.INI parameters that you may use:

Update Task (View Index Updates)

If the systems resources allow it, additional Update tasks may be started on the server, by adding the parameter, Updaters=[number], to the NOTES.INI. As a rule of thumb, do not exceed a maximum of one Update task per CPU. For example, to enable two Update tasks, apply this parameter: Updaters=2

Full Text Index Updates

A separate thread can be created solely for full text updates. This separate thread will only be responsible for updating full text indexes, so view updates will continue to occur even if a large full text index is being rebuilt. Consider applying this NOTES.INI parameter if many full text indexes are present on a Domino server and sufficient CPU resource remain:
UPDATE_FULLTEXT_THREAD=1

Maintaining view indexes

The NOTES.INI parameter DEFAULT_INDEX_LIFETIME_DAYS=[number of days] allows administrators to set a default lifetime for database view indexes if none was selected by the database designer in the Design View Attributes dialog box. The default value for this parameter is 45. Setting this to a lower value can have two benefits: Save Disk Space Reduce amount of work Update task must perform

Creating view indexes is disk intensive and requires CPU and memory resource, so it is important for Administrators and designers to strike a balance. It is not recommended to set this value to lower than 14 (two weeks).

176

4.4.2. Disable Transaction Logging For Certain Databases

Turning transaction logging may impact your enabled DAOS features relative to mail. In most cases, disabling Transaction Logging is not recommended because you lose the benefits of fast server restart. Disabling transaction logging on a database will cause Fixup to run on that database, creating the potential for slow restart. Furthermore, you will not be able to perform incremental backups using the transaction logs for that database. For some databases, however, it might not be necessary to enable transaction logging. This is because it generates additional transaction log traffic on the disk and causes the transaction log drive to fill up quicker. Examples of databases that are suitable for disabling of transaction logs are: Mail boxes (mail.box) - Do not disable Transactional Logging on Mail.boxes if you are running DAOS. Server Log (log.nsf) Monitoring Results (statrep.nsf) Statistics Mail-in (statmail.nsf) Web Administration (webadmin.nsf) Schedule (clubusy.nsf or busytime.nsf) Cluster Database Directory (cldbdir.nsf)

Table 1 highlights NOTES.INI parameters to force new instances of the certain system databases created on server startup to have transaction logging disabled. They do not disable transaction logging on existing databases, or if they are created manually.
NOTES.INI Parameter MailBoxDisableTXNLogging=1 Log_DisableTXNLogging=1 Schedule_DisableTXNLogging=1 System Database Mail.box (See note below) Log.nsf Clubusy.nsf or busytime.nsf

Table 1: NOTES.INI parameters to disable database transaction logging. Note: Do not disable Transactional Logging on Mail.boxes if you are running DAOS.

4.4.3. Replication
If you have more than one Domino server in your environment then you will need to set up replication. The following tips should help optimize the time and resource requirements to replication data around your environment.

Multiple Replicator Tasks

The database replicator (replica) task on a server is responsible for handling scheduled replication requests, as configured in server connection documents. The Cluster Replicator (clrepl) task is event driven and responds to changes in a database. These changes are then replicated to other cluster members. Each replicator task only replicates with one server at a time, therefore the first replication must complete before the next one can start. The number of concurrent replicator tasks running can be set with the following NOTES.INI parameter:

177

Replicators=[number]

In environments where more than two servers participate in the cluster, additional clrepl tasks can be enabled. The number of concurrent clrepl tasks running can be set with NOTES.INI parameter: Cluster_Replicators=[number]

Administrators should monitor Replica.Cluster Domino statistics. For instance, the WorkQueueDepth Domino statistic indicates the number of changes waiting to be replicated to cluster members. If it is continually increasing, enable additional clrepl tasks. The following statistics give indication on the current, average and peak Cluster work queue depth: Replica.Cluster.WorkQueueDepth Replica.Cluster.WorkQueueDepth.Avg Replica.Cluster.WorkQueueDepth.Max

Replication Triangulation
When a client or server replicates with a remote server, it keeps a log of the name of the remote server and the time and date in the Replication History. Domino uses the replication history to determine which documents to scan for changes during the next replication. The purpose of Replication Triangulation is to make each server aware of every other server which maintains a replica of the same database, and which has had a successful replication. In a large environment (hundreds of servers in a domain), the number of replication history events to maintain can cause a significant performance impact to the Domino server. Maintaining replication triangulation history for databases which exist on hundreds or thousands of servers is too expensive. This can manifest itself in the form of increased CPU activity for the replica task. You can disable replication triangulation with the following server NOTES.INI parameters (available in Domino 7.0 and later). NSF_REPLHIST_NO_TRI=1 REPL_NO_WS_TRI_HIST=1 REPL_NO_REMOTE_TRI_HIST=1

For local replicas, use Notes Client NOTES.INI parameters: NSF_REPLHIST_NO_TRI=1 [This will prevent existing triangulated entries from being read] REPL_NO_WS_TRI_HIST=1 [This will prevent new triangulated entries from being written]

Note: After setting the NOTES.INI parameters, the replication history must be purged from each replica of the databases affected. Further information is available in IBM technote 1270104.

178

4.4.4. Internal Caches

In Domino, caches are used to store frequent lookups in memory, to prevent the data being continually read from disk. Since the mechanical hard-disk is commonly the slowest component of the server, caches help improve performance and make a more responsive user experience. The following sections provide Administrators with tips to ensure cache sizes are properly set.

NLCache
NLCache is the name lookup cache. The default is 16MB before Domino 8.5.2. Beginning in Domino 8.5.2, the default is 64MB. It can be increased as needed up to 4GB. To determine if you need to increase the NLCache, use the show stat database or show NLcache console command. For example: Database.NAMELookupCacheCacheSize = 16,447,205 (current size of cache) Database.NAMELookupCacheLookups = 1,879,903 Database.NAMELookupCacheMaxSize = 16,777,216 (default maximum size) Database.NAMELookupCacheMisses = 1,362,746 A relatively high number of misses compared to lookups indicate that you should increase the maximum allowed cache size. Administrators should increase the size of the NLCache in increments, so try doubling the default to begin with, if necessary. Modify using the ini parameter: NLCache_Size= For example, set 67108864, which sets NLCache_Size to 64MB.

Group Cache
When the server needs to lookup the members of a group (For example, in the event of an authentication request) it first checks the Group Cache. It will store results in the Group Cache to optimize performance. Groups are invalidated during updates or when cache is full. The default size is 4MB and it can be increased to 15MB. If the cache needs to rebuild frequently because not enough data can be cached, this can slow down group lookups. Therefore, frequent or very large group updates can slow down server. Verify GroupCache statistics with Show stat net NET.GroupCache.Hits = 155 NET.GroupCache.Misses = 10 NET.GroupCache.NumEntries = 9 NET.GroupCache.Size = 65,406 NET.GroupCache.Used = 2,716 NET.GroupCache.Misses indicates the number of times a group was not found in the cache and so had to be read from disk. Administrators should increase the size of the Group Cache in increments until the number of misses reduces. Modify size with the NOTES.INI parameter: Group_Cache_Size= For example set 15360 , which sets Group_Cache_Size to 15MB.

179

Unified Buffer Manager (UBM)

The UBM (formerly referred to as the NSF Buffer Pool) is used to increase performance by caching disk I/O requests to all databases on a server. The UBM is typically the largest contribution of shared memory usage on the server. In earlier releases of Domino, the maximum size of the UMB was automatically calculated as 3/8th's of the physical RAM in a server. In a server with many gigabytes of RAM, this could result in an unacceptably high UMB upper-limit. Consider a server with 4GB RAM installed: Max UMB Size = (4 x 3/8) = 1.5GB. If the UMB were to grow to its maximum size of 1.5GB, there is a risk that shared memory may exceed process limits and result in a server instability. Domino 8 now enforces a maximum UBM size of 512 MB default on Windows32, Linux, AIX, Solaris, i5/OS and z/OS platforms. On 64-bit Domino, the maximum UBM size is 1024 MB. Therefore, there is no need to take action to properly size the UBM in Domino 8. For additional details on this new behavior, see IBM technote #1268988. For recommendations on setting NSF_BUFFER_POOL_SIZE_MB, read IBM technote 1286171.

NSF_DbCache_MaxEntries
The NSF_DbCache_MaxEntries determines the number of databases that a server can hold in its database cache at one time. If your server has sufficient memory, you can improve the performance of the server by increasing the number of databases that Lotus Domino can cache in memory at one time. The default value is 25 or the NSF_Buffer_Pool_Size divided by 300 KB, whichever value is greater. You should monitor the Database.DbCache.Hits statistic on your server. This indicates the number of times a database open request was satisfied by finding the database in cache. A high value indicates database cache is working effectively. If the ratio of Database.DbCache.Hits to InitialDbOpen is low, you might consider increasing NSF_DbCache_Maxentries. For detail information on how to set the number of databases cached simultaneously in Lotus Domino, read IBM technote 1279893.

NSF Monitor Pool

The Monitor Pool caches event monitors which include server and client mail rules. If set too low the Monitor Pool can become full. Side effects include causing new calendar notices not to display in the Calendar Mini View. The default pool size is 40 MB. The maximum value for this pool is 256/512 MB. To see the current values review stats Database.MonitorPool.Event.Used = 52309 Database.MonitorPool.Monitors.Used = 1184 Database.MonitorPool.Size = 41943040 Check the LOG.NSF for the error Insufficient memory NSF monitor pool is full to know if you should increase this pool for the Domino server. Many customers find a value

180

between 80 and 100 MB to work best. Increase this value if you are approaching the maximum by using the NOTES.INI setting:
NSF_MONITOR_POOL_SIZE_MB=

4.4.5. Multiple Mail Boxes

Additional mailboxes are required on a server only when the amount of mail traffic prevents the router from being able to effectively access the mail box files and access conflicts are encountered. Access conflicts occur when other threads or processes lock the mailbox, while another entry tries to access the mailbox and is denied. The typical number of mail boxes on a mail server ranges from one to four, depending on the volume of mail. IBM technote 1148438 provides Administrators with the procedure to determine whether additional mailboxes are required on busy servers. If you feel you are in need of another mailbox. Change the server configuration to 2 mail boxes.

4.4.6. Tips for Server Based Mail Rules

Keep the number to the absolute minimum, based on business requirements. Rules require processing by the mail router each time it delivers mail. Therefore the fewer rules, the fewer server resources are required. Place rules most likely to be triggered at the top of list and use the Stop processing further mail rules option. Searching the memo body is CPU intensive and will cause and increase in the CPU used by the SMTP task Use the following NOTES.INI parameter to cap the number of mail rules per user's mailfile or server mail.box:
MailMaxFilters=<1 to 100>

This IBM Case study provides Administrators with an indication to the impact mail rules have on a server.

4.4.7. Tuning User Sessions

There are two NOTES.INI parameters that define how many client user sessions a server can have concurrently open and also how long a session remains open. Keeping sessions open requires memory on the server, whereas opening a new session consumes a small amount of CPU resource. Therefore a balance must be achieved. Server_MaxSessions=[number] (Maximum number of concurrent open sessions) Server_Session_Timeout=[number minutes] (maximum duration of idle activity before the Domino server terminates the session)

A timeout value of 30-45 minutes is recommended for most customers. For more information read IBM technote 1089879.

181

4.4.8. Domino Configuration Tuner (DCT)

The Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration analysis for more robust installations and better performance of Domino servers. Make sure you are up-to-date with suggested changes from Lotus Domino by periodically updating DCT. For more information, read "Domino Configuration Tuner (DCT) provides easy-to-use self-service configuration" - IBM technote 4019358.

4.5. Tuning for Virtualized Environments

As organizations continue to try to optimize and leverage their hardware and infrastructure environments, virtualization is becoming a larger footprint for many mail and application server deployments. Virtualization can be one method of reducing TCO (total cost of ownership). This article provides some best practices and guidance for deploying Domino servers on a virtualized VMware architecture. It is worthwhile pointing out that by virtualization alone, it is unlikely that a TCO reduction will be achieved. This is because the number of Domino servers remains the same and so does the Administration effort required. Furthermore there is another (host) operating system to tune and maintain. Organizations may find TCO reductions can be realized more effectively by implementing new Domino features, such as DAOS and ID Vault, or Domino server consolidation (hosting more users on the same server).

4.5.1. The Pros and Cons of Virtualization

The potential benefits of deploying Domino in a virtualized environment include: Organizations can reduce the physical footprint required for supporting a server consolidation effort. You can run two mail servers on one physical server. Organizations can leverage a higher amount of a server's total resources. Migrating from physical hardware to virtualized hardware has little impact on the users. Whereas migrating users to consolidated Domino servers can require from template changes to hard-coded server references, and/or changes to users' desktop configuration. From a disaster recovery perspective, it can allow for an image to be brought up on another VMWare server very quickly and communicate with the Domino Data, DAOS and Transaction logging storage as if it were on the same physical server.

These potential benefits, however, do come with costs. Organizations must consider the following facts: There is a hardware resource overhead to provide the virtualization. More Domino server virtual machines (VMs) typically mean higher disk and network I/O, channelled to a single physical server. Some configurations can lead to dynamic guest resource allocation. This can lead to resources being shared across VMs.

182

4.5.2. Static Resources

A key to vitalizing your Domino servers is to ensure that the architecture within the virtualized systems is static and dedicated. For example Disk I/O for Transaction logs, DAOS, users mail files and networking functions should all be dedicated for each Domino server. This is in order to optimize the I/O performance channels and minimize the impact that the virtualization may have any virtualized node (or guest). Ensure hardware resources are isolated and dedicated to the VM guests. Do not use VMware configurations such as memory over commit, ballooning or dynamic VM balancing as this can lead to stability / performance related problems for the Domino application.

4.5.3. Best Practices for Guest VM

The following points are designed to be templates for a starting point in sizing Domino mail servers.

Virtual CPUs
The number of vCPUs per VM depends on the number of users to be supported. Try not to over-provision vCPUs of Domino guest. 4 CPUs should be fine in most cases.

Virtual RAM
RAM for Domino allows it to cache more data. The operating system does I/O caching too, which improves the overall performance. IBM Software Services for Lotus recommends 4GB of RAM per Domino Server. Improved memory limits in 64-bit OS helps cache more data, and thus avoid disk IO. Reduces response times, and hence increasing the number of users. Increase VM memory when running in 64-bit guest OS. Recommend each VM to have 8 GB allocated for guest OS and Domino Application.services.

Storage
Storage architecture may be a bottleneck due to the number of servers and volumes per LUN. To optimize the storage architecture, do not use shared LUNs. This means each volume (such as notesdata or transaction log drive) is mapped to one set of spindles and are unique to each Domino server. Use Fiber Channel disks when ever possible, 4Gb or better is recommended. 2Gb is supported but has limited bandwidth for Domino Mail servers. Align partitions Use VMFS and use Virtual Center to create partitions

Use separate, dedicated LUNs for OS/Domino, data and transaction logs Separate the IO at physical disk level, not simply logical LUNs

183

Make sure these LUNs have enough spindles to support the IO demands Fewer spindles or too many VMDK files on single VMFS LUN can substantially increase disk IO latencies

RAID configuration Best performance using RAID 1+0 for Data, RAID 0 for Log Raid 5 can cause write queues to build up on slower Storage solutions i.e. iSCSI, Hardware Storage or software-based RAIDs.

Networking Configurations
To optimize the network access, do not share physical network interfaces controllers (NIC) with different Domino servers. Use dedicated NICs based on the network traffic: Use separate NICs for mail and cluster replication traffic. Use Enhanced VMXNET 2 or 3 driver or higher with TSO and Jumbo Frames support. Use NIC Teaming & VLAN Trunking if available - Note, Network teaming not always the best way.

Note: Co-located VMs outperform physical 1Gbps network speed

VM Time Synchronization
Use VMware Tools time synchronization within the virtual machine Enable NTP daemon to sync with external NTP source (using vSphere client) Disable OS Time Service Windows: w32time service Linux: NTP daemon

Resource Reservation
Do not set limits for Domino resources. As the Domino servers are virtualized, they should be done as Static resources for each Domino Mail Server.

Ballooning
Never use Ballooning for Domino servers. Domino allocates and uses the memory on startup based on what it sees. This means Domino servers will build cache pools, based on the total memory physically allocation. If you use ballooning you can remove or compromise these configurations and cause Domino to slowdown, hang or crash. In production environments ballooning should always be 0 for optimal performance.

184

4.6. Domino on Windows Tips

This article provides tips for Administrators running Windows-based Domino servers.

4.6.1. System Page Pool

Large environments with hundreds of gigabytes of Domino data can be affected by a common issue with the Windows. Domino maintains a database cache to improve performance by keeping recently used files open for quicker access. These caches are stored in the Windows page pool and may still remain in the page pool even if the database is closed by the client. If many large databases are opened, it is possible to exceed the page pool limit and cause problems for the Domino server. Read IBM technote 1093511 for more information. To try and alleviate the issue on Windows 2003 (32-bit) servers, perform the following steps to modify two(2) registry keys(these steps were taken from Microsoft document Q312362). 1. Start Registry Editor (Regedt32.exe) 2. Locate and then click the following key in the registry: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management

3. On the Edit menu, click Add Value, and then add the following registry value: Value name: PoolUsageMaximum Data type: REG_DWORD Base: Decimal Value data: 60 Setting the value at 60 informs the Memory Manager to start the trimming process at 60 percent of PagedPoolMax rather than the default setting of 80 percent. If a threshold of 60 percent is not enough to handle spikes in activity, reduce this setting to 50 percent or 40 percent. Value name: PagedPoolSize Data type: REG_DWORD Base: Hex Value data: 0xFFFFFFFF Setting PagedPoolSize to 0xFFFFFFFF allocates the maximum paged pool in lieu of other resources to the computer (~500 MB). Caution: The 0xFFFFFFFF PagedPoolSize setting is not recommended for use on 32-bit Windows Server 2003-based computers that have 64GB of RAM. This will potentially bring the Free System PTE entry down and can cause continuous reboot of the computer. For this configuration, carefully choose a value based on the requirements and available resources.

185

4.6.2. Other Tuning Tips for Windows Servers

The Domino Server Performance Troubleshooting Cookbook is focussed on Windowsbased Domino servers and provides further reading for Administrators.

4.6.3. Additional references

The Knowledge Collection: Hardware or Operating System error - Insufficient system resources on the Domino Wiki goes into detail about Windows Paged Pool issues. It discusses the areas we mentioned as well as others. There are also Windows 2008 specific articles that describe common performance problems if you are not running with the fixes: IBM technote 144982 "Windows 2008 64-bit can cause a significant CPU increase and I/O degradation when Domino opens many databases" IBM technote 1427348 "Windows 2008 64-bit hangs/slows dramatically when Domino server is shut down"

4.7. Domino on Linux Tips

Domino for AIX and for Linux share some characteristics that allow some consistency in the approach to tuning on these platforms.

4.7.1. Monitoring Server Resources

4.3 Establishing a Performance Baseline already discusses the importance of establishing the baseline prior to undertaking tuning changes. The nmon tool is designed to monitoring and analyzing performance data, such as CPU and memory utilization, disk I/O metrics and kernel statistics. The nmon performance tool can capture data to a text file for later analysis and graphing for reports. The output is in a spreadsheet format (.csv).

4.7.2. Operating System Limits

The default setting for maximum number of open files allowed is usually set too low. If the value set for this parameter is too low, errors might occur when opening files or establishing connections. Since this value limits the number of file descriptors that a server process might open, a value that is too low prevents optimum performance. Check /etc/security/limits.conf and append the nofile entry as follows: notes soft nofile 49152 notes hard nofile 49152

186

Note: In the above example, notes is the user account that Domino runs under. Set this according to your environment. Similar to open files, the maximum number of processes/threads can be set too low. The nproc attribute represents the maximum number of processes each user can create. Check /etc/security/limits.conf and append the nproc entry as follows: notes soft nproc 12500 notes hard nproc 12500

Note: In the above example, notes is the user account that Domino runs under. Set this according to your environment.

4.7.3. Linux Services

(Specific to Red Hat Enterprise Linux): The updatedb command updates the locate database. This process is very disk intensive and so should only be run out of business hours. Verify that the mlocate.cron script (typically /etc/cron.daily/mlocate.cron) runs out of normal business hours. Disable these services to prevent port conflicts, if enabled on the Domino server. httpd/apache (HTTP) sendmail (SMTP)

4.7.4. TuneKrnl
Domino for Linux comes with a tool 'tunekrnl' that runs automatically when Domino starts to set dynamic kernel parameters. Allow the Domino server to automatically calibrate kernel parameters.

4.7.5. Troubleshooting and Debug Tips

On Linux, ensure you have installed the latest version of gdb (GNU Project Debugger). On AIX, ensure you have installed the latest version of dbx (the AIX standard command-line debugger). On AIX, ensure you have installed the latest version of procstack (Anoth AIX debugger. Prints the hexadecimal addresses and symbolic names for all the threads in the process).

Note: On AIX, Domino may use both dbx and procstack depending on the situation. Both should be up to date versions.

187

4.7.6. Disabling concurrent I/O and direct I/O on Domino servers on AIX
The CIO feature is not supported for use with Domino servers, do not enable this option on file systems that Domino accesses. For further information read this Infocenter article.

4.7.7. Tuning Java for Domino on AIX

The following IBM technote 1451336, contains guidance on preventing excessive CPU usage by Agent Manager threads.

4.7.8. Perfpmr for AIX

Perfpmr is a tool for AIX that gathers performance specific data from your system. It is a shell script that runs other scripts that execute various tools to get information on system performance. It is useful to review the information Perfpmr collected to troubleshoot your performance issue or have IBM Support to help debug your system. You can run it when you have performance problems, during the normal period, or before and after system configuration changes or fixpack upgrades. For more information on Perfpmr and a list of useful AIX tools, read Performance Monitoring Tips and Techniques

4.8. Domino on IBM i Tips

4.8.1. Overview
Getting started with running Domino on IBM i? Here are some things you should be aware of when running Domino on IBM i On IBM i every Domino installation is a partitioned install. The Domino product code is always stored separately from the Domino server data. You will find the Domino product code located in the /QIBM/ProdData/Lotus/DominoXXX directory where XXX represents your release, for example /QIBM/ProdData/Lotus/Domino852. Domino on IBM i supports running multiple releases. For example, when planning your upgrade you could upgrade your test Domino partitioned server (dpar) to 8.5.2 while leaving your primary dpar at 8.5.1. For more information about multi-versioning refer to an old, but still relevant, IBM Redbooks publication Lotus Domino 6 MultiVersioning Support on the IBM eServer iSeries Server. Domino installs and upgrades can be separate steps on IBM i. This means you can install the product during the business day and then upgrade the server at a later time. This will reduce the outage window required for the upgrade. The command then used to upgrade the server is UPDDOMSVR. For more information on the UPDDOMSVR command refer to UPDDOMSVR InfoCenter Topic. All Domino data must be owned by the QNOTES user profile for proper operation. This is particularly important for all objects in the Domino server data directory. To

188

update the owner on an IFS file use the CHGOWN command, to update the owner of native objects use the CHGOBJOWN command. IBM i is an EBCIDIC system. To prevent performance problems when transferring data between platforms always use binary FTP - in other words, do not drag and drop Domino databases from a Windows environment. For information on using FTP refer to the technote Using FTP to transfer files to/from the Domino Data directory on IBM i. IBM i uses a different file locking mechanism than Windows. Thus, to prevent file access and data corruption issues, do not map a network drive to a Domino server's data directory. There are graphical user interfaces available to use with the IBM i operating system including System i Navigator and IBM Systems Director. If you want to learn about the native CL commands most used by a Domino administrator refer to Multimedia: Introduction to IBM i CL commands for Domino administrators. For information on backup and recovery specific to IBM i refer to Backup and recovery strategies education modules available via IBM Support Assistant. Be aware of network changes or problems. Slow response in your network will cause delays and errors for your users. Also, Domino must have access to a functioning DNS server at all times in order to route mail. If the DNS server IP address changes in your environment you must also update the DNS server entry within IBM i using the CHGTCPDMN command.

4.8.2. Performance
When getting started analyzing system performance: Ensure the system you are using is designed to support your workload. To determine the minimum system requirements for your workload use the IBM Workload Estimator. To compare systems for Domino workload the system Mail and Calendar Users (MCU) rating must be used. CPW is for traditional workloads and should not be used for Domino. To determine the MCU rating for your current system refer to Appendix D of the Performance Capabilities manuals. Version 5, Release 4 Version 6, Release 1 IBM i 7.1

Response time is the best indicator of system performance. The guidelines presented in this section are general guidelines limits where most customers start reporting a response time degradation in their environment. Since each system is unique the exact threshold where response time will be affected may be higher or lower depending on your hardware and system usage.

189

Monitoring Tools
There are multiple tools available for collecting system performance. Collection services IBM i Job Watcher IBM i Disk Watcher IBM Systems Director Navigator

A discussion on the details of these tools and how to use them is beyond the scope of this article. For an introduction on gathering performance statistics refer to Chapter 4 of the IBM Redbooks Domino 6 for iSeries Best Practices Guide.

CPU
Keep CPU utilization at 85% or below for a single processor system Keep CPU utilization at 90% or below for a multi-processor system Be aware that a system can be CPU bound without showing a CPU utilization of 100%. How can this be? Imagine a system with 4 processors. If one processor is pegged, but other processors are idle, the overall system CPU utilization will be 25%. It is possible to have a single threaded, CPU intensive operation be running slowly and thus CPU bound without an obvious indication that CPU is the issue. You should review the CPU utilization for a single thread. If one thread is driving 100% of one processor or 25% overall CPU in our example, then your system is CPU constrained. Hint: Use the IBM Performance Tools for i5/OS WRKSYSACT command to see the CPU utilization at the thread level. By default all Domino jobs run with the same job priority, priority 20. There are cases when you may want to alter the priority of some tasks. One example would be to try and manage which jobs receive more CPU on a CPU constrained system. Changing the run priority can be done by following the instructions in the technote Changing the Priority of a Server Job Permanently on iSeries (IBM i). By changing the job priority you can or force intensive processes such as view index updates, agents or adminp to have lower priority over user interfacing tasks such as server and http.

Memory
Machine pool faulting rate and paging rate should be at or below 5 faults and 10 pages per second. The memory pool for your Domino servers (the BASE pool by default) should stay at or below a Non-DB fault/pages rate of 100/300 per processor. So for 5 dpars running on a single logical partition or system with two full active processors should have enough memory to keep the number of non-db faults/pages per second of 200/600. Determining the correct amount of memory for a system can be challenging. One simple rule of thumb is to check your largest database and most complex views. You should have a minimum of 2 times the amount of space needed for the views in that

190

application dedicated for the Domino server. For example, if you have an application database with multiple views that total 2GB in size you should have a minimum of 4GB dedicated to the Domino server in order for optimal memory performance during view updates. On a memory constrained system, until more memory can be purchased, reducing the size of the NSF buffer pool can improve performance. Be aware that reducing the NSF buffer pool may also reduce the number of router threads and you may need to manually increase the number to prevent mail routing slow downs. For information on how router allocates threads refer to technote 1093562.

Disk
When reviewing disk I/O and percent busy rates you want to see the disks to be 35% or less busy. Avoid running out of disk! Ideally your drives should not be 90% or more utilized. The space used (%used) should be consistent across all drives. When balancing drives the Domino servers should be ended. The protection/mirroring status should be active, not degraded as a status of degraded may indicate an under performing drive or possible disk and hardware errors.

Domino Tasks
There are things specific to Domino that can help or affect overall performance. Stay up to date on ODS level. It is not required that you update to ODS level 51 when you upgrade to Domino 8.5. However; there have been a number of performance improvements that will affect server performance and thus it is important to upgrade as soon as possible. Disable any unnecessary tasks. Are you running SMTP on your administration or hub servers? Are you running COLSRV400 on multiple dpars? Are you running DECS and not using it? Each and every task running under the Domino server takes some system resources to use so eliminating unnecessary tasks will improve overall system performance. Transaction logging has evolved to be a valuable resource for all customers. However; there is a performance impact to enabling transaction logs. In most implementations a CPU increase of 3% is seen, for example CPU used by the Domino server is 38% before transaction logging will be approximately 41% after. There is also typically an increase in the I/O rates between 5-15%. Thus if the percent busy rate of your disks is normally 10% busy you may see your drive %busy rate at 10.5 - 11.5% busy after transaction logging is enabled.

For other overall performance information refer to Sizing Large-Scale Domino Workloads on iSeries.

191

4.9. Tuning Tips for the Domino HTTP Server

The default settings are a "one size fits all" approach. While the settings will work as shipped, making some minor changes based on how you use the Domino server can increase the performance and efficiency of your server. Here is a list of things you can do to optimize your server.

4.9.1. HTTP Server Threads

In order for you server to function properly it must be able to handle all of the requests it receives. Domino provides a number of statistics to help you determine the health of the Domino web server. For example, to determine the peak number of threads used by the web server review the statisticDomino.Threads.Active.Peak. By default, the number of active threads is 40. However; if you are using all available threads and have available memory for additional threads you can increase this value in the server document, Internet Protocols... HTTP tab, Number of active threads parameter as shown in figure 1.

The Domino web server queues requests as they come in so they can be processed by the HTTP server threads. In some cases changing the queue method used can improve performance. For more information refer to technote 1201715: HTTP thread queue implementation in 6.x can cause performance issues for some setups.

192

4.9.2. HTTP Requests

When tuning your server, it helps to understand what requests are being processed on your servers. You can do this by logging Domino web server requests. Thus, you can access this information "after the fact" by reviewing the access logs and/or the web log database domlog.nsf. You enable logging for your server in Internet Protocols tab of the server document as shown in figure 2.

In order to see which requests are being processed in "real-time" you can use the Domino command tell http show thread state. This will present you a list of all active threads and the URL currently being processed if the thread is not idle. Below are two example entries and you can see that the first thread is idle and the second thread is opening the Inbox for User One (uone.nsf). 11/18/2010 12:09:28 Http Worker Thread ID [1c7]: [Thread State is Idle] 11/18/2010 11:18:12 Http Worker Thread ID [2425]: Working session [9a732]: Session State [Processing Request] : GET /mail/uone.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries&PresetFields= DBQuotaInfo;1,FolderName;($Inbox),hc;%24Sender1%7C%2498,UnreadOnly;1&TZType =UTC&Start=1&Count=23&resortdescending=1 HTTP/1.1 If your server is struggling to process all requests due to database contention, view index rebuilds or database corruption, you will see all requests showing the same database, view or document. This is a fast and simple way for you as an administrator to identify potential database issues. In other cases, you may see that all of the pending requests are agents. By default, only 1 agent can process at a time. However; if the agents running on your server are thread-safe, then you may modify the configuration of the server to allow multiple web agents to run concurrently. To make the change, open the server document and access the Internet protocols... Domino Web Engine tab. Change Run web agents and web services concurrently to Enabled as shown in figure 3.

4.9.3. JVM Heap

The default JVM Heap can be modified using the NOTES.INI parameter HTTPJVMMaxHeapSize. If you are using XPages, you will want to increase the size of the heap at 8.5.2 and higher. If you are not using XPages and running 8.5.1, you will want to decrease the default size. More information can be found in technote 1377202: What is the HTTPJVMMaxHeapSize notes.ini parameter in Domino 8.5 and what should it be set to?

193

4.9.4. Database Performance

When working with the Domino web server and applications, the same techniques to improve database performance from a Lotus Notes client can be used on the web. For a list of the most common reasons for slow database performance refer to technote 1174563: Knowledge Collection/Troubleshooting Guide: Known causes of slow database performance.

194