Batch Best Practices

An Oracle White Paper
J uly 2011
Batch Best Practices
Oracle Utilities Application Framework

Oracle Utilities Appli cation Framework - Batch Best Practices

Introduction ......................................................................................... 3
Caveat ............................................................................................. 3
Conventions used in this whitepaper .............................................. 4
Batch concepts .................................................................................... 4
Batch Program ................................................................................ 4
Batch Controls ................................................................................. 5
Batch Overview ................................................................................... 5
COBOL Versus J ava Processes ..................................................... 6
Executing Batch J obs ...................................................................... 6
Scheduler Submission Overview .................................................... 7
Configuration File Hierarchy ............................................................ 8
Execution Modes ........................................................................... 10
CLUSTERED Mode internals ........................................................ 15
Socket issues in CLUSTERED mode ........................................... 18
Clustering using Unicast or Multicast ............................................ 18
Migrating from DISTRIBUTED to CLUSTERED mode.................. 20
CLUSTERED Mode Operations .................................................... 21
Testing for multi-cast ..................................................................... 22
Demonstration Single machine setup ........................................... 22
Timed Batch .................................................................................. 22
CLUSTERED Mode recommendations ......................................... 23
Threadpools and Database Recycling ................................ 24
EXTENDED Mode ............................................................... 25
Operational Best Practices ................................................................ 28
Parameter Guidelines ................................................................... 28
Use Of Batch Control For Parameters .......................................... 29
Multiple Batch Controls for a Batch Program ................................ 29
Take Defaults For Parameters ...................................................... 30
Commonly Used Configuration Settings ....................................... 30
Setting The Error Tolerance .......................................................... 31

Multi-threading Guidelines ............................................................ 32
Altering Commit Interval and Cursor Reinitialization time ............. 33
Online Daemon or Standalone Daemon ....................................... 35
Online Submission Guidelines ...................................................... 36
Use of Spacenames ...................................................................... 36
Maximum Execution Attempts ....................................................... 37
Number of Threads Per Threadpool ............................................. 37
Designing Your Threadpools ......................................................... 38
COBOL Memory Optimization for Batch ....................................... 40
Example Clustered Mode scenarios ............................................. 41
Monitoring Background processes ................................................ 49
J MX Monitoring ............................................................................. 49
Database Connection information ....................................... 50
Commit Strategies ............................................................... 50
Flushing the Batch Cache ................................................... 52
Online Submission Alternative Threadpools ....................... 52
Scheduling Best Practices ................................................................ 53
Optimize the schedule ................................................................... 53
Use a third party batch scheduler ................................................. 54
Scheduler Implementation Guidelines .......................................... 55

3
Introduction
One of the major components of the Oracle Utilities Application Framework is the ability to execute
background processes.
Typically a foreground process (a.k.a. online or Web Service transaction), performs operations on a
single instance of an object within the product. Maintaining a person's contact details or processing a
single payment are two such examples. Background processing, by contrast, performs operations on
multiple instances of an object. This coins the term batch where the background process processes
batches of objects. The term background processing implies that the processing is performed in the
background with little or no user interaction.
The product ships with a preset number of background processes that may be used and configured to
perform the necessary business functions for your site. These background processes can be extended
(just like the rest of the product functionality) and custom background processes can be added.
This white paper outlines the common and best practices used for the background processing (a.k.a.
batch) component of the Oracle Utilities Application Framework. The advice in this whitepaper is
based upon Oracle internal studies and customer feedback around the world. This information is
provided to guide other sites in implementing or maintaining the product in production.
This document is a companion document to the Batch Operations And Configuration Guide or Batch
Server Administration Guide for your product and the Performance Troubleshooting Guidelines
Batch Troubleshooting
Note: For publishing purposes, the word "product" will be used to be denote all Oracle Utilities Application Framework
based products.
(KB Id: 560382.1) documents.
Note: This document may describe features that are not provided by the original product but may be delivered as a
subsequent patch. Refer to the Batch Operations And Configuration Guide or Batch Server Administration Guide
Caveat
for
your product for relevant patch numbers.
While all care has been taken in providing this information, implementation of the practices outlined in
this document may NOT guarantee the same level of (or any) improvement. Not all practices outlined
in this document will be appropriate for your site. It is recommended that each practice be examined in
light of your particular organizational policies and use of the product. If the practice is deemed


4
beneficial to your site, then consider implementing it. If the practice is not appropriate (e.g. for cost
and other reasons), then it should not be considered.
Conventions used in this whitepaper
The advice in this document applies to any product based upon Oracle Utilities Application
Framework versions 2.1 and above. Refer to the installation documentation to verify which version of
the framework applies to your version of the product. For publishing purposes the specific facilities
and instructions for specific framework versions will be indicated with icons:

Advice or instructions marked with this icon apply to Oracle Utilities Application
Framework V2.1 based products and above.



Note: Advice in this document is primarily applicable to the latest version of the Oracle Utilities Application Framework
at time of publication. Some of this advice may apply to other versions of the Oracle Utilities Application Framework
and may be applied at site discretion.
Note: In some sections of this document the environment variable $SPLEBASE (or %SPLEBASE%) is used. This
denotes the root location of the product install. Substitute the appropriate value for the environment used at your site.
Batch concepts
The following section outlines some basic concepts when configuring and executing the batch
component of the Oracle Utilities Application Framework.
Batch Program
The main component of the background process is the batch program. This program contains the
logic to select the batch of records that the process will perform actions upon. The batch of records is
progressively passed to the relevant objects to complete processing. In essence, the program acts as a
driver to push individual records to the relevant objects. Objects in the product are shared across all
modes of access to maximize reuse.
Apart from the logic to decide the subset of records, the batch program contains the following
additional information:
The batch program contains the code necessary to interface to the framework to automatically
manage its individual execution and restart (if necessary).

5
Some of the batch programs contain the code necessary to multi-thread the processing. In
essence the program determines which slice or subset of the overall data it needs to process.
Not all programs support multi-threading (most do, except the extract to file type processes).
The product ships with a preset number of batch programs associated with background processes that
may be used and configured to perform the necessary business functions for your site. These
background processes can be extended (just like the rest of the product functionality) and custom
batch programs can be added.
Batch Controls
A batch program and its parameters must be defined in metadata prior to initial execution. This data is
stored in a Batch Control object within the meta-data component of the framework.
The Batch Control contains the definition of the batch program and the following additional
information:
A Batch Code used as an identifier. This is used by the framework to identify the job
internally and used to denote output.
Basic execution statistics for the last execution, including last execution date and time and the
latest run number which is primarily used for extract processing.
Process specific parameters identify whether a parameter is optional or required and may
provide a site specific default value.
The product ships with a preset number of batch controls associated with background processes that
may be used and configured to perform the necessary business functions for your site. Custom batch
controls can be added as needed.
Batch Overview
The Oracle Utilities Application Framework is a java based framework for running background
processes, online processes and Web Services. The internals of the framework manage the input
channels and direct calls to the appropriate objects within the product to perform the business logic
and database access necessary.
In terms of background processing, the framework wraps the batch program to execute it and manage
it from the operational point of view. This includes the following:
Providing the interface to track the progress of the threads via the Batch Run Tree
transaction.
Providing the infrastructure for the recording of restart checkpoints.
Providing the infrastructure to handle other components like algorithms, developed in either
COBOL or Java.

6
Background processes are executed within a JVM which has loaded the Framework components
necessary for batch execution. Each submission method may have other unique elements, but this
basic summary is true for each.
COBOL Versus J ava Processes
Note: Not all Oracle Utilities Application Framework based products support COBOL. Refer to the installation
documentation with your product for more details. Products not using COBOL should ignore this section.
The Oracle Utilities Application Framework supports both java and COBOL based background
processes. Since the Oracle Utilities Application Framework is Java based, all processes are executed
within a batch JVM (e.g. threadpool). When the process needs to invoke a COBOL based module, the
required! module is loaded and executed internally by the JVM. The results are then passed back to the
main Java objects. This method differs from the one used by the Online and Web Service Adapter
(XAI) in that they use separate child JVMs to isolate COBOL calls. The reason for this difference is
that in Online and Web Services there could be a large number of different COBOL objects called.
Batch is usually limited to a smaller number of COBOL objects.
Given that COBOL and Java live in the same batch JVM, there are a number of concepts which must
be understood:
Any COBOL program that misbehaves (bad data/bad memory management etc) can cause the
failure of the executing batch JVM. This may affect other jobs using the same batch JVM
(threadpool) at the same time (or even future scheduled executions if the batch JVM is not
restarted). Regularly checking the threadpool via JMX and logs is advised to avoid issues.
COBOL programs are attached to the batch JVM as executed. This can increase the memory
footprint over time. COBOL modules cannot be garbage collected using the default method
provided with Java. Over time, a long running batch JVM, that is reused for a lot of batch
process may need to be stopped and restarted to avoid memory issues.
Executing Batch J obs
There are a number of different ways to initiate background processes within the Oracle Utilities
Application Framework. The different ways are available to support the different activities typically
seen during an implementation of the product.
The list below summarizes the methods available with the framework:
Interactive Mode This involves running the SPLBATCH[ . sh] command from the
system prompt for each thread of each background process. A JVM is created for each thread
and shuts down after execution of the thread. This mode has limitations in that you cannot
use JMX to monitor the job and, per thread, is the most resource intensive of all submission
methods. This is primarily recommended for developer use only.
Note: This is the equivalent of running the process submi t j ob[ . sh] e THI N.

7
Online Submission An online transaction is provided to allow end users to register their
intent to execute a background process. A batch daemon (as part of the application server or
standalone) polls for registrations of background processes and executes them. This mode is
primarily recommended for non-production use (for example testing).
Batch Scheduler Submission A set of utilities is provided to allow third party schedulers
to execute background processes from the product. These allow a scheduler to micro-manage
the processes on behalf of the product and also allow integration to non-product processes
into your overall schedule.
Note: The execution methods are described in detail in the Batch Operations And Configuration Guide or
Batch Server Administration Guide
Most implementations use the methods in the following ways:
associated with your product.
The Interactive Method (a.k.a. SPLBATCH[ . sh] ) is not used by site personnel. Developers
may consider using it but may use the other methods.
Note: The SPLBATCH[ . sh] utility is not supplied with Oracle Utilities Application Framework V4.0
and above as it has been replaced by the submi t j ob[ . sh] utility.
The Online Submission Method is used by most implementations for non-production
environments where there is no scheduler present. This allows testing personnel to submit
background processes as necessary. The implementation has a choice in terms of how the
execution via the daemon actually occurs. A discussion of the various execution methods are
discussed in Online Daemon Or Standalone Daemon
The Scheduler Submission Method is commonly used for production and a scheduler test
environment (it is expected that the IT group will get familiar with the scheduler and
scheduler before implementing it into production). Guidelines for this method are outlined in
.
Scheduler Implementation Guidelines
Scheduler Submission Overview
.
One of the common practices at product implementations is to implement a batch scheduler to
schedule and execute batch jobs across an enterprise
1

1
Whilst sites will use the implementation of the product to introduce the batch scheduler, it is apparent that
the scheduler is best used for enterprise wide scheduling.
(including product jobs). Therefore the product
includes a set of scheduler submission utilities that allow third party schedulers to execute background
processes. These include a set of command-line utilities and their associated configuration files. The
intent is that site-specific configurations are applied for the desired subset of background processes.

8
The scheduler then executes those processes as necessary across the hardware allocated for this
purpose.
The basic concepts for these utilities can be summarized as follows:
A worker JVM is started (termed a threadpool) using the t hr eadpool wor ker [ . sh]
utility. This starts a JVM and loads the Oracle Utilities Application Framework ready to accept
work. The t hr eadpool wor ker [ . sh] utility uses a set of configuration files to
determine the characteristics of any threadpool it needs to manage. In Oracle Utilities
Application Framework V2.2, it is possible to manage the threadpools via JMX using a JMX
console or the provided j mxbat chcl i ent [ . sh] utility. The threadpool is given a name
which is used by the Oracle Utilities Application Framework to attribute work to it as
directed.
Each thread (or multiple threads) can be submitted to the named threadpool using the
submi t j ob[ . sh] utility. During this process a small submitter JVM is created to initiate
communication between the background process and the threadpool. This JVM is also used
by the submi t j ob[ . sh] utility to communicate back to the scheduler on the outcome of
the execution of the background process.
At the conclusion of the background process the submi t j ob[ . sh] utility returns the
outcome to the scheduler for the appropriate action. This is where an external scheduler can
decide the next action based upon the outcome of previous executions of background
processes. For example, in the case of failure it may stop dependent background processes
and page the appropriate personnel.
Note: The Scheduler Submission Method described in detail in the Batch Operations And Configuration
Guide or Batch Server Administration Guide
Guidelines on how to use this facility are contained in
associated with the product.
Note: Do not attempt to edit the properties files directly. These are rebuilt from the .template files each time
i ni t i al Set up[ . sh] is executed (i.e. when patches are applied) and all changes WILL BE lost. All changes
should be made in the t hr eadpool wor ker . pr oper t i es. t empl at e and
submi t bat ch. pr oper t i es. t empl at e respectively, and initialSetup[.sh] executed after each change.
Refer to the
.
Operations And Configuration Guide or Server Administration Guide
Note: Keep backups of your .template files. Patches and Service packs will overwrite these files with defaults. When this
happens, check the new .templates for any NEW settings which may be necessary for the newly patched environment(s)
prior to restoring your site-specific configuration."
for your product for details of how to
implement custom templates.
Configuration File Hierarchy
A component of the Scheduler Submission Method is the set of configuration files that control the
behavior for the interface and the options that are used by the execution. For maximum flexibility
there is an inbuilt hierarchy for the configuration options:

9
The t hr eadpool wor ker [ . sh] and submi t j ob[ . sh] have internal defaults that are
used if no configuration files exist. These are not recommended to be used, except in
development testing, as these defaults are usually not recommended for production use.
The internal defaults may be overridden by a properties file associated with the utility. The
t hr eadpool wor ker [ . sh] utility uses the file
$SPLEBASE/ et c/ t hr eadpool wor ker . pr oper t i es. The submi t j ob[ . sh]
utility uses the file $SPLEBASE/ et c/ submi t bat ch. pr oper t i es. Default templates
for these files are provided with the product.
For the submi t j ob[ . sh] utility, it is possible to create a job specific configuration file
which will contain only those characteristics unique to a particular background process. This
configuration file is usually named <bat chcode>. pr oper t i es or
<bat chcode>. pr oper t i es. xml . The xml version is primarily provided for character
sets other than Western European
2
The t hr eadpool wor ker [ . sh] and submi t j ob[ . sh] support command line
options to override previous configuration settings.
.
The figure below illustrates the hierarchies:
Internal defaults
threadpoolworker.properties
Command line options
Internal defaults
submitbatch.properties
Job specific properties
(if exist)
Command line options
t hr eadpool wor ker hierarchy submi t j obhierarchy
Overridden by
Overridden by
Overridden by
Overridden by
Overridden by

Figure 1 Configuration hierarchy

2
Western European character set includes USA, Canada, Australia, Europe (except Eastern Europe) and
New Zealand.

10
Execution Modes
Note: The CLUSTEREDexecution mode applies to Oracle Utilities Application Framework V2.2 SP7 and above
only
Note: A new mode has been introduced called EXTENDEDmode that is available for Oracle Utilities Application
Framework V4.1 and above only after applying patch 1173516.
The execution method is specified in both the t hr eadpool wor ker . pr oper t i es file and the
submi t bat ch. pr oper t i es file as runtime defaults and are established at configuration time
with the conf i gur eEnv. sh utility. This can be overridden at runtime using either
submi t j ob[ . sh] and t hr eadpool wor ker [ . sh] with the - e option.
The THI N execution mode executes a single thread of a single job in a single JVM. This is primarily
designed to be used by developers to test their batch processes during initial development and testing
activities. It can be used for other uses in an implementation (testing etc) but is seen as inefficient
compared to other methods due to the fact that a JVM is required per thread per job.
The DI STRI BUTEDexecution mode (also known as Classic mode)
3
Each worker node offers n number of threads to the grid in a specific thread pool (where n is the
number of concurrent threads the worker can run), and creates and takes out a lease on a
THREAD_OFFER entry on the F1_TSPACE_ENTRY database table to register and stay alive as a
participant in the grid. Similarly, a submitter node leases a WORK_OWNER entry on the table to register
its participation. By default, these leases are renewed every 20 seconds by updating a lease expiry
timestamp on the respective row. There is therefore some overhead involved, and it requires adequate
database response times to function properly.
, allows numerous threads from
numerous jobs to be execute by one or more JVM's known as threadpools. A threadpool is made up of
worker nodes that process work as instructed from submitter nodes.
The workers and submitters poll for GRI D_WORK and WORK_ENDED entries respectively by
regularly scanning the F1_TSPACE_ENTRY table on the database. The default interval setting for
this is every 1-second, so if there are a lot of active nodes in the grid, it can amount to a significant
level of database activity (even though just the header information is selected - not the CLOB column
on the table).
A submitter node inserts a GRI D_WORK entry into the table to notify the grid that there's work to be
done. This entry contains the batch code and batch parameters for the job, which enables the worker
that picks up the request to execute it. The worker takes out a lease on this GRI D_WORK entry to
signal its intent to process the work, and continues to initiate the individual threads for the job. Once

3
The DI STRI BUTEDmode is the default mode from Oracle Utilities Application Framework 2.0, 2.1 and
2.2.

11
the threads have ended, the worker inserts a WORK_ENDEDentry into the table, which notifies the
submitter that its work has been processed.
Note: This is a highly simplified overview of the process. In reality, the worker itself also creates GRI D_WORK and
WORK_ENDEDas well as PER_STATE entries for the individual threads when those are initiated. There is also a
HousekeepingDaemon process that has its own GRI D_WORK entry and which periodically monitors the
F1_TSPACE_ENTRY for expired leases, etc., as well as an optional SchedulerDaemon
The figure below summarizes the DI STRI BUTEDprocessing mode:
which looks for submissions
from the online system.
F1_TSPACE_ENTRY
StandardExecuter
THREAD_OFFER
WORK_OWNER
submitter=true
batchCode=J OB1
theadPool=DEFAULT
threadCount=4
GRID_WORK
WORK_ENDED
THREAD_OFFER
WORK_OWNER
GRID_WORK
WORK_ENDED
Submitter Node
StandardExecuter
submitter=true
batchCode=J OB2
theadPool=DEFAULT
threadCount=2
Submitter Node
StandardExecuter
submitter=false
theadPool=DEFAULT
threadOffered=4
theadpoolworker Node 1
StandardExecuter
submitter=false
theadPool=DEFAULT
threadOffered=4
theadpoolworker Node 2

Figure 2 DISTRIBUTED mode
For publishing purposes, the following facilities are not depicted in the above figure:
The PER_STATE rows that contain the Thr eadWor kUni t entries (i.e. the data) for each
submitted thread.
The HouseKeapi ngDaemon. This singleton periodically monitors the
F1_TSPACE_ENTRY table for expired entries and deletes them. It also has a GRI D_WORK
entry on the F1_TSPACE_ENTRY table.
The Schedul er Daemon. This singleton looks for submissions from the online system by
polling the CI _BATCH_J OB table used for online submission. If found, it submits the job
to the grid. It also has a GRI D_WORK entry on the F1_TSPACE_ENTRY table.
The figure below illustrates the flow of data in DI STRI BUTEDexecution mode:

12
J avaJ ob
Executer
CobolJ ob
Executer
SubmitBatch
StandaloneExecuter
DistributedGridNode
DistributedJ obExecuterStub
ThreadPoolWorker
StandaloneExecuter
AbstractGridWork
AbstractBatchWork
DistributedGridNode
SpaceManager
SpaceJ DBC
SpaceChangePoller
ThreadPool
ThreadOfferManager
WorkProcessor
AbstractGridWork
J obExecuterWork
BatchJ obExecuter
DistributedJ obExecuter
J avaJ ob
Executer
CobolJ ob
Executer GRID_WORK
PER_STATE
Read Thread Work
Store Thread Work
Submit Threads
Submit Job
Execute Job
Execute
Thread
Grab Work
Submitter Node Worker Node

Figure 3 DISTRIBUTED process mode
In DI STRI BUTED mode, the submitter node serializes the appropriate Gr i dWor k entry to the
F1_TSPACE_ENTRY table, and then waits for the job completion WORK_ENDEDentry.
The worker performs the job portion of the execution by calling either J avaJ obExecut er or
Cobol J obExecut er , in which the initialization of the job takes place. In the case of
J avaJ obExecut er , the applications get J obWor k method is invoked and the thread work units
that it collects are divided into the thread chunks and stored in F1_TSPACE_ENTRY PER_STATE
rows. The worker then submits the threads by serializing the thread Gr i dWor k entries to
F1_TSPACE_ENTRY GRI D_WORK rows so that a similar path is followed to execute the individual
threads.
Issues with DISTRIBUTED mode
The DI STRI BUTEDmode is currently is the default mode and is in production at a number of sites
and works well for the most part, but there are a few limitations with this mode:
If a worker node drops off unexpectedly, as happens if a program crashes, the submitter node is
not made aware of it immediately. To the user it looks like the job is still in progress. When
the worker is restarted, the submitter does get notified at that point, which is not ideal.
Note: In this case the worker node logs the frequently misunderstood message, "Maximum number of grid
work failures was reached (1)".

13
If a submitter node drops off, the worker nodes are not aware of that and will continue to
process the job to the end. This is a lesser problem, but is also not ideal.
A worker or submitter nodes health is highly dependent on good, consistent database
response. The database, by nature, is volatile and response times can vary, and it causes
intermittent problems that are difficult to troubleshoot. The most common error is one that
indicates that a lease cannot be renewed, and the Oracle Utilities Application Framework can
and does self-correct most of the time, but it very often leads to further problems.
The last execution mode, CLUSTERED
4
Oracle Coherence provides a grid-enabled implementation of the IBM and BEA CommonJ Work
Manager, which is the basis for JSR-237. Using a Work Manager, the product can submit a collection
of work (a job or set of threads) that needs to be executed. The Work Manager distributes that work in
such a way that it is executed in parallel, typically across the grid. In other words, if there are ten work
items submitted and ten servers in the grid, then each server will likely process one work item. Further,
the distribution of work items across the grid can be tailored, so that certain servers (e.g. one that acts
as a gateway to a particular mainframe service) will be the first choice to run certain work items, for
sake of efficiency and locality of data. The application can then wait for the work to be completed, and
can provide a timeout for how long it is willing to wait.
, provides the same facilities as DI STRI BUTEDmode but
eliminates the need for F1_TSPACE_ENTRY and uses the Named Cache and Work Manager
facilities in Oracle Coherence to address the issue.
When a worker starts, the initialization is similar to DI STRI BUTEDmode, except that a Coherence
based cluster node is started by creating a Wor kManager for each thread in the thread pool. In
Coherence these are known as services, and they specify the number of threads that are offered in that
service. This is exactly the same as the existing concept of thread pools. The objects needed for
classic DI STRI BUTEDmode to manage thread pools, poll for work, create and renew leases, and so
on, are not initialized as that is redundant in a Coherence cluster.
An important difference to note here is that a CLUSTEREDworker plays a far more passive role in the
batch grid. A DI STRI BUTEDworker proactively polls for new work, whereas a CLUSTEREDnode
waits for work to be handed to it from a submitter.
A CLUSTEREDsubmitter also goes through the same initialization as DI STRI BUTEDmode and, like
the worker, starts a Wor kManager instance to join the cluster. This Wor kManager is created with
a thread number specification of zero, which indicates to Coherence that it is a client.

4
CLUSTERED mode is only available with Oracle Utilities Application Framework 2.2 and above. For
Oracle Utilities Application Framework 2.2 customers, refer to the Batch Operations and Configuration
Guide for the appropriate patches to apply to enable CLUSTEREDmode.

14
The job is then submitted by scheduling a serializable Work object. This object should be easily
created from the existing J obExecut er Wor k. This is a Wor kManager schedule call, which
serializes the object to its destination worker node and waits for it to finish. This call will perform the
"job" portion of the run by calling the standard J avaJ obExecut er or Cobol J obExecut er
appropriately.
Where the CLUSTERED implementation changes significantly from the DI STRI BUTED
implementation is in the submission of the threads. In DI STRI BUTED mode, a worker submits the
threads, in effect becoming a submitter (i.e. client). In contrast, the CLUSTEREDimplementation will
make this the responsibility of the submitter node, so that it is physically networked with its worker
nodes. This will result in appropriate notifications in the event of nodes "dropping off". The figure
below illustrates the concept:
Cluster
F1_TSPACE_ENTRY
Machine Machine
StandardExecuter
submitter=true
batchCode=J OB1
theadPool=DEFAULT
threadCount=4
Submitter Node
StandardExecuter
submitter=false
theadPool=DEFAULT
threadOffered=4
StandardExecuter
submitter=false
theadPool=DEFAULT
threadOffered=2

Figure 4 CLUSTERED mode

15
CLUSTERED Mode internals
From the batch applications perspective, the underlying technology is completely transparent.
Whether the execution mode is THI N, DI STRI BUTED or CLUSTERED has no effect on the
application programs logic.
The CLUSTERED mode utilizes the Oracle Coherence NamedCache feature as a way of sharing
information with members of the cluster. There are two NamedCaches, one for job submission and
the other for service information. The Bat chCl ust er Cache moderates all insertions and deletions
to these caches.
The process flow for the CLUSTEREDmode is illustrated in the figure below:
Worker Node Submitter Node
BatchWorkManager
SubmitBatch
StandaloneExecuter
ClusteredNode
MemberListener
MapListener
BatchWorkManager
ThreadPoolWorker
StandaloneExecuter
ClusteredNode
MemberListener
MapListener
BatchWorkManager
BatchWork
AbstractGridWork
AbstractBatchWork
J ava
Batch
Work
COBOL
Batch
Work
Join Coherence Cluster
For Thread Pool as Client
BatchClusterCache
Join Coherence Cluster
For Thread Pool(s)
NamedCache NamedCache
Submit job by inserting entry into
submission cache and wait for
completion
E
x
e
c
u
t
e
Job Submission Cache
Job inserted/deleted event
Updates available threads in
service information cache
when work assigns or
completes
Insert Service
Information when
WorkManager is
created
Orchestrates
read/write to the
Named Caches
that are available
to all members
of the cluster
S
c
h
e
d
u
l
e
Service Information Cache

Figure 5 CLUSTERED process flow
The details of the flow are outlined in the subsections below.
Worker Initialization
When a worker starts, it either joins an existing cluster or creates a new one if the cluster name with
which it is started does not exist. Refer to Member Validation below for more details.
The initialization is then similar to DI STRI BUTEDmode, except that a Cl ust er edNode is created
instead of a Di st r i but edGr i dNode. The Clustered node implements the Member Li st ener

16
interface to listen for member events, and the MapLi st ener interface to listen for insertion and
deletion of a NamedCache.
Note: An online application server may also host a batch worker, but it is generally only recommended for test,
development or demonstration purposes. This is done by specifying the property
com. spl wg. gr i d. onl i ne. enabl ed=t r ue in the spl . pr oper t i es file. The configuration
for this is usually done via the conf i gur eEnv[ . sh] utility.
During the initialization process, the Cl ust er edNode creates a Bat chWor kManager for each
thread pool specified in the properties file or command-line argument. With each Bat chManager
created, the Cl ust er edNode is registered as the Member Li st ener so that it will be notified of
any member events, and an entry is inserted into the service information cache. The service
information cache contains all information needed for a particular service (thread pool), for example
what each thread in that particular thread pool is currently running.
Submitter Initialization
When a submitter starts, it checks that there is a threadpool to service its work. If a cluster does not
exist (i.e. no workers have been started for the cluster) or the existing cluster does not offer threads in
the desired threadpool, the message There is no member in the cluster with that pool name is logged and the
submitter is terminated with a non-zero exit code. Refer to Member Validation below for more details.
The Cl ust er edNode then initiates the job submission by inserting an entry into the job submission
cache. The submitter waits for job completion by monitoring the job submission cache. The insertion
of the job into the job submission cache fires an entry-inserted event that gets processed by all
Cl ust er edNodes that are servicing the particular pool name specified for that job submission. A
node acquires the job by being the first to update the job submission entry with its member id. The
node executes the Submi t Bat chRun session executable and schedules each work entry using a
Bat chWor kManager client. Each work entry is only scheduled as a thread becomes available.
Process
When the Bat chWor kManager processes work, it updates the service info cache and the job
submission cache to specify the member id doing the processing and bat chThr eadI d being
processed. When the work is complete, that information is removed. Once all the work is completed,
the job status of the job entry in the submission cache is changed to ended, signaling to the submitter
that the job completed.
If a worker goes down, a member left event is generated and processed by all registered
Cl ust er edNodes. The submitter uses this event to get the pending work list of the member and,
once the job is complete, the submitter uses this pending work list to update the thread status to error
and to report that a worker unexpectedly went down.
If a submitter goes down, a member left event is also generated for all registered Cl ust er edNodes.
All the nodes processing any work for that particular submitter immediately cancel the corresponding

17
threads for the submitter and update each threads status to error. The submitter itself then terminates
with a non-zero exit code.
Member Validation
An important operation is to validate that any node that joins a grid is supposed to join that particular
grid, and not some other. This is critical to ensure that a submitter node is joined to its intended
cluster before submitting a job to run against that environment. It would be disastrous if, for example,
an archiving job meant to run against a test system were inadvertently submitted to the production
system.
To prevent this:
When joining a cluster, a basic handshake protocol is used to validate that the new member is
connected to the same database as the other members in the cluster. The new member inserts
a unique MEMBER_VALI D entry in the F1_TSPACE_ENTRY table and then waits for
confirmation from an existing member that it saw that same entry.
The cluster name (t angosol . coher ence. cl ust er ) is a mandatory property. By
specifying a unique name for each separate environment, accidental discovery between
environments is avoided.
Note: This property is optional for Oracle Coherence, and is therefore not truly mandatory in that an error will
be reported if omitted, but the validation as described above ensures against arbitrary unions in a cluster. A
unique value for it must be specified for multiple environments to separate the clusters and to avoid cluster
validation errors.
Scheduler Daemon
The scheduler daemon is what enables online job submissions to be processed. It polls the
CI _BATCH_J OB table for Pending entries and submits them to the batch cluster by inserting
appropriate entries into the job submission cache and then polling for completion. This process is
exactly as described above for job submitters, however a scheduler daemon also updates the status to
Ended on the CI _BATCH_J OB tables when the job (i.e. last thread) has finished.
Note: This does not indicate success or failure, but merely whether the threads for the job have ended. The job and thread
statuses are held on the CI _BATCH_RUN and CI _BATCH_THD(Batch Run Tree) tables.
The scheduler daemon is a singleton process (i.e. exactly one of them running on the cluster at any one
point) with appropriate failover in the event of the hosting worker dropping off. If this happens,
another worker that is daemon-enabled will become the host. It is therefore advisable to set the
property com. spl wg. bat ch. schedul er . daemon to t r ue (or use the - d Y command-line
option) to enable this failover capability.
An online application server may also host the scheduler daemon, even if it does not host a batch
worker in the cluster. In other words, these properties, in the spl . pr oper t i es file for the online
application server are perfectly acceptable:

18
com. spl wg. gr i d. onl i ne. enabl ed=f al se
com. spl wg. bat ch. schedul er . daemon=t r ue
Socket issues in CLUSTERED mode
To help minimization of packet loss, the operating sockets buffers need to be large enough to handle
the incoming network traffic while your Java application is paused during garbage collection. By default
Oracle Coherence will attempt to allocate a socket buffer of 2MB. If your operating system is not
configured to allow for large buffers Oracle Coherence will utilize smaller buffers. Most versions of
Unix use a very low default buffer limit, which should be increased to at least 2MB.
Starting with Oracle Coherence 3.1 you will receive the following warning if the operating system failed
to allocate the full size buffer.
Uni cast UdpSocket f ai l ed t o set r ecei ve buf f er si ze t o 1428 packet s
( 2096304 byt es) ; act ual si ze i s 89 packet s ( 131071 byt es) . Consul t
your OS document at i on r egar di ng i ncr easi ng t he maxi mum socket buf f er
si ze. Pr oceedi ng wi t h t he act ual val ue may cause sub- opt i mal
per f or mance.
Though it is safe to operate with the smaller buffers it is recommended that you configure your OS to
allow for larger buffers.
TABLE 1 RECOMMENDED OS SETTINGS FOR NETWORKING
CLUSTERING MODE CONFIGURATION SETTINGS
Linux sysct l - w net . cor e. r mem_max=2096304
sysct l - w net . cor e. wmem_max=2096304
Solaris ndd - set / dev/ udp udp_max_buf 2096304
AIX no - o r f c1323=1
no - o sb_max=4194304
Windows Windows does not impose a buffer size restriction by default.
Note: AIX only supports specifying buffer sizes of 1MB, 4MB, and 8MB. Additionally there is an issue with IBM's
1.4.2, and 1.5 JVMs which may prevent them from allocating socket buffers larger then 64K. This issue has been
addressed in IBM's 1.4.2 SR7 SDK and 1.5 SR3 SDK.
Clustering using Unicast or Multicast
Note: This facility applies to Oracle Utilities Application Framework V2.2 SP7 and above only

Note: Network protocols are more effective when the cluster is across more than one machine. On a single machine there is
little difference between the network protocols.

19
One of the configuration questions that need to be considered with the CLUSTEREDexecution mode
is whether you will use multicast or unicast for the threadpools. By default, a multi-cast based
configuration is provided by the configuration files provided with the installed product.
When deciding whether to use multicast or unicast the following advantages and disadvantages should
be considered:
TABLE 2 DAEMON CONSIDERATIONS
CLUSTERING MODE ADVANTAGES DISADVANTAGES
Multicast (default) Only have to submit to one active node in
cluster.
Threadpools can be clustered and Work
Manager can load balance across them.
Threadpools communicate across cluster
Clusters can be shared or dedicated
Cluster nodes can be added dynamically for
load fluctuations
Network traffic between clusters is via
multi-cast address.
Some sites do not like the multicast
protocol
Unicast Can submit to specific nodes (micro
management)
Clusters can be shared or dedicated
Minimal interaction between nodes
Each node has to be defined to other
nodes (no dynamic node support)
Increased configuration requirements
(Well Known Address support)
Nodes should be on different machines
(current limitation only)
When configuring multicast or unicast a number of settings in the
t hr eadpool wor ker . pr oper t i es and submi t bat ch. pr oper t i es should be
considered:
TABLE 3 CLUSTER CONFIGURATION SETTING
CLUSTERING MODE CONFIGURATION SETTINGS
Multicast t angosol . coher ence. cl ust er addr ess
t angosol . coher ence. cl ust er por t
Unicast t angosol . coher ence. l ocal host
t angosol . coher ence. l ocal por t
t angosol . coher ence. wka
t angosol . coher ence. wka. por t
Please refer to the Operations And Configuration Guide or Batch Server Administration Guide
For more information about multi-cast and uni-cast see the following sites:
for
your product for details of these settings.
Discussion of protocols - http://wiki.tangosol.com/display/COH35UG/Network+Protocols
Advanced Configuration of the multi-cast listener -
http://wiki.tangosol.com/display/COH35UG/multicast-listener

20
Advanced Configuration of the uni-cast listener -
http://wiki.tangosol.com/display/COH32UG/unicast-listener
Note: In Oracle Utilities Application Framework V4.1 and above , the t angasol parameters have been
moved to t angasol - coher ence- over r i de. xml . Refer to the Batch Server Administration Guide
Migrating from DISTRIBUTED to CLUSTERED mode

provided with your product for more information.
Note: This facility applies to Oracle Utilities Application Framework V2.2 SP7 and above only

With the advent of the CLUSTERED execution mode, existing customers of the Oracle Utilities
Application Framework using the DI STRI BUTEDexecution mode can migrate to the CLUSTERED
execution mode. To migrate to CLUSTEREDexecution mode the following must be performed:
Execute the conf i gur eEnv[ . sh] utility to alter the configuration files.
Specify the following configuration values:
CONFIGURATION SETTING COMMENTS
Batch RMI Port Default J MX port for threadpool. Must be unique per environment
Batch Mode Change to CLUSTERED
Coherence Cluster Name Name for the Cluster
Coherence Cluster Address Multi-cast address for use by this cluster
Coherence Cluster Port Port number used for multi-cast address
Coherence Mode Mode of execution of Coherance
Refer to the Batch Operations And Configuration Guide or Batch Server Administration
Guide
Execute the i ni t i al Set up[ . sh] utility to reflect the changes in the product. This may
require additional steps to implement the change for selected platforms. Refer to the
with your product for suggested values for these parameters.
Operations And Configuration Guide or Server Administration Guide
Very few changes have been made to t hr eadpool wor ker . pr oper t i es and
submi t bat ch. pr oper t i es. The t angasol . coher ence. * should be set in these
files.
with your product for
additional advice.
Note: In Oracle Utilities Application Framework V4.1 and above , the t angasol parameters
have been moved to t angasol - coher ence- over r i de. xml . Refer to the Batch Server
Administration Guide
Remove any custom com. spl wg. gr i d. execut i onMode settings from any job
specific configuration files (if used).

21
If your site wishes to use unicast rather than multicast then alter the
t hr eadpool wor ker . pr oper t i es and submi t bat ch. pr oper t i es files
manually as outlined in the Batch Operations And Configuration Guide or Batch Server
Administration Guide
You are now migrated from DI STRI BUTEDto CLUSTEREDexecution mode.
provided for your product version.
CLUSTERED Mode Operations

One of the advantages of using CLUSTERED mode is the simplification of operations for the
threadpoolworker and the submitter nodes. The following operations are applicable to CLUSTERED
mode:
If the threadpoolworker JVM is killed or crashes each related submitter node that was running
in that threadpoolworker is immediately terminated with a non-zero return code and the
relevant batch run tree entries are set to Error status. In DI STRI BUTEDmode, a submitter
node waiting for work to finish is not made aware of an event such as a kill or JVM crash until
the relevant threadpoolworker is restarted.
The submitted node can be gracefully ended by killing the submitter node process on the
operating system. The affected threads are systematically cancelled and the relevant batch run
tree entry statuses updated with appropriate messages. This is not possible in
DI STRUBUTEDmode. JMX can be used in DI STRI BUTED, as it also can in CLUSTERED
mode, but to cancel an entire job requires using the JMX calls to find the appropriate
threadpoolworker(s) where the threads are running and cancelling each submitter thread
individually.
Database access is minimized. In DI STRI BUTEDmode the entire grid is controlled through
a single database table (F1_TSPACE_ENTRY). This table continually gets polled by listeners
for newly submitted work and work that ended, as well as lease renewal agents that manage
the leases for the active nodes. This polling does result in a significant number of additional
database calls and grows as more nodes join the grid. In contrast, CLUSTEREDmode uses
shared cache for clustering, which in turn controls membership, so the
F1_TSPACE_ENTRY table is only used in a very minimal capacity.
Lease renewals issues removed. In DI STRI BUTED mode the lease renewals relied upon
good database response. If the database experienced high demand these renewals would error
which could lead to incomplete threads. In CLUSTEREDmode the last renewals are cache
based and do not depend on the database.

22
Testing for multi-cast
If your site is considering using the multicast option for CLUSTERED mode then one of the
recommendations is that you verify your environment is enabled for multicast. Oracle Coherence
supplies a test utility that can be used to test whether your machine is setup for multicast. Refer to
http://download.oracle.com/docs/cd/E15357_01/coh.360/e15723/tune_multigramtest.htm#BHAI
CAGE for more information about testing for multicast.
Demonstration Single machine setup
If you are interesting in demonstrating the CLUSTERED mode or want to avoid supplying the
clustered address and clustered port then it is possible to limit the cluster to a single
machine/environment then the following settings can be set:
TABLE 5 DEMONSTRATION SETTINGS
CONFIGURATION SETTING COMMENTS
t angosol . coher ence. t t l Set to 0
t angosol . coher ence. l ocal host Set to 127. 0. 0. 1
t angosol . coher ence. cl ust er addr ess Remove parameter as this is defaulted
t angosol . coher ence. cl ust er por t Remove parameter as this is defaulted

Note: These settings only apply to a single copy of the product on a single machine. They are suggested for demonstration
or training purposes only and are not recommended for production.
Timed Batch
Note: This facility applies to Oracle Utilities Application Framework V4.0 and above only
One of the features of the Oracle Utilities Application Framework is the ability to support continuous
or timed batch processes. For example, there are monitor processes built into the product. These
monitor processes track status and business rules for specific objects and then process the data
according to the status and object configuration. This is typically is state transition where an object is
moved from one state to another via this monitor process according to the object specification. This
monitor process will probably be more effective in this situation if it ran continuously.
The facility consists of a Batch Control Type of Timed and a series of attributes to indicate the
attributes of the batch process at execution time. The figure below illustrates the additional attributes
available when Batch Control Type is set to Timed:

Figure 6 Timed Batch attributes

23
The term continuous and timed use the same facilities within the batch framework but have different
processing flows:
Continuous Batch This is a style of writing a batch process. The batch logic executes in a
repeated timed loop, with configurable delay interval and an (optional) maximum duration.
The timings are specified as batch parameters, with millisecond resolution. To the outside
world this just looks like a potentially very long-lived batch run. For this mode the batch
control indicates the job is controlled under a timer control.
Timed Batch This type of batch control allows a job to be executed every timer interval.
The timer creates Bat chJ obQueue entries and the batch infrastructure creates the actual
Bat chRun instances to execute the batch process. The site configures the desired interval
between starts (with resolution given in seconds). The system schedules new runs at each
interval if the last instance of the job has completed. If you turn the timer off any running job
finishes normally, but a new one won't be auto-scheduled.
Timed and continuous jobs are controlled by the default threadpool and not by a scheduler. When the
DEFAULT t hr eadpool wor ker starts it will start executing any job of Batch Control Type set to
Timed with the Timer Active set to Yes. This is whether the batch daemon or batch server is enabled or
not.
Note: If you intend to run Timed or Continuous Batch for your any of your monitor jobs then it is recommended that an
instance of the DEFAULT threadpool be made available to execute the timed or continuous jobs.
To stop a timed or continuous running batch job then it is recommended to change the Timer Active
value on the Batch Control to No. The job will not restart the next timer interval.
As continuous or timed batches are run automatically then additional information must be provided on
the batch control:
User The user to use for execution of the batch process. This user must have security access
to the necessary objects accessed by the batch process.
Batch Language The language for any messages.
Email Address (optional) The email address to send the output on completion of execution.
Thread Count Number of threads to allocated to the job
Override Number Records to Commit (optional) Commit interval for job. This applies to
the complete execution of the job regardless of what part of the day it is submitted.
CLUSTERED Mode recommendations
Note: This new facility applies to Oracle Utilities Application Framework V4.0 and above only
One of the new features of the CLUSTEREDapproach is the ability to optimize Coherence for the
batch activities you are performing on the product environment. A new configuration setting has been
added to Oracle Utilities Application Framework V4.0.2 and above, where the mode in which

24
Coherence operates can be set. The setting is t angosol . coher ence. mode set in the
submi t bat ch. pr oper t i es and t hr eadpool wor ker . pr oper t i es. The setting has valid
values of pr od for Production and dev for Development. These modes do not limit access to
features, but instead alter some default configuration settings. For instance, development mode allows
for faster cluster startup to ease the development process.
The Oracle Coherence mode setting should be set using the following guidelines:
During non-production activities not involving a cluster, the mode should be set to dev. It is
recommended to use the development mode for all pre-production activities, such as
development and testing. This is an important safety feature, because Coherence automatically
prevents these nodes from joining a production cluster.
For production and environments where clustering is required (for cluster testing) the mode
should be set to pr od. The production mode must be explicitly specified when using
Coherence in a production environment.
Threadpools and Database Recycling


Note: This advice applies to c3p0 only and therefore only applies to Oracle Utilities Application V2.2.
If at any stage the database is shutdown and restarted (i.e. recycled) any active threadpools will become
zombies. This means the threadpools will not process any work and seem to be looping. This can be
addressed by changing the tolerances for the database connection to reconnect successfully after
recycling.
In the spl . pr oper t i es for the batch component the following parameters need to be changed to
address connectivity for database recycles:
hi ber nat e. c3p0. i dl e_t est _per i od = 10
(a.k.a. c3p0. i dl eConnect i onTest Per i od)
c3p0 will test all idle, pooled but unchecked-out
connections, every 10 seconds. This test is done
asynchronously.
hi ber nat e. c3p0. acqui r e_i ncr ement = 2
(a.k.a. c3p0. acqui r eI ncr ement )
Determines how many Connections a c3p0 pool will
attempt to acquire when the pool has run out of
Connections. (Regardless of acqui r eI ncr ement ,
the pool will never allow maxPool Si ze to be
exceeded.)
hi ber nat e. c3p0. t i meout = 0
(a.k.a. c3p0. maxI dl eTi me)
This is optional. This parameter sets the number of
seconds a connection can remain pooled but unused
before being discarded. Zero means idle connections
never expire. This is suggested for customers who are

25
using Oracle Audit Vault or Oracle auditing facilities
to remove contention on audit tables when
connections are idle.
EXTENDED Mode
Note:This mode is only supported for specific situations and is only available for Oracle Utilities Application Framework
V2.2 and above via Patch 1173571 (FW4.1) and Patch 11683404 (FW2.2) .
In most cases the CLUSTEREDmode is sufficient for most background processing needs but if you
require a large number of submitters then you might want to consider a new mode called EXTENDED.
This mode utilizes the Coherence*Extend features to provide an alternative way for threadpoolworker
and submitters to be setup.
Note: EXTENDEDmode should only be used if there are a large number of submitters used and the CLUSTERED
mode is reporting communication delays. Communication delays can occur when large numbers of submitters complete.
The EXTENDEDmode differs from CLUSTEREDin a number of key ways:
Submitters using EXTENDEDmode do not participate in any cluster.
EXTENDED mode has configuration overhead in that each threadpoolworker must be
configured to start a port for listening for submitters explicitly.
Each submitter is tied to its configured threadpoolworker (though multiple threadpoolworkers
can be configured as failovers if necessary).
Note: The same threadpoolworker can be used for both CLUSTEREDand EXTENDEDmode if desired.
The idea is to define threadpoolworkers in your configuration as proxies and then define the
submitters to point to those proxies to implement the mode. The proxies are defined in the
coher ence- cache- conf i g. xml configuration files and then augment the local
submi t bat ch. pr oper t i es files (using an external configuration file) on each node to point to
the proxy servers defined. The examples below illustrate a single proxy but it is possible to define
multiple proxies to the submi t bat ch. pr oper t i es to failover balance across proxies.
To use EXTENDEDmode the following must be performed:
Change the following entries in the submi t bat ch. pr oper t i es in the
$SPLEBASE/ spl app/ st andal one/ conf i g subdirectory (or
%SPLEBASE%\ spl app\ st andal one\ conf i g on Windows):
Change the com. spl wg. gr i d. execut i onMode to EXTENDED. For
example:
com. spl wg. gr i d. execut i onMode=EXTENDED
Set the location of configuration file used for the mode using the
t angosol . coher ence. cacheconf i g configuration setting. The setting is in
the format:

26
t angasol . coher ence. cacheconf i g=<f ul l pat h>
where
<f ul l pat h> Full path to configuration file on local machine including
configuration file name (usually ext end- cl i ent -
conf i g. xml )
Edit the $SPLEBASE/ spl app/ st andal one/ conf i g/ coher ence-
cacheconf i g. xml (or
%SPLEBASE%\ spl app\ st andal one\ conf i g\ coher ence- cache-
conf i g. xml on Windows) file to add the proxy scheme section. Substitute x.x.x.x
for the local host IP or hostname and yyyy for the port number to listen upon. For
example:

Create a cache configuration file ext end- cl i ent - conf i g. xml to house the
configuration settings. The main change is to implement a Proxy Server replacing the
x.x.x.x with the hostname and yyyy with the port number specified in the
coher ence- cache- conf i g. xml file. For example:

27


28
Note: For a full description of the configuration file see Cache Configuration Elements
If the com. spl wg. bat ch. cl ust er . pol l Ti meI nSeconds is set in both
submitbatch.properties and threadpoolworker.properties, then ensure it is set to an
appropriate value is set for your site. The value must be the same value in both files to prevent
miscommunications. The default value is 10 seconds, if no setting is specified. The internal
housekeeping daemon uses this value (2x it's value to be precise) and the client will poll using
this interval.
Note: This change alters the way that batch programs initialize context as well as ensure SQL statements are closed at
appropriate times. If this causes some custom programs issues then it is recommended to add the
com. spl wg. submi t bat ch. useOl dExi t CodeHandl i ng=t r ue to the
submi t bat ch. pr oper t i es file. This will enforce the older Exit code handling within the execution of the
program.
Note: When executing in EXTENDEDmode a message outlining that the cache configuration file has been loaded will
appear in the threadpoolworker log files.
Operational Best Practices
The following section outlines the common operations best practices based upon customers
experience and internal studies.
Parameter Guidelines
Given the flexibility of the overriding configuration settings, there are a number of permutations and
combinations of configuration files and command line options to suit your sites needs. The following
guidelines may assist in deciding the optimal mix for your site:
Internal defaults should not be relied upon for non-development use. They are provided for
developers to unit test their code for various testing techniques.
The t hr eadpool wor ker . pr oper t i es and submi t bat ch. pr oper t i es files
should represent your sites global parameter settings. See Commonly Used Configuration
Settings
For the submi t j ob[ . sh] utility the provision of a job specific parameter file is not
necessary in most cases. The following are the only exceptions to this rule:
for guidelines of what commonly is set in those files.
If the background process requires additional parameters then a job specific
parameter file is required.
If any parameter on the job must be overridden on a regular basis then a job specific
parameter file should be created and only contain the parameters that are to be
overridden.

29
The command line options should only be used for reruns or off schedule (a.k.a. special runs).
This avoids updating the configuration files for one-off or special processing. The only
exception to this rule is that the Business Date parameter should be specified on the
command line to avoid the past midnight issue. For more details of this see Scheduler
Implementation Guidelines
While it is possible to override most parameters on the command line, it is not desirable to do
so as this is not efficient. Configuration files are designed to minimize the need for command
line overrides unless such overrides are applicable to the particular execution of the
background process.
.
Command line options are the only option to be used if threads in a background process need
different parameters. For example, if the record range is available as a parameter then the
command line options must be used to specify that record range on the command line.
It is recommended to specify the threadpool on the t hr eadpool wor ker [ . sh] utility to
avoid confusion and allow greater control to start a particular threadpool. This can apply to
the submi t j ob[ . sh] utility as well as it allows the background process to be targeted to a
specific named threadpool.
Use Of Batch Control For Parameters
One source of configuration settings is the Batch Control. Typically, most sites configure their Batch
Controls with the default values for their business practices. In particular extra parameters values and
defaults are specified on batch controls to assist in testing.
While the values are stored in the Batch Control they are not used by the Scheduler Submission
Method as the configuration files are the source of the configuration parameters. This information
needs to be extracted to the configuration files from the batch control.
Note: The parameters used by the Batch Control are stored in the CI _BATCH_CTRL_P table which has Batch
Code (BATCH_CD) and Batch Parameter Name (BATCH_PARM_NAME) as keys. The parameter value is stored
in column BATCH_PARM_VAL. It is not recommended to use BATCH_PARM_NAME='MAX- ERRORS'.
Refer to Setting The Error Tolerance
The reason the Batch Control is not referenced is that the Batch Control is managed by the business,
where the configuration files are under IT control. Typically, Change Management principles separate
the responsibility for these elements. The primary reason is to isolate the physical system from un-
intentional changes to parameters by the business.
for details. Remember ANY parameter must be in job specific configuration files in
the format:
com. spl wg. bat ch. submi t t er . sof t Par amet er . <BATCH_PARM_NAME>=<BATCH_P
ARM_VAL>and do not include Blank values.
Multiple Batch Controls for a Batch Program

30
One of the flexible features of the batch control is the fact that a batch program can be referenced on
multiple batch controls. The batch program is a parameter of a batch control. This is useful to use in a
number of situations:
You need to run an execution of a program where you do not want to alter the existing batch
control. The batch control holds the run number for the next execution and there may be a
business situation where you need to run a one-off execution and do not want it to affect the
batch number.
You want to execute a business process a number of times with different parameters. Some
customers use a common extract format (via a common extract program) but run it multiple
times with different parameters to send to individual interface targets. For example, you may
decide a common format for sending collection information to a number of collection
agencies. In this case, a common program would be written and a batch control created per
collection agency that needed the information. This would allow tracking at an individual
collection agency level and a separation of the execution of the process.
You have a new background based business process you want to trial before you replace an
existing background process. This allows parallel execution.
An example where multiple batch controls for a program is used in the base functionality is ConfigLab.
The Synchronize and Compare Source batch controls reference a common batch program with
different parameter values (in fact the batch control is an implied parameter to the program in this
case).
Take Defaults For Parameters
Each background process has a number of standard parameters that control its behavior. In most cases
the default values are sufficient for the needs of most sites.
Note: Environmental settings such userid, file location and file names are exceptions to this guideline.
The following values should not be altered unless otherwise recommended:
Trace Program Start
Trace Program Exit
Trace SQL
Trace Standard Output.
Commonly Used Configuration Settings
Typically not all configuration settings are configured for each site as defaults are sufficient or can be
set at a global level to cover all background processes. The following guidelines should be considered
when setting values:

31
It is not recommended to set MAX- ERRORS at a global level. This should only be specified at
an individual thread of a background process level. Refer to Setting The Error Tolerance
The Userid used for background processes should be specified at a global level (in
submi t bat ch. pr oper t i es configuration file). This userid is used to mark records
processed with this userid and security.
for
more details.
It is not recommended that the default SYSUSER be used as the userid specified in
background processes. SYSUSER is the initial default user for the Oracle Utilities Application
Framework and is only used to input additional user in the initial stages of configuration. Most
customers create a dedicated user record (such as BATCH) to delineate background processes
from online processes.
The pr ompt For Val ues parameter should be set to f al se in the
submi t bat ch. pr oper t i es configuration file. Alternatives are only used for
development purposes.
The execut i onMode parameter should be set to CLUSTERED in the
submi t bat ch. pr oper t i es and t hr eadpool wor ker . pr oper t i es
configuration files. For customers using versions of Oracle Utilities Application Framework
not supporting the CLUSTERED execution mode, then an execut i onMode of
DI STRI BUTEDshould be used. Refer to the "Batch Operations And Configuration Guide
The di st Thr eadPool parameter in the submi t bat ch. pr oper t i es configuration
file should be set to a common site specific default threadpool used in your implementation.
Refer to "
"
supplied with your product for details of this setting.
Designing Your Threadpools
Setting The Error Tolerance
" for more details.
By default, there are two circumstances where a background process will not complete successfully.
First of all there is some technical issue with the environment or program and the other is if all records
processed fail some business rule. The latter condition may not be appropriate for your site. For
example, if all but one payment fails in the payment background processes then the process is still
considered complete.
To support a more realistic tolerance, it is possible to set a limit on the number of errors tolerated
before the process should be cancelled. At the thread level, the MAX- ERRORS parameter
5

5
The parameters MAX- ERRORS, MAX_ERRORS and maxEr r or s are all acceptable names for this
parameter. For publishing purposes MAX- ERRORS will be used.
can be used

32
to specify a thread level error tolerance where the thread will be cancelled when the tolerance is
reached or exceeded.
The default value of MAX- ERRORS is 0 (zero) which turns off the facility. Any appropriate non-zero
value will then become the error tolerance limit for the thread.
Setting the appropriate value for your site will require business approval in line with your sites
organization business practices.
Note: The MAX- ERRORS parameter is applicable to business errors only. System errors (SEVERE errors) will
terminate the process immediately.
Multi-threading Guidelines
Running a background process in multiple threads is almost always faster than running it in a single
thread. The trick is determining the number of threads that is optimal for each process.
A rule of thumb that may be used is to have three (3) threads per core available. For example if you
have a quad core processor, you can run twelve (12) threads to begin your testing.
This is a rule of thumb because the footprint of each process is different ("heavy" versus "light") and is
dependent on the data in your database. Your hardware configuration (i.e., number of processors,
speed of your disk drives, speed of the network between the database server and the application server)
also has an impact on the optimal number of threads. Please follow these guidelines to determine the
optimal number of threads for each background process:
Execute the background process using the number of threads dictated by the rule of thumb
(described above). During this execution, monitor the utilization percentage of your
application server, database server and network traffic.
If you find that your database server has hit 80-100% utilization, but your application server
hasnt one of the following is probably occurring:
There may be a problematic SQL statement executing during the process. You must
capture a database trace to identify the problem SQL.
It is also possible that your commit frequency may be too large. Commit frequency
is a parameter supplied to every background process. If it is too large, the databases
hold queues can start swapping. Refer to Parameters Supplied to Background
Processes
It is normal if you find that your application server has hit 80-100% utilization but
your database server has not. This is normal because, in general, all processes may
become CPU bound and not IO bound. At this point, you should decrease the
number of threads until just under 90-100% of the application server utilization is
achieved. And this will be the optimal number of threads required for this
background process.
for more information about this parameter.

33
If you find that your application server has NOT hit 80-100% utilization, you should
increase the number of threads until you achieve just under 90-100% utilization on
the application server. And remember, the application server should achieve 80-
100% utilization before the database server reaches 100% utilization. If this proves
not to be true, something is probably wrong with an SQL statement and you must
capture an SQL trace to determine the culprit.
Note: For the Windows platform, the CPU should not exceed 70-80% to provide enough additional CPU for the
operating system to process.
Another way to achieve similar results is to start out with a small number of threads and increase the
number of threads until you have maximized throughput. The definition of throughput may differ for
each process but can be generalized as a simple count of the records processed in the Batch Run Tree.
For example, in the Billing background process in product, throughput is the number of bills
processed per minute. If you opt to use this method, it is recommended that you graph a curve of
throughput vs. number of threads. The graph should display a curve that is steep at first but then
flattens as more threads are added. Eventually adding more threads will cause the throughput to
decline. Through this type of analysis you can determine the optimum number of threads to execute
for any given process.
Altering Commit Interval and Cursor Reinitialization time
There are two parameters that can be altered to control the amount of resources a background process
uses when it executes. While most implementations reuse the default values supplied with the product,
it is possible to alter the values to tune the performance of background processes and allow
background processes to be executed during peak application usage periods.
The Commit Interval parameter controls the Unit of work on the background process. It controls after
how many primary objects are processed before a commit point is taken. It controls the amount of
work the database has to do and when a checkpoint is taken for restart purposes. At the commit point
in a background process, all the outstanding work, since the last commit, is reflected on the database
and a checkpoint is written on the batch control record for the job. You usually take the default for the
job but it can be adjusted according to the circumstances.
The higher commit frequency the value specified the less commit points are taken. Typically commit
points can be expensive transactions in a database so most implementations try and minimize them.
The lower value the more commit points. The latter is desirable when you are running background
during online hours as it reduces the resource hit the background process has on online processes.
Specifying a large commit frequency can cause larger than normal rollbacks to be performed by the
database. This can cause a strain on the database and hardware. The longer the unit of work the more
work the database has to do to roll it back in case of a failure.
The commit interval can have advantages and disadvantages depending on the situation and the value:
TABLE 6 COMMIT FREQUENCY CONSIDERATIONS

34
HIGH VALUE FOR COMMIT FREQUENCY LOW VALUES FOR COMMIT FREQUENCY
Less Commits in process (Less checkpoints).
Larger Unit of Work
Lower Concurrency (higher impact on other users)
Longer rollback in case of failure
Can increase throughput on lightly loaded system
More Commits in process (More frequent checkpoints).
Smaller Unit of work
Higher Concurrency (lower impact on other users)
Shorter rollbacks in case of failure
Can allow background to work harmoniously with online.
The second parameter Cursor Reinitialization Time (in minutes) controls how long a solution set is
held by the process. When Oracle database processes a set of records it typically holds a snap shot of
these records to save processing time. If the set is held too long, the records may not reflect the state
of the database so it is a good idea for the Oracle database to maintain the currency of this data
regularly. Within the background process this is controlled by setting this value to prevent the snapshot
being discarded by Oracle database and causing an abort. If the records are held too long an ORA-
1555 "Snaphot too old" error is generated and the process aborts.
The Cursor Reinitialization and Commit interval parameters are tunable parameters to affect the
impact of the background processes on the other processes running and prevent internal database
errors. It is also important to understand their impact to ascertain whether any change is required. The
following rules of thumb apply to setting the values:
It is recommended that the Commit Interval should not be set to one (1) as this value may
causes excessive database I/O and therefore performance degradation.
For light jobs (short duration, single threaded, small numbers of records etc), the default value
for Commit Interval may satisfy your site performance requirements.
For heavy jobs (long duration, multi-threaded, large number of records etc), then a value for
Commit Interval of between 5 (five) to 20 (twenty) is recommended.
The value of the Commit Interval directly affects the size of the redo logs allocated to the
database. The higher the commit interval the larger the redo logs need to be to hold the
inprocess objects. Work with your sites DBA groups to come up with a compromise between
redo logs and commit interval.
During processing of any background process a main object is used to drive the process. For example
in BI LLI NGthe main object is Account. The BI LLI NGprocess loops through the accounts objects
as it processes. For other processes it is other objects that are considered the main object. This main
object type is used to determine when a transaction is complete.
For both Cursor Reinitialization and Commit interval this is important as:
When a certain number of main objects have been processed then a database commit is issued
to the database. This number is the Commit Interval. The larger the commit interval the larger
the amount of work that the database has to keep track of between commit points.
The Cursor Reinitialization parameter is used to minimize issues in the Oracle database where
the unit of work is so large it causes a "Snapshot too old". The Oracle database stores undo
information on the Rollback Segment and the read consistent information for the current

35
open cursor is no longer available. This is primarily caused when the Oracle database recycles
the Rollback Segment storage regularly. In Oracle Utilities Application Framework based
products this is prevented by reinitializing the cursor on a regular basis to prevent an error.
When this timeout, known as the Cursor Reinitialization, is exceeded then at the end of the
current transaction a commit will be issued.
At any time in a process a commit for objects processed may be caused by the reaching the
Commit Interval or the time limit set on Cursor Reinitialization, whichever comes first.
The setting of Commit Frequency and Cursor Initialization has impact on the amount of
memory the JVM memory space allocated to the individual threads. Higher values of both
require more memory to hold the data.
Note: The Cursor Reinitialization parameter only applies to COBOL based batch processes.
Online Daemon or Standalone Daemon
In Oracle Utilities Application Framework V2.1 and above a background processing daemon was
introduced to support the Online Submission Method (and the internal scheduler). This facility
replaced the cr on based daemon in previous versions. The daemon's responsibility is to execute any
background processes that have been registered by the online submission transaction.
Note: If the daemon is enabled in more than JVM, then the grid will ensure that only one daemon is active at any time.
If the JVM running the daemon fails for any reason, other JVM's in the grid will assume the role of the daemon. As
long as at least one JVM is configured to accept the scheduling daemon, the daemon will run on exactly one thread
somewhere among the batch grid JVMs. For example, if there are 3 JVMs configured to accept the scheduling daemon,
the daemon may end up running in one thread in JVM #2 and not at all on JVMs #1 and #3. If JVM #2 goes
down, the scheduler daemon will start running on one thread either in JVM #1 or #3, but not both. This ensures that
the scheduler is always running, if at all possible, and duplicate submissions do not happen.
There are two options to use the daemon:
The daemon can be executed with an existing J2EE Business Application Server shared with
the online. This reserves some capacity from the server to execute the background processes.
The daemon can be run within a dedicated standalone batch server that is not shared with
online. This allows the execution of background processing to be processed on dedicated
server hardware.
While the decision to use an online or standalone daemon is a site specific one there are a number of
factors that should be considered when making this decision:
DAEMON ADVANTAGES DISADVANTAGES
Online Easiest to configure (Installation option).
Does not require separate resources.
Useful for non-production.
If background process misbehaves it can effect
online.
Cannot manage daemon or trace daemon

36
DAEMON ADVANTAGES DISADVANTAGES
directly or via J MX.
Borrows Application Server capacity from
online.
COBOL processing is less efficient as Child
J VM is used for execution.
Standalone Direct management possible (J MX or
command line based).
Does not directly affect online processing.
Better performance of COBOL based
processing
6
Additional configuration and management.

Requires additional resources.
Online Submission Guidelines
The Oracle Utilities Application Framework includes a facility for allowing the online users submit
batch processes as part of their testing activities. Whilst this facility is handy for use for submitting jobs
there are a few guidelines for it's optimal use:
The online submission facility is not recommended for production use. The facility is designed
primarily for training, demonstration and testing use only and has functionality limited to
those uses.
The online submission facility will run all processes in the DEFAULT or LOCAL
7
When navigating to the Batch Run tree from this transaction, all executions of a job are listed
on the selection dialog. By default, the submission records are sorted by Batch Number in
descending order. The first entry in the list is the latest execution of the job, by default.
threadpools
only by default. This behavior can be overridden using the DI ST- THD- POOL batch
parameter. This will need to be added to the Batch Control definition for individual jobs that
require this functionality. This is equivalent to the
com. spl wg. bat ch. submi t t er . di st Thr eadPool submitter parameter.
Use of Spacenames
By default, all batch jobs submitted are run within the same housekeeping space within the batch grid
8

6
If COBOL is used by your product.
.
This behavior is designed for production use to maximize the efficiency and resource usage of the
batch grid. During development of batch code, it may be desirable to execute each developer's
7
The LOCAL threadpool applies to Oracle Utilities Application Framework V2.1 only.
8
It is known as the MAIN space.

37
workload in their own spaces to isolate developers from affecting each other. There is a facility within
the product to override the space used by the batch grid to allow segregation.
To do this the developer must specify the following parameter for the job (in the pr oper t i es file
for the individual job):
com. spl wg. gr i d. spaceName=<spacename>
where <spacename>is the name of the desired space.
For example:
com. spl wg. gr i d. spaceName=TEST1
Note: This parameter is intended as a development and testing aid only. It provides a hard partition between workers.
Each space name has its own HouseKeepingDaemon and is therefore totally separate from workers with different space
names. For production purposes, distributed thread pools are more flexible and should be used instead.
Maximum Execution Attempts
By default the threadpoolworkers attempt execution of a batch process once. There is a parameter that
specifies how many times the worker(s) in the grid should attempt execution of the work submitted by
this submitter. If an application program crashes and brings down the worker JVM with it, this
parameter is designed to prevent any other worker nodes in the grid from picking up this same bad
work request and thereby spreading the poison work around the grid, crashing JVMs along the way and
ultimately bringing the batch grid down completely.
The default for this is 1, but parameter maxExecut i onAt t empt s can be used to override it as
follows:
com. spl wg. bat ch. submi t t er . maxExecut i onAt t empt s=n
where n is a number greater than 0.
The default is set to 1 and it is highly recommended that it should be at that value unless there
instructed by Oracle Support.
Number of Threads Per Threadpool
Each threadpool defined for the product contains a thread limit and is also subject to the java thread
limits imposed by the JVM vendor. To understand these thread limits a few concepts must be
understood.
Each version of java from each java vendor supports a limit of the active threads that can be
supported. This value varies from java version to java version and java vendor to java vendor. Typically
this varies from 150-600 active threads per JVM. This implies that you can run 150+ threads of
background processing but this is not the case.
Java threads tend to be short lived (typically associated with online or web services style work) and the
limit of java seems to support that level. Conversely, background process threads tend to be long lived

38
and therefore take more resource then short lived threads. This can be explained by the fact that online
transactions and web services tend to operate on one object and background process on multiple.
It is recommended that the maximum number of threads you consider per thread pool is 8 for heavy
jobs and 10-15 for light jobs.
Customers using COBOL based objects should considering minimizing the number of threads in each
pool to reduce Access Violation Errors generated by the Microfocus runtime. If your site finds that
this occurs, reducing the number of concurrent threads per threadpool can reduce the occurance of the
problem.
The thread limit is set at a threadpool level as part of the t hr eadpool wor ker [ . sh] utility. This
thread limit is set to prevent the JVM from using more resources than required. Exceeding this limit
will result in background process execution delays as the process waits for an available thread.
Even though this limit is set, it represents the maximum number of potential threads in the threadpool.
Not all background processes have an equal footprint on the system. Some are heavier (use more
resources) than others. The footprint of a particular background process is not measured by the
volume of data but by the thruput of the background process. The heavier background processes tend
to have lower throughput rates than lighter background processes.
The importance of the footprint is related to the number of threads that can be actually executed at any
time within the threadpool. A threadpool will process less heavier background processes and more
lighter background processes. For example, if the threadpool limit is set to ten (10) and you try and run
ten (10) heavy threads then the JVM may run the background processes slower due to threadpool
having capacity issues. If you sent ten (10) lighter threads it may process them adequately.
To determine the optimal number of threads for your threadpools:
During the testing phase, set all the threadpools used to the 10-15 thread limit. Refer to
Allocate each threads to the threadpool up to the limit. Note the run times.
for additional advice.
Decrease the number of threads sent to the pool and note the run times.
This should assist in determining whether the threadpool can optimally handle less or more threads
and the relative footprint of the threads used.
Note: The online daemon DEFAULT threadpool should remain at five (5).
The key to using the t hr eadpool wor ker [ . sh] and submi t j ob[ . sh] utilities is to design
the number and composition of threadpools. The following guidelines have been found from research
with customer and internal studies to assist in this design process:
Separate java based processes from COBOL based processes. This will assist in
micromanaging the threads in case you need to stop threads of a background process. The

39
j mxbat chcl i ent [ . sh] utility can only kill individual java based background processes
so to kill COBOL based background processes it is recommended to use
j mxbat chcl i ent [ . sh] to shutdown the threadpool. This will stop all threads in the
threadpool but is the only way to stop a COBOL based background process within the Oracle
Utilities Application Framework.
Note: j msbat chcl i ent [ . sh] can be used for any batch process regardless of the language used to
write the functionality.
The number of threads you are simultaneously executing at any point in the schedule will
dictate the number of threadpools to be used for your site. For example, if 80 threads are
executing at any time then 8-10 threadpools may be necessary (this is only a rough
calculation).
Group light footprint background processes into a common set small number of threadpools.
Typically background processes that operate on the same type of data are ideal. Consult your
product documentation as the background processes are usually grouped by functional area
already for consideration.
Consider splitting heavy footprint background processes into a number of threadpools. The
best way to determine this is through trial and error by determining the optimal number of
threads per threadpool for that particular background process.
Name your threadpools appropriate for their function. The DEFAULT threadpool is reserved
for the daemon use only and should not be used if the daemon is active in that environment.
This may sound quite complicated but the process can be simplified in a number of ways:
Java based background processes are the easiest to manage so can be grouped together in a
small number of threadpools.
Increasing the number of threads used to limit the threads to smaller units of work is
acceptable for heavy processes as subdividing the workload into small units can lead to an
increase to thruput levels. Smaller units of work reduce the memory and CPU of the JVM by
reducing work queue length.
The large heavy background processes that require multi-threading should be separated in their
own dedicated threadpools. This threadpool can be started prior to the first thread of the
background process starting and shutdown after the last thread has completed.
Note: In most product implementations, the number of multi-threaded background processes usually represents
fewer than 10% of the total number of background processes that need to be managed.
Shut down any threadpools no longer used by your schedule. This may reduce overall
resource usage.
Threadpool Cluster Considerations

40
The CLUSTEREDexecution mode allows for clustering of threadpools. There are a few guidelines
when using this facility that may assist in configuration and operations:
One Cluster per environment The current implementation of CLUSTERED execution
mode only allows one cluster per environment. This is sufficient for the majority of most
needs. The implementation can run a number of threadpools within the cluster (the
threadpools can be clustered [same name] or non-clustered [different names]).
More than one machine Clustering of threadpools (same name) requires more than one
machine to be optimal. It is not possible to cluster a threadpool within a single machine.
Name your cluster appropriately - The name can be up to thirty-two (32) characters to
define the name of the cluster. This is required and must be unique for each environment.
With classic DI STRI BUTEDmode, the batch JVMs for an environment is naturally grouped
because they register themselves through database table F1_TSPACE_ENTRY, but in
CLUSTERED mode the JVMs are joined through a Coherence cache. The cache may be
restricted to one environment or across all environments, so a unique cluster name, along with
address and port, is required to ensure that they are appropriately grouped per environment.
Environments are typically separated by database and/or database user, so a possible
convention may be to use a combination of database name and owner Id as the cluster name,
for example "DEMO.SPLADM" or simply use the environment name, for example,
"TUGBUDEMO".
Use Multicast By default, multicast is enabled in the configuration files. It should be used
unless multicast is deemed inappropriate for your site (some network administrators do not
like multicast) then unicast is supported. Refer to Clustering using Unicast or Multicast
COBOL Memory Optimization for Batch
for
more information.
In Oracle Utilities Application Framework V2.1 and V2.2 and above, a number of configuration
settings were introduced to optimize the memory used by the threadpool based JVMs. These settings
control the behavior of the threadpool based JVMs in terms of their memory usage.
To optimize the settings for background processing it is recommended to set these memory settings to
the following:
Releasing COBOL Memory COBOL programs only release their thread-bound memory
when the thread dies. This thread-bound memory is primarily memory allocated by the
COBOL runtime on the C heap. As threads return to the thread pool and are used again to
process calls to different COBOL programs, the memory footprint may continue to grow as
more and more different COBOL programs are called. In an online processing scenario, this
can cause memory faults in the long run as many COBOL modules are called during the
availability of the product. During background processing, this problem is somewhat reduced
as the number of COBOL modules called is much lower. Therefore the configuration settings

41
that controls this behavior in spl.properties for background processing should be set as
follows (opposite to what is recommended for online):
spl . r unt i me. cobol . r emot e. r el easeThr eadMemor yAf t er EachCal l =f al se
Minimizing Housekeeping When using the DI STRI BUTED mode of execution, the
threadpool worker polls the F1_TSPACE* tables to check for new available jobs. By default
this poll is performed every 1000ms (1 second). This can be inefficient for the threadpool
worker process, therefore it is recommended to change this tolerance to 5000ms (5 seconds)
in production to reduce overheads. This can be implemented by changing the
et c\ t hr eadpool wor ker . pr oper t i es file and adding the configuration setting:
com. spl wg. gr i d. pol l i ng. mi nMi l l i sBet weenCycl es=5000
Example Clustered Mode scenarios
When utilizing CLUSTERED mode there are a number of scenarios that are common in terms of
architecture and configuration. The following sections will outline each common scenario, give an
example and illustrate how to implement the scenario within the configuration files used for the batch
component.
Note: Whilst there are numerous variations available for each scenario, the samples used in this section are generic and
simplified to cover the more pertinent aspects of the specific scenario.
Generic configuration process
Before discussing the scenarios it is important to understand the process of what configuration files to
update to setup the configuration. Primarily there are three steps to consider:
1. Attach to the environment by issuing the spl envi r on[ . sh] command from the relevant
host. If the scenario then this will be repeated for each host and environment to implement the
change.
2. The threadpool can be optionally defined in the t hr eadpool wor ker . pr oper t i es file.
Definition of the threadpool in this file can be skipped if the threadpool is dynamically created using
the options on the t hr eadpool wor ker [ . sh] utility. Refer to the Batch Server
Administration Guide (or Operations and Configuration Guide for V2.x customers) for details of
the options for this utility. The location of the t hr eadpool wor ker . pr oper t i es file varies
with Oracle Utilities Application Framework versions (refer to the Batch Server Administration
Guide or Operations and Configuration Guide
3. In Oracle Utilities Application Framework V4.1 and above, the cluster configuration is defined in
the t angasol - coher ence- over r i de. xml file as opposed to the
t hr eadpool wor ker . pr oper t i es file used by Oracle Utilities Application Framework
4.0.x and below versions of the framework.
) for the location of this file.
The figure below summarizes the steps used in the configuration:

42
Start
Attach to environment(s)
Setup Threadpool Setup Threadpool t hr eadpool wor ker . pr oper t i es
Setup Clustered mode Setup Clustered mode
t angasol - coherence-
over r i de. xml
End
Oracle Utilities Application Framework
V4.0.x and below
Oracle Utilities Application Framework V4.1
and above

Figure 7 Setup of CLUSTERED mode
When making changes, please ensure the configuration files conform to the expected format as
outlined in the Batch Server Administration Guide or Operations and Configuration Guide
Note: For publishing purposes both multicast and unicast examples will be shown.
for the
version of the Oracle Utilities Application Framework used.
Sample Setup
To illustrate the scenarios a sample setup has been devised to cover all the scenarios. The figure below
illustrates the sample used for the examples:

43
host1 (10.1.10.1)
submitter submitter submitter
host2
(10.1.10.2)
threadpool
(SCEN1)
threadpool
(SCEN2)
threadpool
(SCEN2)
threadpool
(SCEN3)
threadpool
(SCEN3)
A:Single threadpool / Single host
B:Multiple threadpools/ Single host
C:Multiple threadpools/ Multiple hosts

Figure 8 Example threadpool setup
The sample consists of two hosts (host1 and host2) which house an identical copy of the product.
There are a number of threadpools (SCEN1, SCEN2 and SCEN3) spread across the hosts.
Note: For publishing we will assume each threadpool has an arbitrary limit of 8 threads.
This setup will illustrate the following scenarios:
TABLE 8 CLUSTERED SCENARIOS
SCENARIO ATTRIBUTES
A Single threadpool on a single host/environment. This is a common scenario for non-production environment such
as development and initial test environments.
B Multiple threadpools on a single host/environment. This is a scenario used for testing and for executing larger
numbers of jobs and threads simultaneous.
C Multiple threadpools across multiple machines but in a single "environment" (such as production). This is a
scenario where the site may have multiple servers for an environment (e.g. production) and want to run jobs across
the machines.
D Multiple different pool combinations. A combination of scenarios A, B and C.
Setup threadpools
The first part of the process is to setup the t hr eadpool wor ker . pr oper t i es file with the
definitions of the threadpools required for tour scenario. This step is optional as the threadpools can
be dynamically created using the t hr eadpool wor ker [ . sh] utility.
For each threadpool on a host a definition of threadpool can exist in
t hr eadpool wor ker . pr oper t i es are in form:
com. spl wg. gr i d. di st Thr eadPool . t hr eads. <pool name>=<t hr eads>
where

44
<t hr eads> Maximum threads for the threadpoolworker
<pool name> Threadpool name
The following table shows the threadpool file parameter entries for each scenario.
TABLE 9 CLUSTERED SCENARIOS
SCENARIO THREADPOOLWORKER.PROPERTIES FILE ENTRIES
A On host1:
com. spl wg. gr i d. di st Thr eadPool . t hr eads. SCEN1=8
B On host1:
C On host1:
D On host1:
host2:
Setup Clustered Mode
The next step is to define the clustered mode information for the host/environment. There are two
approaches to consider. The clustered mode can use multicast or unicast (see Clustering using Unicast
or Multicast for more information about the different modes).
This information is specified in the t angasol - coher ence- over r i de. xml file for Oracle
Utilities Application Framework V4.1x and above products or he
t hr eadpool wor ker . pr oper t i es file for Oracle Utilities Application Framework V4.0.x and
below (including V2.x) customers.
This requires setting up the following parameters:
TABLE 10 CLUSTERED PARAMETERS
MULTICAST UNICAST
Set t angosol . coher ence. cl ust er to the cluster name. Refer
to the Batch Server Administration Guide or Batch Operations and
Configuration Guide
Set t angosol . coher ence. cl ust er to the cluster name.
Refer to the
for recommendations.
Batch Server Administration Guide or Batch
Operations and Configuration Guide
Set t angosol . coher ence. cl ust er addr ess to a unique IP
address for the cluster to use for multicast communications. Refer
to the
for recommendations.
Batch Server Administration Guide or Batch Operations and
Configuration Guide
Set t angosol . coher ence. l ocal por t to the port number
used by this node in the cluster. This must be unique across
all the environments on the machine.
for valid IP ranges.
Set the t angosol . coher ence. cl ust er por t to a unique port
number for the cluster for the environment.
Set t angosol . coher ence. wka to the host name or host IP
address (if DNS resolution is slow) of the hosts in the cluster
Set t angosol . coher ence. di st r i but ed. l ocal st or age to Set t angosol . coher ence. wka. por t to the same value as

45
MULTICAST UNICAST
f al se to minimize heap size used by threadpoolworker J VM's.
See Coherence Best Practices for more information
t angosol . coher ence. l ocal por t if the implementing
single host solution else set the port number to the same
unique number across all hosts on the cluster.

Therefore, if using Oracle Utilities Application Framework V4.0 and below, the
t hr eadpool wor ker . pr oper t i es entries for the scenarios are as follows:
TABLE 11 THREADPOOLWORKER.PROPERTIES CLUSTERED VALUES FOR SCENARIOS
SCENARIO MULTICAST UNICAST
A, B host1:
t angosol . coher ence. cl ust er =FWDEMO. SPLADM
t angosol . coher ence. cl ust er addr ess=239. 128. 0. 10
t angosol . coher ence. cl ust er por t =7810
t angosol . coher ence. di st r i but ed. l ocal st or age=f al se
host1:
t angosol . coher ence. l ocal por t =7810
t angosol . coher ence. wkapor t =7810
t angosol . coher ence. wka=10. 1. 10. 1

C, D host1:
host2:
host1:
t angosol . coher ence. por t 1=7820
t angosol . coher ence. wka1=10. 1. 10. 1
host2:

Therefore, if using Oracle Utilities Application Framework V4.1 and above, the t angasol -
coher ence- over r i de. xml entries for the scenarios are as follows:
TABLE 12 TANGASOL-COHERENCE-OVERRIDE.XML CLUSTERED VALUES FOR SCENARIOS
SCENARIOS A,B MULTICAST
<coher ence>
<cl ust er - conf i g>

46
<ser vi ces>
<ser vi ce i d=" 1" >


<par am- name>l ocal - st or age</ par am- name>
<par am- val ue syst em-
pr oper t y=" t angosol . coher ence. di st r i but ed. l ocal st or age" >f al se</ par am- val ue>


</ ser vi ce>
</ ser vi ces>
<member - i dent i t y>
<cl ust er - name syst em-
pr oper t y=" t angosol . coher ence. cl ust er " >FWDEMO. SPLADM</ cl ust er - name>
</ member - i dent i t y>
<mul t i cast - l i st ener >
<addr ess syst em-
pr oper t y=" t angosol . coher ence. cl ust er addr ess" >239. 128. 0. 10</ addr ess>
<por t syst em- pr oper t y=" t angosol . coher ence. cl ust er por t " >7810</ por t >
</ mul t i cast - l i st ener >
</ cl ust er - conf i g>
</ coher ence>
SCENARIOS A,B UNICAST
<coher ence>
<uni cast - l i st ener >
<wel l - known- addr esses>

47
<socket - addr ess i d=" 1" >
<addr ess syst em- pr oper t y=" t angosol . coher ence. wka" >host 1</ addr ess>
<por t syst em- pr oper t y=" t angosol . coher ence. wka. por t " >7810</ por t >
</ socket - addr ess>
</ uni cast - l i st ener >
<addr ess syst em- pr oper t y=" t angosol . coher ence. l ocal host " >l ocal host </ addr ess>
<por t syst em- pr oper t y=" t angosol . coher ence. l ocal por t " >7810</ por t >
</ coher ence>
SCENARIOS C,D MULTICAST
<coher ence>
<ser vi ces>
<ser vi ce i d=" 1" >


<par am- name>l ocal - st or age</ par am- name>
<par am- val ue syst em-
pr oper t y=" t angosol . coher ence. di st r i but ed. l ocal st or age" >f al se</ par am- val ue>


</ ser vi ce>
</ ser vi ces>
<mul t i cast - l i st ener >
<addr ess syst em-
pr oper t y=" t angosol . coher ence. cl ust er addr ess" >239. 128. 0. 10</ addr ess>
<por t syst em- pr oper t y=" t angosol . coher ence. cl ust er por t " >7810</ por t >

48
</ mul t i cast - l i st ener >
</ coher ence>
SCENARIOS C,D UNICAST
<coher ence>
<uni cast - l i st ener >
<wel l - known- addr esses>
<addr ess syst em- pr oper t y=" t angosol . coher ence. wka1" >host 1</ addr ess>
<por t syst em- pr oper t y=" t angosol . coher ence. wka1. por t " >7810</ por t >
<addr ess syst em- pr oper t y=" t angosol . coher ence. wka2" >host 2</ addr ess>
<por t syst em- pr oper t y=" t angosol . coher ence. wka2. por t " >7820</ por t >
</ wel l - known- addr esses>
</ uni cast - l i st ener >
<addr ess syst em- pr oper t y=" t angosol . coher ence. l ocal host " >l ocal host </ addr ess>
<por t syst em- pr oper t y=" t angosol . coher ence. l ocal por t " >7810</ por t >
</ coher ence>
Starting the threadpools
The key now is to start the threadpools using the threadpoolworker[.sh]. Generally the command:
t hr eadpool wor ker [ . sh]

49
is sufficient to start all the threadpools in the threadpoolworker.properties file but if you are staring
multiple instances of the same pool (SCEN2) then you need to run additional explicit commands for
each instance. For example for SCEN2:
t hr eadpool wor ker [ . sh] p SCEN2=<t hr eads> - i <r mi por t >
where:
<t hr eads> Maximum threads for the threadpoolworker
<r mi por t > RMI port used for JMX. To manage each instance of the pool an unique port
number should be used.
Monitoring Background processes
Once the background process has started there are a number of ways that can be used to monitor the
progress and status of the process:
Batch Run Tree Every time a background process is executed a set of records are
registered within the Oracle Utilities Application Framework to track the restart and progress
of the background process. This information is updated after each commit point to show the
progress of the job.
JMX Monitoring In Oracle Utilities Application Framework V2.2, JMX monitoring was
made available which allows a JMX client (such as j consol e etc) or the provided
j mxbat chcl i ent [ . sh] utility to monitor the progress of individual threads. This is
ideal for live monitoring of the threads executing on a particular threadpool and has more up
to date information than the Batch Run Tree. Refer to the Batch Operations And
Configuration Guide or Batch Server Administration Guide
Log based monitoring Depending on the submission method a number of logs are written
for each job that can be used to find errors and statistics. Refer to the
associated with your product for
more information about the JMX capabilities.
Batch Operations And
Configuration Guide or Batch Server Administration Guide
Database query As information is displayed on the Batch Run Tree is stored in the
database it is possible to retrieve that information and provide historical analysis of the
background process. This information is particularly useful for tracking performance as well as
determining the footprint of individual background processes. Refer to the
associated with your product for
more information.
Performance
Troubleshooting Guidelines Batch Troubleshooting
J MX Monitoring
(My Oracle Support KB Id 560382.1)
for details of this facility.
Note: The JMX capability is only available for Oracle Utilities Application Framework V2.2 and above only.


50
With the implementation of DI STRI BUTEDand CLUSTEREDmode, the ability to actively monitor
and manage individual batch processes is available via Java Management Extensions (JMX) is possible.
This means a JMX console, such as j consol e, can be used on an active threadpool and individual
batch threads to monitor their progress and manage them remotely.
Refer to the Batch Server Administration Guide and Batch Operations and Configuration Guide
Database Connection information
for
your product for more information on how to enable and monitor using JMX.

Note: Database connection tags are only supported for Oracle Utilities Application Framework V2.2 and above only
For Oracle Utilities Application Framework V2.2 customer, install patch #10215923 to implement this functionality.

By default, the database connection information via the column MODULE on the V$SESSI ON view
for batch processes is set to "JDBC Thin Client". This makes the batch sessions harder to differentiate
from other sessions.
The product has been altered to now populate the MODULE column to display the Batch control id for
the connection and "TUGBU I dl e" when the threadpool has an idle connection.
To implement this change the following setting must be added to the hi ber nat e. pr oper t i es
file contained in $SPLEBASE/ spl app/ st andal one/ conf i g (or
%SPLEBASE%\ spl app\ st andal one\ conf i g on Windows):
hi ber nat e. connect i on. r el ease_mode=on_cl ose
The MODULE will now display the batch control on active connections.
Note: For Oracle Utilities Application Framework V4.1 or above, the CLI ENT_I DENTI FI ER is also populated
with the product user configured to execute the process.
Commit Strategies
Note: Commit Strategies were introduced in Oracle Utilities Application Framework V2.1. Versions of Oracle
Utilities Application Framework prior to V2.2 used the Standard Commit Strategy exclusively. To implement any of
the alternative commit strategies ensure that the Oracle Utilities Application Framework is patched to the latest service
pack to include all strategies.
By default, background processes typically commit records every configurable interval (see Commit
Interval for more information). The commit interval defines the size of the work unit in the database
as well as defines the granularity of restart, as the product rolls back to the last commit point on
process error or failure. This is known as the Standard Commit Strategy and is employed by the
majority of background processes in the product.
Whilst the product generally uses the Standard Commit Strategy it is possible for custom background
processes to use alternative strategies. These are either coded within the custom background process
themselves or are configured using the

51
com. spl wg. bat ch. submi t t er . sof t Par amet er . execut i onSt r at egy setting
specified in the properties file used by the background process.
The table below outlines the valid strategies and their attributes:
TABLE 13 COMMIT STRATEGIES
COMMIT STRATEGY PARAMETER VAULE COMMENTS
Standard Commit (default) St andar dCommi t This strategy Process each Unit of work as part of a group of work units
in one database transaction. The standard maxi mumCommi t Recor ds
parameter defines the commit interval, defaulted to 200 if not supplied or
not a number >0. In the event of an exception, the transaction group is
rolled back to the last committed work unit, the exception is logged and
committed, and the successful entries in this transaction group are
reprocessed (up to the current, failed work unit). Processing can
resumed, upon restart, at the first work unit after the one that failed.
This strategy is appropriate for batch processes that can tolerate errors in
the execution.
Single Transaction Si ngl eTr ansact i on Process the entire workload in a single committed transaction. Any
exception will cause a rollback of processed work and will be considered
an unsuccessful thread execution.
This strategy is most appropriate to update processes or interfaces that
cannot tolerate any errors within a run. For example, an interface that
requires a complete set of results should consider this strategy.
Commit Every Unit Commi t Ever yUni t Each successful record processed has its own commit. There is no need
to "back up" and reprocess units rolled back because of an exception.
This strategy can continue to move forward after exceptions. This is
equivalent to a commit interval of one (1).
This strategy is most appropriate to update processes or interfaces that
can tolerate some errors within a run.
Thread Iteration Thr eadI t er at i on If there is a requirement for thread pool workers to select their data at
initialization time and to loop and process until the end of the selection
then using the ThreadIteration commit strategy is recommended.
The application data can come from a database table, one or more flat
files, or any other source that the thread worker requires. The opening,
fetching and closing of the data is left entirely up to the application
program The batch frameworks responsibility is to provide appropriate
context, commit frequency, error handling and restartability, as it does in
the case of the other strategies.
This strategy was introduced to reduce java heap space usage of the
background process and to provide alternatives for restart. When a
thread is restarted after a premature end (for example error or
cancellation), its initialization method will have the opportunity to refresh
the selection of its data for the run. This is in contrast to the existing
model in which the data will always be based on the original selection
when the job was first submitted
The standard maxi mumCommi t Recor ds parameter defines the commit
interval, defaulted to 200 if not supplied. Soft parameter maxEr r or s

52
COMMIT STRATEGY PARAMETER VAULE COMMENTS
controls the number of errors that the program can tolerate. Each error
that is thrown while within this limit causes all updates for that one work
unit to be rolled back. This strategy class uses a J DBC savepoint for each
work unit to avoid also rolling back the successfully processed units of
work when an error is found. If maxEr r or s is overrun, the thread is
aborted.
Continuous Execution

Cont i nuousExecut i on Support continuous batch processes that may run indefinitely. Similar to
Commit Every Unit Strategy. Introduced to support Timed Batch.
Note: Not all background processes support all the commit strategies listed above. If in using an alternative commit
strategy the program fails, using StandardCommit may resolve the issue.
Flushing the Batch Cache
Note: This facility is available as the following patches 12539014 (FW4.1) , 9866988 (FW4.0.2)
and 9900007 (V2.2) .
The online component of the Oracle Utilities Application Framework uses a cache of static data for
performance reasons. The batch component uses a similar cache mechanism (per threadpoolworker)
using a Hibernate data cache. Whilst this cache is automatically refreshed by the product on a regular
basis it can be now manually refreshed by running the F1- FLUSH background process.
It is recommended to run the F1- FLUSH background process for long continuously running
threadpoolworkers to reflect data changes in configuration data.
Online Submission Alternative Threadpools
When using the online batch submission facility in the product, the batch jobs submitted will execute
in the DEFAULT threadpool by default (in V2.1 this is known as the LOCAL threadpool). Whilst, this
behavior is usually sufficient for most sites during testing, it may be desirable to execute the online
submission jobs on an externally started threadpool. Ensure you have started the external threadpool
prior to submitting any jobs.
To do this, you must add the DI ST- THD- POOL parameter to all batch control records you want to
run in the external pool. To save time, you can also provide a default value for this parameter to save
the submitted having to specify it. An example of this setting on the Batch Control is shown below:

Figure 9 Setup of alternative threadpool

53
Scheduling Best Practices
The product contains a background-processing component to process multiple records over a selected
period (daily, nightly, weekly etc). Optimization of this component is critical to the overall success of
the implementation.
Optimize the schedule
During the implementation of the product, the implementation will work with the business to
determine which processes are applicable to your business to determine the sites schedule. Remember
that NO site has implemented the schedule as it is supplied. It is always a relevant subset of the
provided schedule.
Business processes and business activity drive the schedule that the implementation will go live with.
Some of the background processes will be removed and some will be added. Individual background
processes can be removed if business process does not require the process or, for any reason, the
process is not applicable to the business. Custom Processes (typically for interfaces) will be added by
your implementation team.
Therefore the following goals are applicable to schedule optimization within product:
Maximize throughput by limiting process concurrency. Do not run too many processes
simultaneously. The CPU can flood if you run too much at the same time.
Aligning schedule with Bill Cycle and Meter Read Schedules. The Bill cycle and Meter Read
cycles, if used, can influence the schedule. Refer to the Business Process documentation for
billing and meter reading for more details of this.
The following process can be used to optimize the base schedule supplied with the product:
Remove background processes that are not to be implemented. Check with the business if the
process is applicable or needed for a business process. If in doubt, leave it in.
Add custom background processes to schedule. For custom interfaces or custom business
processes, outside the scope of the base product, you will be adding a few custom background
processes. Your implementation team will have details of these processes.
Adjust dependencies for the added and removed background processes. When you take away
and add background processes, the dependencies will change and the overall flow of data will
change.
Run schedule in test as initially documented You need to run the new schedule at a basic
level to get an idea of how it hangs altogether.
Gather elapsed times and throughput rates - You need to get some stats to determine which
background processes will need optimization and how much optimization is really needed.

54
Determine "heavy" background processes Background processes that take a long time (we
will leave that tolerance up to you) need to be determined, as they will greatly affect the overall
schedule. These become candidates for multiple threading.
Now that we have the basic information we can start optimization.
Heavy background processes can be run multi-threaded Consider multiple threading the
heavy background processes but not too much threading as it can drive up contention. See
Multi-threading guidelines
Move scheduling of background processes to minimize number of background processes
running in parallel Reduce contention around heavy background processes by scheduling
other background processes earlier or later. Also try not to run a lot of light background
processes at the same time. It has the same impact that heavy background processes have.
for more information.
When altering the base schedule remember the following:
If in doubt, do NOT leave it out. Keep processes in the schedule that you are not sure about.
Only run multi-threaded if necessary remember too much threading can increase contention
and therefore reduce throughput.
To run background processes during busy business days then reduce the Commit Interval to
increase transaction concurrency.
Use a third party batch scheduler
The product does not have advanced batch-scheduling capabilities typically required by sites. All sites
tend to use a third party tool for scheduling to manage product background processes as well as
external related background processes (such as backups). In choosing a batch scheduling tool that you
can use with product the following must be supported:
UNIX command line support The batch scheduler should be able to run a UNIX
command line. An added bonus is the ability to parameterize the command line.
UNIX return code support The batch scheduler should be able to act upon the value of a
standard UNIX return code. For example the process will return zero (0) for successful
execution and non-zero for not successful.
Manage Job Dependencies The main advantage of a batch scheduler for product is to
manage the relationships between background processes and ensure that background
processes are only started after dependant background processes are completed successfully.
(Optional) Limit the number of batch background processes executing at one time
The ability for a batch scheduler to submit and control the number of active background
processes is ideal to ensure the resources on a machine are maximized not exhausted. This is
not a mandatory requirement but experience has shown that throughput is maximized if the
scheduler manages the number of active background processes competing for the CPU.

55
Oracle Utilities Application Framework V2.x and above includes an internal scheduler
9
It contains no failover facilities.
that can be
used to submit background processes but it has the following limitations:
It can only be run on one server only.
It can only run product background processes.
Customers have used the following tools successfully with product
10
UC-4/Appworx - UC4 Software (
:
http://www.uc4.com/)
BMC Control-M for Distributed Systems - BMC Software (http://www.bmc.com)
CA-AutoSys - Computer Associates (http://www3.ca.com)
Tivoli Workload Scheduler - IBM (http://www-
306.ibm.com/software/tivoli/products/scheduler/)
COSBatch Open Ssystems Management Ltd. (http://www.cosbatch.com/)
Note: Most third party scheduler products use the term job to denote a background process or an individual thread of a
background process.
Once a third party scheduler has been chosen the scheduler, it must be configured for submission of
background processes. The following guidelines have been used by a number of sites to successfully
implement the scheduler:
Create a separate operating system account for the scheduler to use to submit background
processes. Avoid using administration accounts (e.g. r oot ) or the product administration
account. This account should have access to the relevant product security groups.
To run any utility provided with the product, the batch user must execute the
spl envi r on[ . sh] utility to set the context of the executions. This can be achieved two
ways:
The . pr of i l e (or aut oexec. bat on Windows) for the administration account
can automatically call the spl envi r on[ . sh] utility automatically. This is the
preferred method.

9
Not all products have included the scheduler as part of their product. Refer to the relevant product
documentation for details.
10
There is no preferred solution as typically this software is used beyond just the base product.

56
The command line used in the scheduler for each background process (including
t hr eadpool wor ker [ . sh] as well as submi t j ob[ . sh] ) must be prefixed
with the full spl envi r on[ . sh] utility with the c option. Refer to the
"Operations And Configuration Guide
The background processes must be loaded into the scheduler. This can be done manually or
using an import facility provided with the scheduler. In some products the dependency
information is stored in a set of tables. Refer to the
" for your product for details of the
spl envi r on[ . sh] utility.
Framework Administration Guide on the
Internal Scheduler
Note: The Internal Scheduler feature is not available in ALL products.
online help for more information.
Configure the t hr eadpool wor ker . pr oper t i es and
submi t bat ch. pr oper t i es configuration files with your global defaults.
Each background process must use the submi t j ob[ . sh] utility to execute the
background process using the following guidelines:
The batch code must be provided on the command line in the scheduler using the
b option.
The Business Date should be specified on the command line using the d option.
This is to avoid the Past Midnight issue. If the date is not supplied the product will
use the current business date and if any background process spans across midnight or
processes in the schedule across midnight then the business dates used may be
inconsistent and data may not be processed as expected. Specifying the same
business date for all background process avoids this issue. Most schedulers have a
business date function (also known as an operational date) that can be used for this
purpose.
Consider providing the threadpool as using the p option to explicitly allocate a
threadpool to a background process.
Start the threadpools referenced in the background processes using the
t hr eadpool wor ker [ . sh] utility. Optionally they can also be shutdown using the
j mxbat chcl i ent [ . sh] when not used to save resources.
Ensure the threadpool referenced in the submi t j ob[ . sh] command line (or
configuration file) is running before the process is executed and on the same physical
machine. Remote invocation of worker or submitter threads in not supported in the current
release of the Oracle Utilities Application Framework.
For multi-threaded background processes there are a number of options:
It is possible to submit individual threads of a background process as individual jobs
within your scheduler. This will allow micromanagement of the schedule for
dependencies.

57
It is possible to submit all threads in a single job using the t 0 (zero) option. This
will submit all threads simultaneously in the same threadpool which may not be
desirable.
Remember to add non-product jobs that are necessary as part of your schedule such as
backups and interface transfers.

Batch Best Practices
J uly 2011
Author: Anthony Shorten, Principal Product
Manager

Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com
Copyright 2007-2011, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and
the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
means, electronic or mechanical, for any purpose, without our prior written permission.
Oracle and J ava are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license
and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open
Company, Ltd. 1010

Batch Best Practices

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Batch Best Practices

Uploaded by

Copyright:

Available Formats

An Oracle White Paper

You might also like