<Insert Picture Here> AIA Deep Dive Series

AIA Error Handling and Recovery

Varsha Bhatia, Principal Product Manager

Rohit Agarwal, Principal Solution Architect
 Agenda

• Key features
• Architecture
• Fault policy
• Notification setup
• Error notification and details
• Extending Error Handling
• Logging
• Enabling Error Handling and Logging
• Targeted new features: 2.x – 3.0
• Guaranteed Delivery and Message Resubmission

Copyright © 2008, Oracle. All rights reserved.

 Key Features

 Unified Approach
• Across technologies (BPEL, ESB)
• Across integration patterns

 Error Notifications
• Issues error notifications to suitable Actor roles, and FYI roles
• Links to Error Console (BPM Worklist) for error details

 Error Console
• Centralized UI to access error details that are assigned for resolution or for
informational purposes
• Accessible to administrators, and end-users

 Error Logging
• Logs messages non-intrusively in a consistent schema
• Logs can be searched, sorted, and filtered using Oracle Enterprise Manager

Copyright © 2008, Oracle. All rights reserved.

 AIA Error Handling Architecture

ESB BPEL BPEL Error Scenarios

 BPEL partner link errors
ESB Error Topic Catch Block
 BPEL non-partner link errors
Error Listener Async
BPEL Process
BPEL Fault
Policy
 Enterprise Service Bus (ESB)
errors
 3rd Party Errors (e.g. ODI)
ESB Error Handler BPEL Java Action

AIA Common Error

Handler

AIA Error Topic AIA Logger

Copyright © 2008, Oracle. All rights reserved.

 Fault Policies for BPEL/ESB Errors

<Action id="aia-ora-retry">
<retry>
<retryCount>3</retryCount>
<retryInterval>2</retryInterval>
<exponentialBackoff/>
<retryFailureAction ref="aia-ora-java"/>
<retrySuccessAction ref="aia-ora-java"/>
</retry>
</Action>
<Action id="aia-ora-java">
<javaAction className="oracle.apps.aia.core.eh.BPELJavaAction"
defaultAction="ora-rethrow-fault">
<returnValue value="REPLAY" ref="ora-terminate"/>
<returnValue value="RETHROW" ref="ora-rethrow-fault"/>
<returnValue value="ABORT" ref="ora-terminate"/>
<returnValue value="RETRY" ref="aia-ora-retry"/>
<returnValue value="MANUAL" ref="ora-human-intervention"/>

• BPEL Fault Policy files are located at

<SOA_HOME>/bpel/domains/default/config/fault-
policies/AIADefaultPolicy.xml
• ESB Fault Policy file is located at
$AIA_HOME/config
Copyright © 2008, Oracle. All rights reserved.
 Fault Message Schema

Enterprise Message Elements Fault Specific Elements

• EBMID • Fault Message Code

• EBM Name • Fault Message Text
• EBO Name • Severity
• EBOID • Fault Stack
• Verb Code • Faulting Service ID
• Business Scope Reference ID • Faulting Service Implementation
• Business Scope Reference Code
Instance ID • Faulting Service Instance ID
• Enterprise Service Name • Corrective Action
• Enterprise Service Operation • Reporting Date Time
Name
• Sender Reference ID
• Sender Message ID
• Sender Reference Transaction
Code
• Sender Object Identification ID

Copyright © 2008, Oracle. All rights reserved.

 Notification Setup
Notification Setup
 Error routing mapped to parameters
 Preference-based notification to Actor
Role, FYI Role

Copyright © 2008, Oracle. All rights reserved.

 Notification Setup Parameters

• Error Code
• Fault code in BPEL, ESBString string for ESB. ESB has a
concept of error codes. Examples: bpelx:bindingFault,
bpelx:remoteFault
• System Code
• Value is obtained from EBM Header. Examples: SEBL_01,
PORTAL_01
• Service Name
• Name of the BPEL process or ESB service that encounters
error. Examples: CreateCustomerPartyProvABCSImpl,
CustomerPartyEBS
• Process Name
• Value is obtained from EBM Header. Examples: Siebel-
Portal-Get-Account-Balance

Copyright © 2008, Oracle. All rights reserved.

 Error Notification Role Determination

For every error that occurs, the error handler checks for
matches in the following order:
1. SYSTEM_CODE, ERROR_CODE, SERVICE_NAME,
PROCESS_NAME
2. ERROR_CODE, SERVICE_NAME, PROCESS_NAME
3. SERVICE_NAME, PROCESS_NAME
4. SERVICE_NAME

If none of these values are available, the default roles

are fetched from the AIAConfigurationProperties.xml file.
<Property
name="EH.DEFAULT.ACTOR.ROLE">AIAIntegrationAdmin</Property>
<Property name="EH.DEFAULT.FYI.ROLE"></Property>

Copyright © 2008, Oracle. All rights reserved.

 Error Notification and Resolution

Error Notification Example:

Copyright © 2008, Oracle. All rights reserved.

 Error Console
Note: 791273.1
Sample script to
close out all the tasks
that are pending
Leverages SOA Suite Worklist application
• Error tracking
• Starting point for
further error resolution
• Group assignment:
Actor Role, FYI Role
• Re-assignment
• Identify responsible
persons / groups by
querying AIA setup

Copyright © 2008, Oracle. All rights reserved.

Error Details in the Worklist App

Copyright © 2008, Oracle. All rights reserved.

 Extending Default Fault Message Content
Enriching fault content
Step1
- Implement Custom extension
handler. Implement AIA Interface
Copyright © 2008, Oracle.
Enriched Fault Content
- Add custom content to base fault
message to give more complete
error context: Use xsd:anyType
- Functionally rich information
- More effective Error Resolution
- Applicable for BPEL and Mediator
errors
Depth of Content
- Add one, two custom fields or entire
message payload
- View logged content in AIA Log Files
via Enterprise Manager
Step2 Step3
- Register class with Error - Identify error scenario for which
Handling Framework handler needs to be called within
Setup UI
13 All rights reserved.
Extending Default Fault Message Content

Copyright © 2008, Oracle. All rights reserved.

Extending Default Fault Message Content
Example

AIAConfigurationProperties.xml

Copyright © 2008, 15
Oracle. All rights reserved.
 Custom Handling of Errors
Example:
- Step 1:
• Indicate custom error handling
for Order Fallout functionality:
ORDER_FO

- Step 2:
• Indicate custom plus default AIA
Error Handling behavior: Error Type
= ORDER_FO, AIA_EH_DEFAULT

- Step 3:
• Implement extension listener to
subscribe to messages with JMS
Header stamp = ORDER_FO. AIA
will ensure default behavior in
addition
Enabling Custom Handling of Errors
Step1 Step2 Step3
- Define a unique custom filter - Identify error scenario for which - Implement extension listener
code fault message should be to subscribe to AIA Error topic
stamped with custom filter code for messages with custom
in Setup UI filter code

16 All rights reserved.

Copyright © 2008, Oracle.
Custom Handling of Errors

Copyright © 2008, Oracle. All rights reserved.

 Logging Framework

• Trace Logging –
Recording chronological
sequence of events.
- Custom XPath
- Java API

Siebel CRM
On Demand
• Error Logging –
Recording of errors. No
specific configuration
required. Logging done
non-intrusively.

• Use Oracle Enterprise

Manager (OEM) as user
interface to search, sort,
and filter logs.

Copyright © 2008, Oracle. All rights reserved.

Log Search Including AIA Attributes

Copyright © 2008, Oracle. All rights reserved.

 Enabling Error Handling and Logging
To set up error handling and logging:
• Set up Error Notifications and Roles
• Set up Trace Log Levels
• Enable Trace Logging in AIAConfigurationProperties.xml

Copyright © 2008, Oracle. All rights reserved.

 Enable Trace Logging

• Can be done on system level or on service level

• Service level overrides system level
• To be done in AIAConfigurationProperties.xml in
$AIA_HOME/config:

-bash-2.05b$ more AIAConfigurationProperties.xml

<?xml version="1.0" encoding="UTF-8"?>
<AIAConfiguration xmlns="http://xmlns.oracle.com/aia/core/config/V1"
version="1.0">
<!-- System-wide configuration -->
<SystemConfiguration>
<!-- System configuration properties -->
<Property name="TRACE.LOG.ENABLED">true</Property>
[Snip]

• Reload the configuration file afterwards through the admin page

Copyright © 2008, Oracle. All rights reserved.

 Targeted New Features 2.x – 3.0

2.x 3.0
• SOA platform 10.1.3.x • SOA platform 11g
1. Support OC4J JMS 1. Support Weblogic JMS
2. BPEL, ESB Fault Policies 2. Mediator Error Handling
3. Error Log + Trace Log 3. Error Log + Trace Log
framework framework
• No B2B Support • Capture Errors from B2B
• Worklist an integral part of framework • Option to decouple or switch off Worklist
• Log onto BPEL and ESB dashboards • Drill down into EM Control Flow Trace
externally from email notification
• No customization of notification • Customization of email notification content
• Multiple notifications based on number of • Notification aggregation
errors • Automatic error resubmission
• Manual error resubmission

Copyright © 2008, Oracle. All rights reserved.

Guaranteed Delivery and Error Recovery

Copyright © 2008, Oracle. All rights reserved.

 Topics

Concepts
AIA Approach
Source Responsibilities
Sample Flows
Configuration Settings
Error Recovery
AIA Recommends

Copyright © 2008, Oracle. All rights reserved.

 Guaranteed Delivery - Concepts

What is Guaranteed Delivery

• A message initiated from sender system is persisted until
it is successfully delivered to and acknowledged by the
receiver

Sender and Receiver are not necessarily the participating

applications, they are the logical milestones in an
integration scenario
• There could be multiple milestones in a given integration
scenario

The work done between 2 milestones is considered as one

logical unit of work
• This should be idempotent and could use appropriate
compensation handlers to achieve this.

Copyright © 2008, Oracle. All rights reserved.

 Guaranteed Delivery – Concepts …

Global Transaction

Req EBS Prov

ABCS ABCS

Source
Target
Application
Application

Source Integration
Milestone Target
Milestone
Milestone

Copyright © 2008, Oracle. All rights reserved.

 Sample Flow 2
Global Transaction

Req EBS Prov

ABCS ABCS

Source Producer Process Target

Application Application
Source Integration Target
Milestone Milestone Milestone

Target Milestone is with in Target Application transaction space

Copyright © 2008, Oracle. All rights reserved.

 AIA Approach
AQ-JMS for message persistence

Global Transaction
• Enable all services in one single transaction
• No commit point between two milestones

Error Handling
• Any system error will generate a rollback of all preceding
services
• One notification will be sent to Integration Administrator

Error Recovery
• Manual scripts to recover message from queue and resubmit

Copyright © 2008, Oracle. All rights reserved.

 AIA Approach …

Each message with a unique identifier

EBM Header populated for source milestone identifier
Fault Notification shall contain the message identifier and
source milestone identifier of the faulted message
The source milestone identifier will be resolved to a specific
JMS destination (or source application) in AIA
Configuration Properties File
Error Recovery scripts shall make use of these parameters
to recover and resubmit a faulted message

Copyright © 2008, Oracle. All rights reserved.

 Source Milestone Responsibilities

It could be Source Participating Application

Publish outbound messages to an AQ-JMS

• One queue for each Entity

Publish a unique message identifier on the JMS header of each

message
• Default MSGID generated in the Header Variable can also be
used for correlation

Copyright © 2008, Oracle. All rights reserved.

 Fault Flow
Global Transaction

Req EBS Prov

ABCS ABCS

Source
Target
Application
Application
Integration
Source Milestone
Fault Notification Target Milestone
Milestone
Create Task

Copyright © 2008, Oracle. All rights reserved.

 Configuration Settings

Global Transaction
• IDG – Chapter 11 - Enabling Transactions for an Integration Scenario
in Asynchronous Patterns of the Application Integration Architecture
• No commit point between milestones

Error Handling
• IDG - Chapter 14: Implementing Error Handling and Recovery for the
Asynchronous Message Exchange Pattern to Ensure Guaranteed
Message Delivery
• Use bpelx:rollback instead of throw in the catch blocks.
• Java snippet to send notifications for AIA / User Faults.
• Fault policies to contain a filter expression and perform “No Action” in
case of a roll back fault(ORABPEL-02180).
– Filter Expression - <test>$fault.summary/summary[contains(.,
"ORABPEL-02180")]</test>

Copyright © 2008, Oracle. All rights reserved.

 Error Recovery – Schema parameters

– EBM Header / EBMTracking

– Fault / FaultNotification / FaultMessage

• SenderResourceTypeCode
– Indicates the type of resource or system
where the rolled back message is stored,
whether Queue, Topic or Resequence
Store
• SenderResourceID
– Identification of the resource/system of
type SenderResourceTypeCode
• SenderMessageID
– Identification of the message persisted in
the resource/system associated with
type SenderResourceTypeCode

Copyright © 2008, Oracle. All rights reserved.

 Manual Error Recovery script – FP2.3

Manual ant script at $AIA_HOME/Infrastructure/ErrorHandling/util

• ant –buildfile MessageResubmit.xml

Resubmit Utility API,

• oracle.apps.aia.core.eh.resubmit.util.AIAEHResubmissionUtil class.
• For persistence in a topic:
– public boolean resubmitFailedMessageTopic( String topicName, String
topicTableName, String messageID );
• For persistence in a queue:
– public boolean resubmitFailedMessageQueue( String queueName, String
queueTableName, String messageID);
• For persistence in a Resequencer store:
– public boolean resubmitFailedMessageResequencer( String
routingServiceName, String groupID );

Copyright © 2008, Oracle. All rights reserved.

 Auto Error Recovery using EM

EM Targets
• Register all services and Edge applications as “Targets”

AIA Auto Recovery Notification Agent

• Notified when a Target comes back to life (SEBL_01)
• Query AIA_FAULT_TABLE and resubmit all messages
faulted due to this Target

AIA_FAULT_TABLE
• Mapping table for JMS Queue Name, JMS Message ID and
Target instance id

Copyright © 2008, Oracle. All rights reserved.

 Auto Error Recovery – auto update Tasks

AIA Auto Recovery Notification Agent

• update the respective Task after message resubmit
• Comments and status as completed

AIA_FAULT_TABLE updated for WF Task ID

• At Task creation time, map a task Id with JMS Message Id

Copyright © 2008, Oracle. All rights reserved.

 Fault Flow – auto recovery
EM
Global Transaction

Req EBS Prov

ABCS ABCS

Source
Target
Application
Application
Integration
Source Milestone
Update AIA_FAULT_TABLE Target Milestone
Milestone
Fault Notification
Create Task
Update AIA_FAULT_TABLE
with Task ID

WF AIA Auto
Recovery
Notification
Agent

Copyright © 2008, Oracle. All rights reserved.

 Some Do’s and Don’ts
Fault Policy Changes Do Not Require a Restart of Oracle BPEL
Server
• http://host:port/BPELConsole/domain_name/doReloadFaultP
olicy.jsp
Fault policies should not have any retry configured in
Guaranteed Delivery Global Transaction scenario
AQ JMS should be configured for max_retries=>0

BPEL Fault policy details

http://www.oracle.com/technology/products/ias/bpel/pdf/1013
3technotes.pdf
Taxonomy of BPEL Faults
http://www.oracle.com/technology/products/ias/bpel/htdocs/o
rabpel_technotes.tn007.html
Copyright © 2008, Oracle. All rights reserved.
 AIA Recommends

• Application Services being XA-Compliant is the best

option
• Non idempotent Application Services (non-retryable)
need to have corresponding compensatory services
• ABCS invoking non-idempotent application services
should invoke compensatory services in case of roll back
• Application Services implemented via JCA adapters can
leverage session management to facilitate transaction
coordination
• Requestor, EBS and Provider ABCS have to participate in
same transaction
• Avoid log running transactions by setting the transaction
boundaries based on the business requirements.

Copyright © 2008, Oracle. All rights reserved.

 Common Use cases

Integration of AIA Error Handling Framework with

• Trouble Ticket Application
• Monitoring Systems (BAM, OpenView)
• Legacy / 3rd Party application/Custom Integrations

Extending AIA Error Handling

• Custom notifications
• Extending Fault Message
• Turn off notifications only
• Turn off Task Assignments only

Copyright © 2008, Oracle. All rights reserved.

Copyright © 2008, Oracle. All rights reserved.
Copyright © 2008, Oracle. All rights reserved.
 Q&A

Copyright © 2008, Oracle. All rights reserved.