Review Criteria for API-powered Digital

Updated 4/4/2016

Overview
A

Vendor Experience

Architecture

API Gateway

API Analytics and Monitoring

API Security

Developer Portal

Section A - Vendor Experience

A1

Requirement
Please describe your company's strategy around API
management. What percentage of your company's
revenue is derived from API Management?

A2

When was your API management product GA?

A3

What % of the Fortune 100 uses the product?

Is the product telco grade? What % of the largest 12
global telcos use your product?

A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17

Is the product in production with large retail brands?

Can you provide examples of your thought
leadership in the API space?
What kind of experience do you have running a
managed cloud solution at scale for your cloud
customers?
Who are your largest customers in the:
Retail sector
Financial sector
Telecommunications
Healthcare
Who are your most significant NEW customers (of
your API products) in the past 6 months?
How do you onboard and partner with customers for
success?
Do you provide free training for architecture,
development, and operations on your website?
Do you provide online tutorials to help us learn your
product?
community

Section B - Architecture
Requirement

B3

Does your product support public cloud, private

cloud and hybrid deployments?
Is your Private Cloud fully supported On-Premises
(does not have any dependencies on making calls
externally)?
Does your private and public cloud offering use the
same code base?

B4

Does the solution support a hybrid solution, where

traffic management and security policies can be colocated with the API applications, while other API
functions run on the core API management platform
(enterprise gateway) in the managed cloud or onpremises deployment

B5

Does the platform architecture support multi-tenancy

both for both public and private Cloud deployments?

B6

Can multiple teams work independently with runtime

isolation?

B7

How does the platform support a multi-region, multidata center deployment to ensure the highest level
of availability and distribution?

B1
B2

B8

B9
B10
B11
B12

How does your platform integrate into continuous

development and deployment practices?
Explain how your solution supports a scalable
environment and describe what is needed to
provision additional capacity per API / per team / per
region / per organization.
Does your solution provide a centralized UI for multiDC deployments or do we need to manage them
independently?
Does the solution support zero downtime patching
and updates?
Does the solution have the ability to do intelligent
traffic routing to give users the closest point of
presence over wide geographical areas?

Section C - API Gateway

C3

Requirement
Does the product support
OpenAPI (formerly known
as Swagger) to design APIs and generate
documentation?
Does the product facilitate rapid prototyping of mock
APIs?
Does the product help create uniform, consistent,
well-formed APIs, even if the underlying backend
systems weren't built that way?

C4

How are existing SOAP services added?

C1
C2

C5
C6
C7
C8

C9
C10
C11
C12

C13

Can deployments of assets be automated for the

development lifecycle?
Can your platform reference existing assets such as
encryption libraries, schema validation tools, data
validation libraries, etc.
How does your product support threat detection by
detecting fraudulent data injections at the API level?
Please describe your ability to protect from traffic
spikes.
Please describe your ability to manage API
consumption through quotas. Can quotas be setup
both by developers as well as by product managers
post-development? Can they be adjusted at
runtime?
Can quotas be synchronized across multi-region
deployments?
Does the platform support publishing SOAP, REST,
JSON, and XML style services as APIs as well as
JMS?
Please describe process flows for discovering
services in the runtime environment.

C16

Does the product support API mashups?

Please describe your ability to enhance Proxy
functionality through both configuration and code.
Please describe any out of the box functions for
doing traffic mediation, transformation, and security
at the
API Level.
Are
standard
transformations included? (XML to
JSON, JSON to XML, SOAP to REST, REST to
SOAP)

C17

Does the proxy support compression?

C18

Does the proxy support HTTP & HTTPS?

When necessary, can the proxy talk to JMS based
systems?

C14
C15

C19

C20
C21

C22

C23
C24
C25
C26
C27
C28

Are streaming connections supported?

Please describe the debugging tools built into the
platform.
Can the debugging tool show a "before" and "after"
of each policy during replay? Also can the debugging
be performed in an off-line mode to minimize any
overhead to the runtime API traffic.

How is versioning supported?

Are all policies and system configurations stored in
standards based XML with well published schemas
for easy migration/promotion?
Does the proxy support caching?
In addition to an expiration, can the cache be
manipulated programmatically?
Do you support a multi-level cache model ? For
example, is the in-memory cache able to spill over to
the disk?
Does the product support caching based on payload
information and HTTP headers? Is this available via
built-in policies?

C33

Does the proxy rate limiting, quotas, and spike

arrests?
Can behavior change dynamically based upon
factors such as user credentials, location, device
type,...?
Does the proxy support dynamic routing
(orchestrationor intelligent routing to a second
system based upon the response from a first
system)?
Please describe the out-of-the-box backend service
APIs for common application functionality such as
user management, data storage and
synchronization, messaging, and locations.
Does the platform support identity integration with
popular social networks and Internet services and if
so, which ones?

C34

Does the solution allow the storing and querying of

arbitrary schema-less JSON data?

C29
C30
C31

C32

C35
C36
C37

Can data be tagged and queried by location?

Can binary objects such as files and images be
stored in the platform?
Please provide examples of large-scale deployments
using this capability

C38

Does the platform provide user management and

social relationship functionality for building
personalized applications?

C39

Can the platform support push notifications across

various mobile platforms?

C40
C41
C42

C43
C44
C45
C46
C47

Can the core functionality of the platform be

extended
by the customer?
Does
the platform
support extensions using
common languages like Java, Python, or
JavaScript?
Can the platform host and run unmodified Node.js
applications in order to implement custom APIs
without the need for a separate application server?

Does the platform have wizards to generate APIs

from Swagger, SOAP services, and other backend
services?
What are the standard governance features
available in the product?
How does the product support API Lifecycle
governance?
Can your product publish APIs for external and
internal consumers? How are these managed
independently?
How do you manage API visibility and restrict access
to consumers? Is this configuration in the platform or
built as part of the APIs enablement?

Section D - API Analytics

D1

Requirement
Please describe the out-of-the-box reports provided
by the tool.

D4

Does the UI allow for drill down on each of the

charts?
Does the tool provide a wizard for creating custom
reports?
Are there maps for detailing geo-location of API
calls?

D5

Are the analytics collected asynchronously (so as

not to impede runtime traffic)?

D2
D3

D6
D7
D8

Does the analytics data, once collected, provide an

API for easy access and export?
Can the solution be used to provide business level
visibility?
What level of operational visibility can the solution
provide based on API traffic flowing through the
system?

D9
D10
D11

D12
D13
D14
D15

D16
D17
D18
D19
D20
D21
D22
D23
D24
D25
D26
D27
D28

What tools are available out of the box to do various

kinds of trend analysis and inspection of anomalies?
Can reports be created on-demand?
Does the tool support predictive and trend-based
analytics?
Describe how the product gathers contextual
information (information above and beyond the basic
transaction details which helps the business to
understand the transaction in depth). Please specify
third-party APIs and internal enterprise data sources.
Is there a service for attaining business level insights
based on the contextual data?
What metrics and dimensions are supported by the
tool?
Do you provide service performance monitoring,
reporting, and analysis?

If payload data is captured, can this data be used for

reporting?
What are the exception management reporting
capabilities?
Does your product provide end-end visibility and
trending
performance
statistics?
Does
your
solution support
billing based on TPS
and/or aggregate transactions for each
developer/application.
Solution must provide performance management
data with counters per application type and per API
message type.
What level of reporting is available to the developer?
(call latency,
SLA compliance,
other metrics)
Does
the product
provide easy-to-use
custom
reporting capabilities over multiple dimensions and
filters?
Does your product provide the ability to report using
the payload of the messages?
Does your product provides the ability to easily
integrate with other systems, for instance through
API calls?
Does your product provide capabilities to create
custom dashboards to perform root-cause analysis?
Does your product provide flexibility to extend the
functionality and implement attribute specific runtime
enforcements for API?
Are all of your billing and developer usage data
available via an API to allow an easy integration with
existing systems?
Does the product provide the ability to inspect the
payload and retrieve payload data to create custom
metrics to be included in custom reports?

D29

Does the solution provide the ability to perform

synthetic transaction testing from different global
locations?

Section E - API Security

E5

Requirement
How is single-sign on supported across all the roles
involved in the lifecycle in your product?
What are the standard industry security certifications
available for your product?
Do you use open standards to delegate
authentication capabilities to your tenants?
Explain the mechanisms you use to support API
security (e.g. tokens, encryption, policy systems).
Please describe the security / policy enforcement
options when some assets might require additional
security in a cloud/on-premises infrastructure.

E6
E7

Please describe your expertise with OAuth (including

major customers you have supported).
Which versions of OAuth are supported?

E8

Are LDAP and AD supported?

E9

Does the product support both secure channels and

secure payloads?

E1
E2
E3
E4

E10
E11

Does the proxy provide support for CORS?

Does the proxy protect against XML or JSON
attacks?

E12

Are all of these security features available as selfservice via configuration (not coding)?

E13
E14
E15
E16
E17

How does the solution handle role based access

controls to ensure different members of the API
team can perform their roles effectively without
affecting other teams?
Is your public cloud offering PCI DSS level 1 and
level 2 certified?
Is your public cloud offering HIPAA compliant?
Can the product be extended to support
custom/proprietary implementations?
Can APIs be secured at the operation level? (Ex:
can do GET, but not POST or PUT)

Section F - Developer Portal

F1
F2
F3
F4
F5
F6
F7

Requirement
How are assets manifested in the developer portal
for developer use?
Please describe how the tool facilitates on-boarding.
Is this portal available as a completely on-premises
solution?
Does the solution provide interactive documentation
to allow API consumers to easily try out published
APIs?
Does each developer (or team) get their own
personalized metrics?
Is the registration form customizable?
Can the customer customize, skin, and modify the
portal without vendor involvement?
Does the portal leverage standard CMS
technologies to ensure easy to find skill sets and
pre-existing modules?

F10

Does the tool provide the ability to revoke or

suspend developer keys?
Does the solution support a B2B2D type model
which allows enterprises to let their partners manage
their own pool of developers and their access to the
enterprises APIs?
Please describe the ability for the platform to support
monetization. What are the various revenue models
supported?

F11

Are the pricing models configurable without coding?

F12

Does the platform integrate with third-party payment

systems?

F8

F9

Review Criteria for API-powered Digital Business Platforms

Details
APIs are a critical part of our company strategy moving forward. It
is important to us that whoever we partner with considers API
management a core part of their business.
We're interested in the track record of your company in API
management.
In addition to the product features mentioned above, would like to
understand the real world experience you have had with large scale
deployments on your API management platform.
Knowing the uptime requirements of a telco, it is important to know
that the platform meets these stringent criteria.
High-profile brands that trust your platform would say a lot for the
robustness and performance of your product.
APIs, social, and mobile are fast moving topics. We would like to
work with a vendor who leads the space.
While many vendors are now offering cloud-based versions of their
products, it is critical that the chosen vendor has demonstrated real
world experience with large scale customers running in the cloud.
We would like to know more about your real world experience.

We would like to know more about your market momentum.

Details
Depending on present and future
project requirements, we may
need one or both of the deployments to be supported. In this case,
cloud is understood to mean a vendor managed cloud. Hybrid is
defined as a local gateway with management functions and
analytics in the cloud.

To help with a flexible deployment model that reduces latency since

traffic management and security happens closer to the application,
avoids synchronous call-outs in the main message path, and
protects the last mile.
The ability to run a multi-tenant environment can be important
when dealing with multiple lines of business and/or partners. Is the
cloud installation a true multi-tenant environment? Does the exact
same functionality exist when deployed on premises?
An enterprise SDLC (software development life cycle) can be a
complicated process with many constituents. The ability for diverse
teams to have their own view of the platform with logical separation
of all policies and configurations is very important. The ideal tool
will allow a centrally managed platform to support development
teams across the enterprise. The ideal tool will work with industry
popular CI/CD tools such as Maven or Jenkins.
Geographical redundancy is important both for high availability and
also for latency and performance considerations. We need to
understand how an instance deployed in one physical data center
interacts and collaborates with an instance deployed at another
data center.
The operations teams already have workflows, processes, and
scripts to perform their work. Does your platform integrate well with
these existing tools? Can the platform be run via the command
line? Via scripts?

In today's world, traffic bursts happen. We need to know that our

capacity can scale along with these dynamic fluctuations in traffic.
Ease of management is one of the day-to-day considerations in
choosing a platform such as this. How can the tool ease
management over and contribute to overall productivity?
For critical applications and geographically dispersed user base,
how can the platform be administered so as not to incur any
downtime for developers, partners, and users?
For latency sensitive applications, intelligent routing to the nearest
point of presence can be very important.

Details

Can the services support other protocols and how is complex data
transformation handled?
How hard is it to incorporate into existing development standard
tools? What development tools are required to develop and deploy
with your platform?
IT has invested in middleware, and how can your platform use
these assets?

For example getCustomerInfo API would require multiple back-end

calls to be made to multiple systems and each system supports
different protocols (for example SOAP web service, JSON service
and direct database call). Does this require custom development or
is it supported by configurations?
Please highlight which proxy features cannot be accomplished via
simple configuration.

In order to reuse existing systems or to talk with legacy systems, it

is important that the platform can perform these transformations.
Can messages be both sent and received by the proxy in a
compressed format? This will save bandwidth and reduce latency
in some situations.
Previous generations of software built hard-wired connectors into
their tools. To avoid these brittle connections, can the platform
perform all functionality over standard HTTP?
In the event of the existence of back-end system based upon JMS,
can requests be placed into the correct queue?

For long running transactions or large payloads, can the proxy

stream traffic?
Distributed systems are more complex than client server systems.
What tools does the platform possess which will help us to isolate
issues and solve them faster?

This functionality can be crucial during forensics or during preproduction

policy.
To minimizetesting
impactoftoa developers
and users, versioning needs to
be flexible. Versioning refers to both the version on the API (as part
of the URI) as well as the versions of the policies themselves.
Lastly, versioning refers to minimizing the impact on operations
through obviating the need to maintain multiple versions of a
service.
A standard format like XML allows for easy transformation and
manipulation in a variety of tools.
Caching at the proxy minimizes hits against the back end systems.
While it is important to be able to set a cache to expire a certain
point in time, it is also necessary to invalidate or refresh the cache
via standard API calls to reflect changes in back end systems.
In-memory cache is very fast, but has limitations of size. The ability
to perform multi-level caching is important for heavy caching
situations.
To optimize caching, the platform should be able to cache based
on many types of information, including data contained within the
payloadto
ofdata
the message.
Access
and load on back-end systems must be
configurable and controllable. The ability to block based on sheer
traffic volume is important as are the finer grained controls of rate
limits (messages/time interval) and quotas (raw # of requests
permitted).
In the dynamic world of APIs and mobile applications it is often
necessary for the platform to make dynamic decisions based upon
various pieces of information contained within the inbound request.
In the dynamic world of APIs and mobile applications it is often
necessary for the platform to make dynamic decisions based upon
the current conditions.
Most modern apps require functionality that is missing from existing
backend systems. By providing this functionality out-of-the-box, the
platform speeds time to market for all apps and reduces complexity
in the environment.
Most apps require some social component. By providing this
functionality out-of-the-box, the platform speeds time to market for
all apps and reduces complexity in the environment.
To achieve maximum flexibility, does the platform allow for arbitrary
queries and storing dynamic data (beyond pre-configured SQL-like
schemata)?
Location based service are becoming more and more prevalent.
Geotagging data provides great power to the platform and covers a
gap in most legacy systems.
While it is crucial to store plain text, many modern apps allow for
image uploads (and other binary types).
We would like to understand more about the real world experience
with this part of the platform.

This type of functionality is often absent from legacy systems, yet

required
modernisapplications.
This
typeby
of most
functionality
often absent from legacy systems, yet
required by most modern applications. It is crucial for the system to
be able to communicate with users in a manner in which they are
familiar.
In the interest of minimizing professional services and increasing
time to market, can the above mentioned database functionality be
achieved via configuration (not coding)?
If we are to perform these activities ourselves, the platform needs
to support commonly used technologies.
With the increasing popularity of Node.js, it would be useful to have
this capability built into the platform and not require yet another tool
to
introduced
into the
In be
order
for API teams
to environment.
be agile, rapidly configure/build and
deploy APIs, it's important to have OOTB wizards that can generate
APIs from Swagger docs, SOAP services and other back-end APIs.
It should provide for check-box capability to secure APIs using API
keys, OAuth and be able to enforce CORS and other commonly
expected policies.

Details
The reports in this list should require no configuration. Normally
these will include basic traffic, usage, and performance information.
Drill down analytics allows for quick triage of the health of an API
program and assists in rapid troubleshooting during anomalous
conditions.
No vendor can provide every report we need out of the box. The
platform should have a wizard for easy creation of custom reports.
Many decisions in an API program are based upon the location of
users. The platform should have geo-location reporting built in.
The single greatest factor in the user satisfaction of an app is its
response time. Are the analytics collected in such a way as to not
impact response time?
We are not interested in creating a data silo. The collected
analytics data must be accessible for merging with other business
intelligence tools.
Beyond operational level and developer level metrics, how does
the platform provide visibility to the business?
Beyond simple graphs of traffic, what visibility would an ops team
gain from using the platform?

The tool needs to both provide visibility into trends (to prepare for
capacity bursts or product demand, for example) and to allow
inspection if anomalies are detected.
Do reports need to configured before launching the system? Can
reports be constructed on demand as the need arises (like after
viewing surprising traffic)?
After the fact forensics are important, but the ability to spot trends
in advance is crucial in today's environment.

Transaction data, viewed in a vacuum, is of limited use. Customer

behavior changes greatly based upon their location, the weather,
the type of device being used, etc....
If needed, do you provide the services of data scientists to analyze
this contextual information and report back to the business with
actionable insights?
The tool must support a variety of analytics use cases without
requiring additional programming

For example, can this data query be completed: query the list of
customer ids (part of the API payload) that falls into segment vegi
(again part of API payload) that called the order/create API (API
metadata) during the last seven days.

Details

OAuth is one of the most widely used forms of authentication for

consumer or partner facing apps. We would like to understand both
the product capabilities with regards to OAuth as well as real world
experience.
LDAP and active directory are the most common forms of
authentication in use today. This functionality should be accessible
with no coding.
Different
types of APIs and different types of data require different
types of security. Sometimes a secure SSL connection will be
sufficient. Sometimes the payload will need to be encrypted as
well.
CORS (Cross-origin resource sharing) is a standard mechanism
that allows JavaScript XMLHttpRequest (XHR) calls executed in a
web page to interact with resources from non-origin domains.
CORS is a commonly implemented solution to the "same-origin
policy" that is enforced by all browsers.
As part of a defense in depth strategy, does the platform help in
protecting against modern attack vectors such as XML?
In an effort to minimize the need for professional services and to
accelerate time to market, are all of the above mentioned security
features available via standard policies/configuration?
Auditing and compliance processes dictate that RBAC (Role Based
Access Control) is supported by enterprise platforms. The allows
for an audit trail and administrative accountability. It also aids in the
SDLC by limiting the potential for one team's work to interfere with
the work of another team.
Many APIs require (or eventually require) payment processing as
part of the monetization strategy. PCI certification is necessary.

Details
What additional development is required and what features are
supported?
Developer and partner productivity depends on an efficient on
boarding experience. How does the tool ease this friction?
While documentation is important, experience shows that a
developer's time to value is greatly improved with interactive tools.
To assist developers and teams, will they get their own view of the
metrics
related
to any
they have
registered?
Corporate
policies
mayapplication
dictate thatwhich
we collect
certain
pieces of
information when onboarding a new developer. The data fields in
the registration process need to be configurable to capture these
fields.

As a follow up to the previous question, if we are to be able to

perform this work on our own, the portal will need to be based on
standard technologies.
In the event of an expired contract with a developer or when an
abnormal situation occurs, the platform must allow for both the
disabling and revocation of individual app keys.
Large partners require the ability to maintain the existing
relationships with their own developers. How does the platform
support this second-level relationship?
Some of the APIs will need to be monetized. Given that there are
multiple ways to monetize an API, does the platform allow for
mixing and matching of these models?
Can the financial models be created through configuration only or
do they require custom coding?
Once the metering has been performed, it will be necessary to
pass the transaction to a payment processor. The platform should
be able to connect to these processors (including CDRs).

tal Business Platforms