9. High Availability¶
The AMPS JavaScript Client provides an easy way to create highly-available
applications using AMPS, via the Client
class. In other clients
the highly available version of the client is called HAClient
,
however in JavaScript the Client
class has all the functions of
Client
, providing both regular and highly available modes.
Among its regular functions, Client
provides protection against
network, server, and client outages.
The Client class in the JavaScript library combines
functionality of both Client and HAClient classes
of other AMPS client libraries. |
Using high availability (HA) functions of the Client
allows
applications to automatically:
- Recover from temporary disconnects between client and server.
- Failover from one server to another when a server becomes unavailable.
Because the Client
can automatically manage failover and
reconnection, 60East recommends using the high availability (HA)
functions for applications that need to:
- Ensure no messages are lost or duplicated after a reconnect or failover.
- Persist messages and bookmarks in memory for protection against client failure.
You can choose how your application uses HA features. For
example, you might need automatic reconnection, but have no need to
resume subscriptions or republish messages. The high availability
behavior in Client
is provided by implementations of defined
interfaces. You can combine different implementations provided by 60East
to meet your needs, and implement those interfaces to provide your own
policies.
Some of these features require specific configuration settings on your AMPS instance(s). This chapter mentions these features and describes how to use them from the AMPS JavaScript client. You can find full documentation for these settings and server features in the User Guide.
Reconnection with Client¶
This description provides a high-level framework for understanding the
components involved in failover with the Client
. The components
are described in more detail in the following sections.
The Client
reconnect handler performs the following steps when
reconnecting:
Calls the
ServerChooser
to determine the next URI to connect to and theAuthenticator
to use for that connection.If the connection fails, calls
getError()
on theServerChooser
to get a description of the failure, sends an error to the error handler, and stops the reconnection process.Calls the
DelayStrategy
to determine how long to wait before attempting to reconnect, and waits for that period of time.Connects to the AMPS server. If the connection fails, calls
reportFailure()
on theServerChooser
and begins the process again.Logs on to the AMPS server. If the connection fails, calls
reportFailure()
on theServerChooser
and begins the process again.Calls
reportSuccess()
on theServerChooser
.Receives the bookmark for the last message that the server has persisted. Discards any older messages from the
PublishStore
.Republishes any messages in the
PublishStore
that have not been persisted by the server.Re-establishes subscriptions using the
SubscriptionManager
for the client. For bookmark subscriptions, the reconnect handler uses theBookmarkStore
for the client to determine the most recent bookmark, and resubscribes with that bookmark. For subscriptions that do not use a bookmark, theSubscriptionManager
simply re-enters the subscription, meaning that it is entered at the point at which theClient
reconnects.
The ServerChooser
, DelayStrategy
, PublishStore
,
SubscriptionManager
, and BookmarkStore
are all extension points
for the Client
. You can adapt the failover and recovery behavior
by setting a different object for the behavior you want to customize on
the Client
or by providing your own implementation.
Choosing Store Durability¶
The Client
provides recovery after disconnection using Stores.
As the name implies, stores hold information about the state of the
client. There are two types of store:
- A bookmark store tracks received messages, and is used to resume subscriptions.
- A publish store tracks published messages, and is used to ensure that messages are persisted in AMPS.
The AMPS JavaScript client provides a memory-backed version of each
store. The store interface is public, and an application can create
and provide a custom store as necessary. A Client
can use
a memory backed store for protection:
- Memory-backed stores provide disconnection recovery from AMPS by storing messages and bookmarks in your process’ address space. This is the highest performance option for working with AMPS in a highly available manner. The trade-off with this method is there is no protection from a crash or failure of your client application. If your application is terminated prematurely or, if the application terminates at the same time as an AMPS instance failure or network outage, then messages may be lost or duplicated.
The store interface is public, and an application can create and provide
a custom store as necessary. While clients provide convenience methods
for creating memory-backed Client
objects with the appropriate
stores, you can also create and set the stores in your application code.
The Client
provides convenience methods for creating clients and
setting stores. You can also construct a Client
and set the store
implementations you choose.
In this example, we create several clients. The first client has the
full set of HA features, such as: supports automatic connection failover and
reconnection, uses memory stores for both bookmarks and publishes, and
automatically re-subscribes in case of a failover. The second client
does not set a store for publishes, which means that AMPS will not store
outgoing messages, but is using the bookmark store to track incoming
messages. The final client does not specify any stores, and so has no
persistence for published messages or bookmark subscriptions, but
takes advantage of the automatic failover and reconnection in the
Client
.
// Failover, Bookmark and Publish stores, re-subscription
var memoryClient = amps.Client.createMemoryBacked('memory-backed');
// No publish store, no failover, memory-backed bookmark store
var subscriberClient = new amps.Client('subscriber');
subscriberClient.bookmarkStore(new amps.MemoryBookmarkStore());
// Failover behavior only.
var failoverClient = new amps.Client('failover');
var serverChooser = new amps.DefaultServerChooser();
serverChooser.add('ws://localhost:9000/amps/json');
serverChooser.add('ws://localhost:9100/amps/json');
serverChooser.add('ws://localhost:9200/amps/json');
failoverClient.serverChooser(serverChooser);
failoverClient.delayStrategy(new amps.FixedDelayStrategy());
Example 9.1: Client creation example
Connections and the ServerChooser¶
In the high availability mode, Client
attempts to keep itself
connected to an AMPS instance at all times, by automatically reconnecting
or failing over when it detects that the client is disconnected. When
you are using the Client
directly, your disconnect handler usually
takes care of reconnection. Client
with HA classes on the other hand,
can provide a disconnect handler that automatically reconnects to the
current server or to the next available server.
To inform the Client
of the addresses of the AMPS instances in
your system, you pass a ServerChooser
instance to the Client
.
ServerChooser
acts as a smart enumerator over the servers available:
Client
calls ServerChooser
methods to inquire about what
server should be connected, and calls methods to indicate whether a
given server succeeded or failed.
The AMPS JavaScript Client provides a simple implementation of
ServerChooser
,called DefaultServerChooser
, that provides very
simple logic for reconnecting. This server chooser is most suitable for
basic testing, or in cases where an application should simply rotate
through a list of servers. For most applications, you implement the
ServerChooser
interface yourself for more advanced logic, such as
choosing a backup server based on your network topology, or limiting the
number of times your application should try to reconnect to a given
address.
To connect to AMPS, you provide a ServerChooser
to Client
and
then call Client.connect()
to create the first connection:
var memoryClient = new amps.Client('server-chooser-demo');
// primary.amps.xyz.com is the primary AMPS instance, and
// secondary.amps.xyz.com is the secondary
var chooser = new amps.DefaultServerChooser();
chooser.add('ws://primary.amps.xyz.com:12345/amps/fix');
chooser.add('ws://secondary.amps.xyz.com:12345/amps/fix');
memoryClient.serverChooser(chooser);
memoryClient
.connect()
.then(function() {
...
return memoryClient.disconnect();
})
.catch(...);
Example 9.2: DefaultServerChooser creation example
Client
remains connected to the server until disconnect()
is
called. Client
automatically attempts to reconnect to your server
if it detects a disconnect and, if that server cannot be connected,
fails over to the next server provided by the ServerChooser
. In this
example, the call to connect()
attempts to connect and login to
primary.amps.xyz.com
, and resolves if that is successful. If it
cannot connect, it tries secondary.amps.xyz.com
, and continues
trying servers from the ServerChooser
until a connection is
established. Likewise, if it detects a disconnection while the client is
in use, then Client
attempts to reconnect to the server with which
it was most recently connected; if that is not possible, then it moves
on to the next server provided by the ServerChooser
.
The default ServerChooser
simply provides the next URL in the
sequence. This strategy works for many applications. If you need a
different strategy, you can implement your own logic for failover by
creating a class derived from ServerChooser
.
Setting a Reconnect Delay and Timeout¶
You can control the amount of time between reconnection attempts and
set a total amount of time for the Client
to attempt to reconnect.
The AMPS JavaScript client includes a method for setting a delay strategy on
a client, Client.delayStrategy()
. This method accepts an instance
of any type that provides the methods getConnectWaitDuration()
and
reset()
, as described in the API documentation.
While you can easily implement your own delay strategy, the client also provides two delay strategies:
FixedDelayStrategy
provides the same delay each time theClient
tries to reconnect.ExponentialDelayStrategy
provides an exponential backoff until a connection attempt succeeds.
To use either of these classes, you simply create an instance, set the
appropriate parameters, and install that instance as the delay strategy
for the Client
. For example, the following code sets up a
reconnect delay that starts at 200ms and increases the delay by 1.5
times after each failure. The strategy allows a maximum delay between
connection attempts of 5 seconds, and will not retry longer than 60
seconds.
var client = new amps.Client('delay-strategy-demo');
client.delayStrategy(
new amps.ExponentialDelayStrategy({
initialDelay: 200,
maximumDelay: 5000,
backoffExponent: 1.5,
maximumRetryTime: 60000
})
);
Implementing a Server Chooser¶
As described above, you provide the Client
with connection strings
to one or more AMPS servers using a ServerChooser
. The purpose of
a ServerChooser
is to provide information to the Client
.
A ServerChooser
does not manage the reconnection process, and
should not call methods on the Client
.
A ServerChooser
has two required responsibilities to the
Client
:
Tells the
Client
the connection string for the server to connect to. If there are no servers, or theServerChooser
wants the connection to fail, theServerChooser
returnsnull
.To provide this information, the
ServerChooser
implements thegetCurrentUri()
method.Provides an
Authenticator
for the current connection string. This is especially important for installations where different servers require different credentials or authentication tokens must be reset after each connection attempt.To provide the authenticator, the
ServerChooser
implements thegetCurrentAuthenticator()
method.
The Client
calls the getCurrentUri()
and
getCurrentAuthenticator()
methods each time it needs to make a
connection.
Each time a connection succeeds, the Client
calls the
reportSuccess()
method of the ServerChooser
. Each time a
connection fails, the Client
calls the reportFailure()
method
of the ServerChooser
. The Client
does not require the
ServerChooser
to take any particular action when it calls these
methods. These methods are provided for the Client
to do internal
maintenance, logging, or record keeping. For example, a Client
might keep a list of available URIs with a current failure count, and
skip over URIs that have failed more than 5 consecutive times until all
URIs in the list have failed more than 5 consecutive times.
When the ServerChooser
returns a null
from getCurrentUri()
,
indicating that no servers are available for connection, the Client
calls the getError()
method on the ServerChooser
, if one is
provided, and includes the string returned by getError()
in the
generated exception.
Heartbeats and Failure Detection¶
Use of the Client
allows your application to quickly recover from
detected connection failures. By default, connection failure detection
occurs when AMPS receives an operating system error on the connection.
This system may result in unpredictable delays in detecting a connection
failure on the client, particularly when failures in network routing
hardware occur, and the client primarily acts as a subscriber.
The heartbeat feature of the AMPS client allows connection failure to be
detected quickly. Heartbeats ensure that regular messages are sent
between the AMPS client and server on a predictable schedule. The AMPS
client and server both assume disconnection has occurred if these
regular heartbeats cease, ensuring disconnection is detected in a timely
manner. To use heartbeating, call the heartbeat()
method on
Client
:
var memoryClient = new amps.Client('importantStuff');
...
memoryClient.heartbeat(3);
memoryClient.connect().then(...).catch(...);
Example 9.3: Heartbeat example
Method heartbeat()
takes one parameter: the heartbeat interval. The
heartbeat interval specifies the periodicity of heartbeat messages sent
by the server: the value 3
indicates messages are sent on a
three-second interval. If the client receives no messages in a
six-second window (two heartbeat intervals), the connection is assumed
to be dead, and the Client
attempts reconnection. The optional second
parameter of the heartbeat()
method allows the idle period to be
set to a value other than two heartbeat intervals.
Heartbeats are handled asynchronously by the AMPS client. Your application must not flood the execution queue for longer than the heartbeat interval, or the application is subject to being disconnected. |
Considerations for Publishers¶
Publishing with the Client
in the HA mode is nearly identical to
regular publishing; you simply call the publish()
method with your
message’s topic and data. The AMPS client sends the message to AMPS,
and then returns from the publish()
call. For maximum performance,
the client does not wait for the AMPS server to acknowledge that the
message has been received.
When a Client
sets a publish store, the publish store retains a
copy of each outgoing message and requests that AMPS acknowledge that
the message has been persisted. The AMPS server acknowledges messages
back to the publisher. Acknowledgments can be delivered for multiple
messages at periodic intervals (for topics recorded in the transaction
log) or after each message (for topics that are not recorded in the
transaction log). When an acknowledgment for a message is received, the
Client
removes that message from the publish store. When a connection
to a server is made, the Client
automatically determines which
messages from the publish store (if any) the server has not processed,
and replays those messages to the server once the connection is
established.
For reliable publishers, the application must choose how best to handle
application shutdown. For example, it is possible for the network to
fail immediately after the publisher sends the message, while the
message is still in transit. In this case, the publisher has sent the
message, but the server has not processed it and acknowledged it. During
normal operation, the Client
will automatically connect and retry
the message. On shutdown, however, the application must decide whether
to wait for messages to be acknowledged, or whether to exit.
Publish store implementations provide an unpersistedCount()
method
that reports the number of messages that have not yet been acknowledged
by the AMPS server. When the unpersistedCount()
reaches 0
,
there are no unpersisted messages in the local publish store.
For the highest level of safety, an application can wait until the
unpersistedCount()
reaches 0
, which indicates that all of the
messages have been persisted to the instance that the application is
connected to, and the synchronous replication destinations configured
for that instance. When a synchronous replication destination goes
offline, this approach will cause the publisher to wait to exit until
the destination comes back online or until the destination is downgraded
to asynchronous replication.
For applications that are shut down periodically for short periods of
time (for example, applications that are only offline during a weekly
maintenance window), another approach is to use the Client.flush()
method to ensure that messages are delivered to AMPS, and then rely on
the connection logic to replay messages as necessary when the
application restarts.
For example, the following code flushes messages to AMPS, then warns if not all messages have been acknowledged:
var client = amps.Client.createMemoryBacked('ha-publisher');
...
client
.connect()
.then(function() {
// Publish messages
...
/**
* We think we are done, but the server may not
* have received or acknowledged all messages yet.
* Wait for the server to have received all messages.
* The program could also specify a timeout in this
* command to avoid blocking forever if the network
* is down or all servers are offline.
*/
return client.flush();
})
.then(function() {
/**
* Print warning to the console if messages have
* been published but not yet acknowledged as persisted
*/
if (client.publishStore().unpersistedCount() > 0) {
console.log('All messages have been published');
console.log('But not all have been persisted');
}
return client.disconnect();
});
Example 9.4: HAPublisher
In this example, the client sends each message immediately when
publish()
is called. If AMPS becomes unavailable between the final
publish()
and the disconnect()
, or one of the servers that
the AMPS instance replicates to is offline, the client may not have
received a persisted acknowledgment for all of the published messages.
For example, if a message has not yet been persisted by all of the
servers in the replication fabric that are connected with
synchronous replication, AMPS will not have acknowledged the message.
Before shutting down the client, the code does two things:
- The code first flushes messages to the server to ensure that all messages have been delivered to AMPS.
- The code next checks to see if all of the messages in the publish store
have been acknowledged as persisted by AMPS. If the messages have not
been acknowledged, they will remain in the publish store and will
be published to AMPS, if necessary, the next time the client connects.
An application may choose to wait until
unpersistedCount()
returns0
, or (as we do in this case) simply warn that AMPS has not confirmed that the messages are fully persisted. The behavior you choose in your application should be consistent with the high-availability guarantees your application needs to provide.
AMPS uses the name of the Client to determine the
origin of messages. For the AMPS server to correctly
identify duplicate messages, each instance of an
application that publishes messages must use a distinct
name. That name must be consistent across different runs
of the application. |
AMPS provides persisted acknowledgment messages for topics that do not have a transaction log enabled. However, the level of durability provided for topics with no transaction log is minimal. Learn more about transaction logs in the User Guide. |
Considerations for Subscribers¶
Client
provides two important features for applications that
subscribe to one or more topics: re-subscription, and a bookmark store
to track the correct point at which to resume a bookmark subscription.
Resubscription With SubscriptionManager¶
Any asynchronous subscription placed using a Client
is
automatically reinstated after a disconnect or a failover. These
subscriptions are placed in an in-memory SubscriptionManager
, which
is created automatically when the Client.createMemoryBacked()
static
method is called. Alternatively, it can be created and assigned before
the client is connected:
var client = new amps.Client('subscription-manager-demo');
client.subscriptionManager(new amps.DefaultSubscriptionManager());
Example 9.5: Creating and assigning a subscription manager
Most applications will use this built-in subscription
manager, but for applications that create a varying number of subscriptions,
you may wish to implement SubscriptionManager
to store subscriptions in
a more durable place. Note that these subscriptions contain no message data,
but rather simply contain the parameters of the subscription itself (for
instance, the command, topic, message handler, options, and filter).
When a re-subscription occurs, the AMPS JavaScript Client re-executes the
command as originally submitted, including the original topic, options,
and so on. AMPS sends the subscriber any messages for the specified
topic (or topic expression) that are published after the subscription is
placed. For a sow_and_subscribe
command, this means that the client
reissues the full command, including the SOW query as well as the
subscription.
Bookmark Stores¶
In cases where it is critical not to miss a single message, it is important to be able to resume a subscription at the exact point that a failure occurred. In this case, simply recreating a subscription isn’t sufficient. Even though the subscription is recreated, the subscriber may have been disconnected at precisely the wrong time, and will not see the message.
To ensure delivery of every message from a topic or set of topics, the
AMPS Client
can plug in a BookmarkStore
that, combined with the
bookmark subscription and transaction log functionality in the AMPS
server, ensures that clients receive any messages that might have been
missed. The client stores the bookmark associated with each message
received, and tracks whether the application has processed that message;
if a disconnect occurs, the client uses the BookmarkStore to determine
the correct resubscription point, and sends that bookmark to AMPS when
it re-subscribes. AMPS then replays messages from its transaction log
from the point after the specified bookmark, thus ensuring the client is
completely up-to-date.
Client
helps you to take advantage of this bookmark mechanism
through the Client.bookmarkStore()
method and MemoryBookmarkStore
class. When a bookmark store is assigned to the client, When you create
subscriptions, whenever a disconnection or failover occurs, your application
automatically resubscribes to the message after the last message it processed.
To take advantage of bookmark subscriptions, do the following:
Ensure the topic(s) to be subscribed are included in a transaction log. See the User Guide for information on how to specify the contents of a transaction log.
Before connecting, create and assign a bookmark store object to the client.
Use the
Client.bookmarkStore().discard()
method in message handlers to indicate when a message has been fully processed by the application.
The following example creates a bookmark subscription against a transaction-logged topic, and fully processes each message as soon as it is delivered:
var client = new amps.Client('aClient');
client.subscriptionManager(new DefaultSubscriptionManager());
client.bookmarkStore(new amps.MemoryBookmarkStore());
...
client.execute(
// The subscription command
new amps.Command('subscribe')
.topic('myTopic')
.bookmark(amps.Client.Bookmarks.MOST_RECENT)
.subId('MySubId'),
// Message handler discards every message after processing
function(message) {
console.log(message.data);
client.bookmarkStore().discard(message);
}
);
Example 9.6: Client subscription
Storing these bookmarks in the bookmark store allows the application to restart the subscription from the last message processed, in the event of either server failure or disconnect.
For optimum performance, it is critical to discard every
message once its processing is complete. If a message is
never discarded, it remains in the bookmark store. During
re-subscription, Client always restarts the
bookmark subscription with the oldest undiscarded
message, and then filters out any more recent messages
that have been discarded. If an old message remains in
the store, but is no longer important for the
application’s functioning, then the client and the AMPS
server will incur unnecessary network, and CPU activity. |
The command method, subId()
, specifies an identifier to be used for
this subscription. If the subId
is not provided, Client
will
generate one and resolve the Client.execute()
Promise with it, like
most other Client
functions. If you wish to resume a subscription
from a previous point after the application has disconnected, the
application must pass the same subscription ID as before. Passing a
different subscription ID bypasses any recovery mechanisms, creating
an entirely new subscription. When you use an existing subscription ID,
the Client
locates the last-used bookmark for that subscription in
the bookmark store, and attempts to re-subscribe from that point.
Client.Bookmarks.NOW
specifies that the subscription should begin from the moment the server receives the subscription request. This results in the same messages being delivered as if you had invokedsubscribe()
instead, except that the messages will be accompanied by bookmarks. This is also the behavior that results if you supply an invalid bookmark.
Client.Bookmarks.EPOCH
specifies that the subscription should begin from the beginning of the AMPS transaction log (that is, the first entry in the oldest journal file for the transaction log).
Client.Bookmarks.MOST_RECENT
specifies that the subscription should begin from the last-used message in the associatedBookmarkStore
. Alternatively, if this subscription has not been seen before, it instructs the subscription to begin withEPOCH
. This is the most common value for this parameter, and is the value used in the preceding example. By usingMOST_RECENT
, the application automatically resumes from wherever the subscription left off, taking into account any messages that have already been processed and discarded.
When the Client
re-subscribes after a disconnection and
reconnection, it always uses MOST_RECENT
, ensuring that the
continued subscription always begins from the last message used before
the disconnect, so that no messages are missed.
Conclusion¶
With only a few changes, most AMPS applications can take advantage of
the high availability features of the Client
to become more
highly-available and resilient. Using the PublishStore
, publishers
can ensure that every message published has actually been persisted by
AMPS. Using BookmarkStore
, subscribers can make sure that there
are no gaps or duplicates in the messages received. Client
makes
both kinds of applications more resilient to network and server outages
and temporary issues. Though Client
provides useful defaults for
the PublishStore
, BookmarkStore
, SubscriptionManager
,
ServerChooser
, and DelayStrategy
, you can customize any or all
of these to the specific needs of your application and architecture.