5. Error Handling

In every distributed system, the robustness of your application depends on its ability to recover gracefully from unexpected events. The AMPS client provides the building blocks necessary to ensure your application can recover from the kinds of errors and special events that may occur when using AMPS.

Exceptions

Generally speaking, when an error occurs that prohibits an operation from succeeding, AMPS will throw an exception. AMPS exceptions universally derive from AMPS.Client.Exceptions.AMPSException, so by catching AMPSException, you will be sure to catch anything AMPS throws, for example:

using AMPS.Client;
using AMPS.Client.Exceptions;
...
public static void ReadAndEvaluate(Client client)
{
    // read a new payload from the user
    string payload = Console.ReadLine();

    // write a new message to AMPS
    if (!string.IsNullOrEmpty(payload))
    {
        try
        {
            client.publish("UserMessage", @"{ ""data"" : """ + payload + @""" }");
        }
        catch (AMPSException exception)
        {
            Console.Error.WriteLine("An AMPS exception " +
            "occurred: {0}", exception);
        }
    }
}

Example 5.1: Catching an AMPSException

In this example, if an error occurs the program writes the error to Console.Error, and the publish() command fails. However, client is still usable for continued publishing and subscribing. When the error occurs, the exception is written to the console, which implicitly calls the exception’s ToString() method. As with most .NET exceptions, ToString() will convert the Exception into a string that includes a message, stacktrace and information on any “inner” exceptions (exception from outside of AMPS that caused AMPS to throw an exception).

AMPS exception types vary, based on the nature of the error that occurs. In your program, if you would like to handle certain kinds of errors differently than others, then you can catch the appropriate subclass of AMPSException to detect those specific errors and do something different.

public CommandId CreateNewSubscription(Client client)
{
    CommandId id = null;

    // attempt to retrieve a topic name (or regular
    // expression) from the user.
    string topicName;
    while (id == null)
    {
        topicName = AskUserForTopicName();
        try
        {
            Command command = new Command("subscribe").setTopic(topicName);

            // If an error occurs when setting up the subscription whether or not to try again
            // based on the subclass of AMPSException that is thrown. If a
            // BadRegexTopicException, this exception is thrown during subscription to indicate
            // that a bad regular expression was supplied, so we would like to give the user a
            // chance to correct.
            id = client.executeAsync(command, (x)=>HandleMessage(x));
        }
        catch(BadRegexTopicException ex)
        {
            DisplayError(string.Format("Error: bad topic name or regular " +
                                       "expression ’{0}’. The error was: {1}",
                                       topicName,
                                       ex.Message));
                // we’ll ask the user for another topic
        }

        // If an AMPS exception of a type other than BadRegexTopicException is thrown by
        // AMPS, it is caught here. In that case, the program emits a different error
        // message to the user.
        catch(AMPSException ex)
        {
            DisplayError(string.Format("Error: error setting up subscription " +
                                       "to topic ’{0}’. The error was: {1}",
                                       topicName,
                                       ex.Message));

            // At this point the code stops attempting to subscribe to the client by the return
            // null statement.
            return null; // give up
        }
    }
    return id;
}

Example 5.2: Catching AMPSException subclass

Exception Types

Each method in AMPS documents the kinds of exceptions that it throws. For reference, Table 1A contains a list of all of the exception types you may encounter while using AMPS, when they occur, and what they mean.

Exception Handling and Asynchronous Message Processing

When using asynchronous message processing, exceptions thrown form the message handler are silently absorbed by the AMPS C# client by default. The AMPS C# client allows you to register an exception listener to detect and respond to these exceptions. When an exception listener is registered, AMPS will call the exception listener with the exception. See Example 5.6 for details.

Disconnect Handling

Every distributed system will experience occasional disconnections between one or more nodes. The reliability of the overall system depends on an application’s ability to efficiently detect and recover from these disconnections. Using the AMPS C# client’s disconnect handling, you can build powerful applications that are resilient in the face of connection failures and spurious disconnects.

The HAClient class, included with the AMPS C# client, contains a disconnect handler and other features for building highly-available applications. 60East recommends using the HAClient for automatic reconnection wherever possible, as the HAClient disconnect handler has been carefully crafted to handle a wide variety of edge cases and potential failures. This section covers the use of a custom disconnect handler in the event that the behavior of the HAClient does not suit the needs of your application.

AMPS disconnect handling gives you the ultimate in control and flexibility regarding how to respond to disconnects. Your application gets to specify exactly what happens when an exception occurs by supplying a function to Client.setDisconnectHandler(), which is invoked whenever a disconnect occurs.

Example 5.3 shows the basics:

public class MyApp
{
    string _uri;
    public MyApp(string uri)
    {
        _uri = uri;
        Client client = new Client("sampleClient");

        // setDisconnectHandler() method is called to supply a function for use when AMPS
        // detects a disconnect. At any time, this function may be called by AMPS to
        // indicate that the client has disconnected from the server, and to allow your
        // application to choose what to do about it. The application continues on to
        // connect and subscribe to the orders topic.
        client.setDisconnectHandler(AttemptReconnection);
        client.connect(uri);
        client.logon();
        client.subscribe((m) => ShowMessage(m), "orders", 5000) ;
    }

    public void ShowMessage(Message m)
    {
        // display order data to the user
        ...
    }


    // Our disconnect handler’s implementation begins here. In this example, we simply
    // try to reconnect to the original server after a 5000 millisecond pause. Errors
    // are likely to occur here—therefore we must have disconnected for a reason—but
    // Client takes care of catching errors from our disconnect handler. If an error
    // occurs in our attempt to reconnect and an exception is thrown by connect(), then
    // Client will catch it and absorb it, passing it to the ExceptionListener if
    // registered. If the client is not connected by the time the disconnect handler ..
    public void AttemptReconnection(Client client)
    {
        // simple: just sleep and reconnect
        System.Threading.Thread.Sleep(5000);
        client.connect(_uri);
        client.logon();
    }
}

Example 5.3: Supplying a disconnect handler

By creating a more advanced disconnect handler, you can implement logic to make your application even more robust. For example, imagine you have a group of AMPS servers configured for high availability—you could implement fail-over by simply trying the next server in the list until one is found. Example 5.4 failover shows a brief example.

public class MyApp
{
    string[] _uris;
    int _currentUri = 0;

    public MyApp(string[] uris)
    {

        // Here our application is configured with an array of AMPS server URIs to choose
        // from, instead of a single URI. These will be used in the ConnectToNextUri()
        // method as explained below.
        _uris = uri;
        Client client = new Client(...);
        client.setDisconnectHandler(ConnectToNextUri(client));

        // ConnectToNextUri() is invoked by our disconnect handler, TestDisconnectHandler.
        // Since our client is currently disconnected, we manually invoke our disconnect
        // handler to initiate the first connection.
        ConnectToNextUri(client);
    }

    private void ConnectToNextUri(Client client)
    {

        // In our disconnect handler, we invoke ConnectToNextUri(), which loops around our
        // array of URIs attempting to connect to each one. In the invoke() method it
        // attempts to connect to the current URI, and if it is successful, returns
        // immediately. If the connection attempt fails, the exception handler for
        // AMPSException is invoked.
        // In the exception handler, we advance to the next URI, display a warning message,
        // and continue around the loop. This simplistic handler never gives up, but in a
        // typical implementation, you would likely stop attempting to reconnect at some point.
        while(true)
        {
            try
            {
                client.connect(_uris[_currentUri]);

                // At this point the registers a subscription to the server we have connected to.
                // It is important to note that, once a new server is connected, it the
                // responsibility of the application to re-establish any subscriptions placed
                // previously.
                // This behavior provides an important benefit to your application: one
                // reason for disconnect is due to a client’s inability to keep up with the rate of
                // message flow. In a more advanced disconnect handler, you could choose to not re-
                // establish subscriptions that are the cause of your application’s demise.
                client.subscribe((x)=>MySubscriptionHandler(x), "orders", 5000);
                return;
            }
            catch(AMPSException e)
            {
                _currentUri = (_currentUri + 1) % _uris.Length;
                ShowWarning("Connection failed: {0}. Failing over to {1}", e.ToString(), _uris[_currentUri]);
            }
        }
    }
}

Example 5.4: Simple Client Failover Implementation

Using a Heartbeat to Detect Disconnection

The AMPS client includes a heartbeat feature to help applications detect disconnection from the server within a predictiable amount of time. Without using a heartbeat, an application must rely on the operating system to notify the application when a disconnect occurs. For applications that are simply receiving messages, it can be impossible to tell whether a socket is disconnected or whether there are simply no incoming messages for the client.

When you set a heartbeat, the AMPS client sends a heartbeat message to the AMPS server at a regular interval, and waits a specified amount of time for the response. If the operating system reports an error on send, or if the server does not respond within the specified amount of time, the AMPS client considers the server to be disconnected.

The AMPS client processes heartbeat messages on the client receive thread, which is the thread used for asynchronous message processing. If your application uses asynchronous message processing and occupies the thread for longer than the heartbeat interval, the client may fail to respond to heartbeat messages in a timely manner and may be disconnected by the server.

Unexpected Messages

The AMPS C# client handles most incoming messages and takes appropriate action. Some messages are unexpected or occur only in very rare circumstances. The AMPS C# client provides a way for clients to process these messages. Rather than providing handlers for all of these unusual events, AMPS provides a single handler function for messages that can’t be handled during normal processing.

Your application registers this handler by setting the lastChanceMessageHandler for the client. This handler is called when the client receives a message that can’t be processed by any other handler. This is a rare event, and typically indicates an unexpected condition.

For example, if a client publishes a message that AMPS cannot parse, AMPS returns a failure acknowledgement. This is an unexpected event, so AMPS does not include an explicit handler for this event, and failure acknowledgements are received in the method registered as the lastChanceMessageHandler.

Your application is responsible for taking any corrective action needed. For example, if a message publication fails, your application can decide to republish the message, publish a compensating message, log the error, stop publication altogether, or any other action that is appropriate.

Unhandled Exceptions

In the AMPS C# client, exceptions can occur that are not thrown to the user. For example, when an exception occurs in the process of reading subscription data from the AMPS server, the exception occurs on a thread inside of AMPS. Consider the following example:

public class MyApp
{
    ...
    public static void WaitToBePoked(Client client)
    {
        client.subscribe(
            x=>Console.WriteLine("Hey! {0} poked you!", x.UserId),
            "pokes",
            string.Format("/Pokee LIKE '{0}-.*'", System.Environment.UserName),
            5000
        );
        Console.ReadKey();
    }
}

Example 5.5: Where do exceptions go?

In this example, we set up a simple subscription to wait for messages on the pokes topic, whose Pokee tag begins with our user name. When messages arrive, we print a message out to the console, but otherwise our application waits for a key to be pressed.

Inside of the AMPS client, the client creates a new thread of execution that reads data from the server, and invokes message handlers and disconnect handlers when those events occur. When exceptions occur inside this thread, however, there is no caller for them to be thrown to, and by default they are ignored.

In applications where it is important to deal with every issue that occurs in using AMPS, you can set an ExceptionHandler via Client.setExceptionHandler() that receives these otherwise-unhandled exceptions. Making the modifications shown in Example 5.6 to our previous example will allow those exceptions to be caught and handled. In this case we are simply printing those caught exceptions out to the console.

public class MyApp
{
    ...
    public static void WaitToBePoked(Client client)
    {
        client.setExceptionListener(
            ex=>Console.Error.WriteLine(ex));
        client.subscribe(
            x=>Console.WriteLine("Hey! {0} poked you!",
                x.UserId),
            "pokes",
            string.Format("/Pokee LIKE ’{0}-.*’",
                System.Environment.UserName),
            5000);
        Console.ReadKey();
    }
}

Example 5.6: Exception Listener

In this example we have added a call to client.setExceptionHandler(), registering a simple function that writes the text of the exception out to the console. Even though our application waits for a user to press a key, messages to the console will still be produced, both as incoming poke topics arrive, and as issues arise inside of AMPS.

Detecting Write Failures

The publish methods in the C# client deliver the message to be published to AMPS and then return immediately, without waiting for AMPS to return an acknowledgement. Likewise, the sowDelete methods request deletion of SOW messages, and return before AMPS processes the message and performs the deletion. This approach provides high performance for operations that are unlikely to fail in production. However, this means that the methods return before AMPS has processed the command, without the ability to return an error in the event that the command fails.

The AMPS C# client provides a FailedWriteHandler that is called when the client receives an acknowledgement that indicates a failure to persist data within AMPS. To use this functionality, you implement the FailedWriteHandler interface, construct an instance of your new class, and register that instance with the setFailedWriteHandler() function on the client. When an acknowledgement returns that indicates a failed write, AMPS calls the registered handler method with information from the acknowledgement message, supplemented with information from the client publish store if one is available. Your client can log this information, present an error to the user, or take whatever action is appropriate for the failure.

When no FailedWriteHandler is registered, acknowledgements that indicate errors in persisting data are treated as unexpected messages and routed to the LastChanceMessageHandler. In this case, AMPS provides only the acknowledgement message and does not provide the additional information from the client publish store.