9. Out-of-Focus Messages (OOF)

One of the more difficult problems in messaging is knowing when a record that previously matched a subscription has been updated so that the record no longer matches the subscription. AMPS solves this problem by providing an out-of-focus, or OOF, message to let subscribers know that a record they have previously received no longer matches the subscription. The OOF messages help subscribers easily maintain state and remove records that are no longer relevant.

OOF notification is optional. A subscriber must explicitly request that AMPS provide out-of-focus messages for a subscription.

When OOF notification has been requested, AMPS produces an oof message for any record that has previously been received by the subscription at the point at which:

• The record is deleted,
• The record expires,
• The record no longer matches the filter criteria, or
• The subscriber is no longer entitled to view the new state of the record

AMPS produces an oof message for each record that no longer matches the subscription. The oof message is sent as part of processing the update that caused the record to no longer match. Each oof message contains information the subscriber can use to identify the record that has gone out of focus and the reason that the record is now out of focus.

Because AMPS must maintain the current state of a record to know when to produce an oof message, these messages are only supported for SOW topics, conflated topics, and views. The oof option is not supported for bookmark replays.

When AMPS returns an OOF message, the data contained in the body of the message represents the updated state of the OOF message (except as described below). This will allow the client to make a determination as to how to handle the data, be it to remove the data from the client view or to change their query to broaden the filter thresholds. This enables a client to take a different action depending on why the message no longer matches. For example, an application may present a different icon for an order that moves to a status of completed than it would present for an order that moves to a status of cancelled.

When a delta_publish message causes the SOW record to go out of focus, AMPS returns the merged record.

When there is no updated message to send, AMPS sends the state of the record before the change that produced the oof. This can occur when the message had been deleted, when the message has expired, or when an update causes the client to no longer have permission to receive the record.

For a conflated view or a subscription that uses conflation, the data included in the oof message will be the last data that the subscriber received. When both an update to a message and a change that would cause the message to go out of focus happen in the same conflation interval, the subscriber receives an oof notification with the previously-received state of the message. Likewise, if a change that causes a message to go out of focus and a change that causes the message to come back into focus occur within the same conflation interval, the subscriber receives the state of the message at the end of the conflation interval, and does not receive an indication that the message had gone out of focus during the conflation interval.

Usage

Consider the following scenario where AMPS is configured with the following SOW key for the buyer topic:

<SOW>
    <Topic>
        <Name>buyer</Name>
        <MessageType>xml</MessageType>
        <Key>/buyer/id</Key>
    </Topic>
</SOW>

Example 9.1: Topic Configuration

When the following message is published, it is persisted in the SOW topic:

<buyer>
    <id>100</id>
    <loc>NY</loc>
</buyer>

Example 9.2: First Publish Message

A client issues a sow_and_subscribe request for the topic buyer with the filter /buyer/loc="NY" and the oof option set on the request. The client will be sent the messages as part of the SOW query result.

Subsequently, the following message is published to update the loc tag to LN:

<buyer>
    <id>100</id>
    <loc>LN</loc>
</buyer>

Example 9.3: Second Publish Message

The original message in the SOW cache is updated. The client does not receive the second publish message, because that message does not match the filter (/buyer/loc="NY"). This is problematic. The client has a message that is no longer in the SOW cache and that no longer matches the current state of the record. Because the oof option was set on the subscription, however, the AMPS engine sends an oof message to let these clients know that the message that they hold is no longer in the SOW cache. The following is an example of what’s returned:

<?xml version="1.0" encoding="iso-8859-1"?>
<SOAP-ENV:Envelope>
    <SOAP-ENV:Header>
        <Reason>match</Reason>
        <Tpc>buyer</Tpc>
        <Cmd>oof</Cmd>
        <MsgTyp>xml</MsgTyp>
        <SowKey>6387219447538349146</SowKey>
        <SubIds>SAMPS-1214725701_1</SubIds>
    </SOAP-ENV:Header>
    <SOAP-ENV:Body>
        <client>
            <id>100</id>
            <loc>LN</loc>
        </client>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Example 9.4: oof XML Example Message

An easy way to think about the situations where AMPS sends an OOF notification is to consider what would happen if the client re-issued the original sow request after the above message was published. The /client/loc="NY" expression no longer matches the message in the SOW cache and as a result, this message would not be returned.

Example

To help reinforce the concept of OOF messages, and how OOF messaging can be used in AMPS, consider a scenario where there is a GUI application whose requirement is to display all open orders of a client. There are several possible solutions to ensure that the GUI client data is constantly updated as information changes, some of which are examined below; however, the goal of this section is to build up a sow_and_subscribe message to demonstrate the power that OOF notifications add to AMPS.

Client-Side Filtering in a sow_and_subscribe Command

First, consider an approach that sends a sow_and_subscribe message on the topic orders using the filter /Client="Adam":

AMPS completes the sow portion of this call by sending all matching messages from the orders SOW topic. AMPS then places a subscription whereby all future messages that match the filter get sent to the subscribing GUI client.

Figure 9.1: sow_and_subscribe example

As the messages come in, the GUI client will be responsible for determining the state of the order. It does this by examining the State field and determining if the state is equal to “Open” or not, and then updating the GUI based on the information returned.

This approach puts the burden of work on the GUI and, in a high volume environment, has the potential to make the client GUI unresponsive due to the potential load that this filtering can place on a CPU. If a client GUI becomes unresponsive, AMPS will queue the messages to ensure that the client is given the opportunity to catch up. The specifics of how AMPS handles slow clients is covered in the section called Slow Clients.

AMPS Filtering in a sow_and_subscribe command

The next step is to add an additional ’AND’ clause to the filter. In this scenario we can let AMPS do the filtering work that was previously handled on the client. This is accomplished by modifying our original sow_and_subscribe to use the following filter:

/Client = "Adam" AND /State = "Open"

Similar to the above case, this sow_and_subscribe will first send all messages from the orders SOW topic that have a Client field matching “Adam” and a State field matching “Open.” Once all of the SOW topic messages have been sent to the client, the subscription will ensure that all future messages matching the filter will be sent to the client.

Figure 9.2: State Filter in a sow_and_subscribe

There is a less obvious issue with this approach to maintaining the client state. The problem with this solution is that, while it initially will yield all open orders for client “Adam”, this scenario is unable to stay in sync with the server. For example, when the order for Adam is filled, the State changes to State=Filled. This means that, inside AMPS, the order on the client will no longer match the initial filter criteria. The client will continue to display and maintain these out-of-sync records. Since the client is not subscribed to messages with a State of “Filled,” the GUI client would never be updated to reflect this change.

OOF Processing in a sow_and_subscribe command

The final solution is to implement the same sow_and_subscribe query which was used in the first scenario. This time, we use the filter requests only the State that we’re interested in, but we add the oof option to the command so the subscriber receives OOF messages.

/Client = "Adam" AND /State = "Open"

AMPS will respond immediately with the query results, in a similar manner to a sow_and_subscribe (Figure 9.3) command.

Figure 9.3: sow_and_subscribe with oof enabled

This approach provides the following advantage. For all future messages in which the same Open order is updated, such that its status is no longer Open, AMPS will send the client an OOF message specifying that the record which previously matched the filter criteria has fallen out of focus. AMPS will not send any further information about the message unless another incoming AMPS message causes that message to come back into focus.

In Figure 9.4 the Publisher publishes a message stating that Adam’s order for MSFT has been fulfilled. When AMPS processes this message, it will notify the GUI client with an OOF message that the original record no longer matches the filter criteria. The OOF message will include a Reason field with it in the message header, defining the reason for the message to lose focus. In this case the Reason field will state match since the record no longer matches the filter

Figure 9.4: OOF message

AMPS will also send OOF messages when a message is deleted or has expired from the SOW topic.

We see the power of the OOF message when a client application wants to have a local cache that is a subset of the SOW. This is best managed by first issuing a query filter sow_and_subscribe which populates the GUI, and enabling the oof option. AMPS informs our application when those records which originally matched no longer do, at which time the program can remove them.