3. Subscribe Publish and Subscribe

AMPS is a publish and subscribe message delivery system, which routes messages from publishers to subscribers. “Pub/Sub” systems, as they are often called, are a key part of most enterprise message buses, where publishers broadcast messages without necessarily knowing all of the subscribers that will receive them. This decoupling of the publishers from the subscribers allows maximum flexibility when adding new data sources or consumers.

Publish and Subscribe

Figure 3.1: Publish and Subscribe

AMPS can route messages from publishers to subscribers using a topic identifier and/or content within the message’s payload. For example, in Figure 3.1, there is a Publisher sending AMPS a message pertaining to the LN_ORDERS topic. The message being sent contains information on Ticker “IBM” with a Price of 125, both of these properties are contained within the message payload itself (i.e., the message content). AMPS routes the message to Subscriber 1 because it is subscribing to all messages on the LN_ORDERS topic. Similarly, AMPS routes the message to Subscriber 2 because it is subscribed to any messages having the Ticker equal to “IBM”. Subscriber 3 is looking for a different Ticker value and is not sent the message.

Topics

A topic is a string that is used to declare a subject of interest for purposes of routing messages between publishers and subscribers. Topic-based Publish and-Subscribe (e.g., Pub/Sub) is the simplest form of Pub/Sub filtering. All messages are published with a topic designation to the AMPS engine, and subscribers will receive messages for topics to which they have subscribed.

Topic Based Pub/Sub

Figure 3.2: Topic Based Pub/Sub

For example, in Figure 3.2, there are two publishers: Publisher 1 and Publisher 2 which publish to the topics LN_ORDERS and NY_ORDERS, respectively. Messages published to AMPS are filtered and routed to the subscribers of a respective topic. For example, Subscriber 1, which is subscribed to all messages for the LN_ORDERS topic will receive everything published by Publisher 1. Subscriber 2, which is subscribed to the regular expression topic ".*_ORDERS" will receive all orders published by Publisher 1 and 2.

Regular expression matching makes it easy to create topic paths in AMPS. Some messaging systems require a specific delimiter for paths. AMPS allows you the flexibility to use any delimiter. However, 60East recommends using characters that do not have significance in regular expressions, such as forward slashes. For example, rather than using northamerica.orders as a path, use northamerica/orders.

AMPS does not restrict the characters that can be present in a topic name. However, notice that topic names that contain regular expression characters (such as . or *) will be interpreted as regular expressions by default, which may cause unexpected behavior.

Topics that begin with /AMPS are reserved. The AMPS server publishes messages to topics that begin with /AMPS as described in (EVENT_TOPICS). Some versions of the AMPS client libraries may internally publish to /AMPS/devnull. Your applications should not publish to topics that begin with /AMPS, and publishes to those topics may fail.

Regular Expressions

With AMPS, a subscriber can use a regular expression to simultaneously subscribe to multiple topics that match the given pattern. This feature can be used to effectively subscribe to topics without knowing the topic names in advance. Note that the messages themselves have no notion of a topic pattern. The topic for a given message is unambiguously specified using a literal string. From the publisher’s point of view, it is publishing a message to a topic; it is never publishing to a topic pattern.

Subscription topics are interpreted as regular expressions if they include special regular expression characters. Otherwise, they must be an exact match. Some examples of regular expressions within topics are included in Table 3.1.

Topic Behavior
^trade$ matches only “trade”.
^client.* matches “client”, “clients”, “client001”, etc.
.*trade.* matches “NYSEtrades”, “ICEtrade”, etc.

Table 3.1: Topic Regular Expression Examples

For more information regarding the regular expression syntax supported within AMPS, please see the Regular Expression chapter in the AMPS User Guide.

AMPS can be configured to disallow regular expression topic matching for subscriptions. See the AMPS Configuration Guide for details.

Filtering Subscriptions By Content

One thing that differentiates AMPS from classic messaging systems is its ability to route messages based on message content. Instead of a publisher declaring metadata describing the message for downstream consumers, the publisher can simply publish the message content to AMPS and let AMPS examine the native message content to determine how best to deliver the message.

The ability to use content filters greatly reduces the problem of oversubscription that occurs when topics are the only facility for subscribing to message content. The topic space can be kept simple and content filters used to deliver only the desired messages. The topic space can reflect broad categories of messages and does not have to be polluted with metadata that is usually found in the content of the message. In addition, many of the advanced features of AMPS such as out-of-focus messaging, aggregation, views, and SOW topics rely on the ability to filter content.

Content-based messaging is somewhat analogous to database queries that include a WHERE clause. Topics can be considered tables into which rows are inserted (or updated). A subscription is similar to issuing a SELECT from the topic table with a WHERE clause to limit the rows which are returned. Topic-based messaging is analogous to a SELECT on a table with no limiting WHERE clause.

AMPS uses a combination of XPath-based identifiers and SQL-92 operators for content filtering. Some examples are shown below:

Example Filter for a JSON message:

(/Order/Instrument/Symbol == 'IBM') AND
(/Order/Px >= 90.00 AND /Order/Px < 91.00)

Example Filter for an XML Message:

(/FIXML/Order/Instrmt/@Sym == 'IBM') AND
(/FIXML/Order/@Px >= 90.00 AND /FIXML/Order/@Px < 91.0)

Example Filter for a FIX Message:

/35 < 10 AND /34 == /9

For more information about how content is handled within AMPS and the syntax of AMPS filters, check out AMPS Expressions in the AMPS User Guide.

Tip

Unlike some other messaging systems, AMPS lets you use a relatively small set of topics to categorize messages at a high level and use content filters to retrieve specific data published to those topics. Examples of good, broad topic choices:

trades, positions, MarketData, Europe, alerts

This approach makes it easier to administer AMPS, easier for publishers to decide which topics to publish to, and easier for subscribers to be sure that they’ve subscribed to all relevant topics.