9. State-of-the-World (SOW) Features

The SOW section of the configuration file holds the configuration for the AMPS State of the World.

Element Description
Topic Specifies that AMPS will record distinct messages for this topic in the State of the World. SOW topic definitions are used directly as a last-value cache, and are required for many of the advanced messaging features in AMPS such as out-of-focus notifications and delta messaging. SOW topic definitions can also be used as the underlying topics for views, aggregates, and conflated topics. SOW Topic configuration is described in SOW/Topic.
Queue Defines a message queue. Rather than delivering each message to all matching subscriptions, message queues provide features to help ensure that each message is delivered to and processed by a single subscriber. See the User Guide for a full description of the functionality of message queues. Queue configuration is described in SOW/Queue.
View Defines a View over one or more SOW topics, conflated topics, or other views. A view can perform aggregation, can JOIN multiple topics together. A view can be based on a SOW topic of one message type and project results of a different message type. View configuration is described in SOW/View.
ConflatedTopic Defines a copy of a SOW topic or view that receives current value updates at a specified interval, conflating any changes to values that occur between the scheduled updates. ConflatedTopic configuration is described in SOW/ConflatedTopic.

SOW section configuration elements

The elements within the SOW section are described in detail in the following sections.

SOW/Topic

Creating a Topic element in the SOW configuration section declares that AMPS will maintain state of the world for that topic. When a topic is recorded in the State of the World (SOW), AMPS maintains the most recent value of each distinct message in that topic. In this respect, the SOW for a topic is similar to a table in a relational database.

Notice that AMPS does not require topics to be predeclared. Creating this configuration item is declaring that the State of the World will be maintained for this topic: the configuration does not define a topic. Creating this configuration item is not required to publish messages to the topic.

There are several sets of options required when a topic is added to the State of the World. General options define the topic name and the overall behavior of the State of the World for this topic. Key specification options define how AMPS identifies unique messages within the topic (much like declaring a primary key for a table in a relational database). Growth specification options define how AMPS grows storage for the State of the World for this topic as messages are added.

Example (CHAPTER_NUMBER).(CONFIG_REF_ROOT_SOW_TOPIC) contains a listing of the parameters for a Topic section in the SOW section of an AMPS configuration file. AMPS accepts TopicDefinition as a synonym for Topic, since this was the element name used in versions before 5.0.

This section presents brief descriptions of the options for recording the State of the World for a topic. For full information on the State of the World, see State of the World in the AMPS User Guide.

Element Description
MessageType

Type of messages to be stored. To use AMPS generated SOW keys, the message type specified must support content filtering so that AMPS can determine the SOW key for the message. All of the default message types, except binary, support content filtering. Since the binary message type does not support content filtering, that type can only be used for a SOW when publishers use explicit keys.

See the Message Types chapter in the AMPS User Guide for a discussion of the message types that AMPS loads by default. Some message types (such as Google Protocol Buffers) require additional configuration, and must be configured before using the message type in a SOW topic.
Name

The name of the SOW topic - all unique messages on this topic will be stored in a topic-specific SOW database.

Every SOW requires a method of determining which messages are unique. Several methods are provided within AMPS. See the AMPS User Guide for a discussion on SOW keys, and Key specification options for relevant configuration items.

If no Name is provided, AMPS accepts Topic as a synonym for Name to provide compatibility with versions of AMPS previous to 5.0.

SOW/Topic General Options

A SOW topic must also consider the following options:

Option Description
Storage and Recovery

A SOW topic must either specify the FileName to be used for persisting the topic, or declare that the topic will be in-memory only by setting the Durability to transient.

See Storage and Recovery Definition for details.
Record Identity A SOW topic must define how to determine whether a given message is a distinct record in the topic. Typically, record identity is based on the content of one or more fields in the message content. See Key specification options for details and options.
Memory and File Growth If the SOW topic will contain a large number or records, or if an individual record will exceed the default allocation size, the topic must define the allocation size. See Growth specification options for details.

A SOW topic may also provide the following options (notice, though, that there are restrictions on how some of these options are used with other options.

Option Description
Indexing

A SOW topic can declare additional hash indexes or direct AMPS to create memo indexes before a field is used by an application to improve performance.

See Options for indexing for more details.
Historical Query

A SOW topic can keep message state to allow “point in time” historical queries of current values. Notice that this option is not required for message-by-message replay; recording the topic in a transaction log provides full replay.

See Options for persistence and recovery for more details.
Message Enrichment

A SOW topic can modify the content of messages as they are received from an application.

See Options for enrichment for more details.
Regular Expression Topic Name Matching

A SOW topic can be declared as a regular expression topic, where multiple topic names use the same definition and are stored in the same physical file. This can be useful when converting a system from topic-based routing.

See Options for regular expression topics for more details.

Storage and Recovery Definition

These options specify the storage for the topic and recovery behavior for the topic if the topic is not durably persisted or if the recovery file is not present.

Options for storage and recovery
Element Description
FileName

The file where the State of the World (SOW) data will be stored.

This element is required for SOW topics with a Durability of persistent (the default) because those topics are persisted to the filesystem. This is not required for SOW topics with a Durability of transient.

This element should contain the path and the file name. The path can be either an absolute path or a path relative to the current working directory of the AMPS process.

Within this element, the escape %n will be replaced with the name and message type of the topic. This can be a convenient way to avoid having to retype the topic name in this element.

Two different topics must not share the same file. Two instances of AMPS must not share the same file.
Durability

Defines the data durability of a SOW topic. SOW databases listed as persistent are stored to the file system, and retain their data across instance restarts. Those listed as transient are not persisted to the file system, and are reset each time the AMPS instance restarts.

Notice that when the durability is transient, and the topic is recorded in the transaction log, each time AMPS starts, AMPS will recover the state of the topic from the transaction log. The recovery begins at the RecoveryPoint specified for the topic, which defaults to epoch, the beginning of the transaction log. (For persistent topics, AMPS recovers from the last message written to the SOW topic, or from the RecoveryPoint if the SOW file is removed.)

Default: persistent

Valid values: persistent or transient

When a value of persistent is specified, the FileName element must be present.

Synonyms: Duration is also accepted for this parameter for backward compatibility with configuration prior to 4.0.0.1
RecoveryPoint

For SOW topics that are covered by the transaction log, the point from which to recover the SOW if the SOW file is removed, or if the SOW topic has transient duration.

This configuration item allows two values:

  • epoch recovers the SOW from the beginning of the transaction log
  • now recovers the SOW from the current point in the transaction log

Defaults to epoch.
Expiration

Time for how long a record should live in the SOW database for this topic. The expiration time is stored on each message, so changing the expiration time in the configuration file will not affect the expiration of messages currently in the SOW.

AMPS accepts interval values for the Expiration, using the interval format described in the AMPS Configuration Guide section on units, or one of the following special values:

  • A value of disabled specifies that AMPS will not process SOW expiration for this topic, regardless of any expiration value set on the message. In this case, AMPS saves the expiration for the message, but does not process it. The value must be set to disabled (the default) if History is enabled for this topic.
  • A value of enabled specifies that AMPS will process SOW expiration for this topic, with no expiration set by default. Instead, AMPS uses the value set on the individual messages (with no expiration set for messages that do not contain an expiration value).

Expiration must be disabled if History is enabled.

Default: disabled (never expire)

SOW/Topic Persistence and Recovery Options

Key Specification Options

Each SOW topic must define how AMPS will determine which messages are unique. An application can either have AMPS determine the key by specifying one or more Key fields, provide a SOW key with the publish command each time a message is published to AMPS. AMPS also provides the ability to provide a custom SowKey generator with a plugin module.

See the Sow Key Generation section in the AMPS User Guide for for a full discussion. The following table lists the available configuration items for specifying how AMPS determines the SowKey for a message:

Key specification options
Element Description
Key

Specifies an XPath-based indentifier within each message that AMPS will use to generate a SOW key, which determines whether a message is unique. This element can be specified multiple times to create a composite key from the combined value of the specified Key elements.

When one or more Key elements is specified for the SOW, AMPS generates the SOW key for each message. When no Key fields are specified and no KeyGenerator is specified, publishers must explicitly provide the SOW key for each message when the message is published.

60East recommends configuring a Key and having AMPS generate the SOW key for a message unless your application has specific needs that make this impractical.

AMPS automatically creates a hash index for the set of fields specified in the Key elements.

There is no default for this element.
KeyDomain

The seed value for SowKeys used within the topic when AMPS generates the SOW key. The default is the topic name, but it can be changed to a string value to unify SowKey values between different topics.

For example, if your application has a ShippingAddress SOW and a CreditRating SOW that both use /customerID as the SOW key, you can use a KeyDomain to ensure that the generated SowKey for a given /customerId is identical for both SOW topics. This does not affect how AMPS processes the SOW topics, but can make correlating information from different SOW topics easier in your application.

This option can only be specified when one or more Key fields are specified. When a SOW key generator module is used, or the publisher must send a SOW key, this option is not valid.

Default: the name of the SOW topic.
KeyGenerator

Specifies the SOW key generator module to use for this topic. When this configuration element is present, AMPS calls the specified module to generate a SOW key for each incoming message.

Default: no SOW key generator module. When there is no SOW key generator module specified, AMPS uses the specified Key fields if the Key fields are provided. If no generator is specified and no Key fields are specified, AMPS requires publishers to set a SOW key on each message published.

A KeyGenerator element contains the following elements:

Element Description
Module

Required within a KeyGenerator element.

The name of the module. This module must be loaded elsewhere in the configuration file.
Options Contains one or more XML elements. These elements are provided to the key generator module as options. The options provided depend on the key generator. The creator of the key generator module must document the options for that module.

SOW/Topic Key Specification Options

Memory and File Growth Options

The SOW topic configuration also specifies how the SOW file is allowed to grow. See the AMPS User Guide for details on SOW growth. The configuration options for controlling how the file is allocated and how the file grows are listed below:

Allocation and growth options
Element Description
SlabSize

The size of each allocation for the SOW file, as a number of bytes. When AMPS needs more space for the SOW, it requests this amount of space from the operating system. This effectively sets the maximum message size that AMPS guarantees can be stored in the SOW. This size includes headers set by AMPS on the message.

60East recommends setting this value only if you will be storing messages larger than the default SlabSize or if performance or capacity testing indicates a need to tune SOW performance. If you plan to store messages larger than the default setting, 60East recommends a starting value of several times the maximum message size. For example, if your maximum message size is 2MB, a good starting point for SlabSize would be 8MB.

If it becomes necessary to tune the SlabSize, see the Best Practices section of the AMPS User Guide and Capacity Planning section of the AMPS User Guide for a full discussion about tuning the SlabSize.

Default: 5MB
InitialSlabCount

The number of SOW slabs that AMPS will allocate on startup.

Default: 1

Maximum: 1024

DEPRECATED:

RecordSize

This parameter is deprecated beginning in AMPS 5.0. Use SlabSize parameter instead. Size (in bytes) of a SOW record for this topic.

Default: 512

DEPRECATED:

InitialSize

This parameter is deprecated beginning in AMPS 5.0. Use the InitialSlabCount parameter instead. Initial size (in records) of the SOW database file for this topic.

Default: 2048

DEPRECATED:

IncrementSize

This parameter is deprecated beginning in AMPS 5.0. Use the SlabSize parameter instead. Number of records to expand the SOW database (for this topic) by when more space is required.

Default: 1000

SOW/Topic Growth Specification Options

Indexing Options

AMPS automatically creates a memo index for a field within a SOW topic when that field is used (for example, is used in a view or is queried by an application).

In addition, AMPS automatically creates a hash index (the primary key index) for the combination of fields used to define the SOW key. The options in this section allow you to manage index creation for a SOW topic.

Options for indexing
Element Description
HashIndex

AMPS provides the ability to do fast lookup for SOW records based on specific fields.

When one or more HashIndex elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup, and are kept up to date as records are added, removed, and updated.

The HashIndex element contains a Key element for each field in the hash index.

AMPS uses a hash index when a query uses a exact matching for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions.

AMPS automatically creates a hash index for the set of fields specified in the set of Key fields for the SOW, if those fields are specified.
Index

AMPS supports the ability to precreate memo indexes for specific fields using the Index configuration option.

When one or more Index elements are provided, AMPS creates memo indexes for any field specified in an Index element on startup, before a query that uses that field runs. Otherwise, AMPS indexes each field the first time a query uses the field. Adding one or more Index configurations to a SOW/Topic can improve retrieval performance the first time a query that contains the indexed fields runs for large SOW topics.
ExpectedKeyCountHint

For SOW topics that will contain a large number of distinct keys, providing an expected key count allows AMPS to pre-size the data structure that holds the key. This can provide a performance improvement for publishers by avoiding cases where AMPS has to resize the data structure.

There is no default for this value. When no value is provided, AMPS does not pre-size data structures for the SOW.

When provided, the value of this option should be the power of 2 closest to the maximum number of keys the topic is expected to hold. For example, a topic that is expected to hold 500,000,000 distinct records could set a value of 536870912 (2 ^ 29).

SOW/Topic Indexing Options

Historical Query

A SOW topic can, optionally, maintain the ability to query current values at a specific point in time. To specify this, include a History element in the topic configuration. The History element must include the following options:

History configuration options
Element Description
History/Window

Required when a History element is present.

For a historical SOW, the length of time to store history. For example, when the value is 1w, AMPS will store one week of history for this SOW.

Used within the History element.

Default: By default, AMPS does not expire historical SOW data.
History/Granularity

Required when a History element is present.

For a historical SOW, the granularity of the history to store. For many applications, it is not necessary for AMPS to store all of the updates to the SOW. This parameter sets the resolution at which AMPS will save the state of a message. A value of 0s or equivalent specifies that AMPS will preserve every update within the Window.

For example, when you set a granularity of 1m, AMPS will save the state of the message no more frequently than once per minute, even when the state of the message is updated several times a minute.

Used within the History element.

SOW/Topic/History Options

Message Enrichment

AMPS can modify a message as it is published to the SOW topic from an application. See State of the World Message Enrichment in the AMPS User Guide for details.

Message enrichment options
Element Description
Preprocessing

When present, specifies the message enrichment to be performed before AMPS determines the SOW key for the message.

The Preprocessing element must contain one or more Field elements that specify the enrichment to perform.
Enrichment

When present, specifies the message enrichment to be performed after AMPS determines the SOW key for the message.

The Enrichment element must contain one or more Field elements that specify the enrichment to perform.

SOW/Topic Preprocessing/Enrichment Specification Options

Regular Expression Topics

AMPS can store the last values for a set of topics that match the same naming pattern and use the same configuration in the same physical file. See State of the World Regular Expression Topics in the AMPS User Guide for details.

Regular expression topic options
Element Description
Pattern

When present, declares this topic to be a regular expression topic and specifies the pattern to use to determine if a topic is part of the regular expression topic.

Regular expression topics have the benefits and limitations described in State of the World Regular Expression Topics in the AMPS User Guide.

There is no default for this element.

When this element is present, the Topic cannot specify History, Preprocessing, or Enrichment.

SOW/Topic Regular Expression Topic Specification Options

An example of a SOW configuration looks like the following:

<SOW>
    <!-- Simple SOW topic definition -->
    <Topic>
        <Name>orders</Name>
        <Key>/orderId</Key>
        <MessageType>nvfix</MessageType>
        <FileName>./sow/%n.sow</FileName>
    </Topic>

    <!-- SOW with hash indexes -->
    <Topic>
        <Name>customers</Name>
        <Key>/customerId</Key>
        <MessageType>json</MessageType>
        <FileName>./sow/%n.sow</FileName>

        <!-- Index for exact match on customerName -->
        <HashIndex>
            <Key>/customerName</Key>
        </HashIndex>

        <!-- Index for exact match on company -->
        <HashIndex>
             <Key>/company</Key>
        </HashIndex>

        <!-- Index for exact match on *both*
            customerName and company.
         -->
        <HashIndex>
            <Key>/customerName</Key>
            <Key>/company</Key>
        </HashIndex>

    </Topic>

    <!-- Historical SOW -->
    <Topic>
        <Name>catalog</Name>
        <Key>/sku</Key>
        <MessageType>json</MessageType>
        <FileName>./sow/%n.sow</FileName>
        <History>
            <Window>7d</Window>
            <Granularity>15m</Granularity>
        </History>
    </Topic>

    <!-- SOW with enrichment

        Add a /fullName field constructed
        from /firstName and /lastName. -->
    <Topic>
        <Name>sales-reps</Name>
        <Key>/employeeId</Key>
        <MessageType>bflat</MessageType>
        <FileName>./sow/%n.sow</FileName>
        <Key>/id</Key>
        <Enrichment>
            <Field>CONCAT(/firstName, " ", /lastName) AS /fullName</Field>
        </Enrichment>
    </Topic>
</SOW>

SOW Topic Configuration

SOW/Queue, SOW/LocalQueue, and SOW/GroupLocalQueue

This section lists configuration parameters for queues.

The Queue tag, the LocalQueue tag and the GroupLocalQueue tag are used to configure message queues.

When an AMPS queue is defined with the Queue tag, the queue will be a distributed queue. To make a queue that is limited to the local instance, use the LocalQueue configuration element. To make a queue that is restricted to a subset of instances, in a set of replicated instances (typically all in the same group), use the GroupLocalQueue tag.

AMPS accepts QueueDefinition as a synonym for Queue.

The following configuration items apply to queues regardless of whether they are fully distributed, local, or distributed within a replication group:

Element Description
Name

The name of the queue topic. This name is the name that consumers subscribe to.

If no Name is provided, AMPS accepts Topic as a synonym for Name in the Queue definition.
MessageType
 The message type of the queue.
UnderlyingTopic

A topic name or regular expression for the topic that contains the messages to capture in the queue. These topics must be recorded in a transaction log, and all must be of the same message type as the queue.

If an UnderlyingTopic is not provided, the UnderlyingTopic defaults to the Name of the queue.
DefaultPublishTarget

The topic to publish to when an application publishes a message to the queue. For simplicity, AMPS allows applications to publish messages to the queue, and for those messages to be routed to one of the underlying topics.

The DefaultPublishTarget must be one of the topics included in the queue.

This element is required if the UnderlyingTopic contains regular expression characters. Otherwise, the UnderlyingTopic is a single topic and this element is optional and defaults to the UnderlyingTopic.
LeasePeriod

The amount of time that a subscriber has ownership of the message before the message is returned to the queue. For at-least-once delivery semantics, the consumer must process and acknowledge the message within this lease period, or the message may be provided to another subscriber.

The LeasePeriod is measured from the time that AMPS sends the message to the subscriber. Set the LeasePeriod to account for round trip network latency as well as the expected processing time for the subscribers.

Default: infinite (no expiration)
Semantics

The delivery semantics to use for this queue. Regardless of the delivery semantics, AMPS queues deliver a given message to a single subscriber at a time. When a subscriber fails to process a message (that is, the connection to the subscriber closes before the message is acknowledged, or the lease expires before the message is acknowledged), the semantics specify how the failure is handled.

  • at-least-once With these semantics, you can guarantee that a message has been processed by at least one subscriber, as described in the introduction to Queues in the AMPS User Guide. With this value, a subscriber must explicitly remove the message from the queue once the message is processed. If the subscriber connection closes before acknowledging the message, the subscriber returns the message to the queue or the lease expires, the AMPS server can deliver the message again to allow another attempt to process the message.
  • at-most-once With these semantics, AMPS removes the message from the queue immediately when AMPS sends the message. This allows you to guarantee that no more than one subscriber will process the message, even if the subscriber that receives the message fails without acknowledging the message.

Default: at-least-once
MaxBacklog

The maximum number of outstanding, unacknowledged messages in the queue at any one time. This parameter allows you to set limits on the number of pending messages from the queue overall. When the queue reaches the MaxBacklog, no incoming messages are delivered from the queue until a message is removed from the queue (either by expiring, or being acknowledged by a client). This parameter allows you to avoid overwhelming clients during periods of heavy activity.

Notice that this does not set a limit of any sort on the capacity of the queue. This parameter allows you to limit the number of messages that the queue will make available to subscribers at a given time, but does not restrict the capacity of the queue to track messages.

This backlog number is applied per instance of the queue. That is, each instance of AMPS that hosts an instance of a replicated queue will deliver messages up to the MaxBacklog if messages are available on that instance.

Default: infinite
MaxPerSubscriptionBacklog

The maximum number of outstanding, unacknowledged messages in the queue for an individual subscription. This parameter allows you to avoid overwhelming a single subscriber during a period of heavy activity.

Subscribers can declare the maximum number of messages that the subscription is prepared to lease at a given time. This maximum defaults to 1 when there is no maximum explicitly specified for a subscription. AMPS will lease the number specified in the subscription or the maximum set for the queue, whichever is lower.

Notice that this does not set a limit of any sort on the capacity of the queue. This parameter allows you to limit the number of messages that the queue will make available for a single subscription at a given time, but does not restrict the capacity of the queue to track messages.

Default: 1
TargetQueueDepth

The target number of messages to keep active state on.

Providing this option limits the amount of state that AMPS keeps for unacknowledged messages. In most circumstances, the limit will be the number of messages specified in this parameter. If this instance receives a transfer request for a message that is in the transaction log, but outside the currently active set of messages, AMPS will process the queue until it reaches the message to be transferred.

Messages that are present in the transaction log, but for which AMPS does not currently have active state, are not deliverable from the queue topic, cannot be queried, cannot be deleted using a sow_delete by filter, and so on. They are not yet in the set of messages that AMPS considers to be an active part of the queue.

This option is not set by default. When this option is not set, all messages in the queue are active.

This option is typically unnecessary if the FileBackedMetadata option is enabled. In that case, AMPS will process the full state of the queue, but state that is not currently in use can be paged out to the file.

This option cannot be set if Priority or BarrierExpression is specified.

When this parameter is set, it should be set to a number that accounts for several seconds of traffic to the queue, considering replication and acknowledgment speeds. For example, if a queue is typically processed at a rate of 2500 messages per second within minimal replication delays, a reasonable minimum target value might be 12500 or 15000.

Notice that, if subscribers use content filters to selectively process the queue, if no currently active messages match the subscription, no messages will be delivered (even if there are matching messages in the queue that are not currently active).

Default: unset (no limit) Minimum: 1000
Expiration

The length of time a message can remain in the queue before AMPS considers the message undeliverable.

Messages may expire while a subscriber has a lease on the message. AMPS does not send an additional notification in this case.

A publisher can override this setting by providing an expiration value on an incoming message. If an individual message in the transaction log has an expiration value, AMPS will use that expiration value for the message rather than the default for the queue.

Default: infinite
Filter

An AMPS Filter that is applied to the UnderlyingTopic. When a Filter is specified, only messages matching the Filter appear in the queue.

By default, there is no filter and all messages from the UnderlyingTopic are presented in the queue.
RecoveryPoint

This option allows you to specify the point at which AMPS begins reviewing the transaction log to recover the state of the queue when AMPS restarts and there is no existing information about the state of the queue. By default, AMPS reviews the full log to determine the contents and state of the queue.

If AMPS has a record of the last point in the transaction log at which all previous messages are acknowledged (stored in the queues.ack file), and the recovery point specified in the configuration file has not changed, AMPS will recover from that point rather than the configured RecoveryPoint, since using the cached information will reduce recovery time.

The RecoveryPoint can be one of the following:

  • epoch - Recovery begins at the beginning of the transaction log
  • now - Recovery begins at the time AMPS starts queue recovery, so only new messages are added to the queue
  • creation - Recovery begins at the time the queue was created
  • AMPS bookmark - When an AMPS bookmark is provided, AMPS starts recovery at the specified bookmark.
  • ISO-8601 timestamp - When a timestamp is provided, AMPS starts recovery at the specified timestamp. The timestamp must be provided in the format AMPS uses for timestamp bookmarks.

Default: epoch
FairnessModel

AMPS provides different methods to distribute messages across active subscriptions:

  • fast - AMPS delivers to the first subscription found that can process the message
  • round-robin - AMPS distributes to the next subscription found that can process the message
  • proportional - AMPS delivers to the subscription with the lowest ratio of active messages to available backlog

Each instance of AMPS independently manages the fairness model for subscriptions on that instance. Fairness model information is not replicated across instances.

Default: proportional for at-least-once queues, round-robin (the only valid delivery model) for at-most-once queues
Leasing

Ownership model for leased messages. AMPS supports the following models:

  • strict - AMPS allows a client to acknowledge (sow_delete) only messages that are leased to the client or currently unleased. If a client acknowledges a message leased to another client, there is no effect.
  • sublet - AMPS allows any client to acknowledge any message, regardless of whether another client has a lease on the message.

Default: sublet
MaxDeliveries

Specifies an upper bound to the number of times AMPS may deliver a queue message before automatically expiring it. For example, if AMPS delivers a message twice and MaxDeliveries is set to 2, the message will be expired if the subscriber disconnects or unsubscribes before acknowledging it.

This counter is reset if the server restarts and the counter is not replicated to other instances.

Default: no maximum (0).
MaxCancels

Specifies a limit to the number of times a subscriber may cancel a lease on a message before it is expired. For example, if a message is canceled for the second time and MaxCancels is set to 2, AMPS automatically expires the message instead of returning it to the message queue.

This counter is reset if the server restarts and the counter is not replicated to other instances.

Default: no maximum (0).
Priority

Specifies the order in which messages will be distributed from the queue. When present, this element constructs a value that specifies the priority for messages in the queue. Higher priority messages are delivered first, regardless of the order in which messages have been published to the queue.

The contents of the element can be either the name of a field in the message or an AMPS expression. Either way, the result is treated as an unsigned long value.

For example, to order message delivery based on the /priority field of the message, specify the following element:

<Priority>/priority</Priority>

To order message delivery based on the product of the /price field and the /quantity field in the message, specify the following element:

<Priority>/price * /qty</Priority>

Default: there is no default for this value. When not provided, the delivery order is the order in which messages were processed by this instance of AMPS.

This option cannot be specified on a queue if BarrierExpression or TargetQueueDepth is specified.
BarrierExpression

Specifies the filter used to identify a barrier message (synchronization point) for this queue. When a message matches this expression, it will be delivered to all current subscribers on the queue when every previous message in the queue has been acknowledged. This provides a simple, reliable way to synchronize workers.

<BarrierExpression>/isSyncPoint IS NOT NULL</BarrierExpression>

The BarrierExpression filter is evaluated when AMPS adds the message to the in-memory state of the queue, and is not re-evaluated (unless the in-memory state of the queue is rebuilt after an instance restart).

Default: there is no default for this value. When not provided, no messages are considered to be barrier messages

This option cannot be specified on a queue if Priority or TargetQueueDepth is specified.
DeferredAckExpiration

Specifies the amount of time for AMPS to retain information about an acknowledgment (sow_delete) message received when the corresponding message is not in the queue. This can occur during failover, when messages are received over replication, or in cases where an application that uses a publish store has been offline for an extended period of time.

This element is configured as an interval, for example, 15m or 2h.

The default value is generally recommended. However, in cases where an instance may receive large volumes of acknowledgments for messages that are not currently in the queue, and are not expected to arrive, setting this to a lower value may somewhat reduce the memory required for managing these acknowledgments.

Default: 1d
FileBackedMetadata

Specifies whether AMPS should persist metadata about the queue in the journal directory. This could reduce the active memory footprint of an AMPS instance in cases where a queue has a large number of messages, but it is not being actively consumed. In cases where the queue has a large number of unacknowledged messages when AMPS is restarted, this may also improve recovery time.

This option can be unset or set to a value of enabled.

Default: unset, which indicates that queue metadata will not be persisted

Queue configuration elements

When a queue is defined as a GroupLocalQueue, the following configuration item must be provided:

Element Description
InitialOwner

For a group local queue, provides the instance Name of the instance that will own the message when the message is first published.

This configuration element is required for a GroupLocalQueue. It is not supported for fully distributed queues or for local only queues (that is, a queue defined with the Queue or LocalQueue element).

All instances in a replicated system that define this queue must define the same InitialOwner.

Notice that AMPS does not validate that the InitialOwner exists and is reachable through replication from the current instance. When configuring a GroupLocalQueue, take care to validate the name and ensure that a replication path exists to and from that instance.

This element is required. There is no default for this element.

When a queue is defined as a GroupLocalQueue, the following optional configuration element may be provided.

Element Description
GroupLocalQueueDomain

For a group local queue, provides a way to allow a queue that is hosted in instances that are in different replication groups to be identified as the same queue and function as a single distributed queue.

By default, a GroupLocalQueue is identified using the queue Name and the Group of the AMPS instance that hosts the queue. Queues with the same identity are considered to be instances of the same distributed queue.

When this element is defined, the GroupLocalQueue is identified using the queue Name and the GroupLocalQueueDomain rather than the Group of the instance that hosts the queue.

This element was introduced in version 5.3.4. Previous versions do not consider the Group of the instance or the GroupLocalQueueDomain: instances of the queue with the same Name are considered to be the same distributed queue in those versions regardless of the Group that hosts the queue.

Additional configuration elements for GroupLocalQueue

The following configuration snippet shows one way to configure a queue.

<!-- Notice that the topics to use for the queue (ORDERS_.*) must be
    recorded in a transaction log. -->
<SOW>
    <Queue>
        <Name>MQ</Name>
        <MessageType>json</MessageType>
        <UnderlyingTopic>ORDERS_.*</UnderlyingTopic>
        <DefaultPublishTarget>ORDERS_DIRECT</DefaultPublishTarget>
        <LeasePeriod>60s</LeasePeriod>
        <Expiration>1d</Expiration>
        <MaxBacklog>3</MaxBacklog>
    </Queue>
  </SOW>

The following configuration snippet shows the configuration for a GroupLocalQueue. Any instance of the queue in the group named CONSUME_INSTANCES or that uses the GroupLocalQueueDomain tag of CONSUME_INSTANCES will be treated as an instance of the same queue.

<SOW>
    <GroupLocalQueue>
        <Name>GroupQueue</Name>
        <MessageType>json</MessageType>
        <UnderlyingTopic>ORDERS_.*</UnderlyingTopic>
        <DefaultPublishTarget>ORDERS_DIRECT</DefaultPublishTarget>
        <GroupLocalQueueDomain>CONSUME_INSTANCES</GroupLocalQueueDomain>
    </GroupLocalQueue>
</SOW>

SOW/View

Table 9.6 contains a listing of the parameters for a View section in the SOW section of an AMPS configuration file. For backward compatibility, AMPS accepts ViewDefinition as a synonym for View.

Element Definition
MessageType

The message type of the view. This does not need to be the same type as any of the topics in the aggregation, but does need to be a message type that supports views.

The message type must be one of the message types configured for the instance. AMPS includes fix, xml, nvfix, json, bson and bflat message types with full support for views. You can also use any custom message type defined for the configuration file, provided that the message type supports views.

Notice that the binary message type does not specify a fixed format for the message contents, so that message type cannot be used in a view.

Other message types provided with AMPS may have limitations in their support for views. see the Message Types chapter in the AMPS User Guide for a discussion of the limitations.
Name

Defines the topic name for this view.

If no Name is provided, AMPS accepts Topic as a synonym for Name to provide compatibility with versions of AMPS previous to 5.0.
UnderlyingTopic
 Defines the SOW topic or topics on which this view is based. This element can contain a single topic name, or any number of Join elements.
Projection/Field
 Defines what the view will contain. This element can be specified multiple times to compose a complex view. Complex expressions that use aggregation functions and conditional branching can also be used.
Grouping/Field
 Defines how the records in the underlying topic will be grouped. This is analogous to a SQL GROUP BY clause.
KeyDomain The seed value for SowKeys used within this topic. The default is the topic name, but it can be changed to a string value to unify SowKey values between different topics.
Join

Within an UnderlyingTopic, each Join specifies two topics to join together to create the view, as well as the relationship between those topics.

An UnderlyingTopic can have any number of Join specifications. For more information on Join specifications, see the Join expression in the AMPS User Guide.
Conflation

Defines whether AMPS will attempt to conflate updates to the view.

This item accepts two values:

  • none - No conflation. AMPS fully processes every update to the view, in publication order.
  • inline - Conflation is active. AMPS conflates updates to the same underlying record where possible. See Inline Conflation in the AMPS User Guide for details.

Default: none
Filter

Defines a filter for the view. When provided, only messages from the underlying topic that match this filter will be included in the view.

This option is only valid when the view uses a single UnderlyingTopic. When the view contains a Join specification, this option may not be used.

Default: No filter, which includes all messages from the UnderlyingTopic.
HashIndex

AMPS provides the ability to do fast query lookup for records in a view based on specific fields.

When one or more HashIndex elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup, and are kept up to date as records are added, removed, and updated.

The HashIndex element contains a Key element for each field in the hash index.

AMPS uses a hash index when a query uses an exact string match for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions, or for numeric comparisions.

AMPS automatically creates a hash index for the fields specified in the Grouping for the view.
JoinNullEquivalency

Specifies whether the Join expressions in this view should consider NULL values to be equivalent. When this is enabled, missing values, empty strings, and NULL values will be considered to match in a Join expression. When this is disabled, these values will not match.

Default: disabled
FileName File location to store view data. Unused in this version of AMPS.

Table 9.6: SOW/View

<SOW>
    <!-- Single topic aggregation -->
    <Topic>
        <Topic>/ett/order</Topic>
        <MessageType>fix</MessageType>
        <Key>/orderId</Key>
    </Topic>
    <View>
        <MessageType>nvfix</MessageType>
        <Topic>TOTAL_VALUE</Topic>
        <UnderlyingTopic>/ett/order</UnderlyingTopic>
        <Projection>
            <Field>/109</Field>
            <Field>SUM(/14 * /6) AS /71406</Field>
        </Projection>
        <Grouping>
            <Field>/109</Field>
        </Grouping>
    </View>

    <!-- Single topic aggregation with filter -->
    <Topic>
        <Name>orders</Name>
        <MessageType>json</MessageType>
        <Key>/orderId</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <View>
        <Name>CompleteByRegion</Name>
        <UnderlyingTopic>orders</UnderlyingTopic>
        <MessageType>json</MessageType>
        <Projection>
            <Field>COUNT(/orderId) AS /completedOrders</Field>
            <Field>/region AS /region</Field>
        </Projection>
        <Grouping>
            <Field>/region</Field>
        </Grouping>
        <Filter>/status = 'complete'</Filter>
    </View>

  <!-- Single topic aggregation with a hash index
       for faster query. -->
  <Topic>
      <Name>source-for-hash-sample</Name>
      <MessageType>json</MessageType>
      <Key>/id</Key>
      <FileName>./sow/%n.sow</FileName>
  </Topic>
  <View>
      <Name>hash-sample-view</Name>
      <MessageType>json</MessageType>
      <UnderlyingTopic>source-for-hash-sample</UnderlyingTopic>
      <Projection>
         <Field>SUM(/qty) as /quantity</Field>
         <Field>/customerName</Field>
         <Field>/orderType</Field>
       </Projection>
       <Grouping>
         <Field>/customerName</Field>
         <Field>/orderType</Field>
       </Grouping>
       <!-- provide fast query for exact matches on
            *either* name or orderType -->
       <HashIndex>
          <Key>/name</Key>
       </HashIndex>
       <HashIndex>
          <Key>/orderType</Key>
       </HashIndex>
   </View>

    <!-- Project from one message type to another -->
    <Topic>
        <Name>example</Name>
        <MessageType>json</MessageType>
        <Key>/id</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <View>
        <!-- notice that nvfix topic named 'example' is not the same topic as the json
            topic named 'example' -->
        <Name>example</Name>
        <MessageType>nvfix</MessageType>
        <UnderlyingTopic>[json].[example]</UnderlyingTopic>
        <Projection>
            <Field>[json].[example]./id AS /id</Field>
        </Projection>
        <Grouping>
            <Field>[json].[example]./id</Field>
        </Grouping>
    </View>

   <!-- JOIN topics -->
   <Topic>
        <Name>ORDERS</Name>
        <MessageType>nvfix</MessageType>
        <Key>/OrderID</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <Topic>
        <Name>COMPANIES</Name>
        <MessageType>nvfix</MessageType>
        <Key>/CompanyId</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <View>
        <Name>TOTAL_COMPANY_VOLUME</Name>
        <UnderlyingTopic>
            <Join>[ORDERS]./Tick = [COMPANIES]./Tick</Join>
        </UnderlyingTopic>
        <MessageType>nvfix</MessageType>
        <Projection>
            <Field>[COMPANIES]./CompanyId</Field>
            <Field>[COMPANIES]./Tick</Field>
            <Field>[COMPANIES]./Name</Field>
            <Field>SUM([ORDERS]./Shares) AS /TotalVolume</Field>
        </Projection>
        <Grouping>
            <Field>[ORDERS]./Tick</Field>
        </Grouping>
    </View>
</SOW>

Example 9.2: View Example

SOW/ConflatedTopic

AMPS provides the ability to create ongoing snapshots of a SOW topic, called conflated topics (also called topic replicas in previous releases of AMPS). Conflated topics are updated on an interval, and store a snapshot of the current state of the world at each interval. This helps to manage bandwidth to clients that do not act on each update, such as a client UI that refreshes every second rather than with every update.

For compatibility with previous AMPS versions, AMPS allows you to use ReplicaDefinition as a synonym for ConflatedTopic.

Element Description
Name

String used to define the name of the conflated topic. While AMPS doesn’t enforce naming conventions, it can be convenient to name the conflated topic based on the underlying topic name. For example, if the underlying topic is orders, it can be convenient to name the conflated topic orders-C.

If no Name is provided, AMPS accepts Topic as a synonym for Name to provide compatibility with versions of AMPS previous to 5.0.
UnderlyingTopic

String used to define the SOW topic which provides updates to the conflated topic. This must exactly match the name of a SOW topic.

When the underlying topic is a regular expression topic, this must match the Name of the topic rather than the Pattern of the topic. In this case, the conflated topic will conflate all of the topics in the underlying regular expression topic into individual topics. The names of the individual topics are set using the TopicFormat element in this case.
MessageType
 The message format of the underlying topic. This MessageType must be the MessageType of the provided UnderlyingTopic.
Interval

The frequency at which AMPS updates the data in the conflated topic.

Default: 5s

Minimum: 100ms
Filter Content filter that is applied to the underlying topic. Only messages that match the content filter are stored in the conflated topic.
HashIndex

AMPS provides the ability to do fast lookup for SOW records based on specific fields.

When one or more HashIndex elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup, and are kept up to date as records are added, removed, and updated.

The HashIndex element contains a Key element for each field in the hash index.

AMPS uses a hash index when a query uses exact matching for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions.

AMPS automatically creates a hash index for the set of fields specified in the set of Key fields for the UnderlyingTopic, if that topic provides Key fields.

SOW/ConflatedTopic Parameters

If the underlying topic of a ConflatedTopic is a regular expression topic, an additional configuration parameter is required to specify the topic name that AMPS will use for each conflated topic.

Element Description
TopicFormat

String used to format the name of the conflated topics when the underlying topic is a regular expression topic.

When the underlying topic is a regular expression topic, AMPS will create a conflated topic for each topic within the regular expression topic. This element specifies how AMPS will construct the name of the conflated topics produced.

The TopicFormat must contain the string %n, which is replaced with the name of the topic in the underlying regular expression topic.

For example, if the TopicFormat is %n-C, when the underlying regular expression topic contains a topic named /my/orders, AMPS will produce a conflated topic with the name /my/orders-C.

There is no default for this element. This element cannot contain characters that are significant for regular expressions (such as ., ^, and so on).

SOW/ConflatedTopic Parameters when the UnderlyingTopic is a regular expression topic

<ConflatedTopic>
    <Topic>FastPublishTopic-C</Topic>
    <MessageType>nvfix</MessageType>
    <UnderlyingTopic>FastPublishTopic</UnderlyingTopic>
    <Interval>5s</Interval>
    <Filter>/region = 'A'</Filter>
</ConflatedTopic>
<ConflatedTopic>
    <Topic>LongIntervalTopic-C</Topic>
    <MessageType>json</MessageType>
    <UnderlyingTopic>FastPublishTopic</UnderlyingTopic>
    <Interval>120s</Interval>
    <HashIndex>
       <Key>/order/status</Key>
    </HashIndex>
</ConflatedTopic>
<ConflatedTopic>
   <Topic>ConflateUnderlyingRegex</Topic>
   <MessageType>bflat</MessageType>
   <UnderlyingTopic>TheRegexTopic</UnderlyingTopic>
   <Interval>20s</Interval>
</ConflatedTopic>