9. State-of-the-World (SOW) Features¶

The SOW section of the configuration file holds the configuration for the AMPS State of the World.

Element	Description
`Topic`	Specifies that AMPS will record distinct messages for this topic in the State of the World. SOW topic definitions are used directly as a last-value cache, and are required for many of the advanced messaging features in AMPS such as out-of-focus notifications and delta messaging. SOW topic definitions can also be used as the underlying topics for views, aggregates, and conflated topics. SOW `Topic` configuration is described in SOW/Topic.
`Queue`	Defines a message queue. Rather than delivering each message to all matching subscriptions, message queues provide features to help ensure that each message is delivered to and processed by a single subscriber. See the User Guide for a full description of the functionality of message queues. Queue configuration is described in SOW/Queue.
`View`	Defines a View over one or more SOW topics, conflated topics, or other views. A view can perform aggregation, can JOIN multiple topics together. A view can be based on a SOW topic of one message type and project results of a different message type. View configuration is described in SOW/View.
`ConflatedTopic`	Defines a copy of a SOW topic or view that receives current value updates at a specified interval, conflating any changes to values that occur between the scheduled updates. ConflatedTopic configuration is described in SOW/ConflatedTopic.

Table 9.1: SOW section configuration elements

The elements within the SOW section are described in detail in the following sections.

SOW/Topic¶

SOW/Topic

Creating a Topic element in the SOW configuration section declares that AMPS will maintain state of the world for that topic. When a topic is recorded in the State of the World (SOW), AMPS maintains the most recent value of each distinct message in that topic. In this respect, the SOW for a topic is similar to a table in a relational database.

Notice that AMPS does not require topics to be predeclared. Creating this configuration item is declaring that the State of the World will be maintained for this topic: the configuration does not define a topic. Creating this configuration item is not required to publish messages to the topic.

There are several sets of options required when a topic is added to the State of the World. General options define the topic name and the overall behavior of the State of the World for this topic. Key specification options define how AMPS identifies unique messages within the topic (much like declaring a primary key for a table in a relational database). Growth specification options define how AMPS grows storage for the State of the World for this topic as messages are added.

Example 9.1 contains a listing of the parameters for a Topic section in the SOW section of an AMPS configuration file. AMPS accepts TopicDefinition as a synonym for Topic, since this was the element name used in versions before 5.0.

This section presents brief descriptions of the options for recording the State of the World for a topic. For full information on the State of the World, see State of the World in the AMPS User Guide.

Element	Description
FileName	The file where the State of the World (SOW) data will be stored. This element is required for SOW topics with a `Durability` of `persistent` (the default) because those topics are persisted to the filesystem. This is not required for SOW topics with a durability of `transient`.
MessageType	Type of messages to be stored. To use AMPS generated SOW keys, the message type specified must support content filtering so that AMPS can determine the SOW key for the message. All of the default message types, except `binary`, support content filtering. Since the `binary` message type does not support content filtering, that type can only be used for a SOW when publishers use explicit keys. See the Message Types chapter in the AMPS User Guide for a discussion of the message types that AMPS loads by default. Some message types (such as Google Protocol Buffers) require additional configuration, and must be configured before using the message type in a SOW topic.
Name	The name of the SOW topic - all unique messages on this topic will be stored in a topic-specific SOW database. Every SOW requires a method of determining which messages are unique. Several methods are provided within AMPS. See the AMPS User Guide for a discussion on SOW keys, and Key Table for relevant configuration items. If no `Name` is provided, AMPS accepts `Topic` as a synonym for `Name` to provide compatibility with versions of AMPS previous to 5.0.
`HashIndex`	AMPS provides the ability to do fast lookup for SOW records based on specific fields. When one or more `HashIndex` elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup, and are kept up to date as records are added, removed, and updated. The `HashIndex` element contains a `Key` element for each field in the hash index. AMPS uses a hash index when a query uses a exact matching for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions. AMPS automatically creates a hash index for the set of fields specified in the set of `Key` fields for the SOW, if those fields are specified.
`Index`	AMPS supports the ability to precreate memo indexes for specific fields using the `Index` configuration option. When one or more `Index` elements are provided, AMPS creates memo indexes for any field specified in an `Index` element on startup, before a query that uses that field runs. Otherwise, AMPS indexes each field the first time a query uses the field. Adding one or more `Index` configurations to a `SOW/Topic` can improve retrieval performance the first time a query that contains the indexed fields runs for large SOW topics.
`RecoveryPoint`	For SOW topics that are covered by the transaction log, the point from which to recover the SOW if the SOW file is removed, or if the SOW topic has `transient` duration. This configuration item allows two values: `epoch` recovers the SOW from the beginning of the transaction log `now` recovers the SOW from the current point in the transaction log Defaults to `epoch`.
`Expiration`	Time for how long a record should live in the SOW database for this topic. The expiration time is stored on each message, so changing the expiration time in the configuration file will not affect the expiration of messages currently in the SOW. AMPS accepts interval values for the Expiration, using the interval format described in the AMPS Configuration Guide section on units, or one of the following special values: A value of `disabled` specifies that AMPS will not process SOW expiration for this topic, regardless of any expiration value set on the message. In this case, AMPS saves the expiration for the message, but does not process it. The value must be set to `disabled` (the default) if `History` is enabled for this topic. A value of `enabled` specifies that AMPS will process SOW expiration for this topic, with no expiration set by default. Instead, AMPS uses the value set on the individual messages (with no expiration set for messages that do not contain an expiration value). Default: `disabled` (never expire)
`Durability`	Defines the data durability of a SOW topic. SOW databases listed as `persistent` are stored to the file system, and retain their data across instance restarts. Those listed as `transient` are not persisted to the file system, and are reset each time the AMPS instance restarts. Notice that when the durability is `transient`, and the topic is recorded in the transaction log, each time AMPS starts, AMPS will recover the state of the topic from the transaction log. The recovery begins at the `RecoveryPoint` specified for the topic, which defaults to `epoch`, the beginning of the transaction log. (For `persistent` topics, AMPS recovers from the last message written to the SOW topic, or from the `RecoveryPoint` if the SOW file is removed.) Default: `persistent` Valid values: `persistent` or `transient` Synonyms: `Duration` is also accepted for this parameter for backward compatibility with configuration prior to 4.0.0.1
`History`	Enable historical query for this SOW. This element contains a `Window` and `Granularity` element. When the `History` element is present, historical query is enabled for this sow. Otherwise, AMPS does not enable historical query and does not store the historical state of the SOW. `Expiration` must be `disabled` when `History` is enabled.
`Window`	For a historical SOW, the length of time to store history. For example, when the value is `1w`, AMPS will store one week of history for this SOW. Used within the `History` element. Default: By default, AMPS does not expire historical SOW data.
`Granularity`	For a historical SOW, the granularity of the history to store. For many applications, it is not necessary for AMPS to store all of the updates to the SOW. This parameter sets the resolution at which AMPS will save the state of a message. A value of `0s` or equivalent specifies that AMPS will preserve every update within the `Window`. For example, when you set a granularity of `1m`, AMPS will save the state of the message no more frequently than once per minute, even when the state of the message is updated several times a minute. Used within the `History` element.
`ExpectedKeyCountHint`	For SOW topics that will contain a large number of distinct keys, providing an expected key count allows AMPS to pre-size the data structure that holds the key. This can provide a performance improvement for publishers by avoiding cases where AMPS has to resize the data structure. There is no default for this value. When no value is provided, AMPS does not pre-size data structures for the SOW. When provided, the value of this option should be the power of 2 closest to the maximum number of keys the topic is expected to hold. For example, a topic that is expected to hold 500,000,000 distinct records could set a value of `536870912` (`2 ^ 29`).
`Preprocessing`	When present, specifies the message enrichment to be performed before AMPS determines the SOW key for the message. The `Preprocessing` element must contain one or more `Field` elements that specify the enrichment to perform.
`Enrichment`	When present, specifies the message enrichment to be performed after AMPS determines the SOW key for the message. The `Enrichment` element must contain one or more `Field` elements that specify the enrichment to perform.

Table 9.2: SOW/Topic General Options

Each SOW topic must define how AMPS will determine which messages are unique. An application can either have AMPS determine the key by specifying one or more Key fields, provide a SOW key with the publish command each time a message is published to AMPS. AMPS also provides the ability to provide a custom SowKey generator with a plugin module.

See the Sow Key Generation section in the AMPS User Guide for for a full discussion. The following table lists the available configuration items for specifying how AMPS determines the SowKey for a message:

Element Description

Key

Specifies an XPath within each message that AMPS will use to generate a SOW key, which determines whether a message is unique. This element can be specified multiple times to create a composite key from the combined value of the specified Key elements.

When one or more Key elements is specified for the SOW, AMPS generates the SOW key for each message. When no Key fields are specified and no KeyGenerator is specified, publishers must explicitly provide the SOW key for each message when the message is published.

60East recommends configuring a Key and having AMPS generate the SOW key for a message unless your application has specific needs that make this impractical.

AMPS automatically creates a hash index for the set of fields specified in the Key elements.

There is no default for this element.

KeyDomain

The seed value for SowKeys used within the topic when AMPS generates the SOW key. The default is the topic name, but it can be changed to a string value to unify SowKey values between different topics.

For example, if your application has a ShippingAddress SOW and a CreditRating SOW that both use /customerID as the SOW key, you can use a KeyDomain to ensure that the generated SowKey for a given /customerId is identical for both SOW topics. This does not affect how AMPS processes the SOW topics, but can make correlating information from different SOW topics easier in your application.

This option can only be specified when one or more Key fields are specified. When a SOW key generator module is used, or the publisher must send a SOW key, this option is not valid.

Default: the name of the SOW topic.

KeyGenerator

Specifies the SOW key generator module to use for this topic. When this configuration element is present, AMPS calls the specified module to generate a SOW key for each incoming message.

Default: no SOW key generator module. When there is no SOW key generator module specified, AMPS uses the specified Key fields if the Key fields are provided. If no generator is specified and no Key fields are specified, AMPS requires publishers to set a SOW key on each message published.

A KeyGenerator element contains the following elements:

Element Description
Module
The name of the module. This module must be loaded elsewhere in the configuration file.
Options Contains one or more XML elements. These elements are provided to the key generator module as options. The options provided depend on the key generator. The creator of the key generator module must document the options for that module.

Table 9.3: SOW/Topic Key Specification Options

The SOW topic configuration also specifies how the SOW file is allowed to grow. See the AMPS User Guide for details on SOW growth. The configuration options for controlling how the file is allocated and how the file grows are listed below:

Element Description

SlabSize

The size of each allocation for the SOW file, as a number of bytes. When AMPS needs more space for the SOW, it requests this amount of space from the operating system. This effectively sets the maximum message size that AMPS guarantees can be stored in the SOW. This size includes headers set by AMPS on the message.

60East recommends setting this value only if you will be storing messages larger than the default SlabSize or if performance or capacity testing indicates a need to tune SOW performance. If you plan to store messages larger than the default setting, 60East recommends a starting value of several times the maximum message size. For example, if your maximum message size is 2MB, a good starting point for SlabSize would be 8MB.

If it becomes necessary to tune the SlabSize, see the Best Practices section of the AMPS User Guide and Capacity Planning section of the AMPS User Guide for a full discussion about tuning the SlabSize.

Default: 5MB

InitialSlabCount

The number of SOW slabs that AMPS will allocate on startup.

Default: 1

Maximum: 1024

DEPRECATED:

RecordSize

This parameter is deprecated beginning in AMPS 5.0. Use SlabSize parameter instead. Size (in bytes) of a SOW record for this topic.

Default: 512

DEPRECATED:

InitialSize

This parameter is deprecated beginning in AMPS 5.0. Use the InitialSlabCount parameter instead. Initial size (in records) of the SOW database file for this topic.

Default: 2048

DEPRECATED:

IncrementSize

This parameter is deprecated beginning in AMPS 5.0. Use the SlabSize parameter instead. Number of records to expand the SOW database (for this topic) by when more space is required.

Default: 1000

Table 9.4: SOW/Topic Growth Specification Options

An example of a SOW configuration looks like the following:

<SOW>
    <!-- Simple SOW topic definition -->
    <Topic>
        <Name>orders</Name>
        <Key>/orderId</Key>
        <MessageType>nvfix</MessageType>
        <FileName>./sow/%n.sow</FileName>
    </Topic>

    <!-- SOW with hash indexes -->
    <Topic>
        <Name>customers</Name>
        <Key>/customerId</Key>
        <MessageType>json</MessageType>
        <FileName>./sow/%n.sow</FileName>
        <HashIndex>
            <Key>/customerName</Key>
        </HashIndex>
        <HashIndex>
            <Key>/zipCode</Key>
            <Key>/customerType</Key>
        </HashIndex>
    </Topic>

    <!-- Historical SOW -->
    <Topic>
        <Name>catalog</Name>
        <Key>/sku</Key>
        <MessageType>json</MessageType>
        <FileName>./sow/%n.sow</FileName>
        <History>
            <Window>7d</Window>
            <Granularity>15m</Granularity>
        </History>
    </Topic>

    <!-- SOW with enrichment Add a /fullName field constructed
        from /firstName and /lastName. -->
    <Topic>
        <Name>sales-reps</Name>
        <Key>/employeeId</Key>
        <MessageType>bflat</MessageType>
        <Enrichment>
            <Field>CONCAT(/firstName, " ", /lastName) AS /fullName</Field>
        </Enrichment>
    </Topic>
</SOW>

Example 9.1: SOW Topic Configuration

SOW/Queue, SOW/LocalQueue, and SOW/GroupLocalQueue¶

This section lists configuration parameters for queues.

The Queue tag, the LocalQueue tag, and the GroupLocalQueue tag are used to configure message queues.

When an AMPS queue is defined with the Queue tag, the queue will be a distributed queue. To make a queue that is limited to the local instance, use the LocalQueue configuration element. To make a queue that is restricted to a subset of instances in a set of replicated instances (typically all in the same group), use the GroupLocalQueue tag.

AMPS accepts QueueDefinition as a synonym for Queue.

The following configuration items apply to queues regardless of whether they are fully distributed, local, or distributed within a replication group:

Element	Description
Name	The name of the queue topic. This name is the name that consumers subscribe to. If no `Name` is provided, AMPS accepts `Topic` as a synonym for `Name` in the `Queue` definition.
MessageType	The message type of the queue.
`UnderlyingTopic`	A topic name or regular expression for the topic that contains the messages to capture in the queue. These topics must be recorded in a transaction log, and all must be of the same message type as the queue. If an `UnderlyingTopic` is not provided, the `UnderlyingTopic` defaults to the `Name` of the queue.
`DefaultPublishTarget`	The topic to publish to when an application publishes a message to the queue. For simplicity, AMPS allows applications to publish messages to the queue, and for those messages to be routed to one of the underlying topics. The `DefaultPublishTarget` must be one of the topics included in the queue. This element is required if the `UnderlyingTopic` contains regular expression characters. Otherwise, the `UnderlyingTopic` is a single topic and this element is optional and defaults to the `UnderlyingTopic`.
`LeasePeriod`	The amount of time that a subscriber has ownership of the message before the message is returned to the queue. For `at-least-once` delivery semantics, the consumer must process and acknowledge the message within this lease period, or the message may be provided to another subscriber. The `LeasePeriod` is measured from the time that AMPS sends the message to the subscriber. Set the `LeasePeriod` to account for round trip network latency as well as the expected processing time for the subscribers. Default: `infinite` (no expiration)
`Semantics`	The delivery semantics to use for this queue. Regardless of the delivery semantics, AMPS queues deliver a given message to a single subscriber at a time. When a subscriber fails to process a message (that is, the connection to the subscriber closes before the message is acknowledged, or the lease expires before the message is acknowledged), the semantics specify how the failure is handled. `at-least-once` With these semantics, you can guarantee that a message has been processed by at least one subscriber, as described in the introduction to Queues in the AMPS User Guide. With this value, a subscriber must explicitly remove the message from the queue once the message is processed. If the subscriber connection closes before acknowledging the message, the subscriber returns the message to the queue, or the lease expires, the AMPS server can deliver the message again to allow another attempt to process the message. `at-most-once` With these semantics, AMPS removes the message from the queue immediately when AMPS sends the message. This allows you to guarantee that no more than one subscriber will process the message, even if the subscriber that receives the message fails without acknowledging the message. Default: `at-least-once`
`MaxBacklog`	The maximum number of outstanding, unacknowledged messages in the queue at any one time. This parameter allows you to set limits on the number of pending messages from the queue overall. When the queue reaches the MaxBacklog, no incoming messages are delivered from the queue until a message is removed from the queue (either by expiring, or being acknowledged by a client). This parameter allows you to avoid overwhelming clients during periods of heavy activity. Notice that this does not set a limit of any sort on the capacity of the queue. This parameter allows you to limit the number of messages that the queue will make available to subscribers at a given time, but does not restrict the capacity of the queue to track messages. This backlog number is applied per instance of the queue. That is, each instance of AMPS that hosts an instance of a replicated queue will deliver messages up to the `MaxBacklog` if messages are available on that instance. Default: `infinite`
`MaxPerSubscriptionBacklog`	The maximum number of outstanding, unacknowledged messages in the queue for an individual subscription. This parameter allows you to avoid overwhelming a single subscriber during a period of heavy activity. Subscribers can declare the maximum number of messages that the subscription is prepared to lease at a given time. This maximum defaults to 1 when there is no maximum explicitly specified for a subscription. AMPS will lease the number specified in the subscription or the maximum set for the queue, whichever is lower. Notice that this does not set a limit of any sort on the capacity of the queue. This parameter allows you to limit the number of messages that the queue will make available for a single subscription at a given time, but does not restrict the capacity of the queue to track messages. Default: `1`
`Expiration`	The length of time a message can remain in the queue before AMPS considers the message undeliverable. Messages may expire while a subscriber has a lease on the message. AMPS does not send an additional notification in this case. A publisher can override this setting by providing an expiration value on an incoming message. If an individual message in the transaction log has an expiration value, AMPS will use that expiration value for the message rather than the default for the queue. Default: `infinite`
`Filter`	An AMPS `Filter` that is applied to the `UnderlyingTopic`. When a `Filter` is specified, only messages matching the `Filter` appear in the queue. By default, there is no filter and all messages from the UnderlyingTopic are presented in the queue.
`RecoveryPoint`	This option allows you to specify the point at which AMPS begins reviewing the transaction log to recover the state of the queue when AMPS restarts and there is no existing information about the state of the queue. By default, in this case AMPS reviews the full log to determine the contents and state of the queue. If AMPS has a record of the last point in the transaction log at which all previous messages are acknowledged (stored in the `queues.ack` file), and the recovery point specified in the configuration file has not changed, AMPS will recover from that point rather than the configured `RecoveryPoint`, since using the cached information will reduce recovery time. The `RecoveryPoint` can be one of the following: `epoch` - Recovery begins at the beginning of the transaction log `now` - Recovery begins at the time AMPS starts queue recovery, so only new messages are added to the queue `creation` - Recovery begins at the time the queue was created AMPS bookmark - When an AMPS bookmark is provided, AMPS starts recovery at the specified bookmark. ISO-8601 timestamp - When a timestamp is provided, AMPS starts recovery at the specified timestamp. The timestamp must be provided in the format AMPS uses for timestamp bookmarks. Default: `epoch`
`FairnessModel`	AMPS provides different methods to distribute messages across active subscriptions: `fast` - AMPS delivers to the first subscription found that can process the message `round-robin` - AMPS distributes to the next subscription found that can process the message `proportional` - AMPS delivers to the subscription with the lowest ratio of active messages to available backlog Each instance of AMPS independently manages the fairness model for subscriptions on that instance. Fairness model information is not replicated across instances. Default: `proportional` for `at-least-once` queues, `round-robin` for `at-most-once` queues
`Leasing`	Ownership model for leased messages. AMPS supports the following models: `strict` - AMPS allows a client to acknowledge (`sow_delete`) only messages that are leased to the client or currently unleased. If a client acknowledges a message leased to another client, there is no effect. `sublet` - AMPS allows any client to acknowledge any message, regardless of whether another client has a lease on the message. Default: `sublet`
`MaxDeliveries`	Specifies an upper bound to the number of times AMPS may deliver a queue message before automatically expiring it. For example if AMPS delivers a message twice and `MaxDeliveries` is set to `2`, the message will be expired if the subscriber disconnects or unsubscribes before acknowledging it. This counter is reset if the server restarts, and the counter is not replicated to other instances. Default: no maximum (`0`).
`MaxCancels`	Specifies a limit to the number of times a subscriber may cancel a lease on a message before it is expired. For example if a message is canceled for the second time and `MaxCancels` is set to 2, AMPS automatically expires the message instead of returning it to the message queue. This counter is reset if the server restarts, and the counter is not replicated to other instances. Default: no maximum (`0`).
`Priority`	Specifies the order in which messages will be distributed from the queue. When present, this element constructs a value that specifies the priority for messages in the queue. Higher priority messages are delivered first, regardless of the order in which messages have been published to the queue. The contents of the element can be either the name of a field in the message or an AMPS expression. Either way, the result is treated as an `unsigned long` value. For example, to order message delivery based on the `/priority` field of the message, specify the following element: <Priority>/priority</Priority> To order message delivery based on the product of the `/price` field and the `/quantity` field in the message, specify the following element. <Priority>/price * /qty</Priority> Default: there is no default for this value. When not provided, the delivery order is the order in which messages were processed by this instance of AMPS. This option cannot be specified on a queue if `BarrierExpression` is specified.
`BarrierExpression`	Specifies the filter used to identify a barrier message (synchronization point) for this queue. When a message matches this expression, it will be delivered to all current subscribers on the queue when every previous message in the queue has been acknowledged. This provides a simple, reliable way to synchronize workers. <BarrierExpression>/isSyncPoint IS NOT NULL</BarrierExpression> The `BarrierExpression` filter is evaluated when AMPS adds the message to the in-memory state of the queue, and is not re-evaluated (unless the in-memory state of the queue is rebuilt after an instance restart). Default: there is no default for this value. When not provided, no messages are considered to be barrier messages This option cannot be specified on a queue if `Priority` is specified.
`DeferredAckExpiration`	Specifies the amount of time for AMPS to retain information about an acknowledgment (`sow_delete`) message received when the corresponding message is not in the queue. This can occur during failover, when messages are received over replication, or in cases where an application that uses a publish store has been offline for an extended period of time. This element is configured as an interval, for example, `15m` or `2h`. The default value is generally recommended. However, in cases where an instance may receive large volumes of acknowledgments for messages that are not currently in the queue, and are not expected to arrive, setting this to a lower value may somewhat reduce the memory required for managing these acknowledgments. Default: `1d`

Table 9.5: Queue configuration elements

When a queue is defined as a GroupLocalQueue, the following configuration item must be provided:

Element Description

InitialOwner

For a group local queue, provides the instance Name of the instance that will own the message when the message is first published.

This configuration element is required for a GroupLocalQueue. It is not supported for fully distributed queues or for local only queue (that is, a queue defined with the Queue or LocalQueue element.

All instances in a replicated system that define this queue must define the same InitialOwner.

Notice that AMPS does not validate that the InitialOwner exists and is reachable through replication from the current instance. When configuring a GroupLocalQueue, take care to validate the name and ensure that a replication path exists to and from that instance.

This element is required. There is no default for this element.

Table 9.6: Additional configuration elements for GroupLocalQueue

The following configuration snippet shows one way to configure a queue.

<!-- Notice that the topics to use for the queue (ORDERS_.*) must be
    recorded in a transaction log. -->
<SOW>
    <Queue>
        <Name>MQ</Name>
        <MessageType>json</MessageType>
        <UnderlyingTopic>ORDERS_.*</UnderlyingTopic>
        <DefaultPublishTarget>ORDERS_DIRECT</DefaultPublishTarget>
        <LeasePeriod>60s</LeasePeriod>
        <Expiration>1d</Expiration>
        <MaxBacklog>3</MaxBacklog>
    </Queue>
</SOW>

SOW/View¶

Table 9.6 contains a listing of the parameters for a View section in the SOW section of an AMPS configuration file. For backward compatibility, AMPS accepts ViewDefinition as a synonym for View.

Element	Definition
MessageType	The message type of the view. This does not need to be the same type as any of the topics in the aggregation, but does need to be a message type that supports views. The message type must be one of the message types configured for the instance. AMPS includes `fix`, `xml`, `nvfix`, `json`, `bson` and `bflat` message types with full support for views. You can also use any custom message type defined for the configuration file, provided that the message type supports views. Notice that the `binary` message type does not specify a fixed format for the message contents, so that message type cannot be used in a view. Other message types provided with AMPS may have limitations in their support for views. see the Message Types chapter in the AMPS User Guide for a discussion of the limitations.
Name	Defines the topic name for this view. If no `Name` is provided, AMPS accepts `Topic` as a synonym for `Name` to provide compatibility with versions of AMPS previous to 5.0.
UnderlyingTopic	Defines the SOW topic or topics on which this view is based. This element can contain a single topic name, or any number of `Join` elements.
Projection/Field	Defines what the view will contain. This element can be specified multiple times to compose a complex view. Complex expressions that use aggregation functions and conditional branching can also be used.
Grouping/Field	Defines how the records in the underlying topic will be grouped. This is analogous to a SQL GROUP BY clause.
`KeyDomain`	The seed value for `SowKeys` used within this topic. The default is the topic name, but it can be changed to a string value to unify `SowKey` values between different topics.
`Join`	Within an `UnderlyingTopic`, each `Join` specifies two topics to join together to create the view, as well as the relationship between those topics. An `UnderlyingTopic` can have any number of `Join` specifications. For more information on `Join` specifications, see the Join expression in the AMPS User Guide.
`Conflation`	Defines whether AMPS will attempt to conflate updates to the view. This item accepts two values: `none` - No conflation. AMPS fully processes every update to the view, in publication order. `inline` - Conflation is active. AMPS conflates updates to the same underlying record where possible. See Inline Conflation in the AMPS User Guide for details. Default: `none`
`Filter`	Defines a filter for the view. When provided, only messages from the underlying topic that match this filter will be included in the view. This option is only valid when the view uses a single `UnderlyingTopic`. When the view contains a `Join` specification, this option may not be used. Default: No filter, which includes all messages from the `UnderlyingTopic`.
`HashIndex`	AMPS provides the ability to do fast query lookup for records in a view based on specific fields. When one or more `HashIndex` elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup, and are kept up to date as records are added, removed, and updated. The `HashIndex` element contains a `Key` element for each field in the hash index. AMPS uses a hash index when a query uses an exact string match for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions, or for numeric comparisions. AMPS automatically creates a hash index for the fields specified in the `Grouping` for the view.
`JoinNullEquivalency`	Specifies whether the `Join` expressions in this view should consider `NULL` values to be equivalent. When this is `enabled`, missing values, empty strings, and `NULL` values will be considered to match in a `Join` expression. When this is `disabled`, these values will not match. Default: `disabled`
`FileName`	File location to store view data. Unused in this version of AMPS.

Table 9.6: SOW/View

<SOW>
    <!-- Single topic aggregation -->
    <Topic>
        <Topic>/ett/order</Topic>
        <MessageType>fix</MessageType>
        <Key>/orderId</Key>
    </Topic>
    <View>
        <MessageType>nvfix</MessageType>
        <Topic>TOTAL_VALUE</Topic>
        <UnderlyingTopic>/ett/order</UnderlyingTopic>
        <Projection>
            <Field>/109</Field>
            <Field>SUM(/14 * /6) AS /71406</Field>
        </Projection>
        <Grouping>
            <Field>/109</Field>
        </Grouping>
    </View>

    <!-- Single topic aggregation with filter -->
    <Topic>
        <Name>orders</Name>
        <MessageType>json</MessageType>
        <Key>/orderId</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <View>
        <Name>CompleteByRegion</Name>
        <UnderlyingTopic>orders</UnderlyingTopic>
        <MessageType>json</MessageType>
        <Projection>
            <Field>COUNT(/orderId) AS /completedOrders</Field>
            <Field>/region AS /region</Field>
        </Projection>
        <Grouping>
            <Field>/region</Field>
        </Grouping>
        <Filter>/status = 'complete'</Filter>
    </View>

  <!-- Single topic aggregation with a hash index
       for faster query. -->
  <Topic>
      <Name>source-for-hash-sample</Name>
      <MessageType>json</MessageType>
      <Key>/id</Key>
      <FileName>./sow/%n.sow</FileName>
  </Topic>
  <View>
      <Name>hash-sample-view</Name>
      <MessageType>json</MessageType>
      <UnderlyingTopic>source-for-hash-sample</UnderlyingTopic>
      <Projection>
         <Field>SUM(/qty) as /quantity</Field>
         <Field>/customerName</Field>
         <Field>/orderType</Field>
       </Projection>
       <Grouping>
         <Field>/customerName</Field>
         <Field>/orderType</Field>
       </Grouping>
       <!-- provide fast query for exact matches on
            *either* name or orderType -->
       <HashIndex>
          <Key>/name</Key>
       </HashIndex>
       <HashIndex>
          <Key>/orderType</Key>
       </HashIndex>
   </View>

    <!-- Project from one message type to another -->
    <Topic>
        <Name>example</Name>
        <MessageType>json</MessageType>
        <Key>/id</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <View>
        <!-- notice that nvfix topic named 'example' is not the same topic as the json
            topic named 'example' -->
        <Name>example</Name>
        <MessageType>nvfix</MessageType>
        <UnderlyingTopic>[json].[example]</UnderlyingTopic>
        <Projection>
            <Field>[json].[example]./id AS /id</Field>
        </Projection>
        <Grouping>
            <Field>[json].[example]./id</Field>
        </Grouping>
    </View>

   <!-- JOIN topics -->
   <Topic>
        <Name>ORDERS</Name>
        <MessageType>nvfix</MessageType>
        <Key>/OrderID</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <Topic>
        <Name>COMPANIES</Name>
        <MessageType>nvfix</MessageType>
        <Key>/CompanyId</Key>
        <FileName>./sow/%n.sow</FileName>
    </Topic>
    <View>
        <Name>TOTAL_COMPANY_VOLUME</Name>
        <UnderlyingTopic>
            <Join>[ORDERS]./Tick = [COMPANIES]./Tick</Join>
        </UnderlyingTopic>
        <MessageType>nvfix</MessageType>
        <Projection>
            <Field>[COMPANIES]./CompanyId</Field>
            <Field>[COMPANIES]./Tick</Field>
            <Field>[COMPANIES]./Name</Field>
            <Field>SUM([ORDERS]./Shares) AS /TotalVolume</Field>
        </Projection>
        <Grouping>
            <Field>[ORDERS]./Tick</Field>
        </Grouping>
    </View>
</SOW>

Example 9.2: View Example

SOW/ConflatedTopic¶

AMPS provides the ability to create ongoing snapshots of a SOW topic, called conflated topics (also called topic replicas in previous releases of AMPS). Topic replicas are updated on an interval, and store a snapshot of the current state of the world at each interval. This helps to manage bandwidth to clients that do not act on each update, such as a client UI that refreshes every second rather than with every update.

For compatibility with previous versions of AMPS, AMPS allows you to use ReplicaDefinition as a synonym for ConflatedTopic.

Element	Description
Name	String used to define the name of the conflated topic. While AMPS doesn’t enforce naming conventions, it can be convenient to name the conflated topic based on the underlying topic name. For example, if the underlying topic is `orders`, it can be convenient to name the conflated topic `orders-C`. If no `Name` is provided, AMPS accepts `Topic` as a synonym for `Name` to provide compatibility with versions of AMPS previous to 5.0.
UnderlyingTopic	String used to define the SOW topic which provides updates to the conflated topic. This must exactly match the name of a SOW topic.
MessageType	The message format of the underlying topic. This `MessageType` must be the `MessageType` of the provided `UnderlyingTopic`.
`Interval`	The frequency at which AMPS updates the data in the conflated topic. Default: `5s` Minimum: `100ms`
`Filter`	Content filter that is applied to the underlying topic. Only messages that match the content filter are stored in the conflated topic.
`HashIndex`	AMPS provides the ability to do fast lookup for SOW records based on specific fields. When one or more `HashIndex` elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup, and are kept up to date as records are added, removed, and updated. The `HashIndex` element contains a `Key` element for each field in the hash index. AMPS uses a hash index when a query uses a exact matching for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions. AMPS automatically creates a hash index for the set of fields specified in the set of `Key` fields for the `UnderlyingTopic`, if that topic provides `Key` fields.

Table 9.7: SOW/ConflatedTopic Parameters

<ConflatedTopic>
    <Topic>FastPublishTopic-C</Topic>
    <MessageType>nvfix</MessageType>
    <UnderlyingTopic>FastPublishTopic</UnderlyingTopic>
    <Interval>5s</Interval>
    <Filter>/region = 'A'</Filter>
</ConflatedTopic>
<ConflatedTopic>
    <Topic>LongIntervalTopic-C</Topic>
    <MessageType>json</MessageType>
    <UnderlyingTopic>FastPublishTopic</UnderlyingTopic>
    <Interval>120s</Interval>
    <HashIndex>
       <Key>/order/status</Key>
    </HashIndex>
</ConflatedTopic>

Table Of Contents

Related Topics

9. State-of-the-World (SOW) Features¶

SOW/Topic¶

SOW/Queue, SOW/LocalQueue, and SOW/GroupLocalQueue¶

SOW/View¶

SOW/ConflatedTopic¶