1. Introduction to 60East Technologies AMPS

Welcome to the Advanced Message Processing System (AMPS) from 60East Technologies! AMPS is designed to help you quickly and easily develop and deploy data-intensive applications, with demanding requirements, for low latency and high performance. AMPS takes a nontraditional approach to messaging, storage, and analytics that is designed from the ground up for streaming data and highly-parallelized multicore systems.

AMPS isn’t a traditional database or messaging product. This guide presents a brief introduction to help you understand the capabilities of AMPS and how AMPS operates.

AMPS is widely used for applications such as:

  • Tradeplant operations (including backtesting and historical analysis)
  • Risk calculations
  • Elastic worker farms
  • View servers
  • Message flow integration and “shock absorbers”

AMPS combines a set of capabilities that cut across traditional boundaries between applications that work with data.

AMPS is built around a fast messaging engine that supports both publish and subscribe (fan-out) and queued (competitive consumption) message delivery with full content filtering.

AMPS also provides an integrated database that applications can use as a current value cache, key/value document store, and fully queryable database – or all of these at once. With this database, AMPS includes a built-in aggregation and analytics engine for near-real time analysis of streaming data, including aggregation across multiple topics or message formats.

Integrated message logging provides the ability to record and replay streams of messages with full fidelity.

AMPS is designed from the ground up for enterprise deployment at scale. AMPS provides an extensive set of high-availability features, including integrated replication and automatic failover and recovery for applications. Detailed monitoring and statistics are included from a RESTful interface for ease of data collection and integration with enterprise monitoring and management systems.

Authentication and entitlement capability applies to every operation in AMPS, for fine-grained control over permissions to meet enterprise policy and regulatory requirements. Access to data can be controlled at a topic level, at a message level (content-based security), or at the level of individual fields within a message (limiting the fields a given user has access to view).

60East developed AMPS to serve the needs of some of the most demanding data-intensive applications on the planet. The feature set and capabilities have been engineered for the highest levels of performance, designed for ease of use, and proven in production applications worldwide.

Getting to Know AMPS

AMPS is designed to be a developer friendly product. 60East recommends reading about AMPS with a running instance of AMPS and your development environment of choice available. Although 60East makes every effort to clearly describe how AMPS works, there is no substitute for seeing exactly how a running instance behaves (not to mention the advantages of being able to try out ideas or do quick prototyping while you read).

The table below lists the main parts of the AMPS documentation:

Documentation map
Title Description
Introduction to AMPS

Overview of AMPS functionality.

This is a good place to start if you are new to AMPS or if you are familiar with older versions of AMPS.
AMPS User Guide

Detailed description of AMPS functionality, including guidance and best practices.

If you have detailed questions about how AMPS works, refer to the User Guide.
AMPS Configuration Reference

Guide to AMPS configuration.

This guide shows the configuration file syntax and accepted values. The guide also includes useful examples of configuring commonly-used options.
AMPS Monitoring Reference

Guide to the RESTful monitoring interface and the AMPS statistics database.

Use this guide when creating a monitoring strategy or when collecting statistics about an instance.
AMPS Command Reference Description of the commands sent from an AMPS client to the AMPS server and responses from the server.
Client Language Developer Guide Guide to using a client library to work with AMPS.

This guide uses the spark command line utility for basic examples for simplicity, although a production installation would use an application to perform these functions.

AMPS Concepts

This section describes the overall AMPS approach and the features AMPS provides.

The AMPS messaging system is designed around a few simple principles:

  • Parallelize work and minimize waits and blocking to take full advantage of modern multisocket, multicore systems.
  • Eliminate redundant or unused work by only performing tasks that are necessary to provide the functionality requested by a given operation.
  • Reduce or eliminate cross-system coordination by solving the full range of data delivery and storage problems commonly faced by data-intensive applications.
  • Provide a small, flexible set of commands for ease of use.
  • Provide multiple delivery paradigms supporting both publish-subscribe delivery (many to many) and message queues (single consumption of a message) as well as the ability to query the state of a topic at a point in time.
  • Stay application-focused to provide exactly the capabilities that are heavily used in demanding high-performance applications.
  • Stay hardware aware and build for the future by engineering for next-generation commodity hardware and designing AMPS to fully exploit non-uniform memory access (NUMA), flash-based storage, and high-bandwidth networking.

These concepts are the foundation of how AMPS works and are helpful for understanding how to best use AMPS.

To best take advantage of AMPS, applications typically use the built-in features of AMPS rather than their traditional equivalents.

For example, rather than keeping a separate, independent record of each message published to AMPS for audit purposes, applications most often use one of the data persistence features in AMPS. This speeds development and simplifies deployment by eliminating integration effort, and also solves potential correctness issues which could be caused if messages in persistent storage become inconsistent with the messages provided through the messaging system. With AMPS, the messaging system itself can contain a fully-queryable and replayable record of the system.

As another example, AMPS provides integrated replication rather than relying on an external process. AMPS replication is aware of the format and semantics of the transaction log, the configuration of the instance and the commands sent by publishers. This integration allows AMPS to very efficiently provide a full-fidelity message stream and to provide “self-healing” for an instance to catch up when it has been offline. Further, the message store used for replication (the AMPS transaction log) is also used for durable subscriptions and message replay. Designing and implementing these features together reduces complexity, storage requirements and overhead to enable both capabilities. Within the AMPS server, the implementation uses a sophisticated parallelized algorithm for storage and replay that reduces overall latency and prevents slow consumers or replication destinations from affecting faster consumers. The overall result is to simplify configuration and application development, provide strong consistency and reliability guarantees, and provide the highest possible level of performance.

As a final example, rather than requiring a complex topic structure, requiring applications to oversubscribe and discard messages, AMPS provides both topic filtering and content filtering. AMPS includes an expressive filter grammar to provide precise selection of messages of interest to an individual subscriber. AMPS provides this capability to fully decouple publishers and subscribers. With AMPS, there’s no need to maintain and administer a granular topic structure. Precise filtering and routing improves both network and processor utilization by providing only actionable messages to a subscriber. Likewise, for many applications, there is no need for a publisher to be aware of the processing performed by the subscriber or by AMPS itself.

The examples above highlight just a few of the capabilities AMPS provides and how the AMPS approach simplifies development, administration and operations while providing reliability and performance benefits over conventional systems.

Feature Highlights

Some of the highlights of AMPS features include:

  • Topic based publish and subscribe, including full support for regular expressions to specify topic names.
  • Content filtering based on XPath identifiers (to specify the fields of a message) and SQL-92 (to form a predicate), with added support for Perl-Compatible Regular Expressions (PCRE2).
  • Message queues including content filtering for both publishers and subscribers, configurable strategies for delivery fairness, and truly distributed queues that can efficiently enforce queue semantics and delivery guarantees across a replicated network of AMPS instances.
  • Content-aware messaging support for a wide range of message types, including standard formats such as JSON, FIX, MessagePack, XML, Google Protocol Buffers and BSON. AMPS also supports simple key/value pairs in FIX format (called NVFIX to emphasize that the format uses name/value pairs rather than FIX tags), and a high-performance binary protocol called BFlat. AMPS also supports uninterpreted binary messages, and allows you to create composite message types from existing types to easily combine messages of different types in a single payload.
  • An integrated database and record-aware current value storage (called State of the World, or SOW), with optional historical query capability.
  • Historical replay of message streams, including the ability to preserve the total message ordering across independent topics.
  • Integrated replication and high availability, including automatic resynchronization for instances that fail over.
  • Aggregation and Complex Event Processing (CEP), including the ability to aggregate information across different message streams and message streams of different formats.
  • Advanced messaging capabilities such as atomic query-and-subscribe, incremental (delta) updates, and out-of-focus notifications that tell a subscription when a record no longer matches.
  • Built in statistics and monitoring, with data provided via a standard RESTful interface.
  • Integrated authentication and entitlement across all AMPS features.
  • Actions for automating AMPS functionality, including both routine maintenance tasks and dataflow-aware processing (such as alerting in response to slowdowns or invalid data).
  • Client development kits for popular programming languages such as Java, C#/.NET, C++, Python, JavaScript, and Go.
  • Extensibility API in the AMPS server for adding message types, extending the functions available to the AMPS query language, adding new actions, integrating with enterprise authentication and entitlement systems, and more.

This guide provides an overview of the most commonly used functionality of AMPS, but it is not intended to cover all of the features of AMPS or provide an exhaustive discussion of any individual feature. As mentioned earlier, the AMPS User Guide provides full details about AMPS features.

Software Requirements

The AMPS server is supported on the following platforms:

  • Linux 64-bit (2.6 kernel or later) on x86 compatible processors

Tip

While 2.6 is the minimum kernel version supported, AMPS will select the most efficient mechanisms available to it and as a result, reaps greater benefit from more recent kernel and CPU versions.

The AMPS distribution contains all of the supporting libraries and dependencies needed to run on a typical Linux server installation: no further software is required.

Some utilities provided with the AMPS server have additional dependencies. These utilities are not required to run the server, but can make it easier to troubleshoot and test on the system that hosts the AMPS instance:

  • spark, a basic command line client that supports a subset of AMPS functionality, requires Java 1.7 or later.
  • The utilities for inspecting AMPS files (amps_sow_dump, amps_clients_ack_dump, and so on) require a Python installation.
  • amps-grep requires a Python installation.
  • amps-sqlite3 requires a Python installation and the sqlite3 package for your distribution (often, but not always, installed by default).