1. Overview of AMPS Server Modules¶

Out of the box, AMPS provides a set of capabilities that support most customer scenarios. In some cases, though, you need to extend the behavior of AMPS to tailor how the AMPS instance works to your specific environment. To meet those needs, AMPS provides the ability to load and use extension modules. AMPS extension modules are implemented as shared libraries that are loaded into the AMPS server process at run time. This provides a high-performance way to extend AMPS behavior. Typically, these modules are written in C++.

Many of the features 60East provides in AMPS are implemented as server modules. When you create server modules, you are using the native interfaces that 60East uses to extend AMPS. The AMPS Server SDK contains templates and sample code for each type of module.

You might create an AMPS server module to:

Provide authentication that integrates with the standard authentication framework in your business
Create an entitlement system that follows the data security policies of your business
Create a new administrative action that performs a task unique to your environment
Create a compact message type that is precisely tailored to your messaging needs

This chapter presents an overview of the module interface and describes features common to all modules. The following chapters describe specific topics in implementing server modules.

Prerequisites¶

Extending AMPS involves creating shared objects that are loaded into the AMPS process. To get the most out of this guide, you should be familiar with:

AMPS concepts. To successfully extend AMPS requires a good working knowledge of AMPS functionality as described in the AMPS User Guide. 60East also recommends hands-on experience with AMPS.
Programming in C++ or C. To be successful using this guide, you need a working knowledge of C++ or C.
Developing and debugging server components on Linux operating systems. When extending AMPS, you are creating a server component that runs inside of AMPS. This may require debugging modules that run in a highly-concurrent server environment. For more information, see Debugging, which contains basic information to get you started.

Introduction to AMPS Server Modules¶

The AMPS server contains two kinds of components:

Core engine. The core engine provides the basic functionality of AMPS. The core engine includes message routing, transaction logging, topic and content filtering, state of the world maintenance, subscription management, and so forth. The core engine itself works with highly-optimized AMPS-specific data structures.
Modules. Modules provide the interface between AMPS and external systems. In this release, AMPS provides support for the following types of modules:
- Modules for helping to secure AMPS installations:
  - Authentication modules allow AMPS to verify the identity of clients, for example, by using an external authority. 60East provides a simple sample that shows the mechanics of processing requests, as well as a sample LDAP implementation that demonstrates one way to use an external system to verify identity.
  - Entitlement modules allow AMPS to control access to resources in AMPS. 60East provides a sample entitlement module that allows all permissions to any authenticated user and a sample that reads permission definitions from a file.
  - Authenticator modules provide credentials for AMPS to use on outgoing connections, most typically for replication connections. 60East provides a simple authenticator sample.
- Modules for communicating with applications:
  - Message type modules implement message formats. These modules translate the message bodies received from clients into AMPS-specific data structures so the core engine can filter, compare, project, join, merge, and generate messages. 60East provides implementations of JSON, BSON, FIX, NVFIX, and XML as part of AMPS. The AMPS Server SDK includes both a minimal message type module, and a message type module that includes demonstration implementations of the all of the functionality of message types.
  - Protocol modules handle the headers of messages. 60East provides an implementation of a FIX-like protocol, which is suitable for use with most message types. Because AMPS separates the protocol and message type, you can use the 60East-provided protocols for your message types, which speeds development. Protocol modules work closely with the AMPS internals, and must be kept up to date with each change to AMPS features. It is rarely necessary to develop a protocol module. For more information on protocol modules, contact 60East support.
  - Transport modules provide connections to clients. These connections deliver messages to client programs and receive messages from client programs. The transport module does not interpret messages, but is responsible simply for moving data over the transport. Transport modules work closely with the AMPS internals: for more information on transport modules, contact 60East support.
- Action Modules for performing an action from the AMPS server process, typically used for maintaining the AMPS installation. The AMPS distribution includes a wide variety of action modules for managing AMPS. The AMPS Server SDK includes a sample module that simply writes to the AMPS error log.
- User-Defined Function Modules for extending the AMPS expression language. A user-defined function, or UDF, is called with a set of values from an individual message and returns the results of an operation on that value. For example, you could implement an ISREVERSE() function that takes two strings and returns TRUE if the first string is the reverse of the second.

Several common transports and message types are provided with the AMPS distribution, and the distribution includes the protocol modules used by the AMPS client libraries. If your application requires different message types, will communicate over a different transport, requires authentication, or will enforce granular access to resources, you write an AMPS module to provide that capability to AMPS. The protocol modules provided with AMPS work well for nearly all circumstances, but protocols are included in the extension SDK in the event that no existing protocol works.

The figure below shows how the differences between the transport, protocol, and message type modules. Transport modules handle moving data from place to place. Protocol modules process information about the message intended for the AMPS server. Message type modules interpret the content of the message for use in content filtering, delta messages, and SOW storage.

Figure 1.1: Transport, Protocol, and Message Type

How Do AMPS Modules Work?¶

AMPS modules are implemented as Linux shared objects. Each type of module implements a specific set of functions. You compile your module as a shared object, then tell AMPS where to load the shared object from. At runtime, the AMPS server loads the module and calls the functions that the module implements.

This means that your module, like most shared objects, does not need to implement a main() method. Your module does not need to parse configuration options or use a client library to communicate with AMPS, nor does it need to link to a particular library. The only requirement is that your module implements the functions that AMPS will call and that those functions return the expected results.

All AMPS modules have the same lifecycle. There are four parts of the lifecycle:

Module load and initialization. AMPS loads and initializes the module. This gives you an opportunity to configure any properties of the module that are consistent across contexts. For example, you may load a public key file from the filesystem, or request a set of permissions from a database. AMPS calls the module initialization function once per module from a single thread, although AMPS may initialize multiple modules from different threads simultaneously.

AMPS generally loads modules in the order in which they appear in the Modules definition of the configuration file. However, AMPS does not provide strong guarantees on this ordering, and your modules should not depend on any particular ordering for module load and initialization.
Context initialization. AMPS configures the module for each context in which it is used. Each context is a different use of the module within the AMPS configuration file. For example, the transport module that handles TCP (built into AMPS) can listen on multiple ports. Each port is a different context. An authentication module can configured for different transport, with each transport using different options. In this case, AMPS creates a different context for each transport.

The AMPS server always initializes a context for a module before requesting work from that module. Your module will never receive a request from AMPS for work within a given context unless the context has been initialized. The context initialization method is always called once per context from a single thread, although AMPS may simultaneously initialize multiple contexts from multiple threads.

AMPS does not guarantee a particular ordering for individual context initialization. Although the order generally follows the order of declaration in the configuration file for each type of module, AMPS does not make strong guarantees about precise ordering, and your modules should not depend on any particular ordering for context initialization.

For some types of modules, the AMPS administration interface contains functions that cause a module context to be reinitialized without shutting down AMPS. This is useful in cases where configuration information may have changed – for example, when you need a module that caches permission information to expire the cache immediately. When this happens, AMPS initializes a new context for the module. Once the new context is initialized, AMPS begins using the new context, then destroys the old context.
Servicing requests. Most of the lifetime of the module is spent in this state. When AMPS has work for the module to do – for example, authenticating a new client connection – AMPS calls the appropriate method in the module. When the module has work for AMPS to do – for example, a transport module receiving a new message – the module calls AMPS using the callback provided.

Tip

Your module may service requests on different AMPS threads at the same time. To protect against problems, your module should save any state that will persist from request to request in the context object your module creates.

AMPS calls the method that initializes the module once, and initializes each context once. However, after the module is initialized, multiple threads may be active in the module at the same time. After a context is initialized, a context object may be used by multiple threads at the same time.

AMPS guarantees that each context is destroyed only once, and that the module is shut down only once.
Destruction. When the AMPS server is done with a context, AMPS destroys the context. When AMPS exits, AMPS first destroys the contexts created, then destroys the module. This phase gives the module the opportunity to gracefully shut down and deallocate any resources that need special attention before the module exits. This may include making sure that transport buffers are flushed, or writing out information to a cache file, for example. For some module types, AMPS provides the ability for an administrator to destroy and reload the context. Therefore, it is important for modules to effectively clean up all resources used by a context: a module cannot assume that AMPS is shutting down when a context is destroyed.

AMPS calls specific functions in each part of the lifecycle. The functions called depend on the type of module, as described in the following sections.

If there is no work for your module to do in a certain phase, you can simply return the appropriate status or an empty version of the appropriate object. For example, your administrative action may not need to do anything during module initialization, while your message type may not need to do anything during context initialization. You still need to provide a minimal implementation of the function, or AMPS will exit when it tries to load and use the module.

Creating a Module¶

AMPS modules must be compiled as Linux shared objects and export the appropriate entry points for the module type. Modules are typically implemented in C or C++, although any language that can interoperate with C and create the appropriate shared objects can be used.

To create an AMPS module, you simply include the appropriate header files for the module you are implementing. You then compile the module as a shared object, and include configuration directives for the module in the AMPS configuration file.

The sdk directory of the AMPS distribution contains the headers you need to get started.

Common Header Files¶

amps_module.h This file includes common definitions used throughout AMPS modules.header files common

Common Configuration File Options¶

To use modules in an AMPS installation, you must first load the modules in the Modules section of the configuration file:

<Modules>
    <Module>
        <Name>LDAP-authentication</Name>
        <Library>./lib/lib_ldap_authentication.so</Library>
        <Options>
            <Key1>Value1</Key1>
            <Key2>Value2</Key2>
            <Key3>Value3</Key3>
        </Options>
    </Module>
</Modules>

This element directs AMPS to load the module. Once the module is loaded, AMPS initializes the module and passes the parameters provided to the module – in this case, Key1 = Value1, Key2 = Value2, and Key3 = Value3.

Implementing amps_module_init()¶

The amps_module_init() method has the following signature:

int amps_module_init(amps_module_options options,
                     amps_module_logger logger,
                     amps_module_allocator allocator);

Parameters to the function are:

options: An array of amps_module_options structures. The array is terminated by a structure with a NULL key.
amps_module_logger: A pointer to the logging function AMPS provides for modules.
allocator: A pointer to the memory allocation function AMPS provides for modules. Use this allocator for any memory that your module will return to AMPS, so AMPS can call the correct implementation of free() on the memory.

AMPS calls amps_module_init once for each module loaded, so the function will not be called from multiple threads. In amps_module_init(), you do any global-level initialization your module needs. AMPS modules generally do four things in this function:

Store the provided pointers to the logger and allocator for future use.
Load any external resources required for all instances of the module.
Process the options array, and configure global state based on those options.
Return AMPS_SUCCESS when initialization succeeds.

When you encounter a problem that makes it impossible for your module to do the work it needs to do, return AMPS_FAILURE. AMPS will exit. Your module is responsible for logging details on the problem. Provide as much detail as you can to help troubleshoot the problem.

AMPS does not strictly guarantee the order of initialization for modules, and the server is not yet functional for commands when modules are initialized. At the point of module initialization, the server may not have loaded all of the message types, functions, and so on, so a module should not use the embedded client or message parser during amps_module_init(). Instead, you can register a startup function to run at the end of the initialization process when the main functionality of the server has started.

We recommend that your amps_module_init() method does the minimum work possible to configure the module and ensure that it is possible for the module to run successfully. We recommend doing work that may have inherent intermittent failures when the module context is created or when AMPS issues a request to the module.

For example, a module that does LDAP authentication may need to query different servers for different transports, and those servers may occasionally go offline for maintenance. In this case, returning AMPS_FAILURE if one server was unreachable in initialization would prevent the entire server from starting, even though the problem is intermittent and calls to the other server would succeed. Instead, the authentication module could defer contacting the server until authentication requests arrive, and either return AMPS_FAILURE for the individual request if the server is unreachable, or take some other action depending on the policy the module wants to enforce.

Working With Module Configuration Options¶

AMPS passes configuration options to modules as pointer to an array of amps_option structs. AMPS indicates the end of the array with an option that has a NULL key. Your module processes options until it reaches an option with a NULL key. If no options are specified in the configuration file, the first amps_option in the array will have a NULL key.

The following snippet shows one way to process options:

for (; options->key != NULL; ++options) {
    // process the key/value pair here
}

Depending on the needs of your module, you may choose to store these values for individual contexts to use, or configure global state.

Using amps_logger¶

Your module writes to the AMPS log using the amps_logger pointer provided during initialization.

The following snippet shows how to use amps_logger to write an informational message to the AMPS log:

logger(amps_module_log_level_info, "LDAP Authorization module is initialized.");

The logger provided is guaranteed to produce the correct output when called from multiple threads, so your module does not need to synchronize or protect calls to the logger. The logging levels are described in more detail in the AMPS Configuration Guide. Your module is responsible for creating the string to be logged.

AMPS does not require your module to log, and the exact events logged depend on your environment and the module. Although no logging is required, 60East recommends that you explicitly develop a logging approach.

For example, one approach might be to:

Log an event at amps_module_log_level_info any time your module returns an AMPS_FAILURE that is part of the normal operation of your module. For example, it is generally helpful to log an event at this level if your module refuses an authorization request because the user did not provide the correct password.
Log an event at amps_module_log_level_error any time your module returns an AMPS_FAILURE that is not part of the normal operation of your module. For example, it is generally helpful to log an event at this level if your module refuses an authorization request because the module was unable to reach the server that contains authorization information.
Log an event at amps_module_log_level_critical if your module is unable to initialize, with as much information as you can provide about the failure. AMPS will halt startup if your module does not initialize correctly, so it’s important to provide enough information to be able to quickly diagnose and correct the problem.
Log an event at amps_module_log_level_info when your module is initialized and when a context is initialized. Provide any information that affects the behavior of the module (for example, configuration options, information received from an authorization server, and so on).
Log events at amps_module_log_level_trace for each significant event in the module. For modules, it can be useful to protect trace-level statements so that they are present only in debugging builds of the module, particularly if the events require large amounts of string processing or formatting. (The AMPS server itself contains optimizations to avoid these operations unless trace-level statements are logged, but these optimizations are not available to modules.)
Log an event at amps_module_log_level_developer to provide information for use in developing and debugging the module itself.

The exact logging that you implement depends on your module and environment. Logging that is helpful for some environments may become distracting noise in a different environment. For example, as mentioned above, you might choose to log the entire configuration for the module and detailed information on all events at amps_module_log_level_trace during development and debugging, but reduce those events to amps_module_log_level_developer for modules in production.

Using amps_allocator¶

In some situations, you must allocate memory to be returned to AMPS. For example, some functions allow you to return new information to the client. AMPS is responsible for freeing the memory in these cases, so AMPS provides an allocator for allocating the memory. To allocate the memory for those parameters, use amps_allocator rather than other methods such as the system allocator (malloc) . The typedef for amps_allocator is provided in amps_module.h, with the following signature:

typedef void* (*amps_module_allocator)(unsigned long);

The signature of amps_allocator matches the signature of malloc, so you can substitute amps_allocator in any function that takes a pointer to malloc. The allocator provided is guaranteed to produce the correct results when called from multiple threads, so you do not need to protect or synchronize calls to the allocator.

You do not free memory allocated using the amps_allocator. AMPS handles freeing this memory for memory returned to AMPS. Because you do not free this memory, there is no function provided equivalent to free.

If your module needs to dynamically allocate memory for its own use, the module is responsible for freeing that memory. You allocate that memory with the system allocator (malloc) or equivalent.

Implementing amps_module_terminate()¶

AMPS calls amps_module_terminate when the server shuts down. This gives the module an opportunity to manage any resources that need cleanup before the server exits.

Although AMPS requires that you implement this function, many modules do not need to do cleanup work in this method, and simply return AMPS_SUCCESS.

The function has this signature:

void amps_module_terminate();

This function takes no parameters, and does not allow a return value. AMPS logging is not guaranteed to be active when the AMPS server calls this function in your module, so your module must not use the logger in this function. AMPS calls this function once per module, from a single thread.

Multithreading in AMPS Modules¶

The AMPS server is highly parallelized for performance, and AMPS modules must be work correctly when called from multiple threads at the same time while they are servicing requests. When implementing your module, keep the following guarantees and requirements in mind:

amps_module_init() is called once per module, so only one thread will be active in this function at any time. The module may be used on any number of threads after it is initialized.
The amps_logger and amps_allocator provided to amps_module_init() can be called from multiple threads simultaneously without requiring the module to synchronize calls or protect against concurrent access. The lifetime of these functions is guaranteed to extend past the lifetime of the module, so they are safe to call from context destruction and module termination functions.

The context initialization function for a module is called once per context. Only one thread per context will be active in this function at any time. AMPS reserves the right to initialize more than one context at a time, so more than one thread may enter the context initialization function at a time. The context may be used on any number of threads after it is initialized.

For most modules, AMPS creates a context for each use of the module in the configuration file. For authentication modules, however, AMPS creates a context for each thread listening on the Transport where the authentication module is used.
During the module lifetime, AMPS may call the functions that module provides from any number of threads, and may provide the same context object on multiple threads. The module is responsible for protecting any internal state and the state of the context object as necessary.
The context destruction function for a module is called once per context. Only one thread per context will be active in this function at any time. AMPS reserves the right to destroy more than one context at a time, so more than one thread may enter the context destruction function at a time.
amps_module_terminate() is called once per module, so only one thread will be active in this function at any time. All contexts will be destroyed before this function is called.

Long Running Tasks and AMPS¶

When AMPS calls your module, one of the threads from the AMPS engine is devoted to the work that your module performs. This means that, while the thread is in your module, that thread is not available for other work. This also means that the speed with which your module services requests will directly affect the performance of the AMPS server overall.

For example, if you have an authentication module that uses an external system to check credentials, you must carefully consider the behavior of the module in the event that the remote system is unavailable. Likewise, if you have a message type that relies on an external library for parsing, consider the behavior of the module in the event that AMPS receives a malformed or malicious message that can cause extended parsing times or consume large amounts of memory.

60East recommends setting a timeout and a reasonable failure behavior for operations that may take an extended period of time or that are not under your module’s control. Where possible, 60East recommends setting a reasonable threshold for timeouts, memory consumption, and so forth. The module returns AMPS_FAILURE if a threshold is exceeded, and logs the reason for the failure as a warning or error in the AMPS log.

If your module needs to perform persistent work, you must create and manage the threads that perform that work and allow the thread that calls your module to return to AMPS. For example, if your module will create an embedded client that does extensive processing of messages or that relies on another system such as a RDBMS, 60East recommends creating a background worker thread when the module is initialized and shutting down the thread when the module is destroyed. Failing to return from amps_module_init, a call to an action module, or a message handler in the embedded client in a timely manner can cause AMPS to believe that the thread is stuck and that AMPS is not running normally.

The AMPS Thread Monitor¶

The AMPS thread monitor expects threads created and managed by the AMPS server to periodically report progress. All components within AMPS itself are written with the thread monitor in mind, and typically report progress to the thread monitor before and after operations that may be expensive, or operations that acquire a lock (and, therefore, might block for an extended period of time waiting for the lock).

The server SDK provides a function to report to the AMPS thread monitor that the module is making progress, and should not be considered to have stopped or deadlocked. The function has the following signature.

void amps_thread_monitor_ping();

Calling this function where appropriate can reduce the number of potentially stuck thread warnings in the AMPS log for modules that perform work that takes an extended period of time. When an operation is expected to take more than a few seconds, you can reduce or eliminate warnings by calling amps_thread_monitor_ping() before performing the operation. The thread monitor expects that a thread will ping the thread monitor within approximately 5 seconds of the previous ping.

This function should only be called on a thread created and managed by the AMPS server.

Notice that all this function does is register progress with the thread monitor. Modules that use this function should still avoid long-running operations, since the module is still using a server thread, and a command in the AMPS server or a component of the AMPS server will be waiting for the results of calling the module.

Modules should still avoid doing long-running work on an AMPS thread where possible. If your module needs to make extensive use of amps_thread_monitor_ping(), consider whether the module can defer work, cache work from previous calls to the module, perform work on a different thread, or otherwise improve throughput.

Table Of Contents

Related Topics