25. Operation and Deployment¶
This chapter contains guidelines and best-practices to help plan and prepare an environment to meet the demands that AMPS is expected to manage.
Capacity Planning¶
Sizing an AMPS deployment can be a complicated process that includes many factors including configuration parameters used for AMPS, the data used within the deployment, and how the deployment will be used. This section presents guidelines that you can use in sizing your host environment for an AMPS deployment given what needs to be considered along every dimension: Memory, Storage, CPU, and Network.
Memory¶
Beyond storing its own binary images in system memory, AMPS also tries to store its SOW and indexing state in memory to maximize the performance of record updates and SOW queries.
AMPS needs less than 1GB for its own binary image and initial start up state for most configurations. In the worst-case, because of indexing for queries, AMPS may need up to twice the size of messages stored in the SOW. AMPS maintains a copy of the latest journal file in memory for quick access, and maintains a small amount of metadata for each message in an AMPS queue. The MessageMemoryLimit configured for the instance (or the total of all MessageMemoryLimit settings for each Transport in the instance) specifies the total amount of memory devoted to buffering messages for clients, including conflated subscriptions and aggregated subscriptions.
This puts a general memory consumption estimate for AMPS itself at:
1GB + ( 2S * M ) + ( C * 4096) + ( TMemLimit ) + (J * 2) + (Q * 196 bytes)
Example 25.1: Memory Estimation equation
where:
S = Average message size for topics in the SOWM = Number of messages in all topics in the SOWC = Number of ClientsTMemLimit = Total of all MessageMemoryLimit settings in the instanceJ = JournalSize settingQ = Total number of messages in the queues for the instance
1GB + ( 2 * 1024 * 20,000,000 ) + ( 200 * 4096) + ( 10,000,000,000) +
( 1,000,000,000 * 2 ) + ( 1,000,000 * 196)
Example 25.2: Example memory estimation equation
where:
S = 1024M = 20,000,000C = 200TMemLimit = 10,000,000,000 (10GB)J = 1,000,000,000 (1GB)Q = 1,000,000
An AMPS deployment expected to hold 20 million messages with an average message size of 1KB, that has a total MaxMemoryLimit of 10GB, uses a JournalSize of 1GB, and expects to have a total of 1,000,000 messages active in message queues at any given time and 200 connected clients would consume roughly 53GB.
60East recommends leaving headroom on the system above the capacity estimate for operating system tasks and unexpected traffic. For critical systems that cannot afford downtime even when unexpected events occur, a good recommendation is to allocate 200% of the standard capacity for AMPS, or enough capacity to handle the largest volume increase historically observed for the system. (For example, if the largest volume spike observed was 350% of the typical volume at that time, then planning to be able to handle at least 350% of the typical volume would be important for a critical system).
Storage¶
AMPS needs enough space to store its own binary images, configuration files, SOW persistence files, log files, transaction log journals, and slow client offline storage, if any. Not every deployment configures a SOW or transaction log, so the storage requirements are largely driven by the configuration.
AMPS log files¶
Log file sizes vary depending on the log level and how the engine is
used. For example, in the worst-case, trace logging, AMPS will need
at least enough storage for every message published into AMPS and every
message sent out of AMPS plus 20%.
For info level logging, a good estimate of AMPS log file sizes would
be 2MB per 10 million messages published.
Logging space overhead can be capped by implementing a log rotation strategy which uses the same file name for each rotation. This strategy effectively truncates the file when it reaches the log rotation threshold to prevent it from growing larger.
SOW¶
When calculating the SOW storage, there are a couple of factors to keep
in mind. The first is the average size of messages stored in the SOW,
the number of messages stored in the SOW and the SlabSize defined in
the configuration file. Using these values, it is possible to estimate
the minimum and maximum storage requirements for the SOW:
Min = ( MsgSize * MsgCount ) + ( Cores * SlabSize )
Example 25.3: Minimum SOW Size
where
Min = Minimum SOW SizeMagSize = Average SOW Message SizeMsgCount = Number of Sow MessagesSlabSize = Slab Size for the SOWCores = Number of processor cores in the system
Max = ( MsgSize + SlabSize ) * MsgCount
Example 25.4: Maximum SOW Size
where
Max = Maximim SOW SizeMsgSize = Average SOW Message SizeSlabSize = Slab size for the SOWMsgCount = Number of SOW Messages
The storage requirements should be between the two values above, however
it is still possible for the SOW to consume additional storage based on
the unused capacity configured for each SOW topic. Further, notice that
AMPS reserves the configured SlabSize for each processor core in the
system the first time a thread running on that core writes to the SOW.
For example, in an AMPS configuration file with the SlabSize set to
1MB, the SOW for this topic will consume 1MB per processor core with no
messages stored in the SOW. Pre-allocating SOW capacity in chunks, as a
chunk is needed, is more efficient for the operating system, storage
devices, and helps amortize the SOW extension costs over more messages.
It is also important to be aware of the maximum message size that AMPS guarantees the SOW can hold. The maximum message size is calculated in the following manner:
Max = SlabSize - 64 bytes
Example 25.5: Maximum Message Size allowed in SOW
where
Max = Maximum SOW SizeSlabSize = The configured SlabSize for the SOW
This calculation says that the maximum message size that can be stored
in the SOW in a single message storage is the SlabSize minus 64
bytes for the record header information. AMPS enforces a lower limit of
approximately 1MB: if the maximum size works out to less than 1MB, AMPS
will use 1MB as the maximum size for the topic.
Transaction logs¶
Transaction logs are used for message replay, replication, and to ensure consistency in environments where each message is critical. Transaction logs are optional in AMPS, and transaction logs can be configured for individual topics or filters. When planning for transaction logs, there are three main considerations:
- The total size needed for the transaction log, including in disaster recovery scenarios
- The size to allow for each file that makes up the transaction log
- How many files to preallocate
You can calculate the approximate total size of the transaction log as follows:
Capacity = ( S + 512 bytes ) * N
Example 25.6: Transaction Log Sizing Approximation
where
Capacity = Estimated storage capacity required for transaction logS = Average message sizeN = Number of messages to retain
Size your files to match the aging policy for the transaction log data. To remove data from the transaction log, you simply archive or delete files that are no longer needed. You can size your files to make this easier. For example, if your application typically generates 100GB a day of transaction log, you could size your files in 10GB units to make it easier to remove 100GB increments.
AMPS allows you to preallocate files for the transaction log. For applications that are very latency-sensitive, preallocation can help provide consistent latency. We recommend that those applications preallocate files, if storage capacity and retention policy permit. For example, an application that sees heavy throughput during a working day might preallocate enough files so that there is no need for additional allocation within the working day.
Notice that, if your application uses replication, the AMPS transaction log maintenance actions will not delete unreplicated messages that were initially published to this instance. This means that, when calculating the maximum storage space required, the recovery window for a failure is also important. For example, many systems have a policy of not restarting a failed system until a scheduled maintenance window: if one server in a replicated set of servers could, potentially, be offline for up to 8 hours, then the other servers must be able to store a minimum of 8 hours of journals, even in cases where the normal retention period would be shorter.
Other Storage Considerations¶
The previous sections discuss the scope of sizing the storage, however scenarios exist where the performance of the storage devices must also be taken into consideration.
One such scenario is the following use case in which the AMPS transaction log is expected to be heavily used. If performance greater than 50MB/second is required out of the AMPS transaction log, experience has demonstrated that flash storage (or better) would be recommended. Magnetic hard disks lack the performance to produce results greater than this with a consistent latency profile.
CPU¶
SOW queries with content filtering make heavy use of CPU-based operations and, as such, CPU performance directly impacts the content filtering performance and rates at which AMPS processes messages. The number of cores within a CPU largely determines how quickly SOW queries execute.
AMPS contains optimizations which are only enabled on recent 64-bit x86 CPUs. To achieve the highest level performance, consider deploying on a CPU which includes support for the SSE 4.2 instruction set.
To give an idea of AMPS performance, repeated testing has demonstrated that a moderate query filter with 5 predicates can be executed against 1KB messages at more than 1,000,000 messages per second, per core on an Intel i7 3GHz CPU. This applies to both subscription based content filtering and SOW queries. Actual messaging rates will vary based on matching ratios and network utilization.
Network¶
When capacity planning a network for AMPS, the requirements are largely dependent on the following factors:
- average message size
- the rate at which publishers will publish messages to AMPS
- the number of publishers and the number of subscribers.
AMPS requires sufficient network capacity to service inbound publishing as well as outbound messaging requirements. In most deployments, outbound messaging to subscribers and query clients has the highest bandwidth requirements due to the increased likeliness for a “one to many” relationship of a single published message matching subscriptions/queries for many clients.
Estimating network capacity requires knowledge about several factors, including but not limited to: the average message size published to the AMPS instance, the number of messages published per second, the average expected match ratio per subscription, the number of subscriptions, and the background query load. Once these key metrics are known, then the necessary network capacity can be calculated:
R * S ( 1 + M * S ) + Q
Example 25.7: Network capacity formula
where
R = RateS = Average Message SizeM = Match RatioQ = Query Load
where “Query Load” is defined as:
Mq * S * Qs
where
Mq = Messages per QueryS = Average Message SizeQs = Queries Per Second
In a deployment required to process published messages at a rate of 5000 messages per second, with each message having an average message size of 600 bytes, the expected match rate per subscription is 2% (or 0.02) with 100 subscriptions. The deployment is also expected to process 5 queries per 1 minute ( or 12 queries per second), with each query expected to return 1000 messages.
5000 * 600 B * ( 1 + 0.02 * 100 ) + ( 1000 * 600 B * 1/12 ) ~ 9 MB / s ~ 72 Mb / s
Based on these requirements, this deployment would need at least 72Mb/s of network capacity to achieve the desired goals. This analysis demonstrates AMPS by itself would fall into a 100Mb/s class network. It is important to note, this analysis does not examine any other network based activity which may exist on the host, and as such a larger capacity networking infrastructure than 100Mb/s would likely be required.
NUMA Considerations¶
AMPS is designed to take advantage of non-uniform memory access (NUMA). For the lowest latency in networking, we recommend that you install your NIC in the slot closest to NUMA node 0. AMPS runs critical threads on node 0, so positioning the NIC closest to that node provides the shortest path from processor to NIC.
Linux Operating System Configuration¶
This section covers some settings which are specific to running AMPS on a Linux Operating System.
ulimit¶
The ulimit command is used by a Linux administrator to get and set
user limits on various system resources.
ulimit -c
It is common for an AMPS instance to be configured to consume gigabytes of memory for large SOW caches. If a failure were to occur in a large deployment it could take seconds (maybe even hours, depending on storage performance and process size!) to dump the core file. AMPS has a minidump reporting mechanism built in that collects information important to debugging an instance before exiting. This minidump is much faster than dumping a core file to disk. For this reason, it is recommended that the per user core file size limit is set to 0 to prevent a large process image from being dumped to storage.
ulimit -n
The number of file descriptors allowed for a user running AMPS needs to be at least double the sum of counts for the following: connected clients, SOW topics and pre-allocated journal files. Minimum: 4096. Recommended: 32768, or the value recommended by AMPS in any diagnostic messages, whichever is greater
/proc/sys/fs/aio-max-nr¶
Each AMPS instance requires AIO in the kernel to support at least 16384
plus 8192 for each SOW topic in simultaneous I/O operations. The setting
aio-max-nr is global to the host and impacts all applications. As
such this value needs to be set high enough to service all applications
using AIO on the host. Minimum: 65536. Recommended: 1048576
To view the value of this setting, as root you can enter the following command:
cat /proc/sys/fs/aio-max-nr
To edit this value, as root you can enter the following command:
sysctl -w fs.aio-max-nr = 1048576
This command will update the value for /proc/sys/fs/aio-max-nr and
allow 1,048,576 simultaneous I/O operations, but will only do so until
the next time the machine is rebooted. To make a permanent change to
this setting, as a root user, edit the /etc/sysctl.conf file and
either edit or append the following setting:
fs.aio-max-nr = 1048576
/proc/sys/fs/file-max¶
Each AMPS instance needs file descriptors to service connections and maintain file handles for open files. This number needs to be at least double the sum of counts for the following: connected clients, SOW topics and pre-allocated journal files. This file-max setting is global to the host and impacts all applications, so this needs to be set high enough to service all applications on the host. Minimum: 262144 Recommended: 6815744
To view the value of this setting, as root you can enter the following command:
cat /proc/sys/fs/file-max
To edit this value, as root you can enter the following command:
sysctl -w fs.file-max = 6815744
This command will update the value for /proc/sys/fs/file-max and
allow 6,815,744 concurrent files to be opened, but will only do so until
the next time the machine is rebooted. To make a permanent change to
this setting, as a root user, edit the /etc/sysctl.conf file and
either edit or append the following setting:
fs.file-max = 6815744
/proc/sys/vm/max_map_count¶
AMPS makes extensive use of memory mapped files, and frequently modifies the
maps. The /proc/sys/vm/max_map_count parameter sets the maximum number of
maps that the Linux kernel will allow for a process. If the number of
requested maps exceeds the number of maps in this parameter, memory allocation
operations can fail even when there is sufficient memory available.
This setting is global to the host and applies to all applications, so this needs to be set high enough for the most map-intensive application on the host.
Minimum: 65530 Recommended: 500000
To edit this value, as root you can enter the following command:
sysctl -w vm.max_map_count=500000
This command will update the value for /proc/sys/vm/max_map_count and
allow 500,000 maps to be created, but will only do so until
the next time the machine is rebooted. To make a permanent change to
this setting, as a root user, edit the /etc/sysctl.conf file and
either edit or append the following setting:
vm.max_map_count=500000
Upgrading an AMPS Installation¶
This chapter describes how to upgrade an existing installation of AMPS. The steps presented here focus on upgrading the installation itself, and should be the only steps you need for upgrades that change the HOTFIX version number or the MAINTENANCE version number (as described in Table 25.2).
For changes that update the MAJOR or MINOR version number, AMPS may add features, change file or network formats, or change behavior. For these upgrades, you may need to make changes to the AMPS configuration file or update applications to adapt to new features or changes in behavior.
60East recommends maintaining a test environment that you can use to test upgrades, particularly when an upgrade changes MAJOR or MINOR versions and you are taking advantage of new features or changed behavior.
When the AMPS instance participates in replication, you must coordinate the instance upgrades when upgrading across AMPS versions. AMPS replication works between instances with the same major and minor version number (for example, all AMPS 3.9 releases use the same version of replication, but the 4.0 releases use a different version of replication.) When the AMPS instance participates in replication, you must coordinate the instance upgrades when upgrading across AMPS versions. AMPS replication works between instances with the same major and minor version number (for example, all AMPS 3.9 releases use the same version of replication, but the 4.0 releases use a different version of replication.)
Upgrade Steps¶
Upgrading an AMPS installation involves the following steps:
- Stop the running instance
- If necessary, upgrade any data files or configuration files that you want to retain
- If necessary, update any applications that will use new features
- Install the new AMPS binaries
- Restart the service
As mentioned above, if you are using replication, and the upgrade increments the MAJOR or MINOR version number, you must upgrade all of the instances that replicate at the same time for replication to suceed. This is typically accomplished with a rolling upgrade, where instances are upgraded on a specific schedule to minimize downtime.
Upgrading AMPS Data Files¶
AMPS may change the format and content of data files when upgrading across versions, as specified by the major and minor version number. This most commonly occurs when new features are added to AMPS that require different or additional information in the persisted files. The HISTORY file for the AMPS release lists when changes have been made that require data file changes.
In general, 60East recommends upgrading the data files whenever moving to a new major/minor version and whenever a data file change is mentioned in the HISTORY file.
The AMPS distribution includes the amps_upgrade utility to process
and upgrade data files. The version included with each release of AMPS
upgrades previous versions of the data files to the version of AMPS that
includes the utility. For example, the version of amps_upgrade
included in version 4.1 of AMPS upgrades files to the 4.1 version the
data files.
AMPS versions may upgrade any of the following types of files:
- journals - these files contain the transaction logs for the instance
- clients.ack - this file contains a cache of the last sequence number processed for a publisher
- sow files - these files contain the persisted state of the durable SOW topics for the instance
The amps_upgrade utility handles upgrades for each of these types of
files. Full details on amps_upgrade are available in the AMPS
Utilities Guide.
Best Practices¶
This section covers a selection of best practices for deploying AMPS.
Monitoring¶
AMPS exposes the statistics available for monitoring via a RESTful interface, known as the Monitoring Interface, which is configured as the administration port. This interface allows developers and administrators to easily inspect various aspects of AMPS performance and resource consumption using standard monitoring tools.
At times AMPS will emit log messages notifying that a thread has encountered a deadlock or stressful operation. These messages will repeat with the word “stuck” in them. AMPS will attempt to resolve these issues, however after 60 seconds of a single thread being stuck, AMPS will automatically emit a minidump to the previously configured minidump directory. This minidump can be used by 60East support to assist in troubleshooting the location of the stuck thread or the stressful process.
Another area to examine when monitoring AMPS is the last_active
monitor for the processors. This can be found in the
/amps/instance/processors/all/last_active url in the monitoring
interface. If the last_active value continually increases for more
than one minute and there is a noticeable decline in the quality of
service, then it may be best to fail-over and restart the AMPS instance.
Stopping AMPS¶
To stop AMPS, ensure that AMPS runs the amps-action-do-shutdown
action. By default, this action is run when AMPS receives SIGHUP,
SIGINT, or SIGTERM. However, you can also configure an Action to
shut down AMPS in response to other conditions. For example, if your
company policy is to reboot servers every Saturday night, and AMPS is
not running as a system service (or daemon), you could schedule an AMPS
shutdown every Saturday before the system reboot.
When AMPS is installed to run as a system service (or daemon), AMPS installs shutdown scripts that will cleanly stop AMPS during a system shutdown or reboot.
SOW Parameters¶
Choosing the ideal SlabSize for your SOW topic is a balance between
the frequency of SOW expansion and storage space efficiency. A large
SlabSize will preallocate space for records when AMPS begins writing
to the SOW.
If detailed tuning is not necessary, 60East recommends leaving the
SlabSize at the default size if your messages are smaller than the
default SlabSize. If your messages are larger than the default
SlabSize, a good starting point for the SlabSize is to set it to
several times the maximum message size you expect to store in the SOW.
There are three considerations when setting the optimium SlabSize:
- Frequency of allocations
- Overall size of the SOW
- Efficient use of space
A SlabSize that is small results in frequent extensions of your SOW
topic to occur. These frequent extensions can reduce throughput in a
heavily loaded system, and in extreme cases can exhaust the kernel limit
on the number of regions that a process can map. Increasing the
SlabSize will reduce the number of allocations.
When the SlabSize is large, then the risk of the SOW resize
affecting performance is reduced. Since each slab is larger, however,
there will be more space consumed if you are only storing a small number
of messages: this cost will amortize as the number of messages in the
SOW exceeds the number of cores in the system * the number of
messages that fit into a slab.
To most efficiently use space, set a SlabSize that minimizes the
amount of unused space in a slab. For example, if your message sizes are
average 512 bytes but can reach a maximum of 1.2 MB, one approach would
be to set a SlabSize of 2.5MB to hold approximately 5 average-sized
messages and two of the larger-sized messages. Looking at the actual
distribution of message sizes in the SOW (which can be done with the
amps_sow_dump utility) can help you determine how best to size slabs
for maximum space efficiency.
For optimizing the SlabSize, determine how important each aspect of
SOW tuning is for your application, and adjust the configuration to
balance allocation frequency, overall SOW size, and space to meet the
needs of your application.
Slow Clients¶
As described in Slow Client Management, AMPS provides capacity limits for slow clients to reduce the memory resources consumed by slow clients. This section discusses tuning slow client handling to achieve your availability goals.
Slow Client Offlining for Large Result Sets¶
The default settings for AMPS work well in a wide variety of applications with minimal tuning.
If you have particularly large SOW topics, and your application is disconnecting clients due to exceeding the offlining threshold when the clients retrieving large SOW query result sets, 60East recommends the following settings as a baseline for further tuning:
| Parameter | Recommendation |
|---|---|
MessageMemoryLimit |
This controls the maximum memory consumed by AMPS for client messages. You can increase this parameter to allow AMPS to use more memory to records. Notice, however, that memory devoted to client messages is unavailable for other purposes. Recommended starting point for
tuning large result sets: 10%.
60East recommends tuning the
|
MessageDiskLimit |
The maximum amount of space to consume for offline messages. Recommended starting point for
tuning large result sets: Average
record size * number of expected
records * number of simultaneous
clients, or |
MessageDiskPath |
The path in which to store offline message files. 60East recommends that the message
disk path be hosted on fast,
high-capacity storage such as a
PCIe-attached flash drive. The
available storage capacity of the
disk must be greater than the
configured |
Table 25.1: Client Offline Settings for Large Result Sets
60East recommends that you use these settings as a baseline for further tuning, bearing in mind the needs and expected messaging patterns of your application.
Minidump¶
AMPS includes the ability to generate a minidump file which can be used in support scenarios to attempt to troubleshoot a problematic instance.
The minidump captures thread state information: a snapshot of where in the source code each thread is, the call stack for each thread, and the register information for each frame of the call stack. A minidump also contains basic information about the system that AMPS was running on, such as the processor type and number of sockets. Minidumps do not contain the full internal state of AMPS or the full contents of application memory. They do not contain detailed information about the host system, and have no information about the state of the host or operating system. Instead, minidumps identify the point of failure to help 60East quickly narrow down the issue without generating large files or potentially compromising sensitive data.
Minidumps can be produced much faster than a standard core dump, and use significantly less space since the minidump contains only a small subset of the information a core dump would contain (see Ulimit for more configuration options). Because of this, the AMPS server may produce minidumps for temporary conditions that the server subsequently recovers from, and allows creation of a minidump on demand.
Generation of a minidump file occurs in the following ways:
- When AMPS detects a crash internally, a minidump file will automatically be generated. This includes cases where an AMPS thread or critical internal component has not reported progress for an extended period of time (typically 300 seconds).
- When a user clicks on the
minidumplink in theamps/instance/administratorlink from the administrator console (see the AMPS Monitoring Reference for more information). - By sending the running AMPS process the
SIGQUITsignal. - In response to a configured action.
- If a thread fails to report progress with the AMPS thread monitor for 60 seconds, a minidump will automatically be generated. This should be sent to AMPS support for evaluation along with a description of the operations taking place at the time.
By default the minidump is configured to write to /tmp, but this can be
changed in the AMPS configuration by modifying the MiniDumpDirectory.
60East recommends monitoring the minidump directory.
If minidumps occur, contact 60East support for diagnosis and troubleshooting. Bear in mind that minidumps are often a symptom of a slowdown in the server due to resource constraints rather than an indication that the server has exited.
Once a minidump is submitted to 60East (and acknowledged as received), there is no further need to retain that minidump. 60East recommends removing minidumps when they are no longer needed.