9. Message Type Modules

What’s A Message Type Module?

Message type modules give AMPS the ability to process the contents of messages. AMPS includes implementations of FIX, NVFIX, JSON and XML. To process other types of messages, you implement a message type module.

Message type modules implement four basic sets of capabilities. These sets of capabilities are:

• Topic based routing. These functions provide the basic infrastructure for loading the module. Since topic routing is managed without needing to parse the message, simply allowing a module to be initialized provides the ability to route messages by topic.
• Content filtering and SOW persistence. To provide content filtering, the AMPS module interface specifies a pair of functions that parse the content of the message into the data structures that AMPS uses for filtering. These functions also allow AMPS to find the SOW key in a message so that State of the World databases can function correctly.
• Aggregation, Joins, Views, and Delta Subscribe. The AMPS module interface specifies a set of functions used to create messages from value sets. These functions translate AMPS data structures back into the message format. The functions allow AMPS to create messages of the message type, which enables aggregation, joins and views, and delta subscription. These functions can also enable AMPS status messages (for example, messages to the /AMPS/ClientStatus topic) for the message type.
• Delta publish. This function allows AMPS to accept delta publishes by merging a partial message into an existing message.

Each set of capabilities builds on the previous set. For example, to provide aggregation, a module must also provide content filtering and module initialization. The following table lists the functions for each level of capability.

To enable:	Implement:
Topic based routing	amps_module_init amps_module_terminate amps_message_type_create_context amps_message_type_destroy_context
Content filtering and SOW persistence	amps_message_type_partial_parse_message amps_message_type_parse_message
Aggregation, JOIN, View, Delta Subscribe	amps_message_type_compute_serialize_size amps_message_type_serialize_message
Delta Publish	amps_message_type_delta_publish_merge
Formatting for Log Messages	amps_message_type_trace_message

Table 9.1: Message Type Capabilities and Functions

When to Implement a Message Type Module

Implement a message type module when you need to provide content filtering, aggregation, or delta publishing for an existing native message format that AMPS does not currently support. If you need only topic-based filtering, you can use the binary message type. If you can extract relevant information out of the message body into a preamble, you can use the composite message types to provide filtering (and so forth) over the preamble, and use binary for the original message. In many cases, a composite message type that uses this pattern will outperform a custom message type, since AMPS only needs to parse the preamble.

However, if you have a need to provide a message payload format that is currently in use by a larger system and is not currently supported by AMPS, and if your system requires filtering, views and aggregation, or delta messaging over the entire message rather than a subset of the message, creating a message type module can be a good option.

Message type modules are performance-critical components for AMPS: the speed with which your module parses and serializes messages will directly affect how quickly AMPS can process messages of that type. Pay careful attention to the performance of your module. Minimize the amount of work your message type module performs during parsing, avoid unnecessary allocation or data copies, and minimize parsing when possible.

Caution

Message type modules are performance-critical components of AMPS. The speed of your message type module will directly affect the overall speed and throughput of your AMPS instance when processing messages of that type.

Module Initialization & Topic Based Routing

To provide topic based routing, a module simply needs to provide the functions that allow AMPS to initialize the module. Because topic based routing does not require interpretation of the message body, there is no need to provide functions that parse the message body.

AMPS provides the ability to configure message types globally, when the module loads, or in each use of the message type. Because the same module may be used with different parameters on different transports, AMPS allows you to save the parameters in an message type context.

Your module implements the amps_message_type_create_context function to create a message type context. The function has this signature:

amps_message_type_context amps_message_type_create_context(const amps_module_options options);

The message_type_context type is a pointer to data that your module defines. The AMPS server does not process or interpret this data. The server receives the pointer from your module when the context is initialized, and then provides the pointer back to your module with requests to the module.

If your module is unable to successfully create the context, you return a NULL context and use the amps_logger provided to your module to log information about the problem.

Notice that you cast the pointer to amps_message_type_context before you return it, and cast the pointer back to the type that your application uses after AMPS returns the pointer to you.

AMPS passes configuration options to modules as a pointer to an array of amps_option structs. AMPS indicates the end of the array with an option that has a NULL key. Your module processes options until it reaches an option with a NULL key. If no options are specified in the configuration file, the firstamps_option in the array will have a NULL key.

The following snippet shows one way to process options:

for (; options->key != NULL; ++options) {
    // process the key/value pair here
}

When AMPS exits, AMPS calls amps_authentication_destroy_context with each of the authentication contexts created. This gives your module an opportunity to do any resource management, cleanup or logging necessary before AMPS shuts down. The function has this signature:

int amps_message_type_destroy_context(amps_message_type_context context);

Declaring Supported Operations

By default, AMPS uses the functions that a message type provides to determine the operations that the message type supports. For general purpose message types, this works well. For some types, however, not all of the operations may be appropriate. For example, a message type that supports a specific set of tags may want to support serialization for views, but not for /AMPS/SOWStats messages. For cases like this, AMPS allows a module to explicitly declare the supported operations.

When the function amps_message_type_get_supported_options is present, AMPS calls the function for each context after context initialization returns. The function returns a string that contains a comma-delimited list of options supported by the module for that context. The options are as follows:

Option	Capability
sow_stats	This context supports serializing /AMPS/SOWStats messages
delta	This context supports serializing delta messages.
view	This context supports serializing messages for views.
client_status	This context supports serializing /AMPS/ClientStatus messages
stop_timer	This context supports serializing StopTimer messages.
select	This context supports serializing partial messages when a client uses the select option on a query or subscription.

Table 9.2: Message Type Capabilities

AMPS does not free the memory returned from this function: generally, modules that use this capability return a pointer to a static string. When a module implements this method, AMPS assumes that the module does not support any option that is not returned from the function, regardless of whether the module contains the functions that would support it.

AMPS Identifiers

AMPS locates values in a message using a compact, XPath-based syntax, as described in the AMPS User’s Guide. While creating your message type module, you decide how to interpret values in that syntax. The AMPS server does not enforce any specific semantics for an identifier: your module decides how to interpret the identifier.

There are two guidelines for how your module interprets identifiers:

• Your module should interpret the identifiers in a consistent way, and interpret the same identifier the same way every time. For example, if the module interprets addresses/0 as the first element of an array of addresses at the root level of the message, your module should interpret /addresses/1 as the second element.
• If your module implements message creation, your module should produce output that can be parsed consistently. For example, if your module interprets the message a=b as containing an element of /a with a value of b, your module should not produce a different format when asked to serialize an identifier of /a with a value of b. For simple message types, this is straightforward. For complex hierarchies or complicated translations, we recommend writing test cases to ensure that your module is producing the expected output.

For example, consider a message type that uses a fixed schema, and knows in advance the length and location for a given identifier. In this case, there would be no need for the message type to actually traverse the message to provide values to AMPS.

AMPS Values and Types

AMPS includes a lightweight, efficient type system for values within messages. This type system helps AMPS correctly filter and aggregate messages.

If your module provides content filtering capabilities or any of the capabilities that depend on content filtering, it’s important to set the type of parsed values correctly. Otherwise, AMPS performs comparisons based on the type you provide, which can produce unexpected results. For example, a string comparison "020" < "19" has a different result than a numeric comparison 20 < 19.

The AMPS type system has two basic types:

• String values. These values are represented as a char* pointing to the first character of the string and a length. Notice that, in AMPS modules, strings are not NULL-terminated. This enables string values within AMPS to handle embedded NULLs, provided that the message type module handles these values correctly.

• Numeric values. These values are represented as a union that contains the following C++ types: unsigned long, long, double, or bool.

  When parsing a message, the message type module is responsible for identifying the most appropriate type for the value and setting the value appropriately. When serializing a numeric value, a message type module determines the type of the saved value, retrieves the value, and formats the value for serialization based on the type.

For messages that you parse, your module determines the type of the parsed value and sets the amps_message_value accordingly. For messages provided by AMPS, AMPS provides the value and the type.

Once a parsed value is returned to AMPS, the server handles type conversion as necessary. A module does not have to explicitly provide type conversion, and a message type module cannot set restrictions on type conversion within the AMPS expression engine.

Parsing, Content Filtering and SOW

AMPS represents the contents of a parsed message as a collection of value_set_entry structs. Each value_set_entry contains the value itself, type information about the value, and an XPath-based identifier that represents where the value occurs in the message. Your message type module is responsible for taking the data in the message and parsing it into value_set_entry objects.

AMPS provides your module with the unparsed message, a value set to hold the parsed values, and (optionally) a set of tags that indicate values that AMPS currently needs to find in the message.

The basic outline of AMPS parsing is as follows:

1. Find a tag and value in the message.
2. Create a value_set_entry and set the properties of the entry.
3. Add the value_set_entry to the value set and store the index returned.
4. Use the index returned to indicate the document hierarchy, if necessary.
5. If a set of values to find has been provided, your module can optionally check to see whether all of the values needed have been parsed. If all of the needed values have been parsed, your module can choose to finish parsing.

Message Parsing Example

You decide how to map the message format into XPath in way that is convenient and intuitive for your users. In general,a straightforward mapping is easiest to code and easiest for users to understand.

For example, consider the following simple message format:

messageId=5, customerId=20, notes=Have a nice day.

This message format uses simple, non-hierarchical tag=value pairs. Each pair is separated by a comma. For this message, our message type module will produce three value_set_entry structs that contain the following identifiers and values:

Tag	Value
messageId	5
customerId	20
notes	Have a nice day.

Table 9.3: Flat Format Message Identifiers and Values

A similar approach works for more complicated formats:

{
    "messageId" : 5,
    "customerInfo" : {
        "customerId" : 20,
        "notes" : "Have a nice day."
    }
}

Tag	Parent	Value
messageId		5
customerInfo		{ “customerId” : 20, “notes” : “Have a nice day.” }
customerId	customerInfo	20
notes	customerInfo	Have a nice day.

Table 9.4: Hierarchical Format Example Identifiers and Values

The mapping is similar to the flat message approach. In particular, notice that AMPS does not require a particular parsed value for the customerInfo node. AMPS uses the “leaf” nodes of the hierarchy for comparison on the leaf nodes.

Creating Value Set Entries

The output of your parse function is a set of XPath entries contained in a XPath value set. This section describes the entries and presents the mechanics of creating XPath entries.

For each value in the message, you need to determine the following information:

• Tag This is the name of the value. For the tag, you need a pointer to the start of the tag and the length of the tag.
• Raw data Pointer and length for the raw data. This indicates the position in the message that holds the data for this value set entry.
• Type Type of the value. Within AMPS, all values are typed. It is up to your module to determine how to map the values in a message to the AMPS type system. See AMPS Values and Types for details on the AMPS type system.
• Native value If the value is numeric, the value represented as a C native numeric type (long or double)
• String representation The string representation of the value. For some message types and values, this is the same as the raw data. However, if the data requires escaping, or if the raw data is not suitable for representation as a string, you may need to create a string representation. For example, if your message type sends numeric values as binary values rather than as strings, you can create a string equivalent.
• Index within the value set. This property is assigned after you add a new entry to the value set. You then use this index to provide AMPS information about this value’s place in the document hierarchy

For example, you might parse the following JSON document:

{
    "messageId" : 5,
    "customerInfo" : {
        "customerId" : 20,
        "notes" : "Have a nice day."
    }
}

into the following value set entries:

Tag	Value	Type	Value Set Index (set by AMPS)	Parent	Next Sibling
messageId	5	Long	1	~0	0
customerInfo	{ “customerId” : 20, “notes” : “Have a nice day.” }	String	2	~0	0
customerId	20	Long	3	2	4
notes	Have a nice day.	String	4	2	0

Table 9.5: Value Set Entries

Each of the required pieces of information is a property on the entry object.

Setting Numeric Values

The AMPS server SDK provides functions to help you create XPath entries. To create an entry, you decide the type of value (as described in section AMPS Values and Types. You then call the helper function appropriate for the type.

For example, a message type module that recognizes two numeric types, long and double, might use the following code to set entries on numeric values, based on whether the parser believes the value is a long or a double:

// snippet to show sample use of
// amps_message_value_set_entry_set_<type> functions

// numStr is a C-style string that contains
// the value.

// entry is a pointer to amps_message_set_value_entry

// isDouble is a bool that indicates whether the module
// has parsed the value as a double or a long.
if (isDouble) {
    double d = strtod(numStr, NULL);
    amps_message_value_set_entry_set_double(entry, d);
}
else {
    long l = strtol(numStr, NULL, 10);
    amps_message_value_set_entry_set_long(entry, l);
}

Working With Hierarchical Documents

Value set entries are inherently hierarchical. This means that, for each entry, AMPS requires you to set the identity of the parent node. If an entry has no parent (that is, the entry is at the top level of the message), AMPS requires you to set the parent to the special value ~0.

If the entry has children or siblings, AMPS requires that you set the identity of those nodes, as well. If the entry has children or siblings, AMPS requires that you set the identity of those nodes, as well.

// Sample of setting parent and child nodes.

// root is the top of the document
// child_one is the first child, child_two is the second child
// All nodes are of type amps_message_value_set_entry, and
// are constructed by your parser as it parses.

root.parent = ~0;

// Add entry, child_one, and child_two to the value set
// and set the appropriate pointers.

uint32_t root_handle = amps_message_xpath_update(valueSet, findSet, &root);
uint32_t child_one_handle = amps_message_xpath_update(valueSet, findSet, &child_one);
uint32_t child_two_handle = amps_message_xpath_update(valueSet, findSet, &child_two);

// Now that we have the handles, use them to set up the relationships.

amps_message_xpath_get(valueSet, root_handle)->first_child = child_one_handle;
amps_message_xpath_get(valueSet, child_one_handle)->parent = root_handle;
amps_message_xpath_get(valueSet, child_one_handle)->next_sibling = child_two_handle;
amps_message_xpath_get(valueSet, child_two_handle)->parent = root_handle;

Representing Arrays

The AMPS type system does not provide a distinct type for arrays or repeating values. Instead, repeating values are represented as a collection of value objects with the same path at the same level of the document hierarchy. To represent this, each value in the array has the same parent and the same hash. The next_sibling for each element is set to point to the index for the next element.

Notice that this means that the following two JSON documents have approximately the same representation as parsed documents in AMPS:

{ "contents" : [ {"a" : 1 }, { "a" : 2 }, { "a" : 3} ] }

{ "contents" : "a" : [1, 2, 3] }

In both cases, the documents contain three values for the XPath identifier /contents/a. Those values are 1, 2, 3, in order.

Implementing the Functions

AMPS requires a pair of functions to implement content-based routing. Each function receives an input message and a value set to populate with the parsed contents of the message. Once the value set is populated, or if message parsing fails, the function returns.

These functions produce identical results: the difference is that one function requests that your module parse the message completely, while the other function allows your module to partially parse the message – which can be more efficient for some message types.

To provide content filtering, you implement one or both of the following two functions:

int amps_message_type_parse_message(amps_input_message *inputMessage,
                                    amps_message_xpath_value_set valueSet);

int amps_message_type_partial_parse_message(amps_input_message* inputMessage,
                                            const amps_message_xpath_find_set findSet,
                                            amps_message_xpath_value_set valueSet);

Each function takes the message provided in inputMessage, and populates the provided valueSet with the parsed contents of the message. Once a value_set_entry is populated, you add the entry to the valueSet using the provided amps_message_xpath_update function.

For example, when entry is a populated value_set_entry, a call with the findSet and valueSet passed into partial_parse_message might look like:

amps_message_xpath_update(valueSet, findSet, &entry);

The function returns the number of fields added to the valueSet if parsing succeeds, a negative value if parsing fails for any reason.

Notice that your code does not actually implement comparisons, queries or filtering. AMPS uses the populated valueSet to perform filtering and queries over the messages.

Saving Parsed Values

AMPS has two mechanisms for saving user-defined state when working with messages. These are intended to be used for saving parsed values: while AMPS does not interpret the data you save in the messages, AMPS manages the lifetime of the data in a way that’s consistent with treating this data as parsed state.

Maintaining State for a Value Set

Some message types may be expensive to parse, and for those types, a message type module may need to parse the message once, and maintain a parsed representation of the message. AMPS allows modules to store user-defined data as a part of a value set and to retrieve this data from the value set. AMPS also provides a function so to release the reference to the data, so that the module can free the memory for the data.

To set user-defined data on a value set, use the amps_message_xpath_set_user_data function.

void amps_message_xpath_set_user_data(amps_message_xpath_value_set valueSet_, void* userData_);

As the signature implies, this function takes an amps_message_xpath_value_set and an arbitrary pointer, and stores that pointer in the value set. To retrieve the value, use the amps_message_xpath_get_user_data function.

void* amps_message_xpath_get_user_data(amps_message_xpath_value_set valueSet_);

For example, an implementation of one of the parse functions might check to see whether the message has been previously parsed this way:

...

parsed_message_representation *msg = amps_get_user_data(valueSet);

if (0 == msg)
{
    msg = custom_parser(inputMessage);
}

// use msg to update the valueSet

...

// store msg for possible future use

amps_message_set_user_data(valueSet, (void*)msg);

...

When a value set contains user data, AMPS calls amps_message_type_free_user_data_function with the data when AMPS no longer uses it. For example, assuming that the parse_message function given earlier has a corresponding free_message function to release memory, the module might implement that function as follows:

void amps_message_type_free_user_data_function(void* msg)
{
    // AMPS is done with the message, so release it.
    free_message((parsed_message_representation*) msg);
}

Maintaining State for an Individual Value

AMPS also allows you to store state in an individual value. The amps_message_value_set structure contains three pointers that can be used to save user data for that individual value. For example, your module may want to store a pointer to an object representation of the value. To do this, you simply set one of the user data pointers in the value.

AMPS assumes that these pointers reference memory that is tracked outside of the value itself, typically the parsed representation of the message managed by set user data as described above. AMPS does not provide a mechanism for freeing the memory referenced by an individual value. Instead, AMPS expects that the memory is freed as a part of the call to free_user_data, or otherwise managed by the module.

Aggregation, Joins, Views and Delta Subscribe

To provide aggregation, joins, and views, AMPS must be able to construct messages from value sets. This process is the reverse of parsing process. In this case, AMPS provides a set of XPath identifiers with corresponding values, and your module creates an output message.

Your module implements the following pair of functions:

int amps_message_type_compute_serialize_size(amps_output_message_data *outputMessage);

int amps_message_type_serialize_message(amps_output_message_data *outputMessage);

AMPS calls these functions in sequence to guarantee that the serialization function can write the output message without requiring memory reallocation, regardless of the size of the message.

AMPS follows these steps:

1. AMPS first calls amps_message_type_compute_serialize_size with an amps_output_message_data pointer that contains the value set to be serialized. When AMPS calls this function, the output pointer for the amps_output_message_data points to a buffer of unknown size. Your implementation calculates the size of the message and sets this size in the serialize_size member of the output message.

   Your module can do as much work as necessary to compute the size. If the work your module does would need to be repeated for serialization, you can save the results of that work in a set of user_data pointers held in the outputMessage. AMPS leaves the user_data pointers untouched when it provides the outputMessage to the serialization function. For example, if your module assembles a hierarchical document or escapes values, you can save the results of that work in the outputMessage and use that data for serialization.

2. AMPS then allocates a buffer guaranteed to be large enough to hold the number of bytes returned by amps_message_type_compute_serialize_size. AMPS sets the output pointer of the outputMessage to point to that buffer.

3. AMPS calls amps_message_type_serialize_message, providing the updated outputMessage. Your module writes the serialized message to the output buffer.

The simple message type example in the SDK includes a complete implementation of these functions.

Notice that, in both cases, the provided outputMessage holds the following types of data, which your module assembles into a serialized output message:

• The headers set by the protocol. These are already written to the output message, and the length of the output message is set accordingly. Your module skips past these headers before it begins writing to the output message. You do not modify these headers, but your module must be aware that this data exists and avoid overwriting it.

• Output serialized by other functions (such as delta message functions). This data is in the data message segment provided in the outputMessage. It has not been written to the output message. Your module decides how to write this data to the output message and increments the message length by the number of bytes written. For some message formats, this is a simple copy. For other message formats, your module may need to modify this data. For example, an XML-based message type may need to write the unserialized values before the closing tag in this data.

• Unserialized values. Unserialized values are held in an array of amps_message_value objects pointed to by the values member of the output message. Your serialize function serializes this data to the output message, and increments the message length by the number of bytes written.

  Correctly serializing unserialized values involves a set of common practices across all message types, and is discussed in more detail below.

Serialized messages contain all three types of data. The full sample module in the SDK implements a serializer that correctly handles all three types of data.

Working with Unserialized Values

For unserialized values, your module effectively reverses the parsing process. AMPS provides a set of identifiers and values, and you assemble these values into a message.

AMPS provides unserialized values as instances of the amps_message_value struct. Conceptually, a message value is the combination of an XPath-based identifier or tag, the value of that identifier, and the information necessary to efficiently and correctly serialize the value.

Most values are the result of parsing a published message. These values contain an XPath-based identifier, the tag, to be serialized and the value of the tag. The values provided are the result of parsing the message, as performed by the message type module for the source message.

Other values are provided by AMPS. AMPS generates these values for the status messages AMPS publishes to the /AMPS/.* family of topics. When a value is provided by AMPS, the tag is empty and zero-length. Instead, AMPS provides a constant that your module uses to look up a static string that you define as the name of the tag. Your module looks up the name, formats the tag appropriately, and writes the value.

Serialization Example

For example, consider the hierarchical message discussed above.

XPath-based identifier	Value
/messageId	5
/customerInfo/customerId	20
/customerInfo/notes	Have a nice day.

Table 9.6: Hierarchical Format Example Identifiers and Values

Your module serializes this in the appropriate format for your module. Given the set of tags and values above, a JSON module might serialize the message as follows:

{
    "messageId" : 5,
    "customerInfo" : {
        "customerId" : 20,
        "notes" : "Have a nice day."
    }
}

An XML module might serialize the message this way:

<AMPSMessage>
    <messageId>5</messageId>
    <customerInfo>
        <customerId>5</customerId>
        <notes>Have a nice day.</notes>
    </customerInfo>
</AMPSMessage>

A module that does simple, flat serialization might serialize the message with an implied hierarchy:

messageId=5, customerInfo/customerId=20, customerInfo/notes=Have a nice day.

Notice that your module may receive values that were parsed by another message type. This can happen when, for example, messages from multiple topics are joined together to project a view. Therefore, even if your message type only parses messages with a certain layout (such as messages without a hierarchical structure), your message type may need to serialize messages with a different layout.

Delta Publish

For a delta publish, AMPS needs to be able to receive an update to a message, and merge that update into an existing message to form a complete message. For delta publish, AMPS uses the following process:

1. Parse incoming message by calling amps_message_type_parse_message or amps_message_type_partial_parse_message with the incoming message. Use the value set provided to locate the existing message in the SOW, if one exists.
2. Parse the existing message by calling amps_message_type_parse_message or amps_message_type_partial_parse_message.
3. Merge the parsed messages by calling amps_message_type_delta_publish_merge.
4. Persist the merged message to the SOW, and publish to subscribers as necessary.

The function that AMPS calls to merge messages has the following signature:

int amps_message_type_delta_publish_merge(amps_delta_message_data *pDeltaMessage_);

Merging Messages for Delta Publish

When merging messages, AMPS provides the information for both the original message and an update message. Your module is responsible for creating a serialized message that contains the updated values merged with values that are unchanged from the original message.

AMPS provides your module with a structure that contains the update message, the parsed value set for the update message, the existing message that is being updated, and the parsed value set for the message that is being updated. Your module serializes the unchanged values and adds entries for these values to the value set in the update message.

One simple way to do this is to use the following process to merge messages:

1. Skip past the header information and the new values that have been published in the update.
2. Serialize the existing values that have not been replaced by new values. Your module determines this by checking whether the identifier for the existing value is present in the new value set. If so, the message already contains an updated value.
3. Add the existing values to the value set for the updated message.
4. When all of the existing values have been processed, return the new length of the serialized message.

Following this process, your module creates a merged message that contains values from both the new message and the existing values. For more complicated message formats, such as XML, your module may write the old values & new values into an intermediate format and then serialize the intermediate format.

The following example shows one way to merge messages:

int amps_message_type_delta_publish_merge(amps_delta_message_data *pDeltaMessage_)
{

    // Get the new value set and the old value set.

    amps_message_xpath_value_set newValues = pDeltaMessage_->update_value_set;
    amps_message_xpath_value_set oldValues = pDeltaMessage_->source_value_set;

    // Get pointers to the message data.

    const char *updateMessageData =  pDeltaMessage_->update_data->data;
    const size_t updateMessageDataLength = pDeltaMessage_->update_data->length;

    const char *oldMessageData = amps_message_xpath_get_value_data(oldValues);

    // Start writing after anything that's already been written, such as headers.
    // Notice that AMPS has already reserved space for the message.

    char *output = pDeltaMessage_->output.data + pDeltaMessage_->output.byte_count;
    char *origin = output;

    // Copy the update message verbatim. Notice that, for more complicated message
    // types such as JSON or XML, you may need to do a true merge.

    __builtin_memcpy(output, updateMessageData, updateMessageDataLength);
    output += updateMessageDataLength;

    // Write old values that have not been replaced by new values,
    // and add those values to the value set.

    for (amps_message_value_set_entry* entry =
                amps_message_xpath_begin(oldValues);
       entry != amps_message_xpath_end(oldValues);
       ++entry)
    {

        if (entry->hash == 0) continue;

        size_t valueIndex = amps_message_xpath_find_by_hash(newValues, entry->hash);

        // continue if there's an entry in the new values.

        amps_message_value_set_entry *pValue = amps_message_xpath_get(newValues,valueIndex);

        if (valueIndex != amps_message_value_invalid_index && pValue != NULL && pValue->hash != 0)
        {
            continue;
        }

        amps_message_value_set_entry updateValue = *entry;

        // serialize function (not shown) writes to output and returns
        // the number of characters written. This can be simple for
        // simple message types, or complicated for highly-structured
        // message types.

        updateValue.tag_bytes = entry->tag_bytes;
        updateValue.tag_length = entry->tag_length;

        output += serialize_tag(output, oldMessageData + entry->tag_offset, entry->tag_length);

        updateValue.string_rep = entry->string_rep;
        updateValue.string_rep_length = entry->string_rep;

        updateValue.raw_bytes = entry->raw_bytes;
        updateValue.raw_bytes_length = entry->raw_bytes_length;

        output += serialize_value(output, oldMessageData + entry->offset, entry->length);


        // add the value to the value set

        amps_message_xpath_update(newValues, NULL, &updateValue);

    }

    // set the length of the output message.
    pDeltaMessage_->output.byte_count += (output - origin);

    return pDeltaMessage_->output.byte_count;
}

Providing Readable Messages for Tracing

By default, AMPS includes the verbatim bytes of a message when logging the message. This is a good option for message types that are designed to be easily readable.

If your message type is not easily readable (for example, your message type includes binary data), AMPS provides an interface that allows you to produce a readable version of a message for use in AMPS logging. When your message type module implements this function, AMPS calls the function with a serialized message whenever AMPS needs to produce a readable version of a message.

There is no requirement that you implement this function. If not provided, AMPS logs the verbatim bytes of the message.

The function has the following signature:

int amps_message_type_trace_message(amps_message_type_context  context_,
                                    const  amps_trace_buffer*  pTraceBuffer_,
                                    amps_trace_output_buffer*  pOutputBuffer_);

Your module reads the serialized message provided in the trace buffer, creates a readable version of the message, and copies the readable version of the message into the trace output buffer.

The output buffer contains a pointer (output) to an area of pre-allocated memory, with the size of that area represented by the capacity of the output buffer. Notice that the size of the buffer that is available for your module to use is capacity - length, as the buffer may already contain data. Your module begins writing data at output + length, writes the necessary data, and then updates the value of the length to reflect the number of bytes written.

In the event that the available capacity of the buffer is not sufficient to hold the readable format of the message, return AMPS_FAILURE from the function. AMPS will interpret this as a request for more space in the buffer and call the function again with a larger buffer.

A module should not return AMPS_FAILURE from this function if the message cannot be meaningfully serialized (for example, if an incoming message has corrupt or invalid data). Instead, the module should record the reason for the failure in the buffer and return AMPS_SUCCESS.