3. amps_journal_dump

The AMPS journal dump utility is used in examining the contents of an AMPS journal file for debugging and program tuning. The amps_journal_dump utility is most commonly used as a tool to debug the forensic lifespan of messages that have previously been published to AMPS. The amps_journal_dump tool is used to show that messages exist in a journaled topic, and to show the order the message was received in, and the timestamp associated with the message.

Command Line Options

The amps_journal_dump program has the following options available. These can also be printed to the screen by typing amps_journal_dump -help.

Option Description
Filename of the AMPS journal file.
-h, –help Show the program help message and quit.
-l LIMIT Limit range of output to entries N:M where N is the first entry and M is the last entry. Passing in a single value, M, will return the first M results.
–localtime Display ISO 8601 timestamp in localtime.
–extents Add local and replication extents information at the end of the journal dump.
–no-data Do not display data values in the journal dump output.

Table 3.1: Parameters for amps_journal_dump

Looking at the Output

In this section will examine some sample output from running amps_journal_dump. We will then go over what each of the entries emitted by the program means.

File Name       : AMPS-Sample.0000000000.journal
File Size       : 10485760
Version         : amps::txlog/v8
Extents         : [1623783047000000005:1623783047000002563]
First Timestamp : 20210615T185233.126619Z
Last Timestamp  : 20210615T185234.032520Z
Entry               :                              0
CRC                 :                      175497491
type                :                        publish
flags               :                           none
offset              :                           4096
entry size          :                            256
msg len             :                             10
msg type            :                           json
localTxId           :            1623783047000000005
previousLocalTxId   :                              0
sourceTxId          :             436334058474393276
source              :                              0
client              :           16094765439783135993
clientSeq           :            1623783047000000005
topicHash           :            9914047442222214141
SOW expiration      :                              0
iso8601 timestamp   :        20210615T185233.126619Z
timestamp           :             212490586353126619
previous size       :                              0
topic len           :                              5
topic               : [abcde]
data                : [{"test":1}]
correlation id len  :                              0
correlation id      : []
auth ID len         :                              0
auth ID             : []
rep path len        :                              0
rep path            : []


Total Entries    : 2559
Total Bytes      : 10485760
Remaining Bytes  : 0

Example 3.1: Example of amps_journal_dump output

As is apparent in Example 3.1, the output from amps_journal_dump is split into three sections, a header, a listing of the contents of the journal file and a footer.

The header contains general information about the journal file as it is represented in the filesystem, and state data about the journal file.

Option Description
File Name The name of the file as it appears on the local filesystem.
File Size Number of total bytes allocated by the journal file.
Version This is the version of the formatting used to write the data in the journal file.
Extents A pair of numbers where the first is the number of extents used by the journal file, and the second is the number of blocks allocated for the journal file.

Table 3.2: amps_journal_dump listing header.

The second section of the amps_journal_dump lists each of the entries contained in the journal file, along with all of the meta-data used to track and describe the entry. For the sake of simplicity, Example 3.1 only shows a single listing, but it is more likely that a journal will contain multiple entries.

The exact options present depend on the version of the journal file.

Option Description
Entry A monotonically increasing value representing the order in which the record was inserted into the transaction log file.
CRC The cyclic redundancy check used for error checking the message.
type The AMPS command used in the original message or the type of entry in the journal file.
offset Offset for this entry.
flags Flags for this entry.
entry size The number of bytes allocated to the transaction log record.
msg len The number of bytes consumed by the data segment of the record.
msg type The message type used to format the data segment in the record.
localTxId The monotonically increasing identifier used across all records local transaction log journal files.
previousLocalTxId The monotonically increasing identifier used across all records local transaction log journal files.
sourceTxId The localTxId as it appears on the upstream replication source.
source A unique identifier used to represent the upstream source of the record. If the source is 0 then the transaction originated from the current host.
client The unique identifier associated with the client that published the message recorded in the transaction log.
clientSeq The monotonically increasing sequence identifier from the client.
topicHash The unique identifier for the topic the record was published to.
iso 8601 timestamp The ISO-8601 formatted timestamp representing the time the record was published to the transaction log. Notice that to keep timestamps consistent across instances that may be geographically dispersed, AMPS always timestamps in the UTC timezone.
timestamp | The raw timestamp stored in the AMPS transaction log.
This is a microsecond-precision timestamp.
previous size | Size of the previous entry in the txlog. .
topic len The number of characters in the topic field.
topic The topic for this entry in the transaction log.
data The data for this entry in the transaction log.
auth ID len The number of characters in the authId field.
authID The identifier associated with the authentication token used to enter this command.
correlation id len The length of the correlation ID associated with this command.
correlation id The correlation ID associated with this command.
auth ID len The length of the authentication ID associated with this command.
auth ID The authentication ID associated with this command.
rep path len The length of the replication path associated with this command.
rep path The replication path associated with this command.

Table 3.3: amps_journal_dump sample listing

As seen in Example 3.1, the final section contains general usage information about the data contained in the journal file.

Option Description
Total Entries Total number of journal entries entered into the journal file.
Total bytes The number of reserved bytes consumed by the journal file.
Remaining Bytes The number of unused bytes available out of the total reserved file size.

Table 3.4: amps_journal_dump listing header.

Timestamp formatting

The timestamp format used in amps_journal_dump is formatted by default using the system timezone for its location. To display the time in another timezone, the TZ environment variable can be configured to modify the output.

%> TZ='America/New_York' ./amps_journal_dump A.000000000.journal
%> TZ='Asia/Tokyo' ./amps_journal_dump A.000000000.journal
%> TZ='Europe/London' ./amps_journal_dump A.000000000.journal

NOTE: This will not work on dates prior to 1970.