Building View Server application with AMPS, React, and ag-Grid in JavaScript

Welcome! You are about to learn how to create a View Server -- a modern web application with AMPS, React, and the ag-Grid grid engine. By the end of this tutorial you are going to have a simple View Server web application up and running!

For your reference, here's the list of chapters in this tutorial:

You don't really have to follow the order -- chapters are loosely coupled to demonstrate a particular aspect of working on the view server web application project, however, the best experience can be achieved by completing all chapters in order.

Before We Start

We'll need to have just three things in order to build the app:

  1. Node.js: https://nodejs.org/en/download/

This is the JavaScript runtime that also comes with a package manager for JavaScript dependencies. Node.js will be used for creating a new web app project, running the project in a dev environment, and installing project's dependencies. We recommend Linux users to install Node.js via package manager.

  1. AMPS Server: https://www.crankuptheamps.com/evaluate/

AMPS the server-side software that is in charge of providing all the necessary features for a view server application. We'll be connecting to AMPS from our web application. AMPS is a 64-bit Linux application so we would need to either run it on a Linux box (for example, on a AWS or DigitalOcean instance), or in a virtual machine (for example, VirtualBox).

  1. Visual Studio Code: https://code.visualstudio.com/

While this is optional, we highly recommend it. It's a great looking free code editor with excellent support of JavaScript and React out-of-the-box. It's also free and works on every desktop platform (Windows, MacOS, Linux). Finally, it has a built-in terminal that we can use to execute Node.js commands and run our dev environment.

Chapter 1: A New React Application!

Assuming Node.js is installed, we can start a new React JavaScript project with a single command:

npx create-vite view-server-js --template react

npx is a nice way of using JavaScript applications without installing them. The above command means:

You can read more about vite here.

Once the project is created, let's navigate to the view-server-js directory:

cd view-server-js

Alternatively, you can open the view-server-js directory in Visual Studio Code.

This is the root of our future application and that's where we'll spend most of the time. Any command we execute or file we edit will be done from the project's root directory.

To get started and see the result in a browser:

npm install
npm run dev

The npm install command downloads and installs all the dependencies needed for this project. The dependencies needed are listed in the package.json file. The npm run dev command maintains a special development server that builds and serves the web application as we work on it. It also detects any changes made to the source code and reloads the app automatically in the browser.

Now, the terminal should show you the local address to access the react page, you can also open a browser and navigate to http://localhost:5173.

Chapter 2: Install Application Dependencies

Besides react, which was already installed when we created a new project using create-vite, we will need two other dependencies:

Install the dependencies for the application as follows:

npm install --save amps ag-grid-community ag-grid-react

In the above command, we installed three packages:

The --save modifier tells the package manager to also write these packages as project dependencies. This is useful for automatic installation of dependencies if you decide to share your project with someone, or for automatic deployment.

Chapter 3: React Crash Course

We've just installed React -- but let's take a minute to get familiar with it.

React is a JavaScript library that allows us to easily build modern web applications. The most important things to know about React are:

That's all you need to know about React for this tutorial! If you want, you can quickly learn more about React in this practice tutorial: https://reactjs.org/tutorial/tutorial.html

Chapter 4: First Web Grid in React

Let's create a new React component that we'll call Grid. To do this, create a file named Grid.jsx in the src directory of the project.

In the file, let's first import dependencies for the Grid component:

import { AgGridReact } from 'ag-grid-react'
import { ModuleRegistry, ClientSideRowModelModule, themeAlpine } from 'ag-grid-community'

ModuleRegistry.registerModules([ClientSideRowModelModule])

First, we import the AgGridReact component that we'll be using and also import themes that we want for our grid component to have a nice looking style. We also import the modules we need from ag-Grid and register them appropriately.

Second, we're going to need column definitions for the grid -- notice that this is just an array of objects that describe the columns in our grid:

const columnDefs = [
  { headerName: 'Make', field: 'make' },
  { headerName: 'Model', field: 'model' },
  { headerName: 'Price', field: 'price' }
]

Third, we also need some data for our grid -- for now, we'll just use a sample array of rows/objects:

const rowData = [
  { make: 'Toyota', model: 'Celica', price: 35000 },
  { make: 'Ford', model: 'Mondeo', price: 32000 },
  { make: 'Porsche', model: 'Boxter', price: 72000 }
]

Now we're fully ready to define our Grid component:

const Grid = () => {
  return (
    <div className='ag-container' style={{ height: 400, width: 600 }}>
      <AgGridReact
        rowData={rowData}
        columnDefs={columnDefs}
        theme={themeAlpine}
      />
    </div>
  )
}

export default Grid

In the component function's definition we return what React renders into HTML, so it returns the Grid component's contents.

Notice, that we supply three properties to the AgGridReact component: rowData, columnDefs and theme. The syntax of using React components is very similar to HTML, where properties are similar to HTML attributes. Curly braces allow us to use JavaScript expressions inside the properties.

The two properties of rowData and columnDefs we provided are enough for ag-Grid to render our data in a grid properly. The theme property gives our grid the style we want.

The full src/Grid.jsx file will contain the following:

import { AgGridReact } from 'ag-grid-react'
import { ModuleRegistry, ClientSideRowModelModule, themeAlpine } from 'ag-grid-community'

ModuleRegistry.registerModules([ClientSideRowModelModule])

const columnDefs = [
  { headerName: 'Make', field: 'make' },
  { headerName: 'Model', field: 'model' },
  { headerName: 'Price', field: 'price' }
]

const rowData = [
  { make: 'Toyota', model: 'Celica', price: 35000 },
  { make: 'Ford', model: 'Mondeo', price: 32000 },
  { make: 'Porsche', model: 'Boxter', price: 72000 }
]

const Grid = () => {
  return (
    <div className='ag-container' style={{ height: 400, width: 600 }}>
      <AgGridReact
        rowData={rowData}
        columnDefs={columnDefs}
        theme={themeAlpine}
      />
    </div>
  )
}

export default Grid

Finally, let's use our Grid component in the App (root) component by modifying the src/App.jsx file. Since we don't have a need to pass any properties to our Grid component yet, we can simply provide <Grid />, similar to an HMTL tag without attributes and children.

import Grid from './Grid'
import './App.css'

function App () {
  return (<Grid />)
}

export default App

Nice! We can see the rendered grid in a browser now!

Grid

Chapter 5: Configure and Start AMPS Server

Notice: everything we do with the AMPS Server happens on a Linux machine (either virtual or physical)

Once the AMPS server distribution is downloaded, we need to unpack the archive:

tar xzf AMPS-<version>-Release-Linux.tar.gz

(replace <version> with the actual version of AMPS you downloaded)

That's all you need to install the AMPS Server -- let's set it up! AMPS requires a configuration file in the .xml format. Let's begin by generating a sample configuration file that we will populate with the view server related modifications:

<path-to-dist>/bin/ampServer --sample-config > config.xml

In the above command, we call AMPS with the --sample-config parameter. This will output a sample AMPS configuration which we redirect into the config.xml file.

Add the Market Data topic

The basic idea of a view server applicaiton is to display, in a user friendly way, real-time updates and analytics of the data that is flowing into the AMPS server.

In order to receive the market data flow, we're going to declare a SOW (State-Of-the-World, or database) topic. Messages published to a SOW topic are stored on disk and are uniquely identified by the value of the key field(s), similar to the primary key in SQL databases. If a message is published with the symbol that matches a record already stored in the topic, that record will be replaced with a newer message. You can read more about SOW the feature of AMPS here.

In our case, we're declaring a topic market_data with the key field /symbol. Each time the market value of a symbol ticks, a message will be published to AMPS updating the market value of the symbol. Here's the topic definition -- we'll be adding it into the existing sample configuration file (config.xml):

<AMPSConfig>

    <!--
         existing configuration items
         are here ...
     -->

    <SOW>
        <Topic>
            <Name>market_data</Name>
            <MessageType>json</MessageType>
            <FileName>sow/%n.sow</FileName>
            <Key>/symbol</Key>
        </Topic>
    </SOW>

</AMPSConfig>

The topic definition has the following parameters:

There are many more parameters available to configure for a SOW topic. While it's out of the scope of our tutorial, you can read about all of them here.

Set up the Transport for a JavaScript client connection

Publishing and subscribing to AMPS Server is done by connecting to AMPS using client libraries. AMPS provides client libraries for several programming languages, such as Python, C/C++, Java, C#, and JavaScript. While most clients use a TCP connection to AMPS that uses a proprietary format, the JavaScript client operates from a browser over a WebSocket connection. Thus, in order for the AMPS Server to accept connections from a JavaScript client (that our web application will be using), we need to define a websocket transport, if it does not exist already:

<AMPSConfig>

  <!-- other configuration is here -->

  <Transports>

     <!--
          If it does not exist already, Add this Transport
          to the Transports element.
     -->

      <Transport>
          <Name>any-ws</Name>
          <Type>tcp</Type>
          <InetAddr>9008</InetAddr>
          <Protocol>websocket</Protocol>
      </Transport>

  </Transports>

  <!-- no need to modify the rest of the file -->

</AMPSConfig>

The websocket transport is very similar to the TCP transport. To make a new WebSocket transport, we just need:

Now AMPS is configured so we can connect to the port 9008 with our JavaScript applications, or connect to port 9007 with any of the other client libraries.

There are many more parameters available to configure for a Transport. While it's out of the scope of our tutorial, you can read about all of them here.

After adding the transport definition, we're done configuring AMPS. Save the configuration file and then start the AMPS Server in its own terminal window:

<path-to-dist>/bin/ampServer config.xml

That's it, the AMPS server is up and running and we're ready to publish data to the market_data topic!

Chapter 6: Market data generator

Before we start working on the final portion of this tutorial -- the view server application functionality, we need to create a simulator of the market data flow. In this case it will be a simple script that generates and publishes N messages a second of updates to the market_data topic.

Notice that this is a script that will be running using Node.js -- it's not a part of the web application, but rather a script that runs from a command line.

Let's create a scripts directory with the script market_data.cjs in the project directory.

The market_data.cjs file contains:

const { Client, DefaultServerChooser, MemoryPublishStore } = require('amps')

// constants
const HOST = 'localhost'
const PORT = '9008'
const TOPIC = 'market_data'
const PUBLISH_RATE_PER_SECOND = 2000

// -- the next part of the file is concerned with
//    creating interesting sample data

const SYMBOLS = [
  'MMM', 'ABBV', 'ALV', 'GOOGL', 'AMZN', 'AMGN', 'ABI', 'APPL', 'BHP', 'BA', 'BP',
  'BATS', 'CVX', 'CSCO', 'C', 'KO', 'DD', 'XOM', 'FB', 'GE', 'GSK', 'HSBA', 'INTC',
  'IBM', 'JNJ', 'JPM', 'MA', 'MCD', 'MRK', 'MSFT', 'NESN', 'NOVN', 'NVDA', 'ORCL',
  'PEP', 'PFE', 'PM', 'PG', 'ROG', 'RY', 'RDSA', 'SMSN', 'SAN', 'SIE', 'TSM', 'TOT',
  'V', 'WMT', 'DIS'
]

// helper functions
const randInt = (min, max) => Math.floor((Math.random() * (max - min + 1)) + min)
const round = (value, digits) => Math.round((value + Number.EPSILON) * Math.pow(10, digits)) / Math.pow(10, digits)
const timer = async interval => new Promise(resolve => setTimeout(resolve, interval))

// create initial prices
const pricing = {}
SYMBOLS.forEach(symbol => {
  pricing[symbol] = randInt(100, 1200)
})

// make an update message that is plausible enough to be
// interesting

const makeMessage = () => {
  // calculate update before pacing, for faster rates
  const symbol = SYMBOLS[randInt(0, SYMBOLS.length - 1)]
  let lastPrice = pricing[symbol]
  const bid = round(lastPrice - 0.5, 2)
  const ask = round(lastPrice + 0.5, 2)

  // keep market prices larger so that adjustments are proportionally accurate
  if (lastPrice < 100) {
    lastPrice = 100.0
  } else if (lastPrice > 1200) {
    lastPrice = 1200.0
  }

  // bump up to a nickle on each adjustment
  pricing[symbol] = round(lastPrice + randInt(-5, 5) / 100.0, 2)

  return { symbol, bid, ask }
}

// connect to AMPS and publish the data

const publishMarketData = async () => {
  // publish indefinitely at the rate specified
  const rate = 1.0 / PUBLISH_RATE_PER_SECOND * 1000

  // create the server chooser
  const chooser = new DefaultServerChooser()
  chooser.add(`ws://${HOST}:${PORT}/amps/json`)

  // create the HA publisher and connect
  const client = new Client('market_data_publisher')
  client.serverChooser(chooser)
  client.publishStore(new MemoryPublishStore())

  try {
    await client.connect()

    let lastTick = new Date().getTime()
    while (true) {
      const nextTick = lastTick + rate
      client.publish(TOPIC, makeMessage())

      // pace yourself to maintain the publish rate
      while (new Date().getTime() < nextTick) {
        await timer(0.01)
      }

      lastTick = nextTick
    }
  } catch (err) {
    console.error('err: ', err.message.reason)
    client.disconnect()
  }
}

// if running a script
if (typeof require !== 'undefined' && require.main === module) {
  publishMarketData()
}
// if used as a module
module.exports = { publishMarketData }

Don't forget to replace the HOST value with the actual hostname or IP of the box where you're running AMPS.

There are several parts to this script. We generate updates using the makeMessage() function. The publishMarketData() function publishes these updates using the AMPS JavaScript client at the rate specified in the PUBLISH_RATE_PER_SECOND constant (2000 m/s). We'll talk more about the client API later in this tutorial. For this step, just notice that we connect to the <Transport> we configured earlier and publish messages to the State of the World <Topic> we configured earlier.

You can read more about the JavaScript client in the Developer Guide and the API docs.

To run the script, simply execute it the command line:

node scripts/market_data.cjs

Assuming that the AMPS Server is up and the market data simulator is running, you can observe that the data is flowing in the AMPS Server Monitoring interface called Galvanometer. Navigate to the http://<amps_hostname>:8085 in the browser and you'll see the message flow widget:

Galvanometer -- message flow

Chapter 7: Add and Set Up AMPS Client

At this point we have a running AMPS Server with the market data updates published to it. Let's look closer at the way we create and connect the JavaScript client to the AMPS Server.

The client is connecting to AMPS and can publish and subscribe to AMPS topics. Furthermore, we want the client to maintain the connection, automatically re-connect in case of a disconnection, and re-establish all the subscriptions. Basically, we want the client to gracefully handle issues and proceed with the data consumption after recovery.

While all this functionality is optional, it's a good idea to have these features enabled for a view server application:

There are many other client APIs that provide High Availability and Failover handling -- you can read about them here.

Refactoring the App component

Since we can have more than one Grid component that displays data, they might share the same AMPS client object provided by the App (root) component of our application. Thus, the client will become a part of the state of the App component. Here's an outline of the refactored App component, with its state and lifecycle hooks (useState and useEffect, accordingly):

import { useState, useEffect } from 'react'

const App = () => {
  // the state of the component will be an AMPS Client object
  const [client, setClient] = useState()

  useEffect(() => {
    // this function is called when the component was rendered
    // In our particular case it will only be called once

    return () => {
      // this function will be called when component is about to be destroyed
    }
  }, []) // providing an empty list of dependencies ensures the above function is only called once

  // Contents to render
  return ( ... )
}

Notice that we use React Hooks here. Hooks are a new addition in React 16.8. They let you use state and other React features without writing a class. You can read more about available hooks, including ones to manage component's state and lifecycle here.

Client as a part of the App component's state

The App component now maintains state. The component has an AMPS Client object reference, and it will use this Client object to recieve data from AMPS. The client object is undefined initially, thus we do not provide an initial state value to the useState hook call -- we'll initialize it later:

// the state of the component will be an AMPS Client object
const [client, setClient] = useState()

In the rendering section (that's what we return when the component's function is called) of the App component, we describe the component's UI. We can render the component appropriately depending on the state, that is, based off the existence of a fully initialized and ready-to-use client.

We also pass the client object as a property to the Grid component so the component can use the client:

const App = () => {
  // the state of the component will be an AMPS Client object
  const [client, setClient] = useState()

  // ...

  // client is not ready yet, render "Loading..." label only
  if (!client) {
    return (<div>Loading...</div>)
  }

  return (<Grid client={client} />)
}

While the client is initializing, we show a "Loading..." message. Once the client is ready to go, we pass the client to the Grid and return the results.

Notice that the Grid does not have a client prop yet -- we'll address it later in the next chapter.

Create and Connect the Client

To use the Client API first we need to import required classes:

import { Client, DefaultServerChooser, DefaultSubscriptionManager } from 'amps'

The useEffect hook's callback is invoked after the component has been rendered and displayed. We additionally ensure it is only invoked once, when the component is mounted for the first time, by providing an empty list of hook dependencies:

useEffect(() => {

    // this function is called when the component was rendered
    // In our particular case it will only be called once

}, []) // providing an empty list of dependencies ensures the above function is only called once

That's the right place for us to create and connect the AMPS Client, because at this point we know that the component is displayed and in use (that doesn't matter so much for our little sample application, but only doing work when necessary can be vital for a larger application). Once the client is prepared, we change the state of the App component, which triggers re-rendering.

First, we add some constants at the top of the file:

// constants at the top of the file
const HOST = 'localhost'
const PORT = '9008'

And then, within the render hook callback, we add code that creates, connects, and updates the state with the client object:

useEffect(() => {
  // create the server chooser
  const chooser = new DefaultServerChooser()
  chooser.add(`ws://${HOST}:${PORT}/amps/json`)

  // create the AMPS HA client object
  const client = new Client('view-server')
  client.serverChooser(chooser)
  client.subscriptionManager(new DefaultSubscriptionManager())

  // now we can establish connection and update the state
  client.connect().then(() => setClient(client))

  return () => {
    // this function will be called when component is about to be destructed
  }
}, [])

Disconnect the Client

The optional function can be returned by the render hook callback. It is invoked when the component is about to be destructed. A component is destructed when it's not being rendered by a parent component anymore -- that's a good time to disconnect the client from AMPS:

useEffect(() => {
    // ...

    // disconnect the client from AMPS when the component is destructed
    return () => {
      client.disconnect()
    }
}, [])

At this point, our App component (src/App.jsx) should look as follows:

import Grid from './Grid'
import './App.css'
import { useState, useEffect } from 'react'
import { Client, DefaultServerChooser, DefaultSubscriptionManager } from 'amps'

// constants
const HOST = 'localhost'
const PORT = '9008'

const App = () => {
  // the state of the component will be an AMPS Client object
  const [client, setClient] = useState()

  useEffect(() => {
    // create the server chooser
    const chooser = new DefaultServerChooser()
    chooser.add(`ws://${HOST}:${PORT}/amps/json`)

    // create the AMPS HA client object
    const client = new Client('view-server')
    client.serverChooser(chooser)
    client.subscriptionManager(new DefaultSubscriptionManager())

    // now we can establish connection and update the state
    client.connect().then(() => setClient(client))

    // disconnect the client from AMPS when the component is destructed
    return () => {
      client.disconnect()
    }
  }, [])

  // client is not ready yet, render "Loading..." label only
  if (!client) {
    return (<div>Loading...</div>)
  }

  // Contents to render
  return (<Grid client={client} />)
}

export default App

Now we're fully ready to subscribe and display data from AMPS in our view server application!

Chapter 8: Display AMPS Data in the Grid Component

Row data is unique per each Grid component. Before, in our grid we were simply displaying static row data but now we will receive and dynamically update row data from AMPS -- it means we need to remove the existing constant rowData at the top of the src/Grid.jsx and make row data a part of the Grid's state.

Now that we are going to have a state in the Grid component, we will need to import the useState hook. Another hook we'll need is useEffect, in order to properly remove the active subscription when the component is destructed. Finally, we'll need a useRef hook to keep the reference to the subscription id:

import { useState, useEffect, useRef } from 'react'

Similarly to the way we added state to the App component, we will add rowData as Grid's state. We can now refactor the Grid component definition to provide the client prop as an argument:

const Grid = ({ client }) => {
  // the state of the component is the a list of row objects
  const [rowData, setRowData] = useState([])

  // create and keep a reference to the subscription id
  const subIdRef = useRef()

  useEffect(() => {
    return () => {
      // if there's an active subscription at a time of component destruction, remove it
      if (subIdRef.current) {
        client.unsubscribe(subIdRef.current)
      }
    }
  }, [client]) // we only need to invoke the hook callback when a new client prop provided (will be called once)

  // Other parts of the Grid TBD
}

Initially, there's no row data available, hence an empty list argument we send to the useState hook call.

Subscribe to a Topic

The Grid component is now provided with a set up AMPS Client as a property (prop) from its parent component -- App. We're ready to establish a subscription to the topic we need. Recall that the market_data topic is a SOW topic, that is, a database-like topic that is persisted on a disk.

In order to query that topic and receive matching results, we should issue a sow command. If we'd like to receive updates published to that topic, we'd need to issue a subscribe command. Finally, if we want to get the initial dataset and then seamlessly receive updates that occur to the topic (atomic updates), we should issue a combined command -- sow_and_subscribe.

That's exactly what we want for a view server -- display the initial dataset and real-time updates for that dataset.

First, let's import the AMPS Command class and a new module for ag-Grid (in addition to existing imports):

import { Command } from 'amps'
import { /* existing imports here */, ColumnAutoSizeModule } from 'ag-grid-community'

Remember to register the newly added module by adding it to our list of modules:

ModuleRegistry.registerModules([ /* existing modules here */, ColumnAutoSizeModule ]);

Then, let's add code to establish subscription. The right moment to issue the sow_and_subscribe command is when the AgGridReact is fully initialized and ready. This is done in the onGridReady callback property of the AgGridReact component:

<AgGridReact
    // other properties here

    // the provided callback is invoked once the grid is initialized
    onGridReady={async ({ api }) => {
      // resize columns to fit the width of the grid
      api.sizeColumnsToFit()

      // create a command object
      const command = new Command('sow_and_subscribe')
      command.topic('market_data')
      command.orderBy('/bid DESC')
      command.options('oof,conflation=3000ms,top_n=20,skip_n=0')

      try {
        // subscribe to the topic data and atomic updates
        let rows

        // store the subscription id
        subIdRef.current = await client.execute(command, message => {
          switch (message.header.command()) {
            case 'group_begin': // Begin receiving the initial dataset
              rows = []
              break
            case 'sow': // This message is a part of the initial dataset
              message.data.key = message.header.sowKey()
              rows.push(message.data)
              break
            case 'group_end': // Initial Dataset has been delivered
              setRowData(rows)
              break
            case 'oof': // Out-of-Focus -- a message should no longer be in the grid
              rows = processOOF(message, rows)
              setRowData(rows)
              break
            default: // Publish -- either a new message or an update
              rows = processPublish(message, rows)
              setRowData(rows)
          }
        })
      } catch (err) {
        setRowData([])
        console.error('err: ', err)
      }
    }}
/>
Subscription parameters

For our sow_and_subscribe command, we need to provide some parameters to AMPS:

options: the options fine-tune our subscription -- let's go over them in greater detail:

There are many more options and parameters available, but for now that's all we need to use in our example.

Data Flow

Once the above command is executed, we start receiving messages from AMPS. First we will receive the initial dataset, with messages group_begin and group_end around it. After that, we will be receiving updates that occur to the dataset, such as deletes, updates, and new records in the topic. Here's an example flow:

{ group_begin }
sow:     { symbol: 'AMGN', bid: 401.35, ask: 402.35 }
sow:     { symbol: 'JPM', bid: 908.33, ask: 909.33 }
sow:     { symbol: 'MRK', bid: 574.31, ask: 575.31 }
{ group_end }
publish: { symbol: 'RY', bid: 1150.64, ask: 1151.64 } -- new message
oof:     { symbol: 'JPM', bid: 908.33, ask: 909.33 }  -- delete
publish: { symbol: 'MRK', bid: 584.31, ask: 585.31 }  -- update
...

The group_begin message marks the beginning of the initial state of the topic, and the group_end message marks the end of the initial state. A publish message means that new data has arrived, while the oof notification means that a record that we previously received is no longer in scope.

With this flow in mind we handle received messages in our message handler:

Out-Of-Focus (OOF) Messages

When AMPS notifies us that a message is no longer relevant, we remove that message from the grid. The processOOF function is declared outside of the Grid component. Its main purpose to take an OOF message with current row data, and return new, adjusted row data:

const processOOF = (message, rowData) => {
  const rowIndex = rowData.findIndex(matcher(message))

  if (rowIndex >= 0) {
    const rows = rowData.filter(({ key }) => key !== message.header.sowKey())
    return rows
  }

  return rowData
}

Notice how we use this function in the message handler of the executed sow_and_subscribe command.

Updates and New Messages

On the other side, when AMPS notifies us that new information has arrived, we use the data in that message to update the grid. Similarly to processOOF, the processPublish function is declared outside of the Grid component, takes a message and current row data and returns new row data:

const processPublish = (message, rowData) => {
  const rowIndex = rowData.findIndex(matcher(message))
  const rows = rowData.slice()

  if (rowIndex >= 0) {
    rows[rowIndex] = { ...rows[rowIndex], ...message.data }
  } else {
    message.data.key = message.header.sowKey()
    rows.push(message.data)
  }

  return rows
}

Notice how we use this function in the message handler of the executed sow_and_subscribe command.

In both cases we try to find the index of the existing row by using a matcher:

const matcher = ({ header }) => ({ key }) => key === header.sowKey()

It's a helper function that is used by the findIndex() method of the Array to determine a match. Our matcher is using the sowKey of a record as the indentifier. The SowKey is a unique identifier of each record in a SOW topic. Naturally, we can use it as the identifier for rows in the grid.

Column Definitions

Since we display fields from the market_data topic, we need to adjust column definitions accordingly:

const columnDefs = [
  { headerName: 'Symbol', field: 'symbol' },
  { headerName: 'Bid', field: 'bid', sort: 'desc' },
  { headerName: 'Ask', field: 'ask' }
]

Notice that the sort setting in the grid is the same value we used in our orderBy setting to AMPS -- /bid DESC.

Set Up the Grid component

Here's the updated version of the Grid component. Notice that instead of passing the static array, we're now using rowData state value for the data in the Grid:

const Grid = ({ client }) => {
  /* existing implementation here */

  return (
    <div className='ag-container' style={{height: 600, width: 600}}>
      <AgGridReact
        theme={themeAlpine}
        columnDefs={columnDefs}
        // we now use state to track row data changes
        rowData={rowData}
        // unique identification of the row based on the SowKey
        getRowId={({ data: { key } }) => key}
        // resize columns on grid resize
        onGridSizeChanged={({ api }) => api.sizeColumnsToFit()}
        // the provided callback is invoked once the grid is initialized
        onGridReady={async ({ api }) => {
          // this is the place where we issue a "sow_and_subscribe" command
        }}
      />
    </div>
    )
}

It's working!

View Server - Final Grid

Chapter 9: Make the Grid Component Reusable

Our component, while working, is mostly hardcoded. Let's convert some Grid parameters (such as width and height) and the subscription-related parameters to be properties ("props") -- this way we'd be able to have flexible control over Grid and most importantly -- reuse it.

const Grid = ({ client, width, height, columnDefs, topic, orderBy, options }) => {
    // ...
};

Notice, we now replace the constant column definitions with definitions supplied as props, even though the object name stays same.

<div className='ag-container' style={{height: height ?? 600, width: width ?? 600}}>
    ...
</div>
// create a command object
const command = new Command('sow_and_subscribe')
command.topic(topic)
command.orderBy(orderBy)
command.options(options)

Now, our component is reusable -- we can create several Grid components in the App component (src/App.jsx) that display data with different settings from separate topics:

const App = () => {
  /* existing implementation here */

  return (
    <div style={{ display: 'flex', justifyContent: 'center', flexWrap: 'wrap' }}>
      <Grid
        client={client}
        columnDefs={[
          { headerName: 'Symbol', field: 'symbol' },
          { headerName: 'Bid', field: 'bid', sort: 'desc' },
          { headerName: 'Ask', field: 'ask' }
        ]}
        topic='market_data'
        options='oof,conflation=3000ms,top_n=20,skip_n=0'
        orderBy='/bid DESC'
      />

      <Grid
        client={client}
        columnDefs={[
          { headerName: 'Symbol', field: 'symbol' },
          { headerName: 'Bid', field: 'bid' },
          { headerName: 'Ask', field: 'ask', sort: 'asc' }
        ]}
        topic='market_data'
        options='oof,conflation=500ms,top_n=50,skip_n=10'
        orderBy='/ask ASC'
      />
    </div>
  )
}

Notice I've added some styling to the root div component -- that's done so that several grids look nicer together on the page. Alternatively, this styling information can be moved to src/App.css instead.

Chapter 10: Look and Feel: Dark Mode and Grid Animations

Functionality is important, but let's also make our application look nice! The plan is:

That might sound like a major project. Luckily, modern frameworks like ag-Grid provide most of this functionality out-of-the-box and we just need to apply it for our grid.

Dark Mode

ag-Grid already provides a dark version of their default theme and it looks nice.

First, add the colorSchemeDark import from ag-grid-community to use the dark theme:

import { /* existing imports here */, colorSchemeDark } from 'ag-grid-community'

Then, in the theme property of the AgGridReact component, apply the dark scheme like so:

theme={themeAlpine.withPart(colorSchemeDark)}

Then, let's adjust the background color of the application to match the grid. We'll modify the src/App.css file to add a simple property:

body {
    background: #181d1f;
}

Grid Title Panel

It's nice to have a header panel where we can have a title for the grid. This will help to distinguish each grid from other grids on the page.

First, we define the style description for the grid header container in the src/App.css:

.grid-header {
    height: 48px;
    line-height: 48px;

    background: #222628;
    color: #fff;
    border: 1px solid #68686e;
    border-bottom: unset;

    font-weight: bold;
    text-align: center;
}

.ag-container {
    margin: 10px;
    margin-bottom: 64px;
}

Then, let's add the grid header container in the rendered section of the Grid component (src/Grid.jsx). Notice that the title is a new optional prop:

const Grid = ({ title, /* other props here */ }) => {
    /* existing implementation here */

    return (
        <div className='ag-container' ...>
            <div className='grid-header'>{title}</div>
            <AgGridReact ... />
        </div>
    );
};

Now we can supply the title in src/App.jsx:

<Grid
    title='Top 20 Symbols by BID'
    /* other properties here */
/>

Enable Row Update Animation

This is the easiest part of our style changes -- we simply add the animateRows prop to the AgGridReact component:

<AgGridReact
    /* other properties here */
    animateRows
/>

You can read more about animation in ag-Grid here.

Value Change Animation

First, let's create a new file, src/helpers.js. We'll use this file for some of the appearance related helper functions without polluting the Grid component. To start with, we'll add a module and these functions to the file:

import { HighlightChangesModule, ModuleRegistry } from 'ag-grid-community'

ModuleRegistry.registerModules([HighlightChangesModule])

function numberValueParser (params) {
  return Number(params.newValue)
}

function numberCellFormatter (params) {
  if (!params || !params.value) {
    return ''
  }

  return params.value.toFixed(2).replace(/(\d)(?=(\d{3})+(?!\d))/g, '$1,')
}

function currencyCellFormatter (params) {
  return '$' + numberCellFormatter(params)
}

export function numCol (column, currency = false) {
  column.resizable = true
  column.valueParser = numberValueParser
  column.cellClass = 'right'
  column.valueFormatter = currency ? currencyCellFormatter : numberCellFormatter
  column.cellRenderer = 'agAnimateShowChangeCellRenderer'
  return column
}

export function curCol (column) {
  return numCol(column, true)
}

I borrowed some of the above functions from the ag-Grid's examples. The most interesting one is the numCol() function. That function injects a column definition with extra properties that format a numeric value as currency, apply some styling and most importantly, enable the show change animation.

Notice that we're using a cellClass value -- this applies a CSS class selector to each cell of that column. The selector definition is added to the src/App.css:

.right {
  text-align: right;
}

Now, let's import the curCol function in the src/App.jsx file and use that function to create column definitions for the Bid and Ask columns:

import { curCol } from './helpers'


/* existing implementation here */


<Grid
 /* existing options here */

  columnDefs={[
    { headerName: 'Symbol', field: 'symbol' },
    curCol({ headerName: 'Bid', field: 'bid', sort: 'desc' }),
    curCol({ headerName: 'Ask', field: 'ask' })
  ]}

 /* existing options continues */
/>

Here's the final result -- looks great!

View Server - Final Grid

Chapter 11: Connection State and Error Handling

The AMPS JavaScript client provides several mechanisms to control the state of the client, such as:

You can read more about error handling here.

Error Handler

The errorHandler is set to react to client-wide events, so we set it in the src/App.jsx when creating the client:

useEffect(() => {
    // ...

    // create the AMPS HA client object
    const client = new Client('view-server')

    // ...

    // report general errors in the error handler, for example, message parsing error,
    // or an error thrown in the message handler
    client.errorHandler(err => console.error('Error: ', err))

    // now we can establish connection and update the state
    client.connect().then(() => setClient(client))

    // ...
}, [])

Connection State Listener

Each Grid component will have its own status bar to display the connection state or subscription errors. This is because two Grid components could use different clients, and each component has its own subscription. Thus, we'll need to add a connection state listener inside of the Grid component (src/Grid.jsx).

In order to display connection state information, we'll add a bottom panel under the grid.

We are updating the state of the component to now include connectionStatus and error labels. Since we initially render Grids after the client is successfully connected, the initial value of the connectionStatus is "Connected". A new bottom panel that will display the connection status and, if necessary, any error information from the subscribe command will be added under the AgGridReact component in the Grid's rendered contents:

const Grid = ({ /* props here */ }) => {
  const [connectionStatus, setConnectionStatus] = useState('Connected')
  const [error, setError] = useState()

  /* existing implementation here */

  return (
    <div className='ag-container' ...>
      ...

      <AgGridReact ... />

      <div className='status-panel'>
        <span style={{ color: connectionStatus === 'Connected' ? 'green' : 'yellow' }}>{connectionStatus}</span>
        <span style={{ float: 'right', color: 'red' }}>{error}</span>
      </div>
    </div>
  )
}

Notice how we use inline CSS styles to color the connection status label depending on the connection status.

Since we've added a new panel below the grid, we also need to add a bit of styling information into src/App.css to make it look nice:

.status-panel {
    height: 32px;
    line-height: 32px;

    background: #222628;
    color: #fff;
    border: 1px solid #68686e;
    border-top: unset;

    font-weight: bold;
    text-align: left;
    padding: 0 8px;
}

Let's add the connection state listener logic to properly handle it in accordance of the Grids lifecycle. We'll add a listener in the render hook (useEffect), before establishing a subscription, and remove it when the component is destructed, in the returned callback of the hook:

import { Client, /* existing import here */ } from 'amps'

/* existing implementation here */

const Grid = ({ client, /* other props here */ }) => {
  /* existing implementation here */

  useEffect(() => {
    // subscribe for the connection events
    const listenerId = client.addConnectionStateListener(state => {
      if (state === Client.ConnectionStateListener.LoggedOn) {
        setConnectionStatus('Connected')
      } else if (state === Client.ConnectionStateListener.Disconnected) {
        setRowData([])
        setConnectionStatus('Reconnecting...')
      }
    })

    return () => {
      /* existing implementation here */

      // remove the connection state listener when the component is destructed
      client.removeConnectionStateListener(listenerId)
    }
  }, [client])

  /* existing implementation continues */
}

Notice: we need to import the Client class from the amps package in order to access connection state listener's state constants.

The state listener implementation is pretty straightforward. When the listener is called, it modifies the connectionStatus state, and that triggers a re-render of the Grid component. When the grid component is unmounted/destructed, we also remove the connection state listener. Before, we were simply unsubscribing.

There are other states reported to the listener that are listed here.

In the catch block that executes if the subscription command did not succeed we now update the error state, which results in re-rendering the error label:

<AgGridReact
  // other properties here

  onGridReady={async ({ api }) => {
    /* existing implementation here */

    try {
      /* existing implementation here */
    }
    catch (err) {
      setError(`Error: ${err.message}`)
    }
  }}
/>

That's it, now we have connection and error tracking in our grids!

Connected State and Error State Reconnecting to AMPS

Chapter 12: Data Flow Optimization: Content Filters

One thing that differentiates AMPS from classic messaging systems is its ability to route messages based on message content. Instead of a publisher declaring metadata describing the message for downstream consumers, the publisher can simply publish the message content to AMPS and let AMPS examine the native message content to determine which consumers should receive the message.

The ability to use content filters greatly reduces the problem of oversubscription that occurs when topics are the only facility for subscribing to message content. In addition, many of the advanced features of AMPS such as out-of-focus messaging, aggregation, views, and SOW topics rely on the ability to filter content.

You can read more about Content Filters here.

AMPS uses a combination of XPath-based identifiers for the message fields and SQL operators for filters. For example, the following message:

{"symbol":"GSK","bid": 292.04,"ask": 293.04,"extra":{"hello":"world"}}

would have identifiers /symbol, /bid, /ask, /extra and /extra/hello. Some of the filter values we could apply that would match the above message are:

/symbol = 'GSK'
/symbol IN ('GSK', 'GOOGL')
/extra/hello BEGINS WITH('wo')
/bid > 200 AND /bid < 300
/ask IS NOT NULL
INSTR_I(/symbol, 'gsk') != 0
LENGTH(/symbol) = 3

You can read more about AMPS Expression language here.

Let's add filter support to the grid by supplying a new filter parameter prop (optional) and then to the command we create and execute in the onGridReady callback of the AgGridReact component (src/Grid.jsx):

const Grid = ({ filter, /* other props here */ }) => {
  /* existing implementation here */

  return (
    /* existing implementation here */

    <AgGridReact
      // other properties here

      onGridReady={async ({ api }) => {
        /* existing implementation here */

        command.options(options)

        if (filter) {
          command.filter(filter)
        }

        /* existing implementation continues */
      }}
    />
    /* existing implementation continues */
  )
}

Now we can supply a filter prop in the App component (src/App.jsx), for example:

<Grid
  /* other options here */
  filter='LENGTH(/symbol) = 3'
/>

That limits the records in the grid to just entries where the symbol is three letters long. Easy!

Modify and Apply Content Filters Dynamically

For grids, it's useful to be able to modify the filter value to look into a different subset of data. Let's add a new component, FilterBar (src/FilterBar.jsx) that can be used in the Grid component to dynamically modify the filter:

import { useState } from 'react'

const FilterBar = ({ value, onValueChange }) => {
  const [inputValue, setInputValue] = useState(value ?? '')

  return (
    <div className='filter-bar'>
      <form>
        <label>
          <input
            type='text'
            value={inputValue}
            onChange={({ target }) => setInputValue(target.value)}
          />
        </label>
        <input type='button' value='Filter' onClick={() => onValueChange(inputValue)} />
      </form>
    </div>
  )
}

export default FilterBar

Basically, it's just an input with a button. Once the button is clicked, the value in the input is reported to the parent by invoking a provided prop onValueChange. The initial value is also provided as a prop.

Before we consider this done, though, let's make it look nice with our dark-themed grid by adding some CSS styling to the src/App.css:

.filter-bar {
    background: #222628;
    padding: 5px;
    border: 1px solid #68686e;
    border-bottom: unset;
}

.filter-bar > form {
    display: flex;
    align-items: flex-start;
    justify-content: flex-end;
}

.filter-bar label {
    width: 100%;
    margin-right: 15px;
}

.filter-bar input {
    border: none;
    text-decoration: none;
    border-radius: 2px;
    color: #fff;
    font-size: 13px;
}

.filter-bar input[type=text] {
    width: 100%;
    background: #253232;
    padding: 3px 5px;
}

.filter-bar input[type=button] {
    cursor: pointer;
    width: 50px;
    background-color: #244b5f;
    padding: 3px 10px;
}

We'll only show the filter bar if it's "enabled" -- that is, if the showFilterBar prop is set. Let's add a new optional prop:

const Grid = ({ showFilterBar, /* other props here */ }) => {
    /* existing implementation here */
};

You can enable the filter bar by supplying a prop to the Grid object in the App component:

<Grid
    /* other options here */
    showFilterBar
/>

Before we can add and use the newly created filter bar component, we need to refactor the Grid component a bit to make it handle content filter modifications.

Here's the outline of the refactored Grid component:

import { /* existing imports here */, useCallback } from 'react'
import FilterBar from './FilterBar'

/* existing elements here */

const Grid = ({ showFilterBar, filter, /* other props here */ }) => {
    /* other hooks here */

    // new filter state value hook
    const [filterInput, setFilterInput] = useState(filter || '')

    // new callback hook
    const sowAndSubscribe = useCallback(async filter => {
      // Implementation TBD
    }, [/* hook dependencies go here */])

    return (
      <div className='ag-container' ...>
        ...

        {showFilterBar && <FilterBar value={filterInput} onValueChange={sowAndSubscribe} />}

        <AgGridReact
          // other properties here

          onGridReady={({ api }) => {
            // resize columns to fit the width of the grid
            api.sizeColumnsToFit()

            // notice that we've moved the subscription code from here
            // and simply call the newly created function instead
            sowAndSubscribe()
          }}
        />

        ...
      </div>
    )
};

As you can see above, we call the sowAndSubscribe() function once the filter value is changed. We need to adjust this code a bit to handle an optional new filter value provided -- we need to unsubscribe from a current subscription (if any) and re-subscribe with another filter. To handle re-subscription we can use the sub id reference hook we've added a while ago in the Chapter 8 of this tutorial.

Here's the full implementation of sowAndSubscribe():

const sowAndSubscribe = useCallback(async filter => {
  if (filter !== undefined) {
    if (filter !== filterInput) {
      setFilterInput(filter || '')
    } else {
      return
    }
  } else {
    filter = filterInput
  }

  // clear previous errors, if any
  if (error) {
    setError('')
  }

  // if we had a running subscription already, we need to unsubscribe from it
  if (subIdRef.current) {
    client.unsubscribe(subIdRef.current)
    subIdRef.current = undefined

    // update state
    setRowData([])
  }

  // create a command object
  const command = new Command('sow_and_subscribe')
  command.topic(topic)
  command.orderBy(orderBy)
  command.options(options)

  if (filter) {
    command.filter(filter)
  }

  try {
    // subscribe to the topic data and atomic updates
    let rows
    subIdRef.current = await client.execute(command, message => {
      switch (message.header.command()) {
        case 'group_begin': // Begin receiving the initial dataset
          rows = []
          break
        case 'sow': // This message is a part of the initial dataset
          message.data.key = message.header.sowKey()
          rows.push(message.data)
          break
        case 'group_end': // Initial Dataset has been delivered
          setRowData(rows)
          break
        case 'oof': // Out-of-Focus -- a message should no longer be in the grid
          rows = processOOF(message, rows)
          setRowData(rows)
          break
        default: // Publish -- either a new message or an update
          rows = processPublish(message, rows)
          setRowData(rows)
      }
    })
  } catch (err) {
    setError(`Error: ${err.message}`)
  }
}, [client, error, filterInput, options, orderBy, topic]) // we list dependencies used in the above function

Now we can apply content filters and dynamically modify them!

Here's an example of two grids -- first has both a content filter and a filter bar, and the second has a content filter but no filter bar:

View Server - Grids with Filter

Chapter 13: Data Flow Optimization: Select Lists and Deltas

The last chapter showed how to use content filters to limit the messages (rows) returned to just the items that the user wants to see. We covered conflation in chapter 8 as a way of making sure that updates only arrive as fast as the grid can process and display them.

We can do more to optimize the data flow, though. Let's take a closer look!

Select Lists

In our market_data topic we have somewhat small messages that only consist of three fields. That's not always the case for all topics. Sometimes messages are quite large and contain hundreds of fields. In cases like that, a web application would typically only need to display a fraction of the available fields (say, 3-10).

Of course, we could have the Grid component only display the required columns and ignore the rest, but that would still waste the bandwidth delivering fields that we never needed. A better solution is to use Select Lists!

AMPS has the ability to allow a subscriber to retrieve only the relevant parts of a message, in the same way that a SQL query can retrieve only specified fields from a table. With select lists, AMPS allows an individual subscription to control which fields are retrieved from a subscription or query. For example, say we only want to receive /symbol and /bid fields from the market_data topic messages and don't want anything else to be included. To do this, the application would include the following option on the command used to retrieve data for the overview:

select=[-/,+/symbol,+/bid]

This would tell AMPS:

You can read more about Select Lists here.

In fact, it's safe to assume that we only want to display fields in the grid for which we provided column definitions to the Grid component. Let's automate this -- we can have the Grid build and supply the select list option to the subscription automatically based on the column defintions.

First, Let's add a new optional prop select:

const Grid = ({ select, /* other props here */ }) => {
  /* existing implementation here */

  const sowAndSubscribe = useCallback(async filter => {
    /* existing implementation here */
  }, [/* existing dependencies */, columnDefs, select])

  /* existing implementation continues */
}

Notice that we now have two more dependencies for the useCallback hook -- columnDefs and select.

To enable the select list feature, supply this new prop to the Grid object in the App component:

<Grid
  /* other options here */
  select
/>

This way, we only use the select list feature if it's enabled. If you know that your grid will show all of the fields in the message, for example, then there's no need to do the extra work of setting a select list, and you can leave it turned off.

Second, let's add another helper function outside of the Grid (src/Grid.jsx) component:

const withSelect = (columnDefs, options = '') => (
  `${options}${options ? ',' : ''}select=[-/,${columnDefs.map(c => '+/' + c.field).join(',')}]`
)

This function builds a select=[] option string based on the column definitions. For example, the following column definitions:

columnDefs={[
  {headerName: 'Symbol', field: 'symbol'},
  curCol({headerName: 'Bid', field: 'bid', sort: 'desc'})
]}

will be converted into the the following option which will be appended to the existing options prop (if any):

select=[-/,+/symbol,+/bid]

Now that we can build the appropriate select list, let's replace the way we supply options to the command object in the sowAndSubscribe() function of the Grid:

/* Replace the existing line that sets options with the following: */

// if we're using the select list feature, let's add that to the options
const opts = select ? withSelect(columnDefs, options) : options
command.options(opts)

Delta Subscription

We can optimize the bandwidth even further, though!

Delta subscribe allows applications to receive only the changed parts of a message when an update is made to a record in the SOW. When a delta subscription is active, AMPS compares the new state of the message to the old state of the message, creates a message for the difference, and sends the difference message to subscribers. Using this approach can simplify processing on the client side, and can improve performance when network bandwidth is the most important constraint.

You can read more about Delta subscriptions here.

All we need to do to add the delta subscribe support to the Grid component, is to modify the way we create the Command object in the sowAndSubscribe() function:

/* Replace the line that creates the Command object with this */

// create a command object
const command = new Command(delta ? 'sow_and_delta_subscribe' : 'sow_and_subscribe')

As with select lists, we only use the delta subscription feature if it's enabled. Let's add a new optional prop:

const Grid = ({ delta, /* other props here */ }) => {
  /* existing implementation here */

  const sowAndSubscribe = useCallback(async filter => {
    /* existing implementation here */
  }, [/* existing dependencies */, delta])

  /* existing implementation continues */
}

Notice that we now have another dependency for the useCallback hook -- delta.

To enable the feature, supply a prop to the Grid object in the App component:

<Grid
  /* other options here */
  delta
/>

Chapter 14: Enable Sorting Controls in the Grid

Currently we have to specify sorting in both column definitions (as the sort field) and by supplying the orderBy prop to the Grid component. Let's re-organize the code so that the orderBy is automatically generated from the column definitions. Furthermore, we can enable sorting for columns so that a view server user will be able to change sorting on fly!

Please notice that the sorting is happening on the server side in AMPS -- that's why we supply the orderBy parameter in the subscription command.

First, we enable the UI sorting controls for the AgGridReact component (in the src/Grid.jsx):

<AgGridReact
  /* other options here */

  // enable sorting from the UI
  multiSortKey='ctrl'
  onSortChanged={() => sowAndSubscribe()}
/>

Notice the mulitSortKey prop. That makes it so that if a user wants to sort by more than one column, they can hold down the Ctrl key while clicking on the column headers to do so.

In the above snippet, we are calling the sowAndSubscribe() function again when the sort order changes. This will unsubscribe from an existing subscription (if any), and establish a new subscription with the new order.

Next, we need to be able to build an orderBy string based on the current sorting settings of the columns in the grid. We'll add a new function, called generateOrderBy(), to the Grid:

const generateOrderBy = columns => (
  columns
    .filter(({ sort }) => sort !== null)
    .sort(({ sortIndex: si1 }, { sortIndex: si2 }) => si1 - si2)
    .map(({ colId, sort }) => `/${colId} ${sort}`)
    .join(',')
)

In order to be able to get the column state from the grid engine, we'll need to store a reference to the column API. Let's declare a useRef hook called columnApiRef and populate it in the onGridReady callback:

import { /* existing imports here */, ColumnApiModule } from 'ag-grid-community'

ModuleRegistry.registerModules([/* existing modules here */, ColumnApiModule])

const Grid = ({ /* props here */ }) => {
  /* other hooks here */

  // new grid column API reference hook to access column state when subscribing to AMPS
  const gridApiRef = useRef()

  return (
    <div className='ag-container' ...>
      ...

      <AgGridReact
        // other properties here

        onGridReady={({ api }) => {
          // store the reference to ag-Grid's Column API object
          gridApiRef.current = api

          /* existing implementation here */
        }}
      />

      ...
    </div>
  )
}

Almost done! The only thing we need to modify in our sowAndSubscribe() function is the way we set the orderBy parameter on the command. We need to update this to use our new function:

/* Replace existing line that sets orderBy with the following: */

command.orderBy(generateOrderBy(gridApiRef?.current?.getColumnState() ?? []))

Simple enough. We call the new function, and we're all set! Don't forget to remove the orderBy prop from the Grid definition and the grid objects in the App component (src/App.jsx).

Here's an example of a Grid that is sorted by Group (descending) and then by Symbol (ascending):

View Server - Grid with Order By

Chapter 15: Secure Connection, Authentication, and Entitlements

Most production systems need some sort of access control. Our tutorial wouldn't be complete without showing you how to set up simple access control and permissions in AMPS.

Access control and permissions in AMPS is designed to be straightforward.

Authentication and entitlements are handled by modules, plugins for the AMPS server.

The AMPS server includes an optional module that can use a RESTful web service for authentication and entitlement. Using this module can be the easiest way to integrate into an existing system.

You can read more about the module here.

We are going to write a simple web service in Node.js that will serve as both Authentication and Entitlement service for the module.

First, let's install the dependency for the script:

npm install --save-dev express express-basic-auth

This package implements a simple HTTP service that understands basic authentication. We'll use that to build our authentication and entitlement service.

Now, let's create a new file auth_service.cjs in the scripts directory:

const app = require('express')()
const basicAuth = require('express-basic-auth')

app.use(basicAuth({
  users: { falken: 'joshua' },
  challenge: true
}))

app.get('/:username', function (req, res) {
  console.log('request: ', req.path)

  res.json({
    logon: true,
    topic: [
      {
        topic: '.*',
        read: true,
        write: true
      }
    ],
    admin: [
      {
        topic: '/amps/administrator/.*',
        read: false,
        write: false
      },
      {
        topic: '.*',
        read: true,
        write: true
      }
    ]
  })
})

app.listen(8080)

In the above snippet, we create a simple HTTP service that only responds to an authenticated query. The only user that can authenticate is falken with the password joshua. That service also responds with a permission document that AMPS module accepts when an authorized GET request from AMPS Server is sent to /joshua. In that document we grant both read and write permissions for any topics, but limit the access to the /amps/administrator path in the Admin API (which we will cover later in this tutorial).

Start the service and keep it running:

node scripts/auth_service.cjs

The detailed description of the permission document format is available here.

Now, let's set up the AMPS Server to enable Authentication with the Web Service Authentication and Entitlements module. Modify the AMPS configuration file (config.xml) to add the module and enable it:

<AMPSConfig>
    <Admin>
        ...
        <WWWAuthenticate>Basic realm="AMPS Admin"</WWWAuthenticate>
    </Admin>

    ...

    <Modules>
        <Module>
            <Name>web-entitlements</Name>
            <Library>libamps_http_entitlement.so</Library>
            <Options>
                <ResourceURI>http://localhost:8080/{{USER_NAME}}</ResourceURI>
            </Options>
        </Module>
    </Modules>

    <Authentication>
        <Module>web-entitlements</Module>
    </Authentication>
    <Entitlement>
        <Module>web-entitlements</Module>
    </Entitlement>

</AMPSConfig>

Notice that we also enable basic authentication for the Admin API and Galvanometer by adding the WWWAuthenticate parameter to the Admin section of the configuration file.

Stop the AMPS server (AMPS does not re-read the configuration file while the server is running).

Restart the AMPS server and the authentication/entitlements are all set up to use the service we just created.

Now, if you try to connect to AMPS with the existing market_data script or the View Server application, the connections to AMPS will fail with an auth failure error. Of course, that's what we want -- it means authentication is enabled!

To make our applications work again, let's add a username and password to the connection strings. AMPS expects a connection string of the form: ws://login:password@localhost:9008/amps/json.

You can read more about the connection string format here.

In src/App.jsx and scripts/market_data.cjs:

/* add these where we define the constants */
const LOGIN = 'falken'
const PASSWORD = 'joshua'
const HOST = 'localhost'
const PORT = '9008'

/* Replace the lines where we create the server chooser
   and add a connection string with: */

// create the server chooser
const chooser = new DefaultServerChooser()
chooser.add(`ws://${LOGIN}:${PASSWORD}@${HOST}:${PORT}/amps/json`)

Everything works again, but now both client and server are authenticated through the web service we've created.

Secure Connection over SSL

The final piece of security is an encrypted connection to AMPS. We achieve this by using the Secure WebSocket protocol (wss). In order to set up a Secure WebSocket, we'll need to modify the Transport definition in the AMPS Server configuration file to include:

You can find the configuration reference for transports here.

If you don't have an SSL certificate, we can create a self-signed certificate for testing purposes. Most organizations provide a signed certificate for production instances of AMPS, so check your local policies for production instances.

To create a self-signed certificate, run the following command on the Linux box in the directory where you have the AMPS Server configuration file located:

openssl req -newkey ed25519 -nodes -keyout key.pem -x509 -days 365 -out cert.pem

Once we have the cert.pem and the key.pem, we can adjust the AMPS configuration file:

<AMPSConfig>

    <!-- existing configuration unchanged -->

    <Transports>

       <!-- existing transports unchanged, add: -->

        <Transport>
            <Name>any-ws</Name>
            <Type>tcp</Type>
            <InetAddr>9008</InetAddr>
            <Protocol>websocket</Protocol>
            <Certificate>./cert.pem</Certificate>
            <PrivateKey>./key.pem</PrivateKey>
        </Transport>
    </Transports>

    <!-- remaining configuration unchanged -->

</AMPSConfig>

Notice: Stop and restart the AMPS Server after modifying the configuration file.

Please keep in mind that self-signed certificates won't work in browsers. If you have a development certificate and want to use it, you can start Chrome in the ignore mode:

google-chrome --ignore-certificate-errors

In Node.js, self-signed certificates can be used if you set the env variable NODE_TLS_REJECT_UNAUTHORIZED to 0. Let's modify the scripts/market_data.js file to include the following permission at the top of the file:

process.env.NODE_TLS_REJECT_UNAUTHORIZED = 0

Note: We strongly suggest never using self-signed certificates in production or allowing a production application to accept a self-signed certificate!

The last thing to do is to modify the connection strings in the src/App.jsx and scripts/market_data.cjs to use the wss:// protocol instead of the ws://:

/* Replace the lines where we create the server chooser
   and add a connection string with: */

// create the server chooser
const chooser = new DefaultServerChooser()
chooser.add(`wss://${LOGIN}:${PASSWORD}@${HOST}:${PORT}/amps/json`)

Note: Restart the market_data.cjs script after modifying.

Now we have an encrypted secure connection to AMPS with enabled Authentication and Entitlements!

You can read more about security in AMPS here.

We also have a blog article and a sample repository if you want to experiment with the Authentication and Entitlements in AMPS, available here.

Chapter 16: AMPS Views and Aggregations

AMPS contains a high-performance aggregation engine, which can be used to project one SOW topic into another and, if you like, compute aggregation as it does so. This feature is similar to the CREATE VIEW functionality found in most SQL engines. The AMPS aggregation engine can join input from multiple topics (of the same or different message types) and can produce output in different message types.

You can find the detailed information about Views and Aggregation here.

View topics are part of the AMPS State of the World, which means that views support delta subscriptions and out of focus (OOF) tracking. A view can also be used as the underlying topic for another view.

In addition, for the limited cases where a view is not practical, AMPS allows an individual subscription to request aggregation and projection on a single SOW topic. You can read more about it here.

To demonstrate this concept, we will add a new field to messages in the market_data topic: /group. For sample purposes, the group value will be from 1 to 10, based off the index of the symbol in the SYMBOLS array of the scripts/market_data.cjs data generator script. There's nothing particularly important about that number, but it lets us demonstrate views by splitting all of the published messages into 10 distinct groups.

Let's modify the makeMessage() function of the script, replacing the existing function with:

const makeMessage = () => {
  // calculate update before pacing, for faster rates
  const index = randInt(0, SYMBOLS.length - 1)
  const symbol = SYMBOLS[index]
  let lastPrice = pricing[symbol]
  const bid = round(lastPrice - 0.5, 2)
  const ask = round(lastPrice + 0.5, 2)

  // keep market prices larger so that adjustments are proportionally accurate
  if (lastPrice < 100) {
    lastPrice = 100.0
  } else if (lastPrice > 1200) {
    lastPrice = 1200.0
  }

  // bump up to a nickle on each adjustment
  pricing[symbol] = round(lastPrice + randInt(-5, 5) / 100.0, 2)

  return { symbol, bid, ask, group: (index % 10) + 1 }
}

Now, let's modify the AMPS Server configuration file (config.xml). Stop the AMPS server if it's running and add a new View definition to the SOW section of the config:

<AMPSConfig>
    <!-- existing config is unchanged -->

    <SOW>
        <!-- add this to the existing SOW configuration -->

        <View>
            <Name>agg_market_data</Name>
            <MessageType>json</MessageType>
            <UnderlyingTopic>market_data</UnderlyingTopic>

            <Projection>
                <Field>SUM(/ask) AS /ask_total</Field>
                <Field>SUM(/bid) AS /bid_total</Field>
                <Field>COUNT(/symbol) AS /count</Field>
                <Field>/group</Field>
            </Projection>

            <Grouping>
                <Field>/group</Field>
            </Grouping>
        </View>

    </SOW>

</AMPSConfig>

There are many more parameters available to configure for a View topic. While it's out of the scope of our tutorial, you can read about all of them here.

In the above defintion we created a View that's based on the market_data topic. The data is grouped by the group field, and we project four fields:

You can read more about aggregate functions available in AMPS here.

An example message from the agg_market_data topic would like this:

{"ask_total":2067.52,"bid_total":2063.52,"count":4,"group":10}

Once the view is defined, stop and restart the AMPS Server again. Now the agg_market_data topic can be queried and subscribed in the same way we did it with the market_data topic. In fact, all the features, such as conflation, deltas, content filtering, Out-of-Focus messages and pagination are also supported!

Let's add another Grid in the src/App.jsx file to test our new view:

<Grid
  title='Aggregated Market Data'
  showFilterBar
  columnDefs={[
    { headerName: 'Group', field: 'group', sort: 'asc' },
    curCol({ headerName: 'Bid Total', field: 'bid_total' }),
    curCol({ headerName: 'Ask Total', field: 'ask_total' }),
    { headerName: 'Symbols in Group', field: 'count' }
  ]}
  client={client}
  topic='agg_market_data'
  options='oof,conflation=3000ms'
/>

Now we have real-time market data aggregated updates!

View Server - Grid with a View Topic

Aggregated subscriptions

In addition to precomputed views and aggregates, AMPS provides the ability for the server to compute an aggregation for an individual subscription. When an application requests an aggregated subscription, rather than providing messages for the subscription verbatim, the AMPS server will calculate the requested aggregates and produce a message that contains the aggregated data.

Let's demonstrate this concept by adding another Grid that would display similar results to the view topic, but is built using an aggregated subscription:

<Grid
  title='Market Data - Aggregated Subscription'
  showFilterBar
  columnDefs={[
    { headerName: 'Group', field: 'group', sort: 'asc' },
    curCol({ headerName: 'Bid Total', field: 'bid_total' }),
    curCol({ headerName: 'Ask Total', field: 'ask_total' }),
    { headerName: 'Symbols in Group', field: 'count' }
  ]}
  client={client}
  topic='market_data'
  options='oof,conflation=3000ms,
           grouping=[/group],
           projection=[
              SUM(/ask) AS /ask_total,
              SUM(/bid) AS /bid_total,
              COUNT(/symbol) AS /count,
              /group
           ]'
/>

Notice that this Grid subscribes to the original market_data topic and not the view topic we created earlier.

AMPS provides aggregated subscriptions as a way to do ad hoc aggregation in cases where a specific aggregate is only needed for a short period time, will only be used by a single subscriber, or must be provided before the server can be restarted with a View configuration. If the aggregation is frequently used, or if multiple subscribers will use the aggregation, consider using a View rather than an aggregated subscription.

You can read more about aggregated subscriptions here.

Here's both grids side by side, displaying filtered data:

View Server - View and Aggregated Subscription

Chapter 17: AMPS Server Logs, Monitoring, and Statistics

An important part of any application, including our view server, is being able to monitor the application and understand what the application is doing in production.

The AMPS server provides many convenient ways of logging and monitoring the instance, including Admin API for integrating with third party monitoring tools, such as Prometheus, DataDog, and ITRS.

Logging

AMPS supports logging to many different targets including the console, syslog, and files. Every error message within AMPS is uniquely identified and can be filtered out or explicitly included in the logger output.

Let's add a trace level logging target to AMPS Server configuration so that we can see more information about what happens to the instance. The trace logging is quite verbose so we will rotate the log file, allowing no more than 200MB of logs to be stored at any one time:

<AMPSConfig>

   <!-- existing configuration here -->

    <Logging>

        <!-- add the Target below to the Logging configuration -->

        <Target>
            <Protocol>file</Protocol>
            <Level>trace</Level>
            <FileName>logs/amps.log</FileName>
            <RotationThreshold>200MB</RotationThreshold>
        </Target>
    </Logging>

    <!-- remaining configuration here -->

</AMPSConfig>

After restarting the AMPS Server, we can tail the logs/amps.log file to see the tail of the trace log:

tail -f logs/amps.log

The file provides informational messages on what AMPS is doing, and traces commands to AMPS and messages published from AMPS. If you ever need to understand what commands are being sent to AMPS, how AMPS is interpreting the commands, or what AMPS is sending to subscribers, the trace log is the best way to find that information. 60East strongly recommends that servers used for development keep a trace log to help developers troubleshoot issues, and answer questions about any errors or unexpected behavior.

You can read more about Logging here.

Monitoring AMPS

The AMPS monitoring interface has two distinct components:

Both Galvanometer and Admin API are available by default and fully support Authentication and Entitlement features of AMPS, along with the HTTPS (secure HTTP over SSL) encryption.

The Admin section in the AMPS Server configuration file specifies the URL of the monitoring tools:

<Admin>
    <InetAddr>localhost:8085</InetAddr>
</Admin>

Navigate to the http://<amps_hostname>:8085 in the browser and you'll see the main page of Galvanometer and the message flow widget in particular:

Galvanometer -- message flow

We give a nice overview of Galvanometer features in our blog. If you would like to learn more about the Admin API interface, the detailed overview is available here.

Statistics Database

AMPS Server provides various statistics that are exposed through the Admin API and Galvanometer. They're also optionally stored in the statistics database in the sqlite3 format. In order to enable statistics recording into a file, add the FileName field to the Admin section of the AMPS Server configuration file:

<AMPSConfig>
    <!-- existing configuration here -->


    <!-- replace the Admin section with the
         one below to keep statistics in a database file -->

    <Admin>
        <InetAddr>localhost:8085</InetAddr>
        <FileName>stats.db</FileName>
    </Admin>

   <!-- remaining configuration here -->

</AMPSConfig>

Recording statistics is useful for historical and investigative purposes, providing an insight on how the AMPS instance was loaded and performed through time, who connected to it and when, and many other statistics.

You can read more about the AMPS Statstics Database here.

Chapter 18: Further steps

We've covered a lot of concepts and features, and gone from the most basic example of how to use AMPS to a fully-featured view server. But this is just a fraction of what AMPS can do for you!

Below is a list of great AMPS features to look into when building your product:

Want to know more about how AMPS can help you build great applications? Still having trouble getting AMPS to play your tune? For help or questions, send us a note at support@crankuptheamps.com.