Market Data Server

The Market Data Server receives market data from feed clients, organizes and persists the data, and provides query and subscription services to clients. It maintains real-time market data snapshots for each security, tracks security technicals (open, high, low, close, volume), and sequences all market data updates to ensure consistent ordering across the system. The server supports multiple market data types including best bid/offer quotes, book quotes, time and sales, and order imbalances.

The Market Data Server integrates with the Service Locator for authentication and a MySQL database for historical data storage. It operates as two distinct services: a registry server for managing security metadata and subscriptions, and a feed server for receiving and distributing market data updates.

Configuration

The Market Data Server is configured via a YAML file with five top-level sections: service_locator, registry_server, feed_server, data_store, and an optional countries filter. A cache_block_size may also be specified. Below is the structure of the configuration file with example values:

service_locator:
  # The address of the Service Locator (host:port).
  address: "10.0.0.5:20000"
  # The account username used by the Market Data Server to authenticate with the Service Locator.
  username: market_data_server
  # The password for the Market Data Server's Service Locator account.
  password: [REQUIRED]

registry_server:
  # Primary network interface and port the Market Data Registry Server binds to.
  interface: "0.0.0.0:20300"
  # List of addresses the registry server is reachable at (for registration with Service Locator).
  addresses: ["198.51.100.5:20300", "10.0.0.5:20300"]

feed_server:
  # Primary network interface and port the Market Data Feed Server binds to.
  interface: "0.0.0.0:20400"
  # List of addresses the feed server is reachable at (for registration with Service Locator).
  addresses: ["198.51.100.5:20400", "10.0.0.5:20400"]

data_store:
  # The address of the MySQL server.
  address: "127.0.0.1:3306"
  # The username used to authenticate with MySQL.
  username: spireadmin
  # The password for the MySQL user.
  password: [REQUIRED]
  # The name of the database schema where data is stored.
  schema: spire

# Optional restriction of market data to specified countries (by ISO code).
countries: ["US", "CA"]

# Number of records to buffer per write to the historical data store (default: 1000).
cache_block_size: 1000

A setup.py script is provided to generate the final config.yml from the config.default.yml template. Usage:

  --local 0.0.0.0                 # Local interface (default: auto-detected IP)
  --world 198.51.100.5            # Global/public interface (optional)
  --address 10.0.0.5:20000        # Service Locator address (default: local_interface:20000)
  --password [REQUIRED]           # Service password for authentication
  --mysql_address 127.0.0.1:3306  # MySQL server address
  --mysql_username spireadmin     # MySQL username
  --mysql_password secretpw       # MySQL password (default: --password if omitted)

Market Data Types

The server processes and distributes the following market data types:

BBO Quotes (Best Bid/Offer) - Top-of-book bid and ask prices with sizes
Book Quotes - Full order book depth with multiple price levels per side
Time and Sales - Executed trade prices, sizes, and timestamps
Order Imbalances - Pre-opening and closing auction imbalance data

In addition to raw market data, the server publishes:

Real-time market data snapshots combining BBO, book depth, and last trade
Security technicals including open, high, low, close, and volume
Sequence numbers ensuring consistent ordering of market data updates
Source tracking to identify which feed client published each data point

All market data updates are assigned monotonically increasing sequence numbers per security and data type. This ensures clients can detect gaps, maintain consistent ordering, and synchronize state reliably.

Storage

Market data is persisted to MySQL with indexed queries supporting:

Efficient storage of high-frequency data streams
Time-bounded historical queries
Per-security and per-data-type lookups

Utility Scripts

A utility script is provided to add or update security metadata in the system.

A backup and restore utility supports bidirectional data transfer between MySQL and SQLite:

  --config config.yml              # Configuration file
  --start "2024-01-01 00:00:00"   # Start time range
  --end "2024-12-31 23:59:59"     # End time range
  --output backup.db               # Export MySQL to SQLite
  --cores 8                        # Number of parallel workers

The script uses multiprocessing to efficiently backup or restore large datasets across multiple securities and venues in parallel.

Management

The Market Data Server is controlled using three operational scripts: start.sh, stop.sh, and check.sh.

start.sh

Exits immediately if the server is already running.
Creates a logs/ directory if necessary.
Moves any existing srv_*.log files into logs/.
Starts the MarketDataServer process in the background.
Reads network interfaces from config.yml and waits until the server is listening on at least one configured address.

This ensures the server is fully initialized before the script exits.

stop.sh

Sends SIGINT to request a graceful shutdown.
Waits for termination using exponential backoff (up to 300 seconds).
Sends SIGKILL if the server fails to stop cleanly.
Appends a forced-termination message to the most recent log file (if applicable).

This guarantees consistent shutdown behavior across normal and exceptional conditions.

check.sh

The check.sh script verifies whether the server is currently running by inspecting the PID recorded in pid.lock and testing whether the associated process exists.

Logging

Upon startup, older log files are moved into the logs/ directory.

Navigation menu