Market Data Server

Market Data Server

The Market Data Server receives market data from feed clients, organizes and persists the data, and provides query and subscription services to clients. It maintains real-time market data snapshots for each security, tracks security technicals (open, high, low, close, volume), and sequences all market data updates to ensure consistent ordering across the system. The server supports multiple market data types including best bid/offer quotes, book quotes, time and sales, and order imbalances.

The Market Data Server integrates with the Service Locator for authentication and a MySQL database for historical data storage. It operates as two distinct services: a registry server for managing security metadata and subscriptions, and a feed server for receiving and distributing market data updates.

Configuration

The Market Data Server is configured via a YAML file that defines network interfaces for both services, database connection, and Service Locator integration. Below is the structure of the configuration file with example values:

---
registry_server:
  # Primary network interface and port the Market Data Registry Server binds to.
  interface: "0.0.0.0:20300"
  
  # List of addresses the registry server is reachable at (for registration with Service Locator).
  addresses: ["198.51.100.5:20300", "10.0.0.5:20300"]

feed_server:
  # Primary network interface and port the Market Data Feed Server binds to.
  interface: "0.0.0.0:20400"
  
  # List of addresses the feed server is reachable at (for registration with Service Locator).
  addresses: ["198.51.100.5:20400", "10.0.0.5:20400"]

data_store:
  # The address of the MySQL server (host:port).
  address: "127.0.0.1:3306"
  
  # The username for authenticating with MySQL.
  username: spireadmin
  
  # The password for the MySQL user.
  password: 1234
  
  # The name of the database schema where data is stored.
  schema: spire

# Countries for which market data is supported.
countries:
  - AU
  - CA
  - HK
  - JP
  - US

service_locator:
  # The address of the Service Locator (host:port).
  address: "10.0.0.5:20000"
  
  # The account username used by the Market Data Server to authenticate with the Service Locator.
  username: market_data_service
  
  # The password for the Market Data Server's Service Locator account.
  password: admin_password
...

Functionality

Market Data Types

The server processes and distributes the following market data types:

• BBO Quotes (Best Bid/Offer) - Top-of-book bid and ask prices with sizes
• Book Quotes - Full order book depth with multiple price levels per side
• Time and Sales - Executed trade prices, sizes, and timestamps
• Order Imbalances - Pre-opening and closing auction imbalance data

Security Tracking

For each security, the server maintains:

• Real-time market data snapshots combining BBO, book depth, and last trade
• Security technicals including open, high, low, close, and volume
• Sequence numbers ensuring consistent ordering of market data updates
• Source tracking to identify which feed client published each data point

Data Sequencing

All market data updates are assigned monotonically increasing sequence numbers per security and data type. This ensures clients can detect gaps, maintain consistent ordering, and synchronize state reliably.

Historical Data Storage

Market data is persisted to MySQL with indexed queries supporting:

• Time range queries by security or venue
• Snapshot queries for initial state loading
• Efficient storage of high-frequency data streams

Utility Scripts

add_security_info.py

A utility script to add or update security metadata in the system:

python add_security_info.py
  --config config.yml       # Configuration file
  --symbol AAPL.NASDAQ      # Ticker symbol
  --name "Apple Inc."       # Display name
  --sector Technology       # Sector (optional)
  --board_lot 100          # Board lot size

dump_data_store.py

A backup and restore utility supporting bidirectional data transfer between MySQL and SQLite:

python dump_data_store.py
  --config config.yml              # Configuration file
  --start "2024-01-01 00:00:00"   # Start time range
  --end "2024-12-31 23:59:59"     # End time range
  --output backup.db               # Export MySQL to SQLite
  --cores 8                        # Number of parallel workers

Or to restore from SQLite to MySQL:

python dump_data_store.py
  --config config.yml
  --start "2024-01-01 00:00:00"
  --end "2024-12-31 23:59:59"
  --input backup.db
  --cores 8

The script uses multiprocessing to efficiently backup or restore large datasets across multiple securities and venues in parallel.

Installation & Setup

A setup.py script is provided to generate the final config.yml from the config.default.yml template.

The script supports the following arguments:

python setup.py
  --local 0.0.0.0           # Local interface (default: auto-detected IP)
  --world 198.51.100.5      # Global/public interface (optional)
  --address 10.0.0.5:20000  # Service Locator address (default: local_interface:20000)
  --password [REQUIRED]     # Service password for authentication
  --mysql_address 127.0.0.1:3306  # MySQL server address
  --mysql_username spireadmin     # MySQL username
  --mysql_password secretpw       # MySQL password (default: --password if omitted)
  --mysql_schema spire            # MySQL schema

Operations

The Market Data Server is controlled using three operational scripts: start.sh, stop.sh, and check.sh.

Log files are generated in the format:

srv_YYYYMMDD_HH_MM_SS.log

Upon startup, older log files are moved into the logs/ directory.

check.sh

The check.sh script verifies whether the server is currently running by inspecting the PID recorded in pid.lock and testing whether the associated process exists.

If the server is not running, it prints:

MarketDataServer is not running.

start.sh

The start.sh script:

• Exits immediately if the server is already running
• Creates a logs/ directory if necessary
• Moves any existing srv_*.log files into logs/
• Starts the MarketDataServer process in the background
• Writes the PID to pid.lock
• Reads network interfaces from config.yml and waits until the server is listening on at least one configured address

This ensures the server is fully initialized before the script exits.

stop.sh

The stop.sh script:

• Reads the PID from pid.lock
• Sends SIGINT to request a graceful shutdown
• Waits for termination using exponential backoff (up to 300 seconds)
• Sends SIGKILL if the server fails to stop cleanly
• Appends a forced-termination message to the most recent log file (if applicable)
• Removes the pid.lock file

This guarantees consistent shutdown behavior across normal and exceptional conditions.