Kamal: Created page with "= Market Data Server = The Market Data Server receives market data from feed clients, organizes and persists the data, and provides query and subscription services to clients...."

2025-12-01T22:09:49Z

Created page with "= Market Data Server = The Market Data Server receives market data from feed clients, organizes and persists the data, and provides query and subscription services to clients...."

New page

= Market Data Server =
The Market Data Server receives market data from feed clients, organizes and persists the data, and provides query and subscription services to clients. It maintains real-time market data snapshots for each security, tracks security technicals (open, high, low, close, volume), and sequences all market data updates to ensure consistent ordering across the system. The server supports multiple market data types including best bid/offer quotes, book quotes, time and sales, and order imbalances.

The Market Data Server integrates with the Service Locator for authentication and a MySQL database for historical data storage. It operates as two distinct services: a registry server for managing security metadata and subscriptions, and a feed server for receiving and distributing market data updates.

== Configuration ==
The Market Data Server is configured via a YAML file that defines network interfaces for both services, database connection, and Service Locator integration. Below is the structure of the configuration file with example values:

<pre>
---
registry_server:
# Primary network interface and port the Market Data Registry Server binds to.
interface: "0.0.0.0:20300"

# List of addresses the registry server is reachable at (for registration with Service Locator).
addresses: ["198.51.100.5:20300", "10.0.0.5:20300"]

feed_server:
# Primary network interface and port the Market Data Feed Server binds to.
interface: "0.0.0.0:20400"

# List of addresses the feed server is reachable at (for registration with Service Locator).
addresses: ["198.51.100.5:20400", "10.0.0.5:20400"]

data_store:
# The address of the MySQL server (host:port).
address: "127.0.0.1:3306"

# The username for authenticating with MySQL.
username: spireadmin

# The password for the MySQL user.
password: 1234

# The name of the database schema where data is stored.
schema: spire

# Countries for which market data is supported.
countries:
- AU
- CA
- HK
- JP
- US

service_locator:
# The address of the Service Locator (host:port).
address: "10.0.0.5:20000"

# The account username used by the Market Data Server to authenticate with the Service Locator.
username: market_data_service

# The password for the Market Data Server's Service Locator account.
password: admin_password
...
</pre>

== Functionality ==

=== Market Data Types ===
The server processes and distributes the following market data types:

* '''BBO Quotes''' (Best Bid/Offer) - Top-of-book bid and ask prices with sizes
* '''Book Quotes''' - Full order book depth with multiple price levels per side
* '''Time and Sales''' - Executed trade prices, sizes, and timestamps
* '''Order Imbalances''' - Pre-opening and closing auction imbalance data

=== Security Tracking ===
For each security, the server maintains:

* Real-time market data snapshots combining BBO, book depth, and last trade
* Security technicals including open, high, low, close, and volume
* Sequence numbers ensuring consistent ordering of market data updates
* Source tracking to identify which feed client published each data point

=== Data Sequencing ===
All market data updates are assigned monotonically increasing sequence numbers per security and data type. This ensures clients can detect gaps, maintain consistent ordering, and synchronize state reliably.

=== Historical Data Storage ===
Market data is persisted to MySQL with indexed queries supporting:

* Time range queries by security or venue
* Snapshot queries for initial state loading
* Efficient storage of high-frequency data streams

== Utility Scripts ==

=== add_security_info.py ===
A utility script to add or update security metadata in the system:

<pre>
python add_security_info.py
--config config.yml # Configuration file
--symbol AAPL.NASDAQ # Ticker symbol
--name "Apple Inc." # Display name
--sector Technology # Sector (optional)
--board_lot 100 # Board lot size
</pre>

=== dump_data_store.py ===
A backup and restore utility supporting bidirectional data transfer between MySQL and SQLite:

<pre>
python dump_data_store.py
--config config.yml # Configuration file
--start "2024-01-01 00:00:00" # Start time range
--end "2024-12-31 23:59:59" # End time range
--output backup.db # Export MySQL to SQLite
--cores 8 # Number of parallel workers
</pre>

Or to restore from SQLite to MySQL:

<pre>
python dump_data_store.py
--config config.yml
--start "2024-01-01 00:00:00"
--end "2024-12-31 23:59:59"
--input backup.db
--cores 8
</pre>

The script uses multiprocessing to efficiently backup or restore large datasets across multiple securities and venues in parallel.

== Installation & Setup ==
A <code>setup.py</code> script is provided to generate the final <code>config.yml</code> from the <code>config.default.yml</code> template.

The script supports the following arguments:

<pre>
python setup.py
--local 0.0.0.0 # Local interface (default: auto-detected IP)
--world 198.51.100.5 # Global/public interface (optional)
--address 10.0.0.5:20000 # Service Locator address (default: local_interface:20000)
--password [REQUIRED] # Service password for authentication
--mysql_address 127.0.0.1:3306 # MySQL server address
--mysql_username spireadmin # MySQL username
--mysql_password secretpw # MySQL password (default: --password if omitted)
--mysql_schema spire # MySQL schema
</pre>

== Operations ==
The Market Data Server is controlled using three operational scripts: <code>start.sh</code>, <code>stop.sh</code>, and <code>check.sh</code>.

Log files are generated in the format:
<pre>
srv_YYYYMMDD_HH_MM_SS.log
</pre>

Upon startup, older log files are moved into the <code>logs/</code> directory.

=== check.sh ===
The <code>check.sh</code> script verifies whether the server is currently running by inspecting the PID recorded in <code>pid.lock</code> and testing whether the associated process exists.

If the server is not running, it prints:
<pre>
MarketDataServer is not running.
</pre>

=== start.sh ===
The <code>start.sh</code> script:
* Exits immediately if the server is already running
* Creates a <code>logs/</code> directory if necessary
* Moves any existing <code>srv_*.log</code> files into <code>logs/</code>
* Starts the <code>MarketDataServer</code> process in the background
* Writes the PID to <code>pid.lock</code>
* Reads network interfaces from <code>config.yml</code> and waits until the server is listening on at least one configured address

This ensures the server is fully initialized before the script exits.

=== stop.sh ===
The <code>stop.sh</code> script:
* Reads the PID from <code>pid.lock</code>
* Sends <code>SIGINT</code> to request a graceful shutdown
* Waits for termination using exponential backoff (up to 300 seconds)
* Sends <code>SIGKILL</code> if the server fails to stop cleanly
* Appends a forced-termination message to the most recent log file (if applicable)
* Removes the <code>pid.lock</code> file

This guarantees consistent shutdown behavior across normal and exceptional conditions.

Market Data Server - Revision history

Kamal: Created page with "= Market Data Server = The Market Data Server receives market data from feed clients, organizes and persists the data, and provides query and subscription services to clients...."