Year in Review (2025): Systems Engineering

View Mode Toggles:

The theme for 2025 was Systems Engineering—not just writing code, but architecting and deploying end-to-end systems which are secure by design, resource-efficient, and robust in data-handling. Each of the following projects pushes this principle in a different direction.

Scalable Architecture for Identity-Linked Data

Overarching the 2025 roadmap was an infrastructure platform engineered to link consumer wearable biometrics with verified national identity layers. This system bridges Polar AccessLink (biometrics) and SingPass MyInfo (identity), solving a critical challenge of ingesting high-volume and identity-sensitive health data—without exposing identity fields to intermediate states or database administrators.

The platform was designed as a strictly event-driven, serverless ecosystem on AWS, emphasizing granular scalability and isolation.

• Identity Federation & Data Privacy (updated): One significant engineering complexity was the handling of NRIC data. A layered security flow was implemented: sensitive data is transmitted between services via secure HTTP POST (TLS)—preventing exposure in browser history or server logs—and is persisted in DynamoDB as an RSA-2048–encrypted payload. This ensures that only designated data custodians holding the private-key can recover the plaintext NRIC, while backend services only process plaintext in volatile memory during the transaction.

• OLTP vs. OLAP Separation: Operational and analytical data planes were strictly separated. Ingestion is write-optimized, asynchronously routing payloads to DynamoDB and S3. Downstream analytical data is read-optimized via a scheduled incremental load process which transforms JSON into columnar Parquet format, and made queryable via Amazon Athena. Large-scale queries can thus be performed downstream without impacting live ingestion workloads, while also preserving cost-efficiency.

• Other areas which have broadly been covered in a previous post include:

  • Asynchronous concurrency, SQS buffering, rate-limiting
  • Infrastructure as Code
  • CI/CD and Containerization
  • Observability and Diagnostics
  • Development Environment and Tooling
  • LLM-Assisted Development

Traffic Engineering for LLM Fleet

llm-control-plane is about managing a distributed fleet of self-deployed LLMs on heterogeneous hardware. A centralized smart proxy and routing layer was constructed.

• Semantic & Hardware-Aware Routing: The system routes traffic intelligently—a WorkloadClassifier categorizes incoming prompts by intent (e.g., reasoning, programming, high-throughput etc.) with a lightweight LLM, enabling the LLMRouter to dispatch requests appropriately. By aligning task complexity with hardware capabilities, the routing layer assigns heavy inferential workloads to high-VRAM clusters and trivial queries to agile, low-overhead instances, ensuring optimal utilization of heterogeneous compute resources. These routing decisions assume that LLM nodes are themselves well-optimized. For example, one class of endpoints is backed by custom Vulkan builds of llama.cpp on Strix Halo, described in this post.

• Streaming Proxy with Protocol Normalization: The core of the system is a custom FastAPI proxy designed to intercept and normalize OpenAI-compatible requests. A notable engineering challenge was unifying divergent model output formats—in particular separating internal “reasoning” traces (chain-of-though) from final text output. A robust stream processor that buffers and detects provider-specific delimiters (including <think> tags or proprietary control tokens) in real-time was implemented. This logic dynamically bifurcates the incoming Server-Sent Events (SSE) stream into distinct “reasoning” and “content” channels for the client for easier and standardized client-side consumption, while an SSEAccumulator simultaneously reconstructs the complete response server-side for conversation history logging (e.g. for context).

• Real-Time Observability Plane: To validate these orchestration decisions, a dedicated observability interface was developed using Shiny for Python. This dashboard was specifically engineered to expose the router’s internal state, visualizing the rationale behind specific dispatches and surface other performance metrics. This mechanism provides transparent verification that complex workloads are correctly landing on high-compute targets, allowing for subsequent feedback-based fine-tuning and troubleshooting of the smart routing.

• Stateless Core with Config-Driven Topology: The control-plane routing layer is stateless at the process level, and all LLM nodes are fully stateless. Endpoints are (re-)discovered and configured via a registry, while conversation state is stored centrally in persistent storage accessible only by the control plane. For each request, the router reconstructs the conversation from storage and sends the full context to a selected LLM node; these LLM nodes never hold or read conversational state themselves. This keeps the entire system horizontally scalable and simplifies failure recovery: any failed LLM nodes can be restarted or replaced without affecting in-memory session state.

• Extensibility and Contract-First Design: The entire system is built around explicit contracts:

  • a contract for how workloads are classified;
  • a contract for how endpoints advertise capabilities; and
  • a contract for how streams (reasoning + content) are emitted and consumed.

  By treating these as stable interfaces, it becomes straightforward to add new workloads (e.g., evaluation, embedding-only, high-token “dump” modes), new endpoints (new nodes, new runtimes), or new visualizations in the observability layer without destabilizing the existing system.

Algorithmic Rigor in Signal Processing

fit-diff is a specialized analytical application for the validation of wearable sensor data. The complexity lies in transforming messy, asynchronous sensor data into a format suitable for statistical comparison—it employs mathematical optimization to reconcile asynchronous hardware clocks and proprietary PPG algorithmic variance.

• Adaptive Step-Size Search: The determine_optimal_shift function implements a search algorithm that iteratively shifts the “test” signal against the “reference.” It calculates an adaptive step size based on the greatest common divisor (GCD) of the sampling intervals of both files, ensuring the shift resolution matches the data density while attempting to minimize computational overhead.

• Loss Function Optimization: The alignment isn’t arbitrary; it minimizes a specific loss function (Mean Absolute Error, Mean Squared Error) or maximizes a correlation function (Pearson, Concordance Correlation Coefficient) to find an appropriate temporal offset.

Ongoing work includes an algorithmic validation engine designed to detect anomalies within asynchronous data streams—without relying on “benchmark” devices. As participants are expected to perform similar activities, physiological patterns are also expected to somewhat converge. By treating heart rate streams as signals, and applying z-normalization and global cross-correlation, signal overlap can be maximized mathematically to establish a baseline of “expected” behavior. This process will serve primarily as a quality gate: rather than just aligning data, potential “outlier” devices or sensor failure incidents are flagged, and subsequently excluded from final analysis.

Robust Data Acquisition via Browser Automation

To correlate physiological metrics with environmental stressors, a custom Robotic Process Automation (RPA) engine was engineered to retrieve weather data locked behind a legacy portal lacking modern API access.

HTTP scraping was insufficient due to the target’s frame-based architecture and session management. A highly-customized and extensible automation layer was thus built using Selenium to autonomously navigates the legacy interface, handling multi-step selection logic for weather stations and parameters, effectively creating a “phantom API” over a GUI-only backend.

The system was designed to operate autonomously in a headless environment, orchestrating the scheduled download of TSV-like files as they are generated. These artifacts are subsequently normalized and synced to an S3 Data Lake, decoupling the downstream analytical layers from the fragility of the acquisition method and ensuring that recent environmental data is always available for correlation.

Some key features of the system include:

• Fluent Interface Pattern: The core Automata class wraps the raw Selenium WebDriver in a Fluent Interface. This allows for method chaining, making the automation scripts readable and declarative. Instead of verbose Selenium boilerplate, the code reads like a set of instructions:

  automata.open(url).select(...).input_text(...).click(...).wait(...).click.(...)
  

  Even logic for logins and navigation is encapsulated in high-level methods.

• Idempotent Merging Logic: The entire pipeline is designed to be re-runnable. The _merge_with_existing_data method checks S3 for existing Parquet partitions, loads them, merges newly retrieved data, and deduplicates based on timestamp and station ID. This ensures that re-running the scraper for overlapping dates does not corrupt the dataset, failed runs will automatically be retried without data loss, and the entire historical dataset can be rebuilt from scratch if absolutely necessary.

Data Lake and Onboarding

This project is primarily a Shiny (Python) application, allowing users to explore AWS Data Lake through a reactive web interface rather than looking at metadata, schemas and even raw data files.

• State Management via Reactive Caching: Race conditions—where UI elements renders before schemas resolves—were fixed by preloading the AWS Glue catalog into memory at startup via initialize_data_cache. Despite the built-in reactivity, the reactive graph was also hardened by enforcing explicit dependencies in analysis functions, ensuring stale computations are invalidated only when the full context is available.

• Hybrid Compute & Dual-Layer Caching: The platform bridges serverless and local compute. Heavy data retrieval is offloaded to AWS Athena, while fine-grained statistical tests (ANOVA, OLS) run in-memory. To support this, a dual-layer caching strategy was implemented: global schema caching for UI responsiveness, and per-session query result caching to prevent redundant S3 scans and preserve Athena concurrency limits.

• Semantic Data Integrity: To address “schema-on-read” ambiguity, instead of relying on automated detection, the system relies on configurations (ui_config.py) to enforce data types. This guarantees that identifiers like numeric IDs are treated as categorical variables, disallowing erroneous regression analyses and ensuring robustness of the automated statistical engine.

• Client-Side Infrastructure Automation: A PowerShell automation script was used to encapsulate complex ODBC connection parameters (e.g., AWS region, IAM profiles, S3 output locations) into a single execution. It ensures every analyst’s local environment is identical, effectively decoupling connectivity troubleshooting from the actual data analysis work.

The theme for 2025 was Systems Engineering—not just writing code, but architecting and deploying end-to-end systems that are secure by design, resource-efficient, and robust in data-handling. Each project that follows pushes that principle in a different direction.

Note: the following sections are almost completely LLM-generated based on the version marked “Original”.

Scalable Architecture for Identity-Linked Biometric Data

The core platform for 2025 was a secure bridge between consumer wearables and verified national identity. In practical terms: it connects Polar (for heart rate and activity) with SingPass MyInfo (for identity), while making sure that sensitive details like NRIC never appear in plain text to anyone operating or maintaining the system.

The platform is built as a collection of micro-services on AWS that only activate when needed, instead of one big always-on server. This makes it easier to scale, isolate problems, and keep the system maintainable over time.

• Hybrid Encryption & Identity Federation Handling NRIC data required extra care. The system leverages TLS-encrypted POST requests for secure transit between components—preventing leakage in logs or browser history—while ensuring that only an asymmetrically encrypted, locked version is stored in the database. A small group of authorised data custodians hold the RSA private key needed to unlock and read the NRIC; backend services see it in readable form only briefly in memory during processing, and never store it in plain text.

• OLTP vs. OLAP Separation The system keeps “live operations” and “analytics” strictly apart. Incoming data is written immediately into a fast operational store (DynamoDB and S3) optimised for many small writes and high concurrency. Separately, a scheduled process transforms this raw JSON into columnar Parquet files and builds a simple “lakehouse” for analysis. This means analysts can query large historical datasets cheaply and quickly (via tools like Athena), without affecting the live ingestion workload.

• Other areas which have broadly been covered in a previous post include:

  • Asynchronous concurrency, SQS buffering, rate-limiting
  • Infrastructure as Code
  • CI/CD and Containerization
  • Observability and Diagnostics
  • Development Environment and Tooling
  • LLM-Assisted Development

Traffic Engineering for LLM Endpoints

This project is about running and orchestrating a private fleet of LLMs across multiple machines. At the centre is a layer that behaves less like a “smart traffic controller” for AI workloads.

• Semantic & Hardware-Aware Routing Incoming requests are first classified by a lightweight model (WorkloadClassifier) that identifies what kind of job they represent—deep reasoning, code generation, quick chat, batch-style throughput, and so on. Based on that classification, an LLMRouter sends the request to the most appropriate backend machine. Heavy, long-running tasks go to nodes with large GPUs and high VRAM; lighter queries go to smaller, faster nodes. This prevents a single demanding request from clogging up resources meant for fast responses, and improves overall throughput and latency. These routing decisions assume that the backend nodes are themselves well-tuned. For example, one class of endpoints is backed by custom Vulkan builds of llama.cpp on Strix Halo, described in this post.

• Streaming Proxy with Protocol Normalization A custom FastAPI proxy acts as the single entry point for clients, speaking an OpenAI-compatible API. Behind it, different backends may have slightly different response formats, especially around “reasoning traces” or chain-of-thought output. The proxy’s stream processor watches the Server-Sent Events (SSE) stream in real time, detecting provider-specific markers (for example, tags like <think>). It splits the stream into two parallel channels:

  • one for internal reasoning traces (which can be logged, analysed, or hidden from end users),
  • one for the final answer sent back to the client.

  At the same time, an SSEAccumulator rebuilds the full response on the server to support conversation history, debugging, and offline analysis.

• Real-Time Observability Plane To make sure routing decisions remain transparent and trustworthy, the system exposes an observability dashboard built with Shiny for Python. Instead of just showing chat messages, this dashboard displays why a particular node was chosen: the classification result, confidence scores, and the hardware profile of the endpoint that handled the request (GPU type, VRAM, etc.). This creates a feedback loop: one can see, in real time, that complex reasoning queries do indeed land on the most capable nodes, while light traffic is efficiently routed elsewhere.

• Stateless Core with Config-Driven Topology The central “traffic controller” is the only part of the system that remembers your past messages. It stores conversation history in a durable, shared store that it can look up when needed. The AI machines behind it don’t keep any memory at all—they simply receive the full context from the controller for each request and send back a reply. Because of this, any AI machine can be added, removed, or restarted at any time without losing conversations or moving fragile in-memory data around.

• Extensibility and Clear-Rule Design The system is built around a few very clear, shared rules for how parts talk to each other:

  • a rule for how incoming requests are labelled (e.g. “needs deep reasoning” vs “simple and fast”);
  • a rule for how each machine describes what it can handle; and
  • a rule for how model responses are split into “thinking process” and “final answer.”

  Because these rules stay stable even as the system evolves, it is easy to plug in new kinds of work, new machines, or new dashboards without breaking what already exists. New pieces just have to follow the same rules to fit in.

Algorithmic Rigor in Signal Processing

fit-diff is a tool built to compare and validate sensor data from different wearables—say, a chest strap vs. a smartwatch—without assuming that either one is perfect.

The main difficulty is that each device records at its own sampling rate and starts at its own time. One cannot simply line up timestamps and subtract; doing so would produce misleading statistics.

• Adaptive Step-Size Search The determine_optimal_shift function tackles this by treating one signal as a reference and sliding the other forward and backward in time. At each small time shift, it computes a score describing how well the two signals line up. The step size isn’t arbitrary: it is based on the greatest common divisor (GCD) of the sampling intervals of both devices. This gives a sensible minimum shift that respects the data’s resolution while keeping the search computationally manageable.

• Loss Function Optimization To decide which time shift is “best,” the algorithm minimises or maximises a chosen metric—for example:

  • minimising Mean Absolute Error or Mean Squared Error, or
  • maximising correlation measures like Pearson or Concordance Correlation Coefficient.

  The result is a principled estimate of how much one device’s timeline should be shifted to align with another’s, enabling fair comparisons of heart rate, cadence, or other metrics.

Future work extends this from alignment to validation. Instead of comparing devices against a designated “gold standard,” it looks at how similar participants’ heart rate signals are under similar activities. By standardising (z-normalising) each time series and applying global cross-correlation, the system can estimate how well a given device’s data matches the overall pattern. Devices with consistently low correlations—i.e., they don’t resemble anyone else’s physiology in comparable conditions—can be flagged as likely outliers or failing sensors, and excluded before formal analysis.

Robust Data Acquisition via Browser Automation

To relate physiology to environmental conditions (such as heat stress), the system needed weather data that was only available through an old, complex web portal. There was no usable API, and traditional HTTP scraping does not work on frames, redirects, or session handling.

To work around this, a dedicated Robotic Process Automation (RPA) engine was built on top of Selenium to runs in a headless environment (no visible browser window), routinely logging into the portal, selecting the right weather stations and parameters, and downloading data files as they become available. Once downloaded, these files are standardised and stored in S3 so that downstream analytics can treat them as a clean, reliable data source without worrying about how they were obtained.

Key design elements include:

• Fluent Interface Pattern The core Automata class wraps Selenium in a more readable, chainable API. Instead of low-level driver calls, automation scripts read almost like a set of instructions:

  automata.open(url).select(...).input_text(...).click(...).wait(...).click(...)
  

  Even multi-step flows such as login, navigation, and file downloads are encapsulated as high-level operations, reducing complexity and making the automation easier to maintain.

• Idempotent Merging Logic The ingestion pipeline is designed to tolerate retries and partial failures. When new data arrives, a merge step checks if there are existing Parquet files in S3 for the same stations and time ranges. It then loads the existing data, combines it with the new batch, and removes duplicates based on timestamp and station ID. This means one can safely re-run the scraper for overlapping periods, restart after failures, or even rebuild the entire historical dataset from scratch—without corrupting data or double-counting records.

Data Lake and Onboarding

This project is essentially a web dashboard that allows users to easily explore and analyze data stored in the cloud (AWS). Instead of requiring users to write code or look at raw, messy files, it provides a visual interface for clicking through data and running statistics.

• Stable & Reliable Interface “Glitches” where the interface attempted to display information prematurely were eliminated by pre-loading the table of contents (catalog) upon startup. The system also ensures that calculations do not execute mid-change; it waits until option selection is completely finished before updating results, effectively preventing transient errors.

• Smart Processing & Speed The platform efficiently splits the workload. It leverages the massive power of the cloud to find and retrieve large chunks of data, while detailed statistical calculations are handled instantly by the application itself. Results are also “saved” for the duration of the session, preventing the waste of time and resources on re-downloading the exact same files.

• Data Type Clarity Automated systems sometimes misinterpret what data represents (for example, treating a “User ID” number as a value to be added or subtracted). Instead of relying on guesswork, the application uses a strict rulebook (ui_config.py) that explicitly defines each piece of data. This prevents nonsensical mathematical errors, such as attempting to calculate the “average” of a User ID.

• One-Click Setup Connecting local environments to the cloud usually involves complex, error-prone configuration. This manual process was replaced with a single automated script. This ensures that every analyst’s computer is set up identically, allowing teams to focus on actual analysis rather than troubleshooting connectivity issues.