Case Study
HDFC Sky
Enterprise Investment & Trading Platform
Project Overview
HDFC Sky is a modern digital investment platform by HDFC Securities — one of India's largest financial services groups. It provides real-time stock market data, F&O derivatives, mutual funds, ETFs, commodities, IPOs, research calls, and educational content to retail investors.
The platform is powered by a custom three-tier architecture: a Node.js cron automation system for data ingestion, a Fastify TypeScript HTTP API for data processing and delivery, and a Next.js frontend backed by Redis/RediSearch for ultra-fast content serving. The engineering effort focused on building a reliable, scalable, and maintainable system capable of ingesting, processing, and serving real-time financial data at scale.
My Role
Engineering Challenges & Solutions
Real-time Market Data
Integrating live stock prices, indices, F&O derivatives, ETFs, and commodity data from multiple external sources (HDFC APIs, Trendlyne, BSE, PTI News) with low-latency requirements during market hours.
Designed a three-tier system: Node.js cron jobs fetch from external APIs → Fastify TypeScript API processes and transforms → Redis/RediSearch serves as the ultra-fast data layer for the Next.js frontend.
Multi-tier Data Architecture
Designing a data pipeline that moves information from external APIs → Node.js automation → Fastify HTTP API → WordPress CMS → Redis cache → Next.js frontend, handling failures at every layer.
Deployed Redis with RediSearch modules, creating JSON document indexes for each content type. Full resync on startup, incremental sync via last_id pagination, heartbeat-based health monitoring.
F&O Derivatives Complexity
Building a complete data model for Futures & Options — contracts, expiry dates, open interest, put/call ratios, implied volatility, max pain, heatmaps — across multiple indices and stocks.
Built a comprehensive F&O module with custom WordPress post types (future, option, fno_pc_ratio, fno_open_int, fno_change_oi, fno_vol_pcr, fno_max_pain, fno_heatmap) and a custom MySQL table for expiry metadata.
RediSearch Performance
Designing and maintaining 20+ RediSearch indexes across content types, optimizing FT.SEARCH and FT.AGGREGATE queries for stock screener with 6000+ items and 20+ filter dimensions.
Created 20+ FT.CREATE indexes with custom schemas for stocks, indices, F&O, ETFs, commodities, mutual funds, articles, news, IPOs, and research calls — enabling full-text search and faceted filtering.
WordPress as Headless CMS
Operating WordPress purely as a headless CMS with 20+ custom post types, 14 taxonomies, ACF metaboxes, WPML multilingual support, and real-time sync to Redis via WordPress hooks.
Built a custom WordPress plugin with 15 include files handling CPT registration, taxonomy management, ACF metaboxes, internal REST API for data sync, and real-time sync via transition_post_status hooks.
Cron Job Orchestration
Managing 15+ scheduled cron jobs with market hours awareness, holiday calendars, concurrency protection, timeout handling, and staggered schedules to prevent API rate limiting.
Implemented node-cron with market hours detection, holiday calendar (Republic Day, Independence Day, Gandhi Jayanti, Christmas), CRONS_ALREADY_RUNNING concurrency protection, and staggered batch schedules.
Kubernetes Deployment Pipeline
Setting up GitLab CI/CD with Docker multi-stage builds, Helm charts, and multi-environment deployment (dev/uat/pre-prod/prod) for three independent services.
Configured multi-stage Docker builds with separate Dockerfiles for each service, GitLab CI pipelines with dev/uat/pre-prod/prod deployment stages, and Helm chart integration for Kubernetes.
Chart Performance
Building interactive financial charts (OHLCV, candlestick, line, area, treemap) with Chart.js and TradingView lightweight-charts that render thousands of data points without jank.
Integrated TradingView lightweight-charts for real-time candlestick OHLCV charts, Chart.js for treemaps and auxiliary visualizations, with zoom plugins and custom data label formatting.
Architecture
Feature Highlights
Performance Engineering
Redis Caching
Full content layer served via Redis RediSearch JSON documents with FT.SEARCH queries, eliminating direct WordPress database queries on every page load.
Incremental Sync
Content changes synced from WordPress to Redis in real-time via WordPress hooks. Full resync only on cache miss or deployment.
CDN Distribution
Static assets served through dedicated CDN (static.hdfcsky.com) with 1-year cache headers for font files and long-lived asset caching.
Standalone Next.js
Built with Next.js standalone output mode for optimized Docker images and self-contained deployment without unnecessary dependencies.
Batch Processing
Data imports processed in configurable batches (25-100 items) with pagination, preventing memory spikes and enabling resumable syncs.
Timeout Management
Generous but bounded timeouts per operation — 120min for full stock imports, 15min for routine syncs, 30s for news — with AbortSignal.timeout patterns.
Data Automation & Cron Jobs
Deployed a Node.js cron automation system with 15+ scheduled jobs running at intervals from 30 seconds to 24 hours, handling data ingestion from HDFC internal APIs, Trendlyne SFTP, BSE indices, and PTI news feeds.
Redis & RediSearch Infrastructure
Deployed Redis with RediSearch modules, creating 20+ FT.CREATE indexes across all content types. The cache layer eliminates direct WordPress database queries, enabling sub-millisecond search and filtering for stock screeners, F&O heatmaps, and content discovery.
Technology Stack
Development Workflow
Engineering Contributions
Architected the complete three-tier backend — Node.js cron automation, Fastify TypeScript HTTP API, and Redis/RediSearch cache layer — enabling real-time financial data delivery to millions of users.
Built 20+ RediSearch indexes with custom schemas for stocks, F&O derivatives, ETFs, commodities, mutual funds, and content types, optimizing query performance for sub-millisecond responses.
Developed the Node.js cron automation system with 15+ scheduled jobs, market hours detection, holiday calendar, concurrency protection, and staggered batch processing.
Designed the WordPress headless CMS plugin with 20+ custom post types, 14 taxonomies, ACF metaboxes, WPML multilingual support, and real-time Redis sync via WordPress hooks.
Integrated TradingView lightweight-charts and Chart.js for interactive financial visualizations — OHLCV candlestick charts, treemaps, line charts — serving thousands of data points per request.
Built the F&O derivatives module covering futures, options, put/call ratio, open interest, change in OI, volume PCR, max pain, and heatmap visualizations.
Configured GitLab CI/CD with Docker multi-stage builds and Helm chart deployments across dev/uat/pre-prod/prod Kubernetes clusters.
Implemented the stock screener with 20+ filter dimensions, pre-computed lists (top gainers/losers, 52-week high/low, multi-bagger stocks), and 6000+ item query capacity.
Set up integration with multiple external data sources — HDFC internal APIs, Trendlyne SFTP + HTTP, BSE indices CSV, PTI news feed — with resilient error handling and retry logic.
Business Impact
Enterprise-grade platform architecture serving thousands of daily active users with real-time financial data during peak market hours.
Scalable three-tier system — cron automation, HTTP API, Redis cache — that handles data ingestion, processing, and delivery without performance degradation.
Multi-source data integration pipeline connecting HDFC internal systems, Trendlyne, BSE, and PTI News into a unified content platform.
Comprehensive market data coverage across stocks, indices, F&O derivatives, ETFs, commodities, and mutual funds.
Production-ready CI/CD pipeline with GitLab, Docker, and Kubernetes enabling reliable multi-environment deployments.
Headless CMS architecture that separates content management from presentation, enabling independent scaling of each tier.
Long-term platform maintainability through TypeScript codebase, modular architecture, and comprehensive documentation.
Lessons Learned
Real-time Data Architecture Is Hard
Serving live financial data through a multi-tier system requires careful thinking about failure modes at every layer — API outages, stale caches, partial syncs, and data consistency across independent services.
Caching Is Not Optional for Market Data
Without Redis/RediSearch, direct WordPress database queries would collapse under the load of stock screeners, F&O heatmaps, and real-time index updates. The cache layer is the most critical architectural decision.
Scheduled Jobs Need Production Discipline
Fifteen cron jobs running on different schedules with overlapping data dependencies requires careful orchestration — market hours, holiday calendars, concurrency locks, timeout management, and failure alerting.
WordPress Can Scale as Headless
Operating WordPress purely as a headless CMS with custom post types, ACF, and real-time sync to Redis proved that WordPress can serve as a scalable content management layer when decoupled from presentation.
Financial Data Needs Domain Expertise
Building F&O derivatives pages, stock screeners, and IPO dashboards required deep understanding of financial instruments — futures vs options, expiry cycles, margin requirements, and market microstructure.
CI/CD Pipeline Quality = Release Confidence
The multi-stage deployment pipeline with dev/uat/pre-prod/prod gates, automated builds, and Helm chart deployments gave the team confidence to release frequently without incident.