Context & Pain Points
Magento 2 emits a search query log for every store search, and at this volume the raw stream is both the most useful merchandising signal available and far too large to keep in the application database. The traffic shape is the hard part: retail spikes arrive on promotions and holidays, so an ingestion path sized for the average drops records exactly when the data matters most. Merchandisers then want to ask questions of the last few minutes, which rules out a nightly batch, while auditors want the same records years later, which rules out keeping everything in a hot search index. One pipeline had to serve a real-time read path and a long archive without paying hot-storage rates for the archive.
What We Had To Solve
- Scaling ingestion buffer rates to handle extreme traffic spikes without data loss, on a stream whose peak is many times its average.
- Optimizing OpenSearch cluster indices to maintain fast query response times under high database volumes, where an index that grows unbounded slowly turns every query into a full scan. Index size, not query complexity, was the variable that mattered most.
- Partitioning and compressing historical search histories for long-term audit archiving, without keeping years of records on storage priced for live search.
- Normalising records at ingest, since Magento search logs arrive with the inconsistencies of real store traffic and an analytics index is only as good as the fields it can group on.
How We Built It
- Deployed Amazon Kinesis Firehose streams as serverless ingestion buffers, batching and compressing records before index storage, so peak traffic is absorbed by the buffer instead of backpressuring the store.
- Engineered serverless pre-processing workers using AWS Lambda to clean and format records on the fly, so every document reaching OpenSearch already has the shape the analytics queries group on. Lambda scales with the Firehose batch rate, so pre-processing capacity follows the stream instead of being sized for peak up front.
- Implemented automated OpenSearch index rotation cycles, keeping each index in a size band that answers queries quickly and letting old indices age out as whole units rather than by document.
- Configured S3 lifecycle policies to transition old logs to Glacier, so the audit archive stays complete and retrievable while costing a fraction of hot storage.
- Aggregated the dashboard-facing metrics into Aurora, so recurring reporting queries read a compact relational summary rather than scanning the raw index every time.
Outcomes That Mattered
500M+ Daily Ingestion
Processed and formatted massive daily record streams without dropping a single log entry, including through promotional traffic spikes.
Sub-Second Search
Achieved sub-second response times on complex search analytics queries against billions of index rows, with index rotation keeping each shard in a size the cluster answers quickly.
60% Storage Reduction
Pruned archiving costs by 60% using gzip compression and automated S3 transition policies, without shortening the retention window.
Outcome
Search logs stream through Amazon Kinesis Firehose into Amazon OpenSearch Service, with AWS Lambda cleaning and reshaping records in flight, and the platform processes over 500 million Magento 2 records daily. Firehose absorbs the spikes as a serverless buffer, batching and compressing before anything reaches an index, so a promotion changes the batch size rather than the success rate. Aurora PostgreSQL holds the aggregated analytics the dashboards read. Indices rotate on a schedule and S3 lifecycle policies move older logs down to Glacier, so recent data stays searchable in sub-second time while the audit archive costs archive money. Merchandisers query the last few minutes of store search behaviour from the same pipeline that feeds the audit archive, so there is one path to maintain instead of a live one and a batch one.