Browse past weeks of engineering reads.
Netflix engineers needed a real-time, unified view of service dependencies across their microservices architecture to enable faster troubleshooting and understand blast radius during incidents.
Netflix needed to deploy and serve large language models at scale within their production environment rather than relying on third-party hosted APIs, while maintaining the reliability and performance standards required for their streaming platform.
Netflix needed to generate personalized homepages at scale where every row, entity ordering, and layout element is customized per user while maintaining low latency and high relevance.
Building a data analysis system that can reliably infer causal relationships from observational data while accounting for hidden biases and confounding variables that automated agents might miss.
Netflix needed to manage governance, access control, and orchestration of millions of data warehouse tables and tens of thousands of scheduled workloads across their data platform.
Netflix needed to simplify and standardize batch compute job queueing and scheduling while transitioning their container infrastructure to be more Kubernetes-native.
Netflix needed to predict and mitigate risks associated with content launches to improve availability and reduce failures during high-impact release events.
Preventing corrupted catalog metadata from reaching millions of Netflix viewers by detecting data transformation failures in production before impact.
Netflix needed to consolidate multiple bespoke data movement connectors across different organizations into a unified, centralized management system for batch data movement operations.
How to design a notification system that intelligently decides when and how to deliver personalized notifications to users with varying urgency and cognitive load requirements.
Netflix needed to enable video editors to have more fine-grained control and predictability over AI-assisted video editing systems for creating promotional content at scale.
Netflix needed to improve the accuracy of video quality assessment metrics beyond VMAF v0 to make better encoding decisions and prevent quality misjudgments that could degrade the member experience.
Building a reliable causal inference system that can accurately determine cause-and-effect relationships in data (like the impact of a show on retention) while accounting for hidden biases and confounding variables that automated agents might miss.
Managing millions of data assets, tables, and tens of thousands of scheduled workloads across Netflix's data platform while maintaining proper access control and execution governance at scale.
Netflix needed to efficiently partition and manage petabytes of time series event data across Cassandra clusters while maintaining millisecond-level query latency and handling dynamic workload changes.
Netflix engineers lacked a real-time, comprehensive view of service dependencies and relationships across their distributed microservices infrastructure, making incident diagnosis and troubleshooting during outages significantly slower.
Netflix needed to efficiently support diverse graph query patterns (OLAP and OLTP) across different use cases with varying performance requirements using a unified abstraction layer.
Netflix needed to predict and mitigate risks associated with content launches to improve reliability and reduce unexpected failures in production.
Netflix needed to detect and prevent corrupted catalog metadata from reaching millions of viewers in production before it impacts the streaming experience.
Netflix needed to consolidate multiple bespoke data movement connectors for Cassandra across different engineering organizations into a unified, centralized management system.
Netflix needed to design a personalized notification system that efficiently decides when and how to notify users with relevant content recommendations without overwhelming them or missing critical engagement opportunities.
Accurately measuring video quality perception to optimize encoding decisions and ensure Netflix members receive the best possible streaming experience across different bitrates and codecs.
Netflix needed to efficiently partition and scale time series data across Cassandra clusters to handle petabytes of temporal event data while maintaining millisecond latency query performance.
Netflix needed a real-time, dynamic way for engineers to understand service dependencies and troubleshoot issues quickly across their complex distributed microservices infrastructure.
Netflix needed a unified abstraction layer to efficiently handle multiple graph query paradigms (OLAP and OLTP) with different performance and functionality requirements across diverse business use cases.
Netflix needed to manage the lifecycle of machine learning models across multiple domains and teams at scale, moving beyond their original single-domain personalization focus.
Netflix needed to automatically evaluate the quality and relevance of show synopses at scale to improve member discovery and engagement.
Netflix needed to efficiently extract and surface key moments from hundreds or thousands of hours of raw video footage for editorial teams to accelerate the creative content production process.
Netflix needed a way to enforce consistent architectural patterns and build standards across tens of thousands of Java repositories in their polyrepo strategy.
Netflix needed to build a scalable, flexible media file processing pipeline that could handle diverse camera formats, workflows, and production requirements while maintaining quick turnaround times for global content production.
Netflix needed to optimize bandwidth utilization and video quality for live streaming events at global scale by moving from constant bitrate to variable bitrate encoding.
Netflix needed to design a domain-independent traffic routing system for their ML model serving infrastructure that could handle personalized experiences at scale across multiple domains while maintaining high availability.
Query performance degradation at massive scale (10+ trillion rows, 15M events/second) where repeated identical queries were consuming excessive resources and impacting latency.
Netflix needed to build reliable operations infrastructure to support live streaming at massive scale, going from one show per month to nine shows per day with tens of millions of concurrent viewers.
Delivering high-quality streaming video across diverse devices and varying network conditions requires efficient video encoding; legacy codecs like H.264 and VP9 were limiting compression efficiency, consuming more bandwidth for equivalent visual quality.
Netflix's relational database ecosystem lacked standardization, with databases spread across RDS Postgres and other technologies, leading to inconsistent functionality, suboptimal performance, and higher total cost of ownership.
Netflix needed reliable orchestration for business-critical cloud operations across teams like Open Connect CDN and Live reliability, but faced operational challenges as Temporal adoption grew since 2021.
Netflix needed scalable, deep machine-level understanding of every piece of content across an expanding catalog (including live events and podcasts) to power recommendations and discovery, but building separate models per content type and modality doesn't scale.
Netflix needed to spin up hundreds of containers in seconds to serve streaming traffic, but after modernizing their container runtime, they hit an unexpected performance bottleneck rooted in CPU architecture that impaired container scaling efficiency.
Netflix needed a custom origin server to bridge its cloud-based live streaming pipelines with its CDN (Open Connect), handling the unique challenges of live content delivery such as low-latency requirements, reliability, and the real-time nature of live streams compared to on-demand content.
Netflix's Ranker service had a video serendipity scoring feature (computing how different a title is from a user's watch history) consuming ~7.5% of total CPU per node, creating a significant performance bottleneck at their enormous scale.
Netflix's localization analytics infrastructure (tracking dubbing, subtitling, and translation across hundreds of languages and regions) could not keep pace with the rapidly growing scale of global content, making it difficult to derive timely insights for content localization decisions.
Generic pre-trained LLMs lack the domain-specific alignment needed for Netflix's production use cases in recommendation, personalization, and search, and the post-training pipeline to fine-tune them doesn't scale efficiently across multiple domain constraints and reliability requirements.
Netflix's Graph Search platform for federated enterprise data required users to write structured queries, limiting accessibility and ease of use despite the system being scalable and configurable.