1. Introduction When you upload an image to Instagram or make a transaction on your favorite brokerage app, the backend system assigns a unique ID (UID) to the newly created object. This ID is typically used as the primary key in some database tables and can be used to retrieve the object…

What if we upload large files directly to the object store from the client side (S3 support this and handles authorization as well)? It would save a lot of server bandwith.

A design for large-scale job schedulers based on my own experience and discussions with other engineers. — 1. Introduction When a limit-sell order or some code is scheduled to run in the future, you need a piece of software like Apache Airflow that runs tasks when certain conditions are met. Because of its wide popularity in the industry, job scheduler has been among the top topics of system design…

1. Introduction Designing chat applications like Slack or Messenger has always been among the top questions asked by system design interviewers. Personally, I’ve run into this problem a few times by now. As new grads/junior SDEa, we are good at implementation, yet designing whole systems is a challenge of its own kind…

Part 2 — Log Replication — Introduction Raft is a consensus algorithm built to orchestrate replicas in a distributed fashion. Designed with understandability in mind, Raft has only a few moving parts and is easy to implement. In the last article, I talked about the basics of Raft and explained the leader-election mechanism. …

Part 1 — Leader Election — Introduction Raft is a consensus algorithm built to orchestrate replicas in a distributed fashion. Designed with understandability in mind, Raft has only a few moving parts and is easy to implement. In this article, we are going to explore the basic components of Raft as well as its leader-election mechanism. Fundamentals Dataflow …

Part 3 -Leaderless Replication — In the modern era of the internet, data replication is everywhere. From bank accounts to Facebook profiles to your beloved Instagram pictures, all data that people deem important are almost for sure to be replicated across multiple machines to ensure data availability and durability. Previously I’ve talked about single-leader replication…

Part 2— Multi Leader Replication — Introduction In the modern era of the internet, data replication is everywhere. From bank accounts to Facebook profiles to your beloved Instagram pictures, all data that people deem important are almost for sure to be replicated across multiple machines to ensure data availability and durability. …

Part 1 — Single Leader Replication — Introduction In the modern era of big data, data replication is everywhere. From bank accounts to Facebook profiles to your beloved Instagram pictures, all data that people deem important are almost for sure to be replicated across multiple machines to ensure data availability and durability. …