Logesh Balasubramanian

Top learnings from my discussion with Kishore Gopalakrishna - Language choice for system development should balance team expertise, hiring potential, and ecosystem maturity. A well-implemented system in a familiar language often outperforms a poorly implemented one in a "faster" language. - Distributed systems are complex - they can fail in unexpected ways, making your own computer useless. This challenge is often underestimated. Understand the failure modes. - When building new systems, design for the long-term (10 years) but plan for the near-term (2-3 years). Implement for current needs while keeping the architecture flexible for future growth. Design for long term, plan for short-medium term. - Testing distributed systems requires going beyond known test cases. Develop test frameworks that generate new scenarios automatically to uncover unforeseen issues. - Cost optimization in data systems often involves trade-offs. Kafka, for example, sacrificed transactionality and exact-once delivery for higher throughput and lower cost per log. Its all about the Trade-offs. You get some, you lose some. - When approaching problems, start with first principles. Understand why something is expensive or inefficient before trying to optimize it. Don't jump on a solution. Understand the real problem first. - Open-source success relies heavily on community building. Foster respectful communication, clear processes for resolving conflicts, and a culture of helpfulness to grow a strong contributor base. There are many communities, but only the ones with good culture thrive. - Real-time analytics can significantly boost user engagement. Even small reductions in data latency can lead to measurable improvements in user experience and product value. - When considering real-time analytics, focus on how it can improve your users' lives rather than just technical capabilities. Many support issues can be eliminated by giving users direct access to real-time data. Product isn't useful if it making users' lives better. - Innovation in data products often comes from changing perspective - instead of asking "Do we need this?", ask "How can this improve our users' experience?" This shift can uncover valuable use cases for real-time analytics. Asking the right questions will help you reach the right answers. - When building data products, consider the long-term evolution of technology. Design choices should anticipate future developments, such as improvements in network speed or storage capabilities. Keep your design extensible. - As users become more data-savvy, there's increasing demand for self-serve analytics tools. Empowering users with direct access to data can reduce support overhead and enable more informed decision-making. Empower users to do more with data quickly. ———————————————————————— In case you haven't watched the episode yet, you can watch it here: youtu.be/SmLlJmU9Daw Also subscribe if you like such discussions.

41

3 Comments

Kicking off a super interesting series on the internals of DragonflyDB - a multi-threaded drop-in replacement of Redis ⚡ On a single machine, it offers 25 times the throughput of Redis as it is optimized to leverage every single ounce of performance from the underlying hardware. The first video of the series is out and it is a quick architecture overview and covers two key design decisions. give it a watch - youtu.be/XbV1LoVsbME The low-level design decisions and internal details will just blow your mind away - right from their design of DashTable (not hash table) to sorted set implementation. Trust me, your perspective of looking at Data structures will change after the series :) ⚡ Subscribe to the channel and follow along; I keep things no-fluff. youtube.com/c/ArpitBhayani #AsliEngineering #SystemDesign #DatabaseInternals

286

5 Comments

Redis, despite being single-threaded for most of its operations, uses I/O multiplexing to handle thousands of client connections efficiently. I/O multiplexing is a technique used in computing to handle multiple input/output operations concurrently, which is particularly useful in environments where we need to manage multiple connections or streams of data at the same time. I/O multiplexing allows a single thread to manage multiple I/O operations by monitoring multiple file descriptors (such as sockets, pipes, etc.) and handling I/O events as they occur. In I/O multiplexing, a program uses system calls to monitor multiple file descriptors simultaneously. When one or more of these file descriptors are ready for some I/O operation (like reading or writing), the program is notified, and it can then process the ready file descriptors. Below are few system calls for I/O Multiplexing 1.select(): This is one of the earliest methods of I/O multiplexing. It monitors multiple file descriptors to see if any of them are ready for I/O operations. 2.poll(): Similar to select(), but more scalable for a large number of file descriptors. 3.epoll(): A more efficient and scalable option available on Linux. It is designed to handle large numbers of file descriptors with less overhead compared to select() and poll(). #systemDesign #backendDevelopement #redis

2

Hot take: Building a business around open-source software is a lost battle. In last few months, we have seen how companies such as HashiCorp, Redis labs have faced the heat when they adopted business source licensing model. It resulted in their alternatives popping up within few days like OpenTofu, Dragonfly etc. There is no moat for such businesses because the underlying software is open-sourced which means new entrants can easily replicate what they are doing. They can sell it at a cheaper price or with a better ecosystem(if they are cloud vendors) or both. These businesses also face the dilemma of whether to merge a major improvement in the open-source or put it behind the SaaS offering. An alternative to this model is to build the business around consulting & training. The business keeps the software open sourced & provide consulting services to clients who want to use it in the most efficient manner. Pivotal has done a great job with Spring framework using this model. This way your revenue stream is not marginalized by new players easily & the core team can focus on improving the software without worrying whether it makes their core revenue model i.e. SaaS offering less attractive. Especially in the non-ZIRP era, companies are bound to reconsider where they are spending. Hence they will choose the option that burns the smallest hole in their pocket. #opensource

8

2 Comments

Why? While discussing system design for both trading apps and stock exchanges, I strongly emphasize that they are weak cases for Cloud. While Cloud is really good for scaling and availability, it comes with a tradeoff for Latency. While temporary 300 ms+ latency is OK for most of the apps, a trader can lose a lot of money in that time frame. I am still of the belief that a broker should run its app as close to the stock exchange as possible and latency should trade off for availability . If a person is not able to place order at the. first step, it's still better than seeing order in "Pending" state and then getting executed while stock price has gone down already. https://lnkd.in/d6N9Rp6G

3

Exactly-once semantics of Apache Kafka (part two) Transactions: atomic writes across multiple partitions We previously explained idempotent producers to help produce messages with no loss and no duplication. https://lnkd.in/dtPq-RE7 We need transactions on downstream processing to establish strong delivery guarantees. What's the problem? Applications potentially deliver inaccurate results when: 1. the app crashes after writing B but before marking A as consumed, the app reproceses A once resumes resulting in duplicate B 2. in a distributed setting, instances crash and are replaced with other members, reading same input and produce same output, these are called zombie instances How Kafka Transactions solve these? Transactions enable atomic writes across multiple partitions and topics, allowing a group of messages all being commited together or none. An example app consuming message A at offset X and writing message B as a result of processing such that B = F(A), and since offset commit is another write to an internal topic, now an atomic read-process-write means writing message B along with committing offset X as a single unit. How Kafka delivers exactly-once stream processing? 1. Kafka offers Streams API shipped with strong delivery guarantees. committing state, output messages and offsets atomically 2. Streams library is unfortunately only available for Java. using local state store backed by Kafka topics called changelogs for rebalancing and recovery 2. ksqlDB on other hand as an optimized database for stream processing enables executing sql-like statements 3. ksqlDB offers stateful aggregations, materialized views, joins and data transformations all backed by topics so again transactions 4. statements can be executed through exposed REST APIs, ksqlDB-cli or ksqlDB server Next we talk about zombie instances that are fenced off using transaciton IDs helping app survive restarts, and then two-phase-commit protocol by transaction coordinator.

18

2 Comments

Observation: a lot of domains put up with ad hoc / limited programming models just to get ease of deployment and 0 serialization boilerplate. (Think: KSQL vs Kafka + normal code) You can have the best of both. Ease of deployment, etc, but with a rich, general-purpose language.

4

Continuing the GenAI work we promised, we are thrilled to introduce 🌟 "bondAI" 🌟 . "bond" as in 🤝 and not "James Bond" (at least not right now). https://lnkd.in/g4ExUs6N This GitHub repo has everything 🤞 to get you started. 1. Python Development Environment Inside Docker. So you do not need to install python and learn pip, venv, etc. 😀 2. FastAPI 3. PostgreSQL 💰 Bonus: 🔸 You can use this template for other purposes as well. It is generic enough to create a REST service based on FastAPI and PostgreSQL. 🔸 We will be adding new features with time. e.g. Logging, Tracing, Configs, etc. It is a work in progress. Kindly report issues and contribute. 🙏 Also, if you like it, you can ⭐ the repo for quick reference and motivating us. Original Post Link (as we post updates in comments): https://lnkd.in/gD-gie6C #AgiledCoder #OpenKnowledge #GenAI #OpenAI #FastAPI, #BondAI, #Python #PostgreSQL

2

🌐 Ever wondered how Discord successfully manages trillions of messages? Their journey from Cassandra to ScyllaDB is a fascinating tale of persistence and innovation! 📈 In 2017, Discord was using Cassandra to handle billions of messages. Fast forward to 2022, and they were grappling with serious performance issues, even with 177 nodes. The solution? Migrating to ScyllaDB. 𝗞𝗲𝘆 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀: - Cassandra’s high-toil system caused unpredictable latency and frequent maintenance troubles. - ScyllaDB, written in C++, offered better performance with no garbage collection issues. - A migration plan was implemented to switch from Cassandra to ScyllaDB, reducing 177 nodes to just 72. - Rust-based data services were used to handle concurrent traffic efficiently, preventing database overload. The result? A significant improvement in latency and overall database performance. Discord's new setup handled the traffic spikes during the World Cup Final flawlessly! ⚽ What challenges have you faced with database migrations? Share your experiences below! 👇 #Database #Cassandra #ScyllaDB #Rust #DataMigration #TechInnovation

4

In Memory Allocation, contiguous allocation assigns each process a single, continuous block of memory. This straightforward approach offers faster access due to its sequential structure but comes with challenges like fragmentation. Contiguous allocation is further categorized into Fixed Partitioning and Variable Partitioning, each with its strengths and limitations. Fixed partitioning divides memory into pre-defined, fixed-size blocks. It’s easy to implement and works well for systems with predictable workloads, but it often leads to internal fragmentation, where unused space within a partition is wasted when processes don’t fully utilize their allocated block. Variable partitioning, adjusts partition sizes dynamically to fit the exact memory needs of processes. This reduces internal fragmentation by ensuring memory is allocated as needed, but it can lead to external fragmentation, where scattered free memory blocks make it challenging to allocate large contiguous spaces over time. 🔮 I keep on writing and sharing my learning across my socials, so if you resonate do follow along. I keep it no fluff; just engineering. #50daysofOS #DrGViswanathan #DrGViswanathanChallenge

24

1 Comment

I am a very noob debugger Things I do when stuck on a problem - Constant Debugging - Running same test over and over again - Stressing - Distractions on social media Things I should do - Walking outdoors - Nap - Close laptop and take a break Are you me ? #softwareengineering #debug

44

10 Comments

Own your promo doc, part 2 out of 2 Part 2 is the story of writing my Senior Engineer to Principal Engineer promo (2013-2014). (Part 1, the story of writing my SDE-II to Senior Engineer promo, is at https://lnkd.in/gMHMzFR4) One thing I wanted to do differently was that the doc should be Forward-Looking (“here’s what I’m going to do” - proactive) rather than the standard “here’s what I did” (reactive). Amazon tends to follow a process called "Working Backwards" in a lot of docs - and I figured it would apply well to promos. I wrote the Forward Looking doc well before I thought I was ready. It didn’t just list the things I had accomplished, but the things I felt I could/should accomplish in the next 12 months. I then shared the doc with a selected group of Directors and Principals around me to seek feedback. The basic question was, “if this document was being discussed a year from now and everything in it was then true, would you be Inclined or Not Inclined? Does it ballpark describe the body of work of a Principal?” It was not a comfortable experiment: I certainly put myself out there and felt very vulnerable. The people I sought feedback from were also very direct and candid and challenged me in uncomfortable ways, but I knew they cared about my growth. Their advice helped me to be focused and deliberate on where I spent my time in the next year. When I felt the doc was good enough, I shared it with my manager. I didn’t get promoted the first time around, but I did get promoted six months after my first failed attempt. That was roughly a year after I initially wrote the doc. This is pretty standard for PE promos - if you get promoted the first time around chances are your manager waited too long! Years later, I was having coffee and catching up with the manager that promoted me to Principal. He said something curious: “You know, I was planning on promoting you, but your doc helped me do this six months earlier!” I had spent about 30 hours of my life writing, revising, incorporating feedback, and thinking about it. I don’t remember what kind of financial benefit getting promoted half a year earlier had for me specifically at the time, but let's use the following thought experiment: just using today’s public numbers from levelsfyi, Amazon L6 averages $405k, Amazon L7 averages 680k, so the difference is $275k/yr. Having L7 compensation six months earlier would mean an additional 138k. Given 30 hours of additional work, that’s an ROI of $5000/hr. I'd say that's a pretty decent ROI for that investment of time! (there were many other benefits to being promoted earlier, but that one is easy to quantify and articulate)

361

21 Comments

Engineers hate Mondays. That's why I'm giving away free food: 0. How Uber Tracks Billions of Trips: https://lnkd.in/da-c_aKc Author: Arpan G. 1. UML Class Diagram Explained with Examples: https://lnkd.in/dcN--x_8 Author: Ashish Pratap Singh 2. How to Approach Scaling Your Database: https://lnkd.in/degTqAxi Author: Nikki Siapno 3. How to Choose Between Batch, Micro-Batch, and Streaming when Building a Data Pipeline: https://lnkd.in/dQWkXVHq Author: Zach Wilson 4. How Deep is your DOM?: https://lnkd.in/dBcPMz8m Author: Maxi Ferreira 5. How Stripe’s Document Databases Supported 99.999% Uptime With Zero-Downtime Data Migrations: https://lnkd.in/dkpd-A4Y 6. Load Balancing Crash Course: https://lnkd.in/dqvGA65U 7. How Uber Developed a System to Manage Flaky Tests at Scale: https://lnkd.in/dcZ_ExhY Still hungry? I am packaging the other deep dives and trends in today's issue of Hungry Minds. Join 18,001+ software engineers from big tech to startups here: https://lnkd.in/dkJiiBnf What are you reading this week? #softwareengineering #coding #programming

214

51 Comments

Reading this research paper on How Deadlocks in Java21 Virtual Threads led to open 18,000+ TCP sockets in CLOSE_WAIT state, bringing down the Tomcat throughput all the way from 1,000 req/s to 0, while thousands of threads stuck in "blank" or parked space - and this makes me go back to Classic Deadlock Problem in Concurrent Computing and OS. Published by Netflix Performance Engineering team. Here's a quick summary of it and what we can learn from this scenario: TLDR: The JVM has spawned thousands of virtual threads (VTs) for each incoming request. Four VTs are pinned to OS threads (since the node has 4 vCPUs), and they are waiting to acquire the lock. A 5th VT, which is unpinned, cannot proceed because the fork-join pool (which manages OS threads) is fully occupied by the pinned VTs. This creates a deadlock-like situation where the 4 pinned threads and the 1 unpinned thread are all competing for the same lock, but the unpinned thread can’t get an OS thread to proceed because it's already full. Worth reading it, give it a go. Full article: https://lnkd.in/dH5bRbcn #performance #java #virutalthreads #distributedsystems

20

1 Comment

How do you become a better programmer while being an SRE? Many SigNoz users and customers are SREs and DevOps engineers. In my conversations with them sometimes they lament the fact they don’t “code” any longer and want to sharpen their edges by writing application code. But they rarely get a chance to do this at their job. Coding is also like building a muscle. If you go months and months without writing any application code, when you do finally need to write code, suddenly you will find that you have lost the muscle. It's like not benching for a year and then trying to bench press. (Infrastructure-as-code is a different muscle entirely, and more config files than code). While strictly speaking SRE job does involve writing code - scripts, yaml, etc. IS CODE. But to those who miss writing application code, here’s some tips on getting started: 1. Pick a language you want to code in first, decide a hobby project and build it from end to end. With AI assistants like ChatGPT and CoPilot easily available now, an easy way is to chat with the LLMs to develop a plan and build the project. They can act as a sparring partner for you and also help in rubber ducking or resolve errors which may come up 2. Contribute to an open source project : One problem with doing something which is not part of your main job is to keep yourself motivated an accountable. I find open source communities to be a great way to help in this. Rather than choosing a hobby project of your own, you can just choose to work on a open source project. Start asking questions, joining community calls and interacting on Github/Slack. You will find problems in the project that need to be solved - and you can raise your hand to solve it. The advantage is you have people in the community to help you if you get stuck and if your PR gets accepted finally then you have a “greener” github profile to show for 🙂 3. Practice Leetcode everyday : For people who want something more piecemeal, leetcode could be great way to start. Agree it is not strictly application code, but it still helps you solve problems with code. And if you agree that writing code is like building a muscle, Leetcode is doing 10 pushups everyday. It is not tough to do - but if you do it everyday, you are on your way. 4. Build sample apps and demo projects for your team : This may not be applicable to all teams, but if you work in a developer focused product (like SigNoz) your team may need sample apps for their docs and demos. Raise your hand when you get such an opportunity. 5. Study books on design patterns and check code structures of open source projects: While this is not as good as “writing code”, if you really can’t get time to put keystrokes to your code editor, just start reading how other open source projects are solving problems. Studying more about design patterns and their application in real projects will help you get the juices flowing and who knows may give you enough motivation to get started 🙂

53

6 Comments

Building a platform that has LinkedIn integration… If Microsoft’s docs were written on paper I would burn them. For some reason getting the integrations and permissions correct has been one of the biggest headaches in this project. Redis? No problem. Stripe for subscriptions and auth flow using 0Auth? Easy. Setting up S3 buckets for photo storage and adding packages to change all photos being saved to webp? Took like an hour. Posting to LinkedIn using their API? Well first you gotta weed through all the irrelevant and old docs they keep around to confuse you. Make sure your app has the correct permissions from the beginning - if you miss something there it’s a whole thing to change it. You’ll need different ones than you think… and if you’re lucky enough to find the hidden errors logged somewhere - they won’t help much. Just not a fun integration for me. #linkedin #softwaredeveloper #aiplatforms

3

2 Comments

🗞️ Grab the latest edition of our Confluent Developer Newsletter! 🗞️ The new roundup includes helpful resources like: - Localized Hindi (हिंदी) captions for Python developers in our #ApacheKafka 101 course! - A 6-minute video on how exactly once-processing works in #ApacheFlink - Upcoming events around the world 🌎 … and much more! Read it below ⬇️

2

🔍 Wondering why Slack recently migrated to a Cellular Architecture? I believe every migration or upgrade in a system aims to leverage advantages while carefully managing trade-offs within its architecture to meet non-functional requirements like 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲, 𝐞𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲, 𝐫𝐞𝐬𝐢𝐥𝐢𝐞𝐧𝐜𝐞, and 𝐟𝐥𝐞𝐱𝐢𝐛𝐢𝐥𝐢𝐭𝐲. Essentially, 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 is 𝐚𝐥𝐥 𝐚𝐛𝐨𝐮𝐭 𝐦𝐚𝐧𝐚𝐠𝐢𝐧𝐠 𝐭𝐫𝐚𝐝𝐞-𝐨𝐟𝐟𝐬. There's no one-size-fits-all design template. Instead, it requires careful analysis, planning, execution, and adaptation, all done with good judgment at the right time and place within the organization. Let's break it down 𝐖𝐡𝐲 Slack did it with Cellular/Modular Architecture; 1️⃣ 𝐒𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: With more users joining every day, Slack needed a system that could handle the increasing demand without slowing down. Cellular Architecture offers the scalability they need to keep everyone connected seamlessly. 2️⃣ 𝐄𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲: By breaking their monolithic system into smaller, more manageable cells, Slack can now optimize each part independently. This means faster response times and better performance for users. 3️⃣ 𝐑𝐞𝐬𝐢𝐥𝐢𝐞𝐧𝐜𝐞: Cellular Architecture improves fault isolation, meaning if one part of the system goes down, it won't bring the whole platform crashing with it. Slack can now ensure uninterrupted service even in the face of challenges. 4️⃣ 𝐅𝐥𝐞𝐱𝐢𝐛𝐢𝐥𝐢𝐭𝐲: Each cell operates as its mini-system, giving Slack the flexibility to experiment and innovate without disrupting the entire platform. They can roll out updates and new features more efficiently than ever before. 👩‍💻 With this migration, Slack is not just upgrading its infrastructure — it's paving the way for a more reliable, scalable, and resilient platform for users worldwide. Read more detailed article in the Slack Engineering blog. #slack #architecture #innovation #scalability #efficiency #resilience #flexibility #techupgrade 

3