Note: I wrote this post for Conviva. You can read the version published on Conviva’s website here

In the last two sections, we went over how TUN devices work, and I showed some local network setup that allows two tun devices to forward packets between each other. In this section, I will go over an overarching sketch of the concurrency architecture for my TCP stack, and create the skeleton for the client/TCP program interaction.

In this interlude post I go over some local network setup to allow forwarding TCP packets between two tun devices. This is a simple local network set up that allows us to test out having two separate user space programs running the custom TCP stack and establishing a connection with one another. The point with this was to have the packets actually hitting my kernel’s network stack, even in this toy implementation. Later on, I can change the local network setup if I want to have the packets be forwarded to the outside world, but I wouldn’t have to change my TUN-based TCP implementation.

As I set out to implement TCP in OCaml, I ran up against a very simple problem: where should it go? There is an existing kernel space tcp, and I want to implement my own as a user space program. The solution to this is TUN/TAP. TUN/TAP allows you to implement a user space program that can read IP packets over a virtual network device.

Introduction I was reading this small summary of 3FS architecture in this blog - 3FS Performance Journal-1. In my opinion, it’s a pretty neat piece of work, and it mentioned that “Management Server” component kept track of all nodes addresses.

Every transactional system must execute, order, validate, and persist transactions.

Introduction

There's better ways to replicate data than Raft.

I am working through the MIT distributed systems lab, and I wanted to share my experience and post on some cool things that I’ve been learning. I really appreciate courses that post all their materials online, so out of respect for the course, I wont be sharing any of my code online. I figure this will take up a big chunk of my time, however, so I want to log my journey through the course. The course is in Golang, so it will be nice to solidify my Golang skills.

Welcome back to part 2 of this series, in which we are implementing a disk backed B+ tree. In this part we will go over the B+ tree in-memory data structure, how to serialize that data structure into a disk layout, and how to deserialize the disk layout back into the in-memory node type.

Here’s a link to the code on GitHub

In this series we will be implementing a disk based B+ tree in OCaml. Here in part 1, we will cover the file and storage managers which we will use for our B+ tree implementation. I will start with a top down introduction that will help give an intuition for what we want out of our file/storage manager design, such that it will seamlessly integrate with the actual algorithm implementation later on. In part 2, I will go over how the B+ tree nodes are laid out on disk and serialized and deserialized between disk and the in memory node representation. Finally, part 3 will cover implementation of the B+ insertion algorithm. Beyond that there will likely be posts on implementing deletion and auxiliary operations built on top of the main insertion and deletion operations.

Introduction

Part 4 covered two big topics: registering optimization passes and setting up a JIT to compile our code on the fly. Challenge #5 - Find Examples for Optimization Passes.

Before moving onto chapter 4, let’s tinker with the code we have so far. I am posting some easy challenges here (and the solutions will be posted as well shortly).

Hello again! In the last blog post: Getting Acquainted with LLVM Codegen Part 1, we split up the code from the tutorial into separate modules, and also got rid of the global LLVM context, moving it locally to the codegen module. This blog post will continue the tutorial, and we will add some extensions to the code.

I am currently looking at chapter 3 of the LLVM Kaleidoscope Language tutorial, where a simple language called Kaleidoscope is implemented to learn about how LLVM works. Here is a link to the tutorial: LLVM Tutorial Chapter 3.

Aurora, Socrates, PolarDB, and Taurus.

Filesystems, file IO, and durability.

Here’s a link to the code on GitHub.

Here’s a link to the code on GitHub

An overview of erasure coding, its trade-offs, and applications in distributed storage systems.

Here’s a link to the code on a GitHub repo

Consistent replication algorithms can be placed on a sliding scale based on how they handle replica failures. Across the three common points on this spectrum, the resource efficiency, availability, and latency are compared, providing guidance for how to choose an appropriate replication algorithm for a use case.

This post covers a course that is part of TiDB’s talent plan on building a simple implementation of Percolator. You can find the course here

I’m writing the posts in this series based on a course I completed. I can’t recommend the course highly enough. If you’re actually interested in really intuiting an LSM engine, I recommend you do the course yourself.

Welcome to the B-Tree tutorial.

I’ve been using Rust at work for the last few months and keep hearing about “fearless concurrency” in Rust. I’m still not entirely sure what that means but I mistakenly assumed that it meant that race conditions were impossible in Rust. This is obviously wrong, but it took me a while to understand why.

This post is migrated version of the one already published on Medium.

Support for searching down the tree in search of a single value.

Opening and closing databases as a BerkeleyDB ABI-compatible library.

This post is a follow up to my previous post about how to implement the Raft consensus protocol in Rust. In the previous post I went through the basics of how to set up the Raft cluster and implement the logic required for the RPC’s.

Our next goal is to be able to walk the tree, and print out the values stored in the B-Tree.

What Is Raft?

Support for parsing and printing information contained in the header for BerkeleyDB pages.

My attempt to decipher the Raft whitepaper and how KRaft implementation adheres to the raft philosophy and techniques.

Welcome to part 2 of the series, in this post, we’ll discuss implementing a new histogram style inspired by Postgres in MySQL for estimating row selectivity.

A more accurate title would’ve been porting Postgres’ conditional selectivity to MySQL, but you’d ask me what’s conditional selectivity then! This blog post series will explain both.

Welcome to my take on the implementation of extendible hash tables (minus the code btw, owning to the educational policies)

MongoDB is an extremely popular no SQL database solution with an interesting cache architecture. Tons of projects use MongoDB in production including ours at GoTo Financial. It has been working perfectly for quite some time until came to a point when it suddenly wasn’t and this small post is all about our misadventures with MongoDB.

Script and tips on how to set up multiple master-replica Redis instances and multiple Redis sentinels to monitor them.

The background and context on why the groupings exist the way they do, and thedifferent sorts of pages you’ll find in this section. But, this is allphilosophical waxing, so quite skippable.

Paxos, Raft, and all their flavors, variations, and alternatives.

In early 2019, some months after completing a rehab program for drug addiction, I was in a very open-minded headspace where I wanted to challenge myself and find ways to improve as a person.

In this post, we will discuss the challenges of programming with locks and how the C++ language offers some useful tools to make it easier. We will start with an example in C and then use C++ to improve upon it in steps. The example APIs are based on real-life APIs from the SerenityOS kernel.

We often hear that making small incremental improvements every day can lead to great things. This popular piece of advice rings true, and it’s a powerful reminder to keep pushing ourselves forward.

This post explores one of my favorite hacks in SerenityOS. I don’t recommend doing this in your codebase, but it has worked for us so far. :^)

Someone once suggested that I set aside a small portion of my income every month. I’ve been doing this for years with an automatic bank transfer so I can’t mess it up.

This post describes in detail how I support myself while working on the SerenityOS project.

This post describes the Ladybird browser, based on the LibWeb and LibJS engines from SerenityOS.

A reminder that macOS does not respect the usual ways of making data durable on disk.

This post describes how we’re going to achieve memory safety in SerenityOS.

When application code provides high-level fault injection points, fuzzing becomes significantly more effective.

Hello friends! :^) Let me tell you a story…

This post describes a compile-time technique for getting the benefits of aggressive inlining in hot code while protecting cool code from its downsides.