Meet Praefect: The Traffic Manager Making Your Git Data Highly Available

Meet Praefect: The Traffic Manager Making Your Git Data Highly Available

This router and transaction manager ensures there are multiple copies of each Git repository available in the event of an outage – no NFS required. Meet Praefect: The traffic manager making your Git data highly available. Scaling infrastructure to make the service highly available is key. At GitLab.

As critical software projects grow, scaling infrastructure to make the service  highly available is key. At GitLab, our biggest struggle in scaling was right in our name: Git.

The trouble with scaling Git

Git is software that is distributed, but not usually run in a ‘highly available cluster,’ which is what GitLab needs. At first, we solved this with a  boring solution, NFS – which exposes a shared filesystem across multiple machines and generally worked. As we’d soon find out, most NFS appliances were for bulk storage and not fast enough. This led to problems with GitLab’s Git access being slow.

To solve the speed problem we built  Gitaly, our service that provides high-level RPC access to Git repositories.

When we started with  Gitaly v1.0, our goal was to remove the need for a network-attached filesystem access for Git data. When that was complete, the next problem to tackle was that all your data is only stored once. So, if you have a server down, or your hard disk dies, or something happens to this one copy, you're in deep trouble until a backup is restored. This is an issue for GitLab.com, but it’s also a big risk for our customers and community.

Back at our  Summit in Cape Town in 2018, the Gitaly team (at the time, that was  Jacob Vosmaer and me) and some other engineers discussed pursuing a fault-tolerant, highly available system for Git data. For about a month we went back and forth about how we would go about it – ranging from wild ideas to smaller iterations towards what we want. The challenge here was that the ultimate aim is always going to be 100% availability, but you’re never going to make that. So let's aim for a lot of nines (three nines being 99.9%, five being 99.999%, etc.) Ideally, we'd be able to iterate to 10 nines if we wanted to.

Eventually we chose the design of a proxy: introduce a new component in the GitLab architecture, which is Praefect, and then route all the traffic through it to Gitaly storage nodes to provide a Gitaly Cluster. Praefect inspects the request and tries to route it to the right Gitaly backend, checks that Gitaly is up, makes sure the copies of your data are up to date, and so on.

git

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Best Practices for Using Git

Git has become ubiquitous as the preferred version control system (VCS) used by developers. Using Git adds immense value especially for engineering teams where several developers work together since it becomes critical to have a system of integrating everyone's code reliably.

7 Best Practices in GIT for Your Code Quality

Git plays a significant role in software development. It allows developers to work on the same code base at the same time. Check out 7 best practices for Git.There is no doubt that Git plays a significant role in software development. It allows developers to work on the same code base at the same time. Still, developers struggle for code quality.

Git Commands You Can Use To Dig Through Your Git History

Git Commands You Can Use To Dig Through Your Git History. In this short article, we’ll be exploring some quick git commands that can help us in digging through our repositories’ history of commits.

Git Rebase Tutorial and Comparison with Git Merge

In this article, I will explain to you a few differences between git merge, git rebase, and the git interactive rebase.I will tell a bit about what pros...

Mirroring Git Changes From One Server to Another Server

Hello all, nowadays most of the development teams using GIT version control, some of you may have a requirement of mirroring your team's git changes from one server to another Git server. This article will help you to achieve the Git mirroring between one server to another server.