Coordinating Goroutines  - errGroup

Coordinating different goroutines to achieve a single goal

Photo by Hunter Harritt on Unsplash

In a previous article I talked about how we can use the context.Context struct to share data and group distinct goroutines. One thing that’s missing from that article, is how to synchronise that group of things. Synchronise concurrent tasks means either wait for all of them to finish before doing something else, or cancelling all of them in case things go wrong. That’s where the sync/errgroup comes to the scene. Here are the official docs for the package. The package name conveys two things: used to synchronise, and we are going to be dealing with errors. That pretty much matches what was mentioned above, right? The name errgroup may sound a bit weird as the mechanism to synchronise goroutines, but bear with me, it will make sense later. Essentially you will be registering goroutines to that group and wait for them to finish, one way or the other. The main struct inside the package is the errgroup.Group.

To create a new Group you can call errgroup.WithContext that will receive a context.Context as parameter and return a pointer to a new group and a new context.

There are two methods you will use in this struct:

• (*errgroup.Group).Go — register a function to the groups and start it as a new goroutine;
• (*errgroup.Group).Wait — waits until all registered goroutines finish or an error occurs in any goroutine of the group. That’s why it uses the error as the synchronisation mechanism. If an error occurs, the context (that one that was returned by the errgroup.WithContext) gets cancelled and the error gets returned. One thing to pay attention here is that the goroutines that have already started executing will continue to do so, so it’s important to get the context passed along to the goroutine and check if that got cancelled inside the goroutine (usually that’s a simple thing to achieve with the select statement);

#software-development #golang

What is GEEK

Buddha Community

Coordinating Goroutines  - errGroup

Coordinating different goroutines to achieve a single goal

Photo by Hunter Harritt on Unsplash

In a previous article I talked about how we can use the context.Context struct to share data and group distinct goroutines. One thing that’s missing from that article, is how to synchronise that group of things. Synchronise concurrent tasks means either wait for all of them to finish before doing something else, or cancelling all of them in case things go wrong. That’s where the sync/errgroup comes to the scene. Here are the official docs for the package. The package name conveys two things: used to synchronise, and we are going to be dealing with errors. That pretty much matches what was mentioned above, right? The name errgroup may sound a bit weird as the mechanism to synchronise goroutines, but bear with me, it will make sense later. Essentially you will be registering goroutines to that group and wait for them to finish, one way or the other. The main struct inside the package is the errgroup.Group.

To create a new Group you can call errgroup.WithContext that will receive a context.Context as parameter and return a pointer to a new group and a new context.

There are two methods you will use in this struct:

• (*errgroup.Group).Go — register a function to the groups and start it as a new goroutine;
• (*errgroup.Group).Wait — waits until all registered goroutines finish or an error occurs in any goroutine of the group. That’s why it uses the error as the synchronisation mechanism. If an error occurs, the context (that one that was returned by the errgroup.WithContext) gets cancelled and the error gets returned. One thing to pay attention here is that the goroutines that have already started executing will continue to do so, so it’s important to get the context passed along to the goroutine and check if that got cancelled inside the goroutine (usually that’s a simple thing to achieve with the select statement);

#software-development #golang

A Goroutines Gotcha

Goroutines are a simple and lightweight concurrency mechanism that is very popular among Golang developers. It is common for Golang programmers to leverage Goroutine semantics to achieve program execution efficiency through parallelism. However, care must be taken when designing Goroutine implementations in order to avoid unintended side-effects. I will highlight one such side effect that I ran into while writing my own Goroutines. As it turns out, this was a documented pitfall that I wasn’t aware of until I ran into it myself.

The program

The logic I needed to code was simple: I had a slice of functions of a certain type that I needed to iterate over and execute. Since these functions were independent of each other, it only made sense to execute each function in its own Goroutine. Simple enough. The code I came up with was similar to the following:

import (
   "fmt"
   "sync"
)

func main() {
   ConcurrentFunctions(func1, func2)
}

func ConcurrentFunctions(fns ...func()) {
   var wg sync.WaitGroup
   for _, fn := range fns {
      wg.Add(1)
      go func() {
         fn()
         wg.Done()
      }()
   }

   wg.Wait()
}

func func1() {
   fmt.Println("I am function func1")
}

func func2() {
   fmt.Println("I am function func2")
}

“ConcurrentFunctions” is a variadic function that takes any number of functions as input, and iterates over these to execute them. The execution of each function is performed in a separate Goroutine. So, what would you expect the above program to print when it is run? “I am function func1” and “I am function func2” (not necessarily in that order), right?

Wrong!

When run , the above program prints the following:

I am function func2
I am function func2

#golang #closure #goroutines #pitfalls

It’s the coordination, stupid

The biggest problem in software is not related to develop but to people coordination:

• Stakeholders talk a different language than developers.
• Developers have different opinions on how code should be written.
• UX designers don’t usually discuss with frontend devs to see how easy is to develop one or another approach.

Communication is a way to solve coordination problems but communicate people inside an organization is a very time-consuming task, perhaps the more time-consuming task you can imagine. But without communication is impossible to achieve the goal, the cost of implementing a different thing than the one our clients want is very high.

So it is like an impossible problem, we need communication but the communication has a big cost. What to do?

Inside the organization

Microservices are just a simple way to split our big codebase in several ones. Those deployable codebases (services) must minimize dependencies between them. Following the Inverse Conway Maneuver this will produce that the team in charge of microservice A will be very independent of the team in charge of microservice B.

Now both teams are autonomous and independent then we have reduced the need of coordination between them, also we have reduced the cognitive load.

If your teams are not in charge of one or some microservices in reality you have a bigger problem than having a monolith (few benefits and poorer performance). You will notice some advantages like fewer conflicts when changing code because it is going to be more difficult to have two devs of different teams working in the same codebase. But coordination problems will be still there, you will start feeling that your speed developing decreases.

The same happens if your microservices are not independent. This will be reflected in teams coordination, two teams in charge of two very coupled microservices will need ways to coordinate their work (more communication between teams then higher costs).

Now imagine that you slice vertically your product and you give the responsibility of each vertical slice to one team. In this scenario we have reduced the coordination between those teams maintaining the quality of our product, because those teams don’t need to talk between them. We can introduce more teams in charge of vertical slices of our product. Slicing more and more vertically our product will force us evolving our architecture to simplify our teams work.

Inside the teams

Probably you have faced the problem of not having tasks to take because all of them depend on others that have not been finished. In this situation the only thing we can do is talking with our team to define a plan to avoid being blocked (coordinate between us with a meeting).

There are several ways to avoid this coordination in form of meetings, at least from a development point of view:

• Trunk Based Development is a way to simplify that problem, because you don’t need to wait until the whole task is finished to share your code.
• Having different independent streams of works allow also to have developers always working in a task.
• Transferring knowledge requires a lot of communication, try pair programming to do it more effectively.
• If you do pairing rotate pairs to avoid silos. If not, create mechanism to avoid the same people working always in the same stories.
• Code reviews requires a lot of communication, make them smaller or even better start pairing.

#software-development #coordination #meetings #technology-radar #microservices

Go: Common Misconceptions About Goroutines

When we hear the term goroutines the very first thing that come to our mind is, it’s way faster than threads, pretty much lightweight as compared to threads and is managed by the Go runtime which doesn’t depend much on the Operating System.

Many of those who had freshly started with Go will be in an assumption that any sequence of actions can be parallelizable by spinning multiple go-routines. But that’s true only to a certain extent and increasing more go-routines doesn’t help after that.

So without much lag, let’s get straight on to the point. In this blog post, I will try to convey some top-level observations on how goroutines work.

As this is intended to be understood for people who have minimal to zero knowledge in Go, I will be taking an example of computing Fibonacci Sequence for 1–40 numbers and return a map with key as number and value as the Fibonacci value of the key.

---

Base Level Calculation without any parallelism

type compute struct {
		sync.RWMutex
		response map[int]int
	}

	func fib(i int) int {
		if i == 0 {
			return 0
		}
		if i == 1 {
			return 1
		}
		return fib(i-1) + fib(i-2)
	}

	func (c *compute) sequentialFibonacci(count int) {
		for i := 0; i < count; i++ {
			c.response[i] = fib(i)
		}
	}
	func main() {
		runtime.GOMAXPROCS(4)
		action := flag.String("action", "sequential", "Specify if fib had to be sequentially computed or parallel")
		flag.Parse()
		startTime := time.Now()
		c := compute{response: make(map[int]int, 40)}
		switch *action {
		case "sequential":
			c.sequentialFibonacci(40)
		case "parallel":
	    panic("not yet implemented")
		}
		fmt.Println("Total time taken ", time.Since(startTime).Milliseconds())
	}

So here we have a flag which asks if the client had to do sequential processing or parallel processing. Considering only sequential is implemented, we are iterating from 0–30 using a for loop and on every iteration, we are computing the Fibonacci of that particular iteration key and storing it into a map.

If you look at the above code, when I was computing for Fibonacci of 1, all the other 39 computations are waiting for the first one to get done.

Similarly, on Fibonacci of 2, the remaining 38 keeps waiting which means our sequential processing is increasing the wait time of the client.

As you can see in the above screenshot it’s taking 990 ms on an average to compute the Fibonacci sequence of first 40 numbers. Let’s try parallelising the task with the help of Goroutines and see how it scales.

#parallel-computing #go #goroutines

Calculating the Bearing Between Two Geospatial Coordinates

When programmatically adjusting a camera to follow a specific path — be it on google street view or a Geographic Information System animation, the programmer may be required to specify the bearing between two points or coordinates of latitude and longitude. In this post, I explain how this may be achieved.
Mathematics
Mathematically the bearing between point a and point b is calculating by taking the inverse tan function of X and Y

#python #bearings #coordinates #gis #geospatial