In this article, we focus on the pipeline pattern in Go programming. Pipeline is no stranger to those who have used Unix/Linux command line. It is a technical method to splice various commands to complete a stronger function. Today, streaming processing, functional programming, and simple API orchestration of microservices by application gateways are all influenced by pipeline technology. Pipeline technology can easily split the code into multiple small modules with high cohesion and low coupling according to the principle of single responsibility, and then it can be easily assembled to complete more complex functions.
HTTP processing
In this Pipeline model, we< Go programming mode: modifier >There was an example in, and we'll revisit it here. In that article, we have a pile of small function codes such as WithServerHead(), WithBasicAuth(), WithDebugLog(), which can be easily organized when we need to implement an HTTP API.
The original code looks like this:
http.HandleFunc("/v1/hello", WithServerHeader(WithAuthCookie(hello)))
http.HandleFunc("/v2/hello", WithServerHeader(WithBasicAuth(hello)))
http.HandleFunc("/v3/hello", WithServerHeader(WithBasicAuth(WithDebugLog(hello))))Through a proxy function:
type HttpHandlerDecorator func(http.HandlerFunc) http.HandlerFunc
func Handler(h http.HandlerFunc, decors ...HttpHandlerDecorator) http.HandlerFunc {
for i := range decors {
d := decors[len(decors)-1-i] // iterate in reverse
h = d(h)
}
return h
}We can remove the continuous nesting and use it as follows:
http.HandleFunc("/v4/hello", Handler(hello,
WithServerHeader, WithBasicAuth, WithDebugLog))Channel management
Of course, if you want to write a generic pipeline framework, it's not easy to use Go Generation, but let's not forget that the two most distinctive artifacts of Go language, Go route and Channel, can also be used by us to construct this kind of programming.
Rob Pike introduced the following programming mode in his blog Go Concurrency Patterns: Pipelines and cancellation.
Channel forwarding function
First, we need an echo() function, which will put an integer array into a Channel and return the Channel
func echo(nums []int) <-chan int {
out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}Then, according to this pattern, we can write this function.
Square function
func sq(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
out <- n * n
}
close(out)
}()
return out
}Odd filter function
func odd(in <-chan int) <-chan int {
out := make(chan int)
go func() {
for n := range in {
if n%2 != 0 {
out <- n
}
}
close(out)
}()
return out
}Summation function
func sum(in <-chan int) <-chan int {
out := make(chan int)
go func() {
var sum = 0
for n := range in {
sum += n
}
out <- sum
close(out)
}()
return out
}Then, our client code is as follows: (Note: you may think that sum(), odd() and sq() are too similar. You can actually pass our previous Map/Reduce programming mode or Go Generation mode (let's merge)
var nums = []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
for n := range sum(sq(odd(echo(nums)))) {
fmt.Println(n)
}The above code is similar to the Unix/Linux command we executed: echo $num | sq | sum
Similarly, if you don't want to have so many nested functions, you can use a proxy function to do it.
type EchoFunc func ([]int) (<- chan int)
type PipeFunc func (<- chan int) (<- chan int)
func pipeline(nums []int, echo EchoFunc, pipeFns ... PipeFunc) <- chan int {
ch := echo(nums)
for i := range pipeFns {
ch = pipeFns[i](ch)
}
return ch
}Then you can do this:
var nums = []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
for n := range pipeline(nums, gen, odd, sq, sum) {
fmt.Println(n)
}Fan in/Out
Another advantage of using Go route and Channel in Go language is that you can write 1-to-many or many to 1 pipeline s, that is, Fan In/ Fan Out. Let's take a look at an example of Fan in:
We want to sum the prime numbers in a long array in a concurrent way. We want to sum the arrays in segments first, and then concentrate them.
Here is our main function:
func makeRange(min, max int) []int {
a := make([]int, max-min+1)
for i := range a {
a[i] = min + i
}
return a
}
func main() {
nums := makeRange(1, 10000)
in := echo(nums)
const nProcess = 5
var chans [nProcess]<-chan int
for i := range chans {
chans[i] = sum(prime(in))
}
for n := range sum(merge(chans[:])) {
fmt.Println(n)
}
}Let's look at the implementation of our prime() function:
func is_prime(value int) bool {
for i := 2; i <= int(math.Floor(float64(value) / 2)); i++ {
if value%i == 0 {
return false
}
}
return value > 1
}
func prime(in <-chan int) <-chan int {
out := make(chan int)
go func () {
for n := range in {
if is_prime(n) {
out <- n
}
}
close(out)
}()
return out
}As we can see,
We first made an array from 1 to 10000,
Then, echo all the arrays into a channel – in
At this time, five channels are generated, and then sum(prime(in)) is called, so the go route of each Sum will start to calculate the Sum
Finally, sum up all the results to get the final result.
The merge code is as follows:func merge(cs []<-chan int) <-chan int { var wg sync.WaitGroup out := make(chan int) wg.Add(len(cs)) for _, c := range cs { go func(c <-chan int) { for n := range c { out <- n } wg.Done() }(c) } go func() { wg.Wait() close(out) }() return out }The structure of the whole program is shown as follows:
Extended reading
If you want to know more about such concurrency related technologies, please refer to the following resources:
- Go Concurrency Patterns – Rob Pike – 2012 Google I/O presents the basics of Go's concurrency primitives and several ways to apply them.
www.youtube.com/watch?v=f6kdp27TYZ... - Advanced Go Concurrency Patterns *– *Rob Pike – 2013 Google I/O covers more complex uses of Go's primitives, especially select.
blog.golang.org/advanced-go-concur... - Squinting at Power Series *– *Douglas McIlroy's paper shows how Go-like concurrency provides elegant support for complex calculations.
swtch.com/~rsc/thread/squint.pdf
(end of the full text) this article is not made by myself. It reprints the left ear mouse blog and its source Cool shell – CoolShell