hi, Hello, I'm haohongfan.
Recently, when I was doing the cortex optimization of prometheus ecology, I encountered a problem of go mod. Let's share it here.
Why do I call the title: how to cheat Go mod? This is very interesting. We'll sell it here first, but it really breaks through the relevant features of Go mod. (well, Cao Da's Go mod ten sins can be added)
Before formally launching this topic, we need to briefly introduce the two projects of cortex and thanos.
Limitations of Prometheus
When it comes to business development, it is basically inseparable from the monitoring system. Prometheus, as the darling of the cloud, graduated from CNCF with excellent design, flexible use and excellent results. It is also the first choice for many companies to do monitoring.
However, Promethues also has its own limitations, among which the most influential is its data high availability scheme and cluster scheme. Monitoring is also an important part of the business system, and the alarm cannot be sent in time due to the downtime of the monitoring system.
Prometheus officials have also proposed a federal solution to solve the cluster problem, but this solution is extremely complex and many problems can not be solved, so it has created two other CNCF sandbox projects: cortex and thanos. Both projects are to solve Promethues' cluster and are highly available.
Because the purpose of the two projects to solve the problem is the same, many functions can be reused with each other, so interesting things happen.
cortex
In other words, because of some requirements, I have to change the relevant code of thanos. When I debug locally, I replace d the thanos that cortex depends on.
replace github.com/thanos-io/thanos => /Users/hhf/goproject/cortex/thanos
When I compile it again, I can't compile it
# github.com/sercand/kuberesolver ../../../go/pkg/mod/github.com/sercand/kuberesolver@v2.1.0+incompatible/builder.go:108:82: undefined: resolver.BuildOption ../../../go/pkg/mod/github.com/sercand/kuberesolver@v2.1.0+incompatible/builder.go:163:32: undefined: resolver.ResolveNowOption
This makes people very helpless. Don't worry. Let's see who this kuberesolver is dependent on.
Let's look at what happened before it was replace d:
▶ go mod graph| grep kuberesolver github.com/weaveworks/common@v0.0.0-20210419092856-009d1eebd624 github.com/sercand/kuberesolver@v2.1.0+incompatible github.com/weaveworks/common@v0.0.0-20210112142934-23c8d7fa6120 github.com/sercand/kuberesolver@v2.1.0+incompatible github.com/weaveworks/common@v0.0.0-20200206153930-760e36ae819a github.com/sercand/kuberesolver@v2.1.0+incompatible github.com/weaveworks/common@v0.0.0-20201119133501-0619918236ec github.com/sercand/kuberesolver@v2.1.0+incompatible github.com/weaveworks/common@v0.0.0-20200914083218-61ffdd448099 github.com/sercand/kuberesolver@v2.1.0+incompatible github.com/weaveworks/common@v0.0.0-20200625145055-4b1847531bc9 github.com/sercand/kuberesolver@v2.1.0+incompatible github.com/thanos-io/thanos@v0.13.1-0.20200731083140-69b87607decf github.com/sercand/kuberesolver@v2.4.0+incompatible
You can see that in the normal version, kuberesolver@2.4.0 Dependent on thanos, kuberesolver@v2.1.0 Dependent on weaveworks.
After replace
▶ go mod graph| grep kuberesolver github.com/weaveworks/common@v0.0.0-20210419092856-009d1eebd624 github.com/sercand/kuberesolver@v2.1.0+incompatible
Isn't it amazing, kuberesolver@v2.4.0 This version disappeared. Because kuberesolver v2.1.0 and v2.4.0 are incompatible, it cannot be compiled after replace.
Gomod replace semantics
In fact, this is not magical. This involves the replace semantics of Go mod, but it is also a feature that is easy to ignore.
replace directives: (https://golang.org/ref/mod#go-mod-file-replace)
replace directives only apply in the main module's go.mod file and are ignored in other modules. See Minimal version selection for details.
In fact, it is very simple. replace is only effective for the main module (that is, your current project). You can make the following summary:
-
The replace ment of the main module does not take effect for the dependent module
-
The replace ment of the go.mod of the dependent module is also ineffective for the main module
Therefore, after replacement, the replacement of thanos that cortex depends on will not take effect. Let's sort out the dependency tree:
-
Main module cortex = > require github.com/weaveworks/common v0.0.0-20210419092856-009d1eebd624
-
weaveworks => requre github.com/sercand/kuberesolver v2.1.0+incompatible
-
Therefore, the overall kuberesolver is only v2.1.0
This logic is consistent with gomod's replace semantics, that is, the compilation after replace is only correct.
Cheat gomod
That's even more amazing. Why can cortex directly require thanos be compiled successfully? According to the gomod replace semantics, this is also compiled, which is correct.
According to the document, we know that replace only acts on the main module, and it will not take effect without the main module. There is no doubt about this.
I did an experiment and put it on https://github.com/georgehao/gomodtestmain , you can try it if you are interested. This can verify that gomod follows gomod replace semantics and MVS (minimum version selection) algorithm.
The problem has basically reached an impasse. How can we break the situation?
Continue to use the go mod graph function to view the dependency tree of thanos that the cortex depends on.
github.com/thanos-io/thanos@v0.19.1-0.20210729154440-aa148f8fdb28 gopkg.in/yaml.v3@v3.0.0-20210107192922-496545a6307 github.com/thanos-io/thanos@v0.13.1-0.20210401085038-d7dff0c84d17 github.com/Azure/azure-pipeline-go@v0.2.2 github.com/thanos-io/thanos@v0.8.1-0.20200109203923-552ffa4c1a0d k8s.io/utils@v0.0.0-20191114200735-6ca3b61696b6 github.com/thanos-io/thanos@v0.13.1-0.20210204123931-82545cdd16fe gopkg.in/yaml.v2@v2.3.0 github.com/thanos-io/thanos@v0.13.1-0.20201030101306-47f9a225cc52 go.uber.org/goleak@v1.1.10 github.com/thanos-io/thanos@v0.13.1-0.20200807203500-9b578afb4763 go.elastic.co/apm/module/apmot@v1.5.0 .... github.com/thanos-io/thanos@v0.13.1-0.20200731083140-69b87607decf github.com/gogo/protobuf@v1.3.1
Because the dependency tree is too long (more than 700 lines), I won't post it. Basically, I can see that cortex depends on thanos N versions. In the last version, we found an interesting thing in go.mod:
require ( github.com/sercand/kuberesolver v2.4.0+incompatible // indirect )
It's been a long time, because thanos is a very old version of gomod require kuberesolver@v2.4.0 , gomod mistakenly thinks that the thanos that cortex depends on is still required kuberesolver@v2.4.0 Although thanos has long been changed to replace kuberesolver, it allows cortex to compile smoothly.
Is this a gomod bug?
Why does cortex rely on so many versions of thanos? This is the problem of reusing the functions of cortex and thanos mentioned at the beginning.
At present, cortex and thanos are basically dependent on:
cortex 1.9.0 -> thanos v0.19.1-0.20210729154440-aa148f8fdb28 thanos v0.19.1-0.20210729154440-aa148f8fdb28 -> cortex v1.8.1-0.20210422151339-cf1c444e0905 cortex v1.8.1-0.20210422151339-cf1c444e0905 -> thanos v0.13.1-0.20210401085038-d7dff0c84d17 ....
The cross reference between cortex and thanos is like a Russian doll, which is a nightmare for gomod. The semantics of go mod replace has been cracked by these two dolls.
How to solve
There are two ways to solve the problem of how to replace cortex thanos. In fact, you know the root of the problem. There are two ways:
-
Due to the gomod MVS algorithm, we directly specify the version of kuberesolver in the main project cortex as v2.4.1
-
Scheme 1 is only applicable to downward compatible projects. If a project does not have this sense of responsibility, it may cause problems. Therefore, a more direct solution is to directly modify thanos go.mod and move the kuberesolver that thanos depends on from replace to require
Welcome to the official account. Learn more about learning materials and pay attention to the official account reply.
-
Reply to 0 and get the Go manual
-
Reply 1: get the Go source code flow chart
