Original link: https://mp.weixin.qq.com/s/1X...
gRPC is a great technology with strict interface constraints and high performance. It is applied in k8s and many microservice frameworks.
As a programmer, learning is right.
I've written some GRC services in Python before. Now I'm going to use Go to experience the original GRC program development.
The feature of this paper is to speak directly with code, and introduce various methods of gRPC through the complete code out of the box.
The code has been uploaded to GitHub. Let's officially start.
introduce
gRPC is a cross language open source RPC framework developed by Google based on Protobuf. gRPC is designed based on HTTP/2 protocol. It can provide multiple services based on one HTTP/2 link, which is more friendly to mobile devices.
introduction
First, let's look at the simplest gRPC service. The first step is to define the proto file, because gRPC is also a C/S architecture. This step is equivalent to clarifying the interface specification.
proto
syntax = "proto3";
package proto;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
Generate gRPC code using the gRPC plug-in built in protocol Gen go:
protoc --go_out=plugins=grpc:. helloworld.proto
After this command is executed, a HelloWorld. XML file will be generated in the current directory pb. Go file, which defines the interfaces of server and client respectively:
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.
type GreeterClient interface {
// Sends a greeting
SayHello(ctx context.Context, in *HelloRequest, opts ...grpc.CallOption) (*HelloReply, error)
}
// GreeterServer is the server API for Greeter service.
type GreeterServer interface {
// Sends a greeting
SayHello(context.Context, *HelloRequest) (*HelloReply, error)
}
The next step is to write the code of the server and the client to implement the corresponding interfaces respectively.
server
package main
import (
"context"
"fmt"
"grpc-server/proto"
"log"
"net"
"google.golang.org/grpc"
"google.golang.org/grpc/reflection"
)
type greeter struct {
}
func (*greeter) SayHello(ctx context.Context, req *proto.HelloRequest) (*proto.HelloReply, error) {
fmt.Println(req)
reply := &proto.HelloReply{Message: "hello"}
return reply, nil
}
func main() {
lis, err := net.Listen("tcp", ":50051")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
server := grpc.NewServer()
//Register the reflection service required by grpcurl
reflection.Register(server)
//Register business services
proto.RegisterGreeterServer(server, &greeter{})
fmt.Println("grpc server start ...")
if err := server.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
client
package main
import (
"context"
"fmt"
"grpc-client/proto"
"log"
"google.golang.org/grpc"
)
func main() {
conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
if err != nil {
log.Fatal(err)
}
defer conn.Close()
client := proto.NewGreeterClient(conn)
reply, err := client.SayHello(context.Background(), &proto.HelloRequest{Name: "zhangsan"})
if err != nil {
log.Fatal(err)
}
fmt.Println(reply.Message)
}
This completes the development of the most basic gRPC service. Next, we will enrich this "basic template" and learn more features.
Flow mode
Next, let's look at the way of streaming. As the name suggests, data can be sent and received continuously.
Flow words are divided into one-way flow and two-way flow. Here we directly take two-way flow as an example.
proto
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
// Sends stream message
rpc SayHelloStream (stream HelloRequest) returns (stream HelloReply) {}
}
Add a stream function "SayHelloStream" to specify the stream characteristics through the "stream" keyword.
HelloWorld. Needs to be regenerated pb. Go file, no more here.
server
func (*greeter) SayHelloStream(stream proto.Greeter_SayHelloStreamServer) error {
for {
args, err := stream.Recv()
if err != nil {
if err == io.EOF {
return nil
}
return err
}
fmt.Println("Recv: " + args.Name)
reply := &proto.HelloReply{Message: "hi " + args.Name}
err = stream.Send(reply)
if err != nil {
return err
}
}
}
Add the SayHelloStream function on the basic template. Nothing else needs to be changed.
client
client := proto.NewGreeterClient(conn)
//Stream processing
stream, err := client.SayHelloStream(context.Background())
if err != nil {
log.Fatal(err)
}
//Send message
go func() {
for {
if err := stream.Send(&proto.HelloRequest{Name: "zhangsan"}); err != nil {
log.Fatal(err)
}
time.Sleep(time.Second)
}
}()
//Receive message
for {
reply, err := stream.Recv()
if err != nil {
if err == io.EOF {
break
}
log.Fatal(err)
}
fmt.Println(reply.Message)
}
The message is sent through a goroutine, and the for loop of the main program receives the message.
Executing the program will find that both the server and the client continue to have printouts.
Validator
Next is the verifier, which is a natural requirement. Because it involves requests between interfaces, it is necessary to verify the parameters properly.
Here we use protocol Gen govalidators and go grpc middleware to implement.
Install first:
go get github.com/mwitkow/go-proto-validators/protoc-gen-govalidators go get github.com/grpc-ecosystem/go-grpc-middleware
Next, modify the proto file:
proto
import "github.com/mwitkow/go-proto-validators@v0.3.2/validator.proto";
message HelloRequest {
string name = 1 [
(validator.field) = {regex: "^[z]{2,5}$"}
];
}
Here, verify the {name} parameter. Normal requests can only be made if they meet the requirements of regularity.
There are other verification rules, such as verifying the size of numbers, which will not be introduced here.
Next, generate * pb.go file:
protoc \
--proto_path=${GOPATH}/pkg/mod \
--proto_path=${GOPATH}/pkg/mod/github.com/gogo/protobuf@v1.3.2 \
--proto_path=. \
--govalidators_out=. --go_out=plugins=grpc:.\
*.proto
After successful execution, there will be one more HelloWorld validator. pb. Go file.
You should pay special attention here. You can't use the previous simple commands. You need to use multiple} PROTOS_ The path} parameter specifies the directory where the proto file is imported.
The official gives two kinds of dependence, one is google protobuf and the other is gogo protobuf. I use the second one here.
Even if you use the above command, you may encounter this error:
Import "github.com/mwitkow/go-proto-validators/validator.proto" was not found or had errors
But don't panic. The probability is the reference path. You must be optimistic about your installation version and the specific path in # GOPATH #.
Finally, the server code transformation:
Import package:
grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware" grpc_validator "github.com/grpc-ecosystem/go-grpc-middleware/validator"
Then add the verifier function during initialization:
server := grpc.NewServer( grpc.UnaryInterceptor( grpc_middleware.ChainUnaryServer( grpc_validator.UnaryServerInterceptor(), ), ), grpc.StreamInterceptor( grpc_middleware.ChainStreamServer( grpc_validator.StreamServerInterceptor(), ), ), )
After starting the program, we use the previous client code to request, and we will receive an error:
2021/10/11 18:32:59 rpc error: code = InvalidArgument desc = invalid field Name: value 'zhangsan' must be a string conforming to regex "^[z]{2,5}$"
exit status 1
Because {name: zhangsan} does not meet the regular requirements of the server, but if you pass the parameter {name: zzz}, you can return normally.
Token authentication
Finally, it's time for authentication. Let's first look at the Token authentication method, and then introduce the certificate authentication.
First transform the server. With the experience of the verifier above, you can write an interceptor in the same way, and then inject it when initializing the server.
Authentication function:
func Auth(ctx context.Context) error {
md, ok := metadata.FromIncomingContext(ctx)
if !ok {
return fmt.Errorf("missing credentials")
}
var user string
var password string
if val, ok := md["user"]; ok {
user = val[0]
}
if val, ok := md["password"]; ok {
password = val[0]
}
if user != "admin" || password != "admin" {
return grpc.Errorf(codes.Unauthenticated, "invalid token")
}
return nil
}
metadata.FromIncomingContext: read the user name and password from the context, and then compare them with the actual data to determine whether they have passed the authentication.
Interceptor:
var authInterceptor grpc.UnaryServerInterceptor
authInterceptor = func(
ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler,
) (resp interface{}, err error) {
//Intercept the normal method request and verify the {Token
err = Auth(ctx)
if err != nil {
return
}
//Continue processing request
return handler(ctx, req)
}
initialization:
server := grpc.NewServer( grpc.UnaryInterceptor( grpc_middleware.ChainUnaryServer( authInterceptor, grpc_validator.UnaryServerInterceptor(), ), ), grpc.StreamInterceptor( grpc_middleware.ChainStreamServer( grpc_validator.StreamServerInterceptor(), ), ), )
In addition to the above verifier, there is also a Token authentication interceptor called authInterceptor.
The last is the transformation of the client. The client needs to implement the {PerRPCCredentials} interface.
type PerRPCCredentials interface {
// GetRequestMetadata gets the current request metadata, refreshing
// tokens if required. This should be called by the transport layer on
// each request, and the data should be populated in headers or other
// context. If a status code is returned, it will be used as the status
// for the RPC. uri is the URI of the entry point for the request.
// When supported by the underlying implementation, ctx can be used for
// timeout and cancellation.
// TODO(zhaoq): Define the set of the qualified keys instead of leaving
// it as an arbitrary string.
GetRequestMetadata(ctx context.Context, uri ...string) (
map[string]string, error,
)
// RequireTransportSecurity indicates whether the credentials requires
// transport security.
RequireTransportSecurity() bool
}
The GetRequestMetadata method returns the necessary information required for authentication. The RequireTransportSecurity method indicates whether to enable the security link. It is generally enabled in the production environment, but it is not enabled here for testing convenience.
Implementation interface:
type Authentication struct {
User string
Password string
}
func (a *Authentication) GetRequestMetadata(context.Context, ...string) (
map[string]string, error,
) {
return map[string]string{"user": a.User, "password": a.Password}, nil
}
func (a *Authentication) RequireTransportSecurity() bool {
return false
}
connect:
conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure(), grpc.WithPerRPCCredentials(&auth))
Well, now our service has the function of Token authentication. If the user name or password is incorrect, the client will receive:
2021/10/11 20:39:35 rpc error: code = Unauthenticated desc = invalid token exit status 1
If the user name and password are correct, you can return normally.
One way certificate authentication
There are two ways of certificate authentication:
- One way authentication
- Two way authentication
Let's first look at the one-way authentication method:
Generate certificate
First, a self signed SSL certificate is generated through the openssl tool.
1. Generate private key:
openssl genrsa -des3 -out server.pass.key 2048
2. Remove password from private key:
openssl rsa -in server.pass.key -out server.key
3. Generate csr file:
openssl req -new -key server.key -out server.csr -subj "/C=CN/ST=beijing/L=beijing/O=grpcdev/OU=grpcdev/CN=example.grpcdev.cn"
4. Generate certificate:
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
One more word, let's introduce the three files contained in the X.509 certificate: key, csr and crt.
- Key: the private key file on the server, which is used to encrypt the data sent to the client and decrypt the data received from the client.
- csr: Certificate Signing Request file, which is used to submit to the certification authority (CA) to sign the certificate.
- crt: a certificate signed by a certification authority (CA), or a developer's self signed certificate, including the information of the certificate holder, the holder's public key, and the signer's signature.
gRPC code
After having the certificate, the rest is to transform the program. The first is the server code.
//Certificate authentication - one way authentication
creds, err := credentials.NewServerTLSFromFile("keys/server.crt", "keys/server.key")
if err != nil {
log.Fatal(err)
return
}
server := grpc.NewServer(grpc.Creds(creds))
Only a few lines of code need to be modified. It's very simple. Next is the client.
Because it is one-way authentication, there is no need to generate a certificate for the client separately, just copy the crt file of the server to the corresponding directory of the client.
//Certificate authentication - one way authentication
creds, err := credentials.NewClientTLSFromFile("keys/server.crt", "example.grpcdev.cn")
if err != nil {
log.Fatal(err)
return
}
conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(creds))
Well, now our service supports one-way certificate authentication.
But it's not over yet. There may be a problem here:
2021/10/11 21:32:37 rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0" exit status 1
The reason is that Go 1.15 began to abandon CommonName, and SAN certificate is recommended. If you want to be compatible with the previous method, you can support it by setting environment variables, as follows:
export GODEBUG="x509ignoreCN=0"
However, it should be noted that since Go 1.17, environment variables are no longer effective and must be implemented through SAN. Therefore, in order to upgrade the follow-up Go version, it is better to support it as soon as possible.
Two way certificate authentication
Finally, let's look at two-way certificate authentication.
Generate certificate with SAN
It's still a certificate, but this time it's a little different. We need to generate a certificate with SAN extension.
What is SAN?
SAN (Subject Alternative Name) is an extension defined in the SSL standard x509. The SSL certificate using the SAN field can extend the domain name supported by this certificate, so that a certificate can support the resolution of multiple different domain names.
Copy the default OpenSSL configuration file to the current directory.
Linux system:
/etc/pki/tls/openssl.cnf
Mac system on:
/System/Library/OpenSSL/openssl.cnf
Modify the temporary configuration file, find the [req] paragraph, and then remove the comments of the following statements.
req_extensions = v3_req # The extensions to add to a certificate request
Then add the following configuration:
[ v3_req ] # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [ alt_names ] DNS.1 = www.example.grpcdev.cn
Multiple domain names can be configured at the [alt_names] location, such as:
[ alt_names ] DNS.1 = www.example.grpcdev.cn DNS.2 = www.test.grpcdev.cn
For the convenience of testing, only one domain name is configured here.
1. Generate ca certificate:
openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key -subj "/CN=example.grpcdev.com" -days 5000 -out ca.pem
2. Generate server certificate:
#Generate certificate openssl req -new -nodes \ -subj "/C=CN/ST=Beijing/L=Beijing/O=grpcdev/OU=grpcdev/CN=www.example.grpcdev.cn" \ -config <(cat openssl.cnf \ <(printf "[SAN]\nsubjectAltName=DNS:www.example.grpcdev.cn")) \ -keyout server.key \ -out server.csr #Signing certificate openssl x509 -req -days 365000 \ -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial \ -extfile <(printf "subjectAltName=DNS:www.example.grpcdev.cn") \ -out server.pem
3. Generate client certificate:
#Generate certificate openssl req -new -nodes \ -subj "/C=CN/ST=Beijing/L=Beijing/O=grpcdev/OU=grpcdev/CN=www.example.grpcdev.cn" \ -config <(cat openssl.cnf \ <(printf "[SAN]\nsubjectAltName=DNS:www.example.grpcdev.cn")) \ -keyout client.key \ -out client.csr #Signing certificate openssl x509 -req -days 365000 \ -in client.csr -CA ca.pem -CAkey ca.key -CAcreateserial \ -extfile <(printf "subjectAltName=DNS:www.example.grpcdev.cn") \ -out client.pem
gRPC code
Next, start to modify the code. First look at the server:
//Certificate authentication - two way authentication
//Read and parse the information from the certificate related files to obtain the certificate public key and key pair
cert, _ := tls.LoadX509KeyPair("cert/server.pem", "cert/server.key")
//Create a new, empty CertPool
certPool := x509.NewCertPool()
ca, _ := ioutil.ReadFile("cert/ca.pem")
//An attempt was made to resolve the incoming PEM encoded certificate. If the parsing is successful, it will be added to CertPool for later use
certPool.AppendCertsFromPEM(ca)
//Build the # TransportCredentials # option based on # TLS #
creds := credentials.NewTLS(&tls.Config{
//Set the certificate chain to allow one or more certificates to be included
Certificates: []tls.Certificate{cert},
//The client's certificate must be verified. The following parameters can be selected according to the actual situation
ClientAuth: tls.RequireAndVerifyClientCert,
//Set the collection of root certificates. The verification method uses the mode set in ClientAuth
ClientCAs: certPool,
})
Look at the client:
//Certificate authentication - two way authentication
//Read and parse the information from the certificate related files to obtain the certificate public key and key pair
cert, _ := tls.LoadX509KeyPair("cert/client.pem", "cert/client.key")
//Create a new, empty CertPool
certPool := x509.NewCertPool()
ca, _ := ioutil.ReadFile("cert/ca.pem")
//An attempt was made to resolve the incoming PEM encoded certificate. If the parsing is successful, it will be added to CertPool for later use
certPool.AppendCertsFromPEM(ca)
//Build the # TransportCredentials # option based on # TLS #
creds := credentials.NewTLS(&tls.Config{
//Set the certificate chain to allow one or more certificates to be included
Certificates: []tls.Certificate{cert},
//The client's certificate must be verified. The following parameters can be selected according to the actual situation
ServerName: "www.example.grpcdev.cn",
RootCAs: certPool,
})
be accomplished.
Python client
As mentioned earlier, gRPC is cross language. At the end of this article, we write a client in Python to request the Go server.
Use the simplest way to achieve:
The proto file uses the proto file of the initial "basic template":
syntax = "proto3";
package proto;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
// Sends stream message
rpc SayHelloStream (stream HelloRequest) returns (stream HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
Similarly, you also need to generate Pb from the command line Py file:
python3 -m grpc_tools.protoc -I . --python_out=. --grpc_python_out=. ./*.proto
After successful execution, HelloWorld will be generated in the directory_ pb2. Py and helloworld_pb2_grpc.py two files.
This process may also report errors:
ModuleNotFoundError: No module named 'grpc_tools'
Don't panic. The package is missing. Just install it:
pip3 install grpcio pip3 install grpcio-tools
Finally, take a look at the Python client code:
import grpc
import helloworld_pb2
import helloworld_pb2_grpc
def main():
channel = grpc.insecure_channel("127.0.0.1:50051")
stub = helloworld_pb2_grpc.GreeterStub(channel)
response = stub.SayHello(helloworld_pb2.HelloRequest(name="zhangsan"))
print(response.message)
if __name__ == '__main__':
main()
In this way, you can request the server-side service started by Go through the Python client.
summary
This paper explains some applications of gRPC from the perspective of actual combat and directly speaking with code.
The content includes simple gRPC service, stream processing mode, verifier, Token authentication and certificate authentication.
In addition, there are other contents worth studying, such as timeout control, REST interface and load balancing. I will take time to continue to improve the REST of this part in the future.
The code in this article has been tested and verified, can be executed directly, and has been uploaded to GitHub. Partners can look at the source code and learn by comparing the content of the article again and again.
Source address:
WeChat public number [programmer Huang Xiaoxie] is the former engineer of ant Java, who focuses on sharing Java technology dry cargo and job search experience. It is not limited to BAT interview, algorithm, computer basis, database, distributed official account, spring family bucket, micro service, high concurrency, JVM, Docker container, ELK, big data, etc. [book] get 20 selected high-quality e-books necessary for Java interview.