Microservices have everything from code to k8s deployment (X. error handling)

Posted by michalchojno on Fri, 18 Feb 2022 06:30:56 +0100

We will use a series to explain the complete practice of microservices from requirements to online, from code to k8s deployment, from logging to monitoring.

The whole project uses the micro services developed by go zero, which basically includes go zero and some middleware developed by relevant go zero authors. The technology stack used is basically the self-developed component of the go zero project team, which is basically the go zero software.

Actual project address: https://github.com/Mikaelemmmm/go-zero-looklook

1. Overview

During our normal development, when the program makes an error, we hope to quickly locate the problem through the error log (then the parameters passed in, including stack information, must be printed to the log), but at the same time, we want to return to the front-end user-friendly and understandable error prompt. If these two points are only through one FMT Error,errors. If new returns an error message, it is definitely impossible to do so, unless the log is recorded at the same time when it returns the error prompt at the front end. In this way, the log will fly all over the sky, the code will be ugly, and the log will be ugly at that time.

Let's think about it. If there is a unified place to record the log, and only one return err is needed in the business code, the error prompt information and log information returned to the front end can be separated from the prompt and record. If it is implemented according to this idea, it's not too cool. Yes, go zero look is handled in this way. Let's take a look.

2. rpc error handling

Under normal circumstances, go zero's rpc service is based on grpc, and the default error returned is grpc's status Error cannot merge our customized errors, and it is not suitable for our customized errors. Its error code and error type are defined to die in the grpc package. ok, if we can use the custom error to return in rpc, then it will be converted to the status of grpc when the interceptor returns uniformly Error, can our rpc err and api err manage our own errors in a unified way?

Let's take a look at the status of grpc What's in the code of error

package codes // import "google.golang.org/grpc/codes"

import (
    "fmt"
    "strconv"
)

// A Code is an unsigned 32-bit error code as defined in the gRPC spec.
type Code uint32
.......

The error code corresponding to the err of grpc is actually a uint32. We define the error ourselves, use uint32, and then turn it into the err of grpc when the global interceptor of rpc returns

So we define the global error code in app/common/xerr

errCode.go

package xerr

// Successful return
const OK uint32 = 200

// The first three represent business and the last three represent specific functions

// Global error code
const SERVER_COMMON_ERROR uint32 = 100001
const REUQES_PARAM_ERROR uint32 = 100002
const TOKEN_EXPIRE_ERROR uint32 = 100003
const TOKEN_GENERATE_ERROR uint32 = 100004
const DB_ERROR uint32 = 100005

// User module

errMsg.go

package xerr

var message map[uint32]string

func init() {
   message = make(map[uint32]string)
   message[OK] = "SUCCESS"
   message[SERVER_COMMON_ERROR] = "The server is wandering,Try again later"
   message[REUQES_PARAM_ERROR] = "Parameter error"
   message[TOKEN_EXPIRE_ERROR] = "token Invalid, please log in again"
   message[TOKEN_GENERATE_ERROR] = "generate token fail"
   message[DB_ERROR] = "Database busy,Please try again later"
}

func MapErrMsg(errcode uint32) string {
   if msg, ok := message[errcode]; ok {
      return msg
   } else {
      return "The server is wandering,Try again later"
   }
}

func IsCodeErr(errcode uint32) bool {
   if _, ok := message[errcode]; ok {
      return true
   } else {
      return false
   }
}

errors.go

package xerr

import "fmt"

// Common fixed error
type CodeError struct {
   errCode uint32
   errMsg  string
}

// Error code returned to the front end
func (e *CodeError) GetErrCode() uint32 {
   return e.errCode
}

// Return to the front-end display end error message
func (e *CodeError) GetErrMsg() string {
   return e.errMsg
}

func (e *CodeError) Error() string {
   return fmt.Sprintf("ErrCode:%d,ErrMsg:%s", e.errCode, e.errMsg)
}

func NewErrCodeMsg(errCode uint32, errMsg string) *CodeError {
   return &CodeError{errCode: errCode, errMsg: errMsg}
}
func NewErrCode(errCode uint32) *CodeError {
   return &CodeError{errCode: errCode, errMsg: MapErrMsg(errCode)}
}

func NewErrMsg(errMsg string) *CodeError {
   return &CodeError{errCode: SERVER_COMMON_ERROR, errMsg: errMsg}
}

For example, we use rpc code when registering users

package logic

import (
    "context"

    "looklook/app/identity/cmd/rpc/identity"
    "looklook/app/usercenter/cmd/rpc/internal/svc"
    "looklook/app/usercenter/cmd/rpc/usercenter"
    "looklook/app/usercenter/model"
    "looklook/common/xerr"

    "github.com/pkg/errors"
    "github.com/tal-tech/go-zero/core/logx"
    "github.com/tal-tech/go-zero/core/stores/sqlx"
)

var ErrUserAlreadyRegisterError = xerr.NewErrMsg("The user is already registered")

type RegisterLogic struct {
    ctx    context.Context
    svcCtx *svc.ServiceContext
    logx.Logger
}

func NewRegisterLogic(ctx context.Context, svcCtx *svc.ServiceContext) *RegisterLogic {
    return &RegisterLogic{
        ctx:    ctx,
        svcCtx: svcCtx,
        Logger: logx.WithContext(ctx),
    }
}

func (l *RegisterLogic) Register(in *usercenter.RegisterReq) (*usercenter.RegisterResp, error) {

    user, err := l.svcCtx.UserModel.FindOneByMobile(in.Mobile)
    if err != nil && err != model.ErrNotFound {
        return nil, errors.Wrapf(xerr.ErrDBError, "mobile:%s,err:%v", in.Mobile, err)
    }

    if user != nil {
        return nil, errors.Wrapf(ErrUserAlreadyRegisterError, "User already exists mobile:%s,err:%v", in.Mobile, err)
    }

    var userId int64

    if err := l.svcCtx.UserModel.Trans(func(session sqlx.Session) error {

        user := new(model.User)
        user.Mobile = in.Mobile
        user.Nickname = in.Nickname
        insertResult, err := l.svcCtx.UserModel.Insert(session, user)
        if err != nil {
            return errors.Wrapf(xerr.ErrDBError, "err:%v,user:%+v", err, user)
        }
        lastId, err := insertResult.LastInsertId()
        if err != nil {
            return errors.Wrapf(xerr.ErrDBError, "insertResult.LastInsertId err:%v,user:%+v", err, user)
        }
        userId = lastId

        userAuth := new(model.UserAuth)
        userAuth.UserId = lastId
        userAuth.AuthKey = in.AuthKey
        userAuth.AuthType = in.AuthType
        if _, err := l.svcCtx.UserAuthModel.Insert(session, userAuth); err != nil {
            return errors.Wrapf(xerr.ErrDBError, "err:%v,userAuth:%v", err, userAuth)
        }
        return nil
    }); err != nil {
        return nil, err
    }

    // 2. Generate token
    resp, err := l.svcCtx.IdentityRpc.GenerateToken(l.ctx, &identity.GenerateTokenReq{
        UserId: userId,
    })
    if err != nil {
        return nil, errors.Wrapf(ErrGenerateTokenError, "IdentityRpc.GenerateToken userId : %d , err:%+v", userId, err)
    }

    return &usercenter.RegisterResp{
        AccessToken:  resp.AccessToken,
        AccessExpire: resp.AccessExpire,
        RefreshAfter: resp.RefreshAfter,
    }, nil
}
errors.Wrapf(ErrUserAlreadyRegisterError, "User already exists mobile:%s,err:%v", in.Mobile, err)

Here we use the go default errors package errors Wrapf (if you don't understand here, check Wrap and wrapf under go's errors package)

The first parameter, ErrUserAlreadyRegisterError, is defined above using xerr Newerrmsg ("the user has been registered") returns a friendly prompt to the front end. Remember that the method under our xerr package is used here

The second parameter is recorded in the server log, which can be written in detail. It doesn't matter at all. It will only be recorded in the server and will not be returned to the front end

Let's see why the first parameter can be returned to the front end, and the second parameter is to record the log

⚠️ [note] in the main method of rpc startup file, we add the global interceptor of grpc, which is very important. If we don't add it, we can't realize it

package main

......

func main() {
    ........

    //RPC log, global interceptor of grpc
    s.AddUnaryInterceptors(rpcserver.LoggerInterceptor)

    .......
}

Let's look at rpcserver Implementation of loggerinterceptor

import (
    ...
    "github.com/pkg/errors"
)

func LoggerInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {
   resp, err = handler(ctx, req)
   if err != nil {
      causeErr := errors.Cause(err)                // err type
      if e, ok := causeErr.(*xerr.CodeError); ok { //Custom error type
         logx.WithContext(ctx).Errorf("[RPC-SRV-ERR] %+v", err)

         //Convert to grpc err
         err = status.Error(codes.Code(e.GetErrCode()), e.GetErrMsg())
      } else {
         logx.WithContext(ctx).Errorf("[RPC-SRV-ERR] %+v", err)
      }
   }

   return resp, err
}

When a request enters the rpc service, first enter the interceptor and then execute the handler method. If you want to handle something before entering, you can write it before the handler method. What we want to deal with is the return result. If there is an error, so we use GitHub under the handler COM / PKG / errors package. This package is often used in go to handle errors. This is not the official errors package, but it is well designed. The official Wrap and Wrapf of go draw on the ideas of this package.

Because when our grpc internal business returns an error

1) if it is our own business error, we will uniformly generate the error with xerr, so we can get the error information defined by us. Because our own error is also uint32, it is uniformly converted to grpc error err = status Error (codes. Code (e.GetErrCode()), e.GetErrMsg()), then the obtained here, e.GetErrCode() is the code we defined, and e.GetErrMsg() is the second parameter of the error we defined earlier

2) However, there is another case where the rpc service is abnormal. The error thrown out at the bottom is a grpc error. In this case, we can just record the exception directly

3. api error

When api called Register in rpc in logic, rpc returned the error message code for the above second steps as follows

......
func (l *RegisterLogic) Register(req types.RegisterReq) (*types.RegisterResp, error) {
    registerResp, err := l.svcCtx.UsercenterRpc.Register(l.ctx, &usercenter.RegisterReq{
        Mobile:   req.Mobile,
        Nickname: req.Nickname,
        AuthKey:  req.Mobile,
        AuthType: model.UserAuthTypeSystem,
    })
    if err != nil {
        return nil, errors.Wrapf(err, "req: %+v", req)
    }

    var resp types.RegisterResp
    _ = copier.Copy(&resp, registerResp)

    return &resp, nil
}

Here is also errors. Using the standard package Wrapf, that is, all errors returned in our business are applicable to the errors of the standard package, but the internal parameters should use the errors defined by xerr

Here are two points to note

1) For example, if the api does not want to handle the error, it will directly return the error message to the front end (if we do not want to handle it, we will directly return the error message to the rpc)

In this case, just write it directly like the above figure, and use err of rpc call directly as errors The first parameter of wrapf is thrown out, but the second parameter is better to record the detailed log you need for later viewing in the api log

2) api service no matter what error message rpc returns, I want to redefine myself and return the error message to the front end (for example, rpc has returned "user already exists". When api wants to call rpc, as long as there is an error, I will return to the front end "user registration failed")

In this case, it can be written as follows (of course, you can put xerr.NewErrMsg("user registration failed") on the top of the code to use a variable, or put a variable here)

func (l *RegisterLogic) Register(req types.RegisterReq) (*types.RegisterResp, error) {
    .......
    if err != nil {
        return nil, errors.Wrapf(xerr.NewErrMsg("User registration failed"), "req: %+v,rpc err:%+v", req,err)
    }
    .....
}

Next, let's look at how to deal with the final return to the front end. We then look at app / usercenter / CMD / API / internal / handler / user / registerhandler go

func RegisterHandler(ctx *svc.ServiceContext) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        var req types.RegisterReq
        if err := httpx.Parse(r, &req); err != nil {
            httpx.Error(w, err)
            return
        }

        l := user.NewRegisterLogic(r.Context(), ctx)
        resp, err := l.Register(req)
        result.HttpResult(r, w, resp, err)
    }
}

It can be seen here that the handler code generated by go zero look is different from the code generated by the default official goctl in two places, that is, when dealing with error handling, we replace it with our own error handling, in common / result / httpresult go

[note] some people will say that every time we use goctl, we have to change it manually. That's not a lot of trouble. Here we use the template function provided by go zero (we need to learn from the official documents if we don't know this yet). Just modify the template generated by the handler. The template file of the whole project is placed under deploy/goctl, Here, the template modified by hanlder is in deploy/goctl / 1.2.3-cli / API / handler tpl

ParamErrorResult is very simple and is dedicated to handling parameter errors

// http parameter error return
func ParamErrorResult(r *http.Request, w http.ResponseWriter, err error) {
   errMsg := fmt.Sprintf("%s ,%s", xerr.MapErrMsg(xerr.REUQES_PARAM_ERROR), err.Error())
   httpx.WriteJson(w, http.StatusBadRequest, Error(xerr.REUQES_PARAM_ERROR, errMsg))
}

Let's focus on HttpResult, the error handling method returned by the business

// http return
func HttpResult(r *http.Request, w http.ResponseWriter, resp interface{}, err error) {
    if err == nil {
        // Successful return
        r := Success(resp)
        httpx.WriteJson(w, http.StatusOK, r)
    } else {
        // Error return
        errcode := xerr.SERVER_COMMON_ERROR
        errmsg := "The server is running away. I'll try again later"

        causeErr := errors.Cause(err) // err type
        if e, ok := causeErr.(*xerr.CodeError); ok {
            // Custom error type
            // Custom CodeError
            errcode = e.GetErrCode()
            errmsg = e.GetErrMsg()
        } else {
            if gstatus, ok := status.FromError(causeErr); ok {
                // grpc err error
                grpcCode := uint32(gstatus.Code())
                if xerr.IsCodeErr(grpcCode) {
                    // Distinguish between user-defined errors and system bottom layer and db errors. Bottom layer and db errors cannot be returned to the front end
                    errcode = grpcCode
                    errmsg = gstatus.Message()
                }
            }
        }

        logx.WithContext(r.Context()).Errorf("[API-ERR] : %+v ", err)
        httpx.WriteJson(w, http.StatusBadRequest, Error(errcode, errmsg))
    }
}

err: log error to log

errcode: the error code returned to the front end

errmsg: friendly error message returned to the front end

Success is returned directly. If an error is encountered, GitHub is also used COM / PKG / errors package to determine whether the error is our own defined error (the error defined in the api directly uses our own defined xerr) or grpc error (thrown out by rpc service). If it is grpc error, it is converted into our own error code through uint32, According to the error code, find the defined error information in the error information defined by ourselves and return it to the front end. If the api error is directly returned to the error information defined by ourselves and can't be found, then return the default error "the server is out of service",

4. Ending

Here, the error handling message has been clearly described. Next, we need to see how to collect and view the error log printed on the server, which involves the log collection system.

Project address

https://github.com/zeromicro/go-zero

https://gitee.com/kevwan/go-zero

Welcome to go zero and star support us!

Wechat communication group

Focus on the "micro service practice" official account and click on the exchange group to get the community community's two-dimensional code.

Topics: Go grpc