Golang concise architecture practice

Posted by nunomira on Sat, 08 Jan 2022 07:34:28 +0100

Project code location: https://github.com/devYun/go-clean-architecture

For reprint, please state the source ~, this article is posted on luozhiyun's blog: https://www.luozhiyun.com/archives/640

Because golang does not have a unified coding mode like java, we, like other teams, adopt Go package oriented design and architecture layering This article introduces some theories, and then carries out subcontracting in combination with previous project experience:

├── cmd/
│   └── main.go //Start function
├── etc
│   └── dev_conf.yaml              // configuration file 
├── global
│   └── global.go //Global variable references, such as database, kafka, etc
├── internal/
│       └── service/
│           └── xxx_service.go //Business logic processing class
│           └── xxx_service_test.go 
│       └── model/
│           └── xxx_info.go//structural morphology
│       └── api/
│           └── xxx_api.go//Interface implementation corresponding to routing
│       └── router/
│           └── router.go//route
│       └── pkg/
│           └── datetool//Time tool class
│           └── jsontool//json tool class

In fact, the above division just simply subcontracts the functions. There are still many problems in the process of project practice. For example:

For function implementation, do I pass through function parameters or structural variables?

Is it safe to use the global variable reference of a database? Is there excessive coupling?

Almost all the code implementation depends on the implementation rather than the interface. Do you want to modify all the implementations when switching MySQL to MongDB?

So now in our work, with more and more code, we feel more and more confused about various init, function, struct and global variables in the code.

Each module is not independent. It seems that it is logically divided into modules, but there is no clear relationship between upper and lower layers. There may be configuration reading, external service invocation, protocol conversion, etc. in each module.

Over time, the calls between different package functions of services slowly evolve into a network structure, and the flow direction and logic sorting of data flow become more and more complex. It is difficult to understand the data flow direction without looking at the code calls.

But as refactoring says: let the code work first - if the code can't work, it can't produce value; Then try to make it better - by refactoring the code, we and others can better understand the code and constantly modify the code as needed.

So I think it's time to change myself.

The Clean Architecture

In the concise architecture, we put forward several requirements for our project:

Independent of the framework. The architecture does not depend on the existence of some feature rich software libraries. This allows you to use these frameworks as tools rather than cramming your system into their limited constraints.
Testable. Business rules can be tested without a UI, database, Web server, or any other external element.
Independent of the user interface. The UI can be easily changed without changing other parts of the system. For example, a Web UI can be replaced with a console UI without changing business rules.
Independent of database. You can replace Oracle or SQL Server with Mongo, BigTable, CouchDB or other things. Your business rules are not bound by the database.
Independent of any external organization. In fact, your business rules don't know anything about the outside world.

The concentric circles in the figure above represent software in various fields. Generally speaking, the deeper you go, the higher your software level. The outer circle is the tactical realization mechanism, and the inner circle is the strategic core strategy.

For our project, code dependency should be from outside to inside, one-way single-layer dependency, which includes code name, or class function, variable or any other named software entity.

For the concise architecture, it is divided into four layers:

Entities: entities
Usecase: Express application business rules, corresponding to the application layer, which encapsulates and implements all use cases of the system;
Interface Adapters: the software in this layer is basically some adapters, which are mainly used to convert the data in use cases and entities into the data used by external systems, such as databases or Web;
Framework & Driver: the outermost circle is usually composed of some frameworks and tools, such as Database, Web framework, etc;

For my project, it is also divided into four layers:

models
repo
service
api

models

It encapsulates various entity class objects, such as those interacting with the database and UI. Any entity class should be placed here. For example:

import "time"

type Article struct {
	ID        int64     `json:"id"`
	Title     string    `json:"title"`
	Content   string    `json:"content"`
	UpdatedAt time.Time `json:"updated_at"`
	CreatedAt time.Time `json:"created_at"`
}

repo

The database operation class is stored here, and the database CRUD is here. It should be noted that there is no business logic code here. Many students like to put business logic here.

If ORM is used, put the code related to ORM operation here; If microservices are used, the codes of other service requests are put here;

service

This is the business logic layer, where all business process processing code should be placed. This layer will decide what code to request from the repo layer, whether to operate the database or call other services; All business data calculations should also be placed here; The input parameters accepted here should be passed in by the controller.

api

Here is the code for receiving external requests, such as the handler corresponding to gin, gRPC, other REST API framework access layers, and so on.

Interface oriented programming

In addition to the models layer, layers should interact with each other through interfaces, not implementations. If you want to call the repo layer with service, you should call the repo interface. When modifying the underlying implementation, our upper base class does not need to be changed. We only need to change the underlying implementation.

For example, if we want to query all articles, we can provide such an interface in repo:

package repo

import (
	"context"
	"my-clean-rchitecture/models"
	"time"
)

// IArticleRepo represent the article's repository contract
type IArticleRepo interface {
	Fetch(ctx context.Context, createdDate time.Time, num int) (res []models.Article, err error)
}

The implementation class of this interface can be changed according to requirements. For example, when we want mysql to store queries, we only need to provide such a base class:

type mysqlArticleRepository struct {
	DB *gorm.DB
}

// NewMysqlArticleRepository will create an object that represent the article.Repository interface
func NewMysqlArticleRepository(DB *gorm.DB) IArticleRepo {
	return &mysqlArticleRepository{DB}
}

func (m *mysqlArticleRepository) Fetch(ctx context.Context, createdDate time.Time,
	num int) (res []models.Article, err error) {

	err = m.DB.WithContext(ctx).Model(&models.Article{}).
		Select("id,title,content, updated_at, created_at").
		Where("created_at > ?", createdDate).Limit(num).Find(&res).Error
	return
}

If we want to change MongoDB to implement our storage another day, we only need to define a structure to implement the IArticleRepo interface.

When implementing the service layer, we can inject the corresponding repo implementation according to our requirements, so there is no need to change the implementation of the service layer:

type articleService struct {
	articleRepo repo.IArticleRepo
}

// NewArticleService will create new an articleUsecase object representation of domain.ArticleUsecase interface
func NewArticleService(a repo.IArticleRepo) IArticleService {
	return &articleService{
		articleRepo: a,
	}
}

// Fetch
func (a *articleService) Fetch(ctx context.Context, createdDate time.Time, num int) (res []models.Article, err error) {
	if num == 0 {
		num = 10
	}
	res, err = a.articleRepo.Fetch(ctx, createdDate, num)
	if err != nil {
		return nil, err
	}
	return
}

Dependency injection DI

Dependency injection is called dependency injection (DI for short). Di was often encountered in java projects before, but many people in go said it was not necessary, but I think it is still necessary in the process of large-scale software development, otherwise it can only be passed through global variables or method parameters.

As for what is DI, it is simply a dependent module, which is injected (i.e. passed as a parameter) into the module when the module is created. If you want to know more about DI, here are some recommendations Dependency injection and Inversion of Control Containers and the Dependency Injection pattern These two articles.

If DI is not used, there are two main inconveniences. One is that the modification of the underlying class needs to modify the upper class. In the process of large-scale software development, there are many base classes, and dozens of files are often modified after a link is changed; On the other hand, unit testing between layers is not convenient.

Because dependency injection is adopted, it is inevitable to write a lot of new in the initialization process. For example, in our project:

package main

import (
	"my-clean-rchitecture/api"
	"my-clean-rchitecture/api/handlers"
	"my-clean-rchitecture/app"
	"my-clean-rchitecture/repo"
	"my-clean-rchitecture/service"
)

func main() { 
	// Initialize db
	db := app.InitDB() 
	//Initialize repo
	repository := repo.NewMysqlArticleRepository(db)
	//Initialize service
	articleService := service.NewArticleService(repository)
	//Initialization api
	handler := handlers.NewArticleHandler(articleService)
	//Initialize router
	router := api.NewRouter(handler)
	//Initialize gin
	engine := app.NewGinEngine()
	//Initialize server
	server := app.NewServer(engine, router)
	//start-up
	server.Start()
}

So for such a piece of code, is there any way we don't have to write it ourselves? Here we can use the power of the framework to generate our injection code.

In go, DI tools are not as convenient as java. Generally, the technical framework mainly includes: wire, dig, fx, etc. Because wire uses code generation for injection, its performance will be relatively high, and it is a DI framework launched by google, so we use wire for injection here.

The requirements for wire are very simple. Create a new wire Go file (the file name can be arbitrary) and create our initialization function. For example, if we want to create and initialize a server object, we can:

//+build wireinject

package main

import (
	"github.com/google/wire"
	"my-clean-rchitecture/api"
	"my-clean-rchitecture/api/handlers"
	"my-clean-rchitecture/app"
	"my-clean-rchitecture/repo"
	"my-clean-rchitecture/service"
)

func InitServer() *app.Server {
	wire.Build(
		app.InitDB,
		repo.NewMysqlArticleRepository,
		service.NewArticleService,
		handlers.NewArticleHandler,
		api.NewRouter,
		app.NewServer,
		app.NewGinEngine)
	return &app.Server{}
}

It should be noted that the annotation in the first line: + build wireinject indicates that this is an injector.

In the function, we call wire Build() passes in the constructor of the type on which the Server depends. Finish wire Execute the wire command after the go file, and a wire will be generated automatically_ Gen.go file.

// Code generated by Wire. DO NOT EDIT.

//go:generate go run github.com/google/wire/cmd/wire
//+build !wireinject

package main

import (
	"my-clean-rchitecture/api"
	"my-clean-rchitecture/api/handlers"
	"my-clean-rchitecture/app"
	"my-clean-rchitecture/repo"
	"my-clean-rchitecture/service"
)

// Injectors from wire.go:

func InitServer() *app.Server {
	engine := app.NewGinEngine()
	db := app.InitDB()
	iArticleRepo := repo.NewMysqlArticleRepository(db)
	iArticleService := service.NewArticleService(iArticleRepo)
	articleHandler := handlers.NewArticleHandler(iArticleService)
	router := api.NewRouter(articleHandler)
	server := app.NewServer(engine, router)
	return server
}

You can see that wire automatically generates the InitServer method for us, which initializes all the base classes to be initialized in turn. Then, in our main function, we just need to call this InitServer.

func main() {
	server := InitServer()
	server.Start()
}

test

We have defined what each layer should do, so we should be able to test each layer separately, even if the other layer does not exist.

models layer: this layer is very simple. Since it does not rely on any other code, it can be tested directly with the single test framework of go;
repo layer: for this layer, since we use MySQL database, we need to mock mysql so that we can test normally even without mysql. I use GitHub COM / data-dog / go sqlmock this library to mock our database;
Service layer: because the service layer depends on the repo layer, and because they are associated through interfaces, I use GitHub COM / golang / mock / go mock repo layer;
api layer: this layer depends on the service layer, and they are associated through interfaces, so gomock can also be used to mock the service layer here. However, it's a little troublesome here. Because our access layer uses gin, we also need to simulate sending requests during single test;

Because we are through GitHub COM / golang / mock / gomock, so you need to perform the following code generation. The generated mock code is put into the mock package:

mockgen -destination .\mock\repo_mock.go -source .\repo\repo.go -package mock

mockgen -destination .\mock\service_mock.go -source .\service\service.go -package mock

The above two commands will help me automatically generate mock functions through the interface.

repo layer test

In the project, because we use gorm as our orm library, we need to use GitHub COM / data-dog / go sql mock with gorm:

func getSqlMock() (mock sqlmock.Sqlmock, gormDB *gorm.DB) {
	//Create sqlmock
	var err error
	var db *sql.DB
	db, mock, err = sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherEqual))
	if err != nil {
		panic(err)
	}
	//Combining gorm and sqlmock
	gormDB, err = gorm.Open(mysql.New(mysql.Config{
		SkipInitializeWithVersion: true,
		Conn:                      db,
	}), &gorm.Config{})
	if nil != err {
		log.Fatalf("Init DB with sqlmock failed, err %v", err)
	}
	return
}

func Test_mysqlArticleRepository_Fetch(t *testing.T) {
	createAt := time.Now()
	updateAt := time.Now()
	//id,title,content, updated_at, created_at
	var articles = []models.Article{
		{1, "test1", "content", updateAt, createAt},
		{2, "test2", "content2", updateAt, createAt},
	}

	limit := 2
	mock, db := getSqlMock()

	mock.ExpectQuery("SELECT id,title,content, updated_at, created_at FROM `articles` WHERE created_at > ? LIMIT 2").
		WithArgs(createAt).
		WillReturnRows(sqlmock.NewRows([]string{"id", "title", "content", "updated_at", "created_at"}).
			AddRow(articles[0].ID, articles[0].Title, articles[0].Content, articles[0].UpdatedAt, articles[0].CreatedAt).
			AddRow(articles[1].ID, articles[1].Title, articles[1].Content, articles[1].UpdatedAt, articles[1].CreatedAt))

	repository := NewMysqlArticleRepository(db)
	result, err := repository.Fetch(context.TODO(), createAt, limit)

	assert.Nil(t, err)
	assert.Equal(t, articles, result)
}

service layer test

Here we mainly use the code generated by gomock to mock the repo layer:

func Test_articleService_Fetch(t *testing.T) {
	ctl := gomock.NewController(t)
	defer ctl.Finish()
	now := time.Now()
	mockRepo := mock.NewMockIArticleRepo(ctl)

	gomock.InOrder(
		mockRepo.EXPECT().Fetch(context.TODO(), now, 10).Return(nil, nil),
	)
	
	service := NewArticleService(mockRepo)

	fetch, _ := service.Fetch(context.TODO(), now, 10)
	fmt.Println(fetch)
}

api layer testing

For this layer, we not only need to mock the service layer, but also need to send httptest to simulate request sending:

func TestArticleHandler_FetchArticle(t *testing.T) {

	ctl := gomock.NewController(t)
	defer ctl.Finish()
	createAt, _ := time.Parse("2006-01-02", "2021-12-26")
	mockService := mock.NewMockIArticleService(ctl)

	gomock.InOrder(
		mockService.EXPECT().Fetch(gomock.Any(), createAt, 10).Return(nil, nil),
	)

	article := NewArticleHandler(mockService)

	gin.SetMode(gin.TestMode)

	// Setup your router, just like you did in your main function, and
	// register your routes
	r := gin.Default()
	r.GET("/articles", article.FetchArticle)

	req, err := http.NewRequest(http.MethodGet, "/articles?num=10&create_date=2021-12-26", nil)
	if err != nil {
		t.Fatalf("Couldn't create request: %v\n", err)
	}

	w := httptest.NewRecorder()
	// Perform the request
	r.ServeHTTP(w, req)

	// Check to see if the response was what you expected
	if w.Code != http.StatusOK {
		t.Fatalf("Expected to get status %d but instead got %d\n", http.StatusOK, w.Code)
	}
}

summary

The above is my summary and Reflection on the problems found in the golang project. Whether it is right or not, it has solved some of our current problems. However, the project always needs continuous reconstruction and improvement, so we can change it next time there is a problem.

I feel there is something wrong with my above summary and description. Please point it out and discuss it at any time.

Project code location: https://github.com/devYun/go-clean-architecture