Talent Plan KV learning camp

About Talent Plan KV training camp, I think it is necessary for me to introduce it. This is a set of open source distributed KV storage practical courses launched by PingCAP company. This course includes four sub projects:

1. Project 1 requires participants to complete a stand-alone KV Server independently
2. Project 2 needs to implement distributed key value database server based on Raft algorithm
3. Project 3 needs to support multiple Raft clusters based on Project 2
4. Project 4 needs to support distributed transactions based on Project 3

The difficulty is stepped. I believe it will be a great gain if all the sub projects of tinykv can be realized and a distributed kv database can be produced. As far as I know, MIT has a set of MIT 6.824 courses, and Ping Cap's launch of this set of tinykv courses is equivalent to making up for the gap in this domestic course.

The project I am currently participating in requires Go language to implement these labs. Maybe when you see this article, you may change to other languages, such as Rust. Of course, it is not important to use what programming language to implement it. The important thing is in the reading part of the paper, which is like you understand the design drawings of a building, Then language is a pile of bricks 🧱 Tools.

• Thesis address: https://github.com/kvbase/raft-thesis-zh_cn

As the organizer requires that the participants should not directly post the answer code of each project in any way, of course, I think it is right to do so, but the organizer encourages the participants to write some ideas about how to solve the problem of lab and share them. In the evening, take time to complete the first lab. this article will write how to get into the pit of Project 1.

Standalone KV Server

The first lab is a typical example of TDD development. If readers don't know what TDD is, go to Google. Project 1 is to implement the API interface reserved by the author based on BadgerDB as the storage engine, and then the official creates a makefile file in the project root directory. Participants only need to make project1 to view the completion of project 1.

BadgerDB is a key value local database based on Log Structured Merge (LSM) Tree developed by dgraph.io and developed by Go.

You need to have a Go mod foundation to develop this project. You don't have to make it up yourself.

After executing make project1, you can see that a lot of exceptions are thrown. The reason for these exceptions is that the unit test written by the official engineer fails to pass. All you have to do is put / tinykv / kV / server / server_ All unit test cases under test.go can implement the functions in the api called.

1. The first thing you want to achieve

package standalone_storage

import (
    "github.com/pingcap-incubator/tinykv/kv/config"
    "github.com/pingcap-incubator/tinykv/kv/storage"
    "github.com/pingcap-incubator/tinykv/proto/pkg/kvrpcpb"
)

// StandAloneStorage is an implementation of `Storage` for a single-node TinyKV instance. It does not
// communicate with other nodes and all data is stored locally.
type StandAloneStorage struct {
    // Your Data Here (1).
}

func NewStandAloneStorage(conf *config.Config) *StandAloneStorage {
    // Your Code Here (1).
    return nil
}

func (s *StandAloneStorage) Start() error {
    // Your Code Here (1).
    return nil
}

func (s *StandAloneStorage) Stop() error {
    // Your Code Here (1).
    return nil
}

func (s *StandAloneStorage) Reader(ctx *kvrpcpb.Context) (storage.StorageReader, error) {
    // Your Code Here (1).
    return nil, nil
}

func (s *StandAloneStorage) Write(ctx *kvrpcpb.Context, batch []storage.Modify) error {
    // Your Code Here (1).
    return nil
}

1. The second thing you want to achieve

package server

import (
    "context"
    "github.com/pingcap-incubator/tinykv/proto/pkg/kvrpcpb"
)

// The functions below are Server's Raw API. (implements TinyKvServer).
// Some helper methods can be found in sever.go in the current directory

// RawGet return the corresponding Get response based on RawGetRequest's CF and Key fields
func (server *Server) RawGet(_ context.Context, req *kvrpcpb.RawGetRequest) (*kvrpcpb.RawGetResponse, error) {
    // Your Code Here (1).
    return nil, nil
}

// RawPut puts the target data into storage and returns the corresponding response
func (server *Server) RawPut(_ context.Context, req *kvrpcpb.RawPutRequest) (*kvrpcpb.RawPutResponse, error) {
    // Your Code Here (1).
    // Hint: Consider using Storage.Modify to store data to be modified
    return nil, nil
}

// RawDelete delete the target data from storage and returns the corresponding response
func (server *Server) RawDelete(_ context.Context, req *kvrpcpb.RawDeleteRequest) (*kvrpcpb.RawDeleteResponse, error) {
    // Your Code Here (1).
    // Hint: Consider using Storage.Modify to store data to be deleted
    return nil, nil
}

// RawScan scan the data starting from the start key up to limit. and return the corresponding result
func (server *Server) RawScan(_ context.Context, req *kvrpcpb.RawScanRequest) (*kvrpcpb.RawScanResponse, error) {
    // Your Code Here (1).
    // Hint: Consider using reader.IterCF
    return nil, nil
}

The corresponding unit test file is tinykv/kv/server/server_test.go, the official engineer of the test case has written it. You only need to implement the underlying code and run through the unit test to complete Project 1. Here, I remind you that Project 1 can be regarded as an adapter mode, but with one layer of encapsulation. When writing the implemented or testing unit tests, you can test them one by one. First, comment out all the unit tests, Then implement a solution, annotate one, run one, good luck!.