golang API request queue

Posted by SauloA on Sun, 31 May 2020 10:17:47 +0200

outline

When a third-party API is called, there are basically restrictions on the access speed limit. When there are multiple third-party APIs, it is not good to control the access speed, which often leads to HTTP 429(Too Many Requests) and then it will be banned for a period of time

In order to cope with this speed limit, a simple request queue is used to control the access speed. After that, HTTP 429 has not been encountered

Realization ideas

First, each request is wrapped into a struct of RequestParam, which contains the address, type, parameter and channel of response

When sending a request, just put the RequestParam in the request queue. After the request is completed, put the response in the corresponding channel

The whole code implementation is simple:

 1  package util
 2  
 3  import (
 4    "fmt"
 5  
 6    apiclient "gitee.com/wangyubin/gutils/api_client"
 7    "gitee.com/wangyubin/gutils/logger"
 8  )
 9  
10  // What is included in the request
11  type RequestParam struct {
12    Api     string
13    Method  string
14    JsonReq interface{}
15    Resp    chan []byte
16  }
17  
18  // Request queue is essentially a channel
19  type RequestQueue struct {
20    Queue chan RequestParam
21  }
22  
23  var queue *RequestQueue
24  
25  // Get queue
26  func GetQueue() *RequestQueue {
27    return queue
28  }
29  
30  // Initialize queue
31  func InitRequestQueue(size int) {
32    queue = &RequestQueue{
33      Queue: make(chan RequestParam, size),
34    }
35  }
36  
37  // Put the request on the queue
38  func (rq *RequestQueue) Enqueue(p RequestParam) {
39    rq.Queue <- p
40  }
41  
42  // Request queue service, waiting to accept and process requests
43  func (rq *RequestQueue) Run() {
44    lg := logger.GetLogger()
45    for p := range rq.Queue {
46      var resp []byte
47      var err error
48      switch p.Method {
49      case "GET":
50        resp, err = apiclient.GetJson(p.Api, p.JsonReq)
51      case "POST":
52        resp, err = apiclient.PostJson(p.Api, p.JsonReq)
53      default:
54        err = fmt.Errorf("Wrong type of METHOD(%s)\n", p.Method)
55      }
56  
57      if err != nil {
58        lg.Err(err).Msg("access api error: " + p.Api)
59        continue
60      }
61      if p.Resp != nil {
62        p.Resp <- resp
63        close(p.Resp)
64      }
65    }
66  
67    lg.Info().Msg("request queue finished!")
68  }

The request here is an apiclient encapsulated by myself, which can be replaced according to the actual situation

In my application scenario, as long as the api is accessed in sequence, HTTP 429 will not appear. If this is too fast, you can try to add some time intervals to the Run() function

1  func (rq *RequestQueue) Run() {
2    lg := logger.GetLogger()
3    for p := range rq.Queue {
4       time.Sleep(1 * time.Second)
5       // ... omitted code
6    }
7  
8    lg.Info().Msg("request queue finished!")
9  }

usage method

It's easy to use. Start it first, and then put RequestParam in the queue at each call place and wait for response

Start queue service

1  func main() {
2      // init request queue and start queue service
3      util.InitRequestQueue(100)
4      queue := util.GetQueue()
5      defer close(queue.Queue)
6      go queue.Run()
7  
8      // Other startup codes
9  }

Use queue service

 1  func Request(param1 string, param2 int) error {
 2   api := "http://xxxx.com"
 3   api = fmt.Sprintf("%s?period=%s&size=%d", api, param1, param2)
 4  
 5   queue := util.GetQueue()
 6   param := util.RequestParam{
 7     Api:    api,
 8     Method: "GET",
 9     Resp:   make(chan []byte, 1),
10   }
11   queue.Enqueue(param)
12  
13   var respData struct {
14     Status string       `json:"status"`
15     Data   []model.Data `json:"data"`
16   }
17   var err error
18   for resp := range param.Resp {
19     err = json.Unmarshal(resp, &respData)
20     if err != nil {
21       lg.Err(err).Msg("unmarshal json error")
22       return err
23     }
24   }
25  
26   fmt.Println(respData) 
27   return  err
28  }

Topics: Go JSON