outline
When a third-party API is called, there are basically restrictions on the access speed limit. When there are multiple third-party APIs, it is not good to control the access speed, which often leads to HTTP 429(Too Many Requests) and then it will be banned for a period of time
In order to cope with this speed limit, a simple request queue is used to control the access speed. After that, HTTP 429 has not been encountered
Realization ideas
First, each request is wrapped into a struct of RequestParam, which contains the address, type, parameter and channel of response
When sending a request, just put the RequestParam in the request queue. After the request is completed, put the response in the corresponding channel
The whole code implementation is simple:
1 package util
2
3 import (
4 "fmt"
5
6 apiclient "gitee.com/wangyubin/gutils/api_client"
7 "gitee.com/wangyubin/gutils/logger"
8 )
9
10 // What is included in the request
11 type RequestParam struct {
12 Api string
13 Method string
14 JsonReq interface{}
15 Resp chan []byte
16 }
17
18 // Request queue is essentially a channel
19 type RequestQueue struct {
20 Queue chan RequestParam
21 }
22
23 var queue *RequestQueue
24
25 // Get queue
26 func GetQueue() *RequestQueue {
27 return queue
28 }
29
30 // Initialize queue
31 func InitRequestQueue(size int) {
32 queue = &RequestQueue{
33 Queue: make(chan RequestParam, size),
34 }
35 }
36
37 // Put the request on the queue
38 func (rq *RequestQueue) Enqueue(p RequestParam) {
39 rq.Queue <- p
40 }
41
42 // Request queue service, waiting to accept and process requests
43 func (rq *RequestQueue) Run() {
44 lg := logger.GetLogger()
45 for p := range rq.Queue {
46 var resp []byte
47 var err error
48 switch p.Method {
49 case "GET":
50 resp, err = apiclient.GetJson(p.Api, p.JsonReq)
51 case "POST":
52 resp, err = apiclient.PostJson(p.Api, p.JsonReq)
53 default:
54 err = fmt.Errorf("Wrong type of METHOD(%s)\n", p.Method)
55 }
56
57 if err != nil {
58 lg.Err(err).Msg("access api error: " + p.Api)
59 continue
60 }
61 if p.Resp != nil {
62 p.Resp <- resp
63 close(p.Resp)
64 }
65 }
66
67 lg.Info().Msg("request queue finished!")
68 }
The request here is an apiclient encapsulated by myself, which can be replaced according to the actual situation
In my application scenario, as long as the api is accessed in sequence, HTTP 429 will not appear. If this is too fast, you can try to add some time intervals to the Run() function
1 func (rq *RequestQueue) Run() {
2 lg := logger.GetLogger()
3 for p := range rq.Queue {
4 time.Sleep(1 * time.Second)
5 // ... omitted code
6 }
7
8 lg.Info().Msg("request queue finished!")
9 }
usage method
It's easy to use. Start it first, and then put RequestParam in the queue at each call place and wait for response
Start queue service
1 func main() {
2 // init request queue and start queue service
3 util.InitRequestQueue(100)
4 queue := util.GetQueue()
5 defer close(queue.Queue)
6 go queue.Run()
7
8 // Other startup codes
9 }
Use queue service
1 func Request(param1 string, param2 int) error {
2 api := "http://xxxx.com"
3 api = fmt.Sprintf("%s?period=%s&size=%d", api, param1, param2)
4
5 queue := util.GetQueue()
6 param := util.RequestParam{
7 Api: api,
8 Method: "GET",
9 Resp: make(chan []byte, 1),
10 }
11 queue.Enqueue(param)
12
13 var respData struct {
14 Status string `json:"status"`
15 Data []model.Data `json:"data"`
16 }
17 var err error
18 for resp := range param.Resp {
19 err = json.Unmarshal(resp, &respData)
20 if err != nil {
21 lg.Err(err).Msg("unmarshal json error")
22 return err
23 }
24 }
25
26 fmt.Println(respData)
27 return err
28 }