You have to work very hard to look effortless!
WeChat search official account [Coding road], together with From Zero To Hero!
preface
The first two articles Go language io package core interface details,Go language io package basic interface details , we have learned the core interfaces, basic interfaces and composite interfaces in the io package, which are basic interface definitions and specifications. In this article, let's take a look at the use of the above interfaces in the io package, including three structures and some methods. We deepen our understanding of interface definitions by studying the source code.
structural morphology
LimitedReader
LimitedReader limits the length of data to be read, and reads n bytes at most. The specific definitions are as follows:
// Reader R, as the underlying reader, is used to read data;
// n record how many remaining bytes can be read (initialized to n, the number of remaining readable bytes will be updated after each data reading)
type LimitedReader struct {
R Reader // underlying reader
N int64 // max bytes remaining
}
correlation method
- Return LimitedReader instance
// Returns the LimitReader instance. r is the underlying Reader, and n is the number and size of bytes to be read
func LimitReader(r Reader, n int64) Reader { return &LimitedReader{r, n} }
- Read method
// The Read method reads the data into the byte array p, and returns the Read byte length and the generated error
// The number of remaining readable bytes N is updated each time the method is called
func (l *LimitedReader) Read(p []byte) (n int, err error) {
// If the number of remaining readable bytes n < = 0, EOF error is returned
if l.N <= 0 {
return 0, EOF
}
// If the provided byte array space is too large, you only need to use n lengths, because it limits the maximum reading of N bytes
if int64(len(p)) > l.N {
p = p[0:l.N]
}
// Read data to byte data p and subtract N from the number of bytes successfully read
n, err = l.R.Read(p)
l.N -= int64(n)
return
}
SectionReader
SectionReader wraps the ReaderAt type and overrides the Read, Seek, and ReadAt methods. The function of SectionReader is to limit the range of reading data. It can only Read a part (or a paragraph) of the original data. It is defined as follows:
// r: ReaderAt instance for reading data
// base: save the starting position of the readable data range, and the variable value will not change
// off: save the current location. The variable value will change every time the data is read
// limit: save the end position of the readable data range, and the variable value will not change
type SectionReader struct {
r ReaderAt
base int64
off int64
limit int64
}
correlation method
- Initializing the SectionReader instance
// The NewSectionReader returns the initialized SectionReader. It needs to provide a ReaderAt instance, as well as the start position off and the data length n to be read, that is, it limits the data range to be read to [off,off+n)
func NewSectionReader(r ReaderAt, off int64, n int64) *SectionReader {
return &SectionReader{r, off, off, off + n}
}
- Read method: reads data within the specified range
// Read the data into the byte array p, and return the read byte length and the generated error
func (s *SectionReader) Read(p []byte) (n int, err error) {
// If the current position exceeds the readable data range, EOF error is returned
if s.off >= s.limit {
return 0, EOF
}
// If the length of the byte array p is greater than the length of the remaining readable data, reduce the length of p to the length of the remaining readable data
if max := s.limit - s.off; int64(len(p)) > max {
p = p[0:max]
}
// Read the data from the off position and update the off position
n, err = s.r.ReadAt(p, s.off)
s.off += int64(n)
return
}
- Seek method: set the value of the SectionReader off variable according to the provided where and offset
var errWhence = errors.New("Seek: invalid whence")
var errOffset = errors.New("Seek: invalid offset")
// The Seek method sets the value of the SectionReader off variable according to where and offset, and returns the length from the start of the readable range and the generated error
func (s *SectionReader) Seek(offset int64, whence int) (int64, error) {
switch whence {
default:
return 0, errWhence
// Start seek from the starting position, that is, based on sectionreader base
case SeekStart:
offset += s.base
// Start seek from the current location, i.e. based on sectionreader off
case SeekCurrent:
offset += s.off
// Start seek from the current location, i.e. based on sectionreader limit
case SeekEnd:
offset += s.limit
}
// If the position of the final offset is before the base, it is an illegal position and error will be returned;
// If no error is returned after limit, you can refer to the definition of Seek method in Seeker interface
if offset < s.base {
return 0, errOffset
}
// Modify the value of off and return the length from base
s.off = offset
return offset - s.base, nil
}
- ReadAt method: Based on the starting position base, calculate the read position offset as base+off according to the input parameter off, and then start from this position to read the data within the readable data range into the byte array p. (independent of the off variable defined in the SectionReader structure)
func (s *SectionReader) ReadAt(p []byte, off int64) (n int, err error) {
// If the input parameter off < 0 or the off size exceeds the readable data range, EOF error is returned
if off < 0 || off >= s.limit-s.base {
return 0, EOF
}
// The starting position of this reading is off + s.base
off += s.base
// If the length of the byte array p is greater than the readable length s.limit-off, the byte array is reduced to the readable length
if max := s.limit - off; int64(len(p)) > max {
p = p[0:max]
n, err = s.r.ReadAt(p, off)
// Since the length of the read data is less than the length len(p) of the original byte array p, refer to the definition of the ReadAt method in the ReaderAt interface and return error
if err == nil {
err = EOF
}
return n, err
}
// If the length of the byte array is less than or equal to the readable length s.limit-off, read the data into p
return s.r.ReadAt(p, off)
}
-
Size method: returns the range size of readable data
// Size returns the length of the readable range func (s *SectionReader) Size() int64 { return s.limit - s.base }
teeReader
teeReader is a packet level private data type because its first letter is lowercase. The function of teeReader is to give the data read by the Reader to the Writer to write to the file, which is equivalent to providing a bridge to connect a Reader and a Writer. teeReader will give all the data read in the Reader to the Writer at one time without buffer. It is defined as follows:
type teeReader struct {
r Reader
w Writer
}
correlation method
- Initialization method: provide a Reader and Writer for initialization
func TeeReader(r Reader, w Writer) Reader {
return &teeReader{r, w}
}
- Read method: the reader reads the data into the byte array p, and then the Writer writes the data in p to the file. p acts as a transit station for the data.
func (t *teeReader) Read(p []byte) (n int, err error) {
// Read the data into the byte array p, and return the read data length and error
n, err = t.r.Read(p)
// According to the definition of the Read method in the Reader interface, the caller should first focus on whether n is greater than 0, rather than err
// If n > 0, it means that the data has been read, and the data part p[:n] read in p is handed over to Writer w to write
if n > 0 {
if n, err := t.w.Write(p[:n]); err != nil {
return n, err
}
}
return
}
Method definition
WriteString
WriteString is used to write a string to a file
// Input parameter: Writer w, used to write string to file; s: String to write
// Return value: the number of bytes successfully written, n, and the possible error
func WriteString(w Writer, s string) (n int, err error) {
// If the incoming writer implements the StringWriter interface, directly call the WriteString method of the StringWriter interface
// Otherwise, the Writer's Write method is called
if sw, ok := w.(StringWriter); ok {
return sw.WriteString(s)
}
return w.Write([]byte(s))
}
ReadAtLeast
ReadAtLeast: read at least min bytes, that is, use Reader r to read at least min bytes into byte array buf
Input parameter
- Reader r: used to read data
- buf []byte: save the read data
- min int: read at least min bytes
Return value
- n: Number of bytes successfully read in
- err: generated error
func ReadAtLeast(r Reader, buf []byte, min int) (n int, err error) {
// If the length of the byte array buff provided by the method is less than min, it is impossible to meet the definition of the method and directly return ErrShortBuffer error
if len(buf) < min {
return 0, ErrShortBuffer
}
// N record the number of bytes currently read, and cycle until the read data bytes n > = min
for n < min && err == nil {
var nn int
nn, err = r.Read(buf[n:])
n += nn
}
// If the read data n > = min, nil is returned regardless of whether an error is generated, because at least min bytes have been read
if n >= min {
err = nil
} else if n > 0 && err == EOF {
//If the read data is less than min bytes, but EOF is generated at the end of the file, the method returns errunexpected EOF error
err = ErrUnexpectedEOF
}
// Other cases: 0 < = n < min, directly return the current err
return
}
Several error s
-
If the length of the byte array buff provided by the method is less than min, it is impossible to meet the definition of the method and directly return ErrShortBuffer error.
-
If the read data is greater than 0 and less than min bytes, but EOF is generated at the end of the file, the method returns errunexpected EOF error
-
If the read data is greater than or equal to min bytes, but an error occurs, because the method has met the need to read at least min bytes, this error will be discarded and nil will be returned
-
Other case s: the number of bytes read is 0, or the data read is less than min bytes, but other non EOF error s are generated. The method returns the corresponding error
ReadFull
ReadFull: use Reader r to read data and fill the byte array buf
Input parameter
- Reader r
- Byte array buf
Return value
- n: The number of bytes successfully read and written to the byte array buf
- err: generated error
func ReadFull(r Reader, buf []byte) (n int, err error) {
// This method directly calls ReadAtLeast(r, buf, len(buf)), which is equivalent to reading the buf byte array full.
return ReadAtLeast(r, buf, len(buf))
}
Referring to the definition of ReadAtLeast, the following conclusions can be drawn:
- If the returned err = EOF, no data must be read, because if the read data n > 0, errunexpected EOF will be returned after encountering EOF
- If n=len(buf) is returned, err must be nil
copyBuffer
copyBuffer is a private method that uses buffer to copy data from Reader to Writer. Method uses a byte array as a buffer. The Reader reads data into the byte array each time, and then the Writer writes the data in the byte array to the file until the Reader finishes reading data or encounters error. Finally, the number of bytes copied and the error generated during copying are returned. If the buffer array buf =nil is passed when this method is called, a temporary byte array will be allocated.
func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
// If the Reader src implements the WriteTo interface, you can call the WriteTo method to complete the copy
if wt, ok := src.(WriterTo); ok {
return wt.WriteTo(dst)
}
// If the Writer implements the ReaderFrom interface, you can call the ReadFrom method to complete the copy
if rt, ok := dst.(ReaderFrom); ok {
return rt.ReadFrom(src)
}
// If the buffer passed by the method is nil, a buffer is allocated
if buf == nil {
// The default is 32 kb
size := 32 * 1024
// If src is a LimitedReader, the buffer size should adapt to the LimitedReader
if l, ok := src.(*LimitedReader); ok && int64(size) > l.N {
// At least 1
if l.N < 1 {
size = 1
} else {
// The maximum size of the LimitedReader cannot be exceeded
size = int(l.N)
}
}
buf = make([]byte, size)
}
// Keep cycling, read data from Reader src to buf, and Writer dst writes the read data to buf
for {
nr, er := src.Read(buf)
// According to the definition of the Read method in the Reader interface, process the data first and then error
// If NR > 0, the dst writes the read data buf[0:nr]
if nr > 0 {
nw, ew := dst.Write(buf[0:nr])
// Write saves the total copied byte size
if nw > 0 {
written += int64(nw)
}
// err encountered during writing, end replication
if ew != nil {
err = ew
break
}
// The number of bytes written is inconsistent with the number of bytes read. End the copy and return ErrShortWrite
if nr != nw {
err = ErrShortWrite
break
}
}
// If Er is encountered during reading= Nil, if ER= EOF, and finally return the error;
// If er = EOF and the logic goes here, it means that the above write does not generate error, then the whole file is copied, the err field is not assigned, and the final returned err = nil
if er != nil {
if er != EOF {
err = er
}
break
}
}
// Returns the total number of bytes copied and the resulting error
return written, err
}
Copy
The function of copy is to copy the data of Reader src to Writer dst, and then return the copied bytes and the encountered error. Copy directly calls the copyBuffer method, but does not provide a buffer array. A temporary array will be used inside the copyBuffer method.
According to the definition of copyBuffer above, if the replication is completed successfully, err=nil in the return value instead of err=EOF, because if EOF is encountered, it indicates that the replication has been completed, and err should be nil. See the analysis of the copyBuffer source code above for the specific logic
func Copy(dst Writer, src Reader) (written int64, err error) {
return copyBuffer(dst, src, nil)
}
CopyBuffer
The difference between copyBuffer and Copy method is that copyBuffer can pass in a byte array as a global buffer; The Copy method does not provide a buffer, but uses the temporary buffer of copyBuffer
func CopyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
// If the incoming byte buffer is not nil, but the length is 0, it is panic directly
if buf != nil && len(buf) == 0 {
panic("empty buffer in io.CopyBuffer")
}
// If the passed in byte array buffer = nil, a temporary buffer is generated using copyBuffer
// If the incoming byte array buf= If nil, len (buf) > 0, buf is used as the global buffer
return copyBuffer(dst, src, buf)
}
CopyN
The function of CopyN is similar to that of Copy. Both are used to Copy data from the Reader to the Writer. However, CopyN limits the number of bytes copied to n. The method returns the length written and the generated error.
func CopyN(dst Writer, src Reader, n int64) (written int64, err error) {
// The Copy method is called to Copy data, but the Reader src is encapsulated into a LimitReader that can read up to n bytes,
// The LimitReader can read up to n bytes, so the copy can complete up to n bytes
written, err = Copy(dst, LimitReader(src, n))
// written == n, indicating successful replication, err=nil
if written == n {
return n, nil
}
// err == nil, indicating that the Copy method is called above to complete the Copy;
// Written < n indicates that LimitReader encountered EOF before reading n bytes. The method did not complete the task of copying n bytes and returned err = EOF
if written < n && err == nil {
// src stopped early; must have been EOF.
err = EOF
}
// For other errors, just return the error directly
return
}
summary
This article analyzes the source code of the three structures and methods in the io package. The main contents are as follows:
-
structural morphology
- LimitedReader: limits the number of bytes read
- SectionReader: limits the range of reads
- teeReader: acts as a bridge between Reader and Writer to complete data transfer
-
method
- WriteString: write string
- ReadAtLeast: read min bytes at least
- ReadFull: fill the passed in byte array
- copyBuffer: use buffer array to copy data from Reader to Writer
- Copy: cannot provide global buffer array. Use copyBuffer to complete the copy
- copyBuffer: you can provide a global buffer array and use copyBuffer to complete replication
- CopyN: copy n bytes with LimitedReader
See here, do you like me, amazed at the simplicity and subtlety of go language interface design? Let's keep learning together!
more
Personal blog: https://lifelmy.github.io/
WeChat official account: long Coding Road