1, Detailed explanation of foundation
1. Traditional IO(BIO)
Traditional IO is stream oriented and synchronous blocking io. Each socket request needs to be processed by a corresponding thread. If there are not enough threads to process, the request is either waiting or rejected. That is, each connection requires the service to be processed by a corresponding thread. This is why traditional IO is inefficient.
2. NIO(NIO 1.0 or New IO or Non Blocking IO)
Because traditional IO is inefficient, it is used in jdk1 4 and later versions provide a set of APIs to specifically operate non blocking I/O, which can replace the standard Java IO API.
Note: the traditional I/O package has been re implemented with NIO. Even if we use traditional IO, it will be more efficient than the original one.
NIO uses the event driven idea and is based on Reactor (explained in this article). When the socket has a stream readable or writable socket, the operating system will notify the reference program to process it accordingly, and then the application will read the stream to the buffer or write it to the operating system. (instead of each request being processed by one thread, one thread constantly polls the status of each IO operation).
NIO supports buffer oriented, channel based IO operations. NIO will read and write files in a more efficient way. It consists of three main parts: Buffers, Channels and selectors. In the follow-up, we will explain one by one.
An important sentence: the first biggest difference between NIO and traditional IO is that IO is stream oriented and NIO is buffer oriented.
3,AIO(NIO 2.0)
Asynchronous non blocking IO, that is, there is no need for a thread like NIO to poll for the state change of all IO operations. After the corresponding state change, the system will notify the corresponding thread to handle it.
At jd.k1 7, Java nio. Four asynchronous channels are added under the channels package:
- AsynchronousSocketChannel
- AsynchronousServerSocketChannel
- AsynchronousFileChannel
- AsynchronousDatagramChannel
The read/write method will return an object with a callback function. When the read/write operation is completed, the callback function will be called directly.
AIO is not the focus of this paper, just a brief introduction.
2, Detailed explanation of the three components of NIO
1. Buffer
Buffer: a container for a specific basic data type. By Java As defined in the NiO package, all buffers (ByteBuffe r, CharBuffer, ShortBuffer, IntBuffer, LongBuffer, FloatBuffer and DoubleBuffer) are subclasses of the buffer abstract class.
In NIO library, all data is processed by buffer. When reading data, it is directly read into the buffer; When writing data, it is also written to the buffer. Any time you access the data in NIO, you operate through the buffer.
Four core attributes in the buffer:
- Capacity: indicates the maximum capacity of data in the buffer. Once declared, it cannot be changed.
- Limit: limit, indicating the size of data that can be manipulated in the buffer. (data cannot be read or written after limit)
- Position: position, indicating the position in the buffer where data is being manipulated.
- Mark: mark, indicating the current position of the record. You can reset() to the location of mark.
Note: 0 < = mark < = position < = limit < = capacity
The buffer is actually an array and provides structured access to data and maintenance of read / write locations. Please refer to the following figure for four core attributes.
There are two core methods for buffer access to data:
- put(): store data into buffer
put(byte b): writes a given order byte to the current position of the buffer
put(byte[] src): writes the bytes in src to the current position of the buffer
put(int index, byte b): writes the specified byte to the index position of the buffer (position will not be moved) - get(): get the data in the cache
get(): read a single byte
get(byte[] dst): batch read multiple bytes into dst
get(int index): read the bytes of the specified index position (position will not be moved)
Just know the common methods:
The code for operating the buffer is as follows. You can have a simple look and understand it easily. We won't operate these codes in person when using NIO in the future.
@Test
public void test1(){
String str = "abcd";
//1. Allocate a buffer of the specified size
ByteBuffer buf = ByteBuffer.allocate(1024);
System.out.println("-----------------allocate() establish----------------");
System.out.println("position = "+buf.position());
System.out.println("limit= "+buf.limit());
System.out.println("capacity = "+buf.capacity());
//2. Use put() to store data into buffer
buf.put(str.getBytes());
System.out.println("-----------------put() storage----------------");
System.out.println("position = "+buf.position());
System.out.println("limit = "+buf.limit());
System.out.println("capacity = "+buf.capacity());
//3. Switch data reading mode
buf.flip();
System.out.println("-----------------flip()----------------");
System.out.println("position = "+buf.position());
System.out.println("limit = "+buf.limit());
System.out.println("capacity = "+buf.capacity());
//4. Use get() to read the data in the buffer
byte[] dst = new byte[buf.limit()];
buf.get(dst);
System.out.println(new String(dst, 0, dst.length));
System.out.println("-----------------get() read----------------");
System.out.println("position = "+buf.position());
System.out.println("limit = "+buf.limit());
System.out.println("capacity = "+buf.capacity());
//5. rewind(): can be read repeatedly
buf.rewind();
System.out.println("-----------------rewind() Repeatable reading----------------");
System.out.println("position = "+buf.position());
System.out.println("limit = "+buf.limit());
System.out.println("capacity = "+buf.capacity());
//6. clear(): clear the buffer However, the data in the buffer still exists, but it is in the "forgotten" state
buf.clear();
System.out.println("-----------------clear()----------------");
System.out.println("position = "+buf.position());
System.out.println("limit = "+buf.limit());
System.out.println("capacity = "+buf.capacity());
System.out.println((char)buf.get());
}
The operation results are as follows:
-----------------allocate() establish---------------- position = 0 limit= 1024 capacity = 1024 -----------------put() storage---------------- position = 4 limit = 1024 capacity = 1024 -----------------flip()---------------- position = 0 limit = 4 capacity = 1024 abcd -----------------get() read---------------- position = 4 limit = 4 capacity = 1024 -----------------rewind() Repeatable reading---------------- position = 0 limit = 4 capacity = 1024 -----------------clear()---------------- position = 0 limit = 1024 capacity = 1024 a
mark(): mark, indicating the position of the current position.
reset(): restore to the location of mark.
The use of mark() and reset() can be understood according to the following code.
@Test
public void test2(){
String str = "abcde";
ByteBuffer buf = ByteBuffer.allocate(1024);
buf.put(str.getBytes());
buf.flip();
byte[] dst = new byte[buf.limit()];
buf.get(dst, 0, 2);
System.out.println(new String(dst, 0, 2));
System.out.println("position1 = "+buf.position());
//mark(): mark position=2
buf.mark();
buf.get(dst, 2, 2);
System.out.println(new String(dst, 2, 2));
System.out.println("position2 = "+buf.position());
//reset(): restore to the location of mark
buf.reset();
System.out.println("position3 = "+buf.position());
//Determine whether there is any remaining data in the buffer
if(buf.hasRemaining()){
//Gets the number of operations that can be performed in the buffer
System.out.println(buf.remaining());
}
}
The above code is only easy to understand. If you understand it, you don't need to look at the code implementation. You won't write these principle level codes manually during real operation.
NIO buffer is divided into indirect buffer and indirect buffer.
1.1 indirect buffer:
When using indirect buffer, that is, allocate the buffer through allocate() method and build the buffer in the memory of JVM. When our program wants to read data from hard disk, it needs the following three steps:
1. First read the data from the physical hard disk into the physical memory
2 then copy the contents to the memory of the JVM
3 then the application can read the content
Both reading and writing are like this. You need to copy this action. The disadvantage is that it is slow and the advantage is safety.
The code is as follows:
/**
* Indirect buffer read / write operation
* @throws IOException
*/
@Test
public void test001() throws IOException {
long statTime=System.currentTimeMillis();
// Read in stream
FileInputStream fst = new FileInputStream("f://1.mp4");
// Write stream
FileOutputStream fos = new FileOutputStream("f://2.mp4");
// Create channel
FileChannel inChannel = fst.getChannel();
FileChannel outChannel = fos.getChannel();
// Allocate a buffer of the specified size
ByteBuffer buf = ByteBuffer.allocate(1024);
while (inChannel.read(buf) != -1) {
// Turn on read mode
buf.flip();
// Write data to channel
outChannel.write(buf);
buf.clear();
}
// Close channel, close connection
inChannel.close();
outChannel.close();
fos.close();
fst.close();
long endTime=System.currentTimeMillis();
System.out.println("Time consuming to manipulate indirect buffers:"+(endTime-statTime));
}
1.2 direct buffer:
Using direct buffer is to create a buffer directly in memory in the application and physical disk. In physical memory, the copying step is omitted. (allocate the direct buffer through the allocateDirect() method and establish the buffer in the physical memory)
/**
* Direct buffer
* @throws IOException
*/
@Test
public void test002() throws IOException {
long statTime=System.currentTimeMillis();
//Create pipe
FileChannel inChannel= FileChannel.open(Paths.get("f://1.mp4"), StandardOpenOption.READ);
FileChannel outChannel=FileChannel.open(Paths.get("f://2.mp4"), StandardOpenOption.READ,StandardOpenOption.WRITE, StandardOpenOption.CREATE);
//Define mapping file
MappedByteBuffer inMappedByte = inChannel.map(MapMode.READ_ONLY,0, inChannel.size());
MappedByteBuffer outMappedByte = outChannel.map(MapMode.READ_WRITE,0, inChannel.size());
//Direct operation on buffer
byte[] dsf=new byte[inMappedByte.limit()];
inMappedByte.get(dsf);
outMappedByte.put(dsf);
inChannel.close();
outChannel.close();
long endTime=System.currentTimeMillis();
System.out.println("Time consuming to operate direct buffer:"+(endTime-statTime));
}
2. Channel
Channel refers to the connection between memory and IO devices (such as files and sockets), that is, the connection between source node and target node. In java NIO, the channel itself is not responsible for storing and accessing data. It is mainly responsible for data transmission in conjunction with the buffer.
Take a vivid and obvious example to illustrate the necessity of channels in the operating system. First take a look at this figure:
If we use traditional io, that is, we do not use channels for data transmission, then our cpu will be responsible for directly calling the io interface and then interacting with the direct buffer / indirect buffer.
If we use a channel, when I/O operations are needed, the CPU only needs to start the channel, and then the CPU can continue to execute its own program, and the channel executes the channel program, that is to say, the channel enables a higher degree of parallelism between the host (CPU and memory) and I/O operations.
The internal operation diagram of the channel is as follows:
The relevant methods and codes of the channel are as follows (in that sentence, you can understand it. These are implemented internally during application):
/*
* 1, Channel: used to connect the source node to the target node. In java NIO, it is responsible for the transmission of data in the buffer. The channel itself does not store data and needs to cooperate with the buffer for transmission.
*
* 2, Main implementation classes of channel
* java.nio.channels.Channel Interface:
* |--FileChannel: Channels for reading, writing, mapping, and manipulating files.
* |--SocketChannel: Read and write data in the network through TCP.
* |--ServerSocketChannel: You can listen to new TCP connections and create a SocketChannel for each new connection.
* |--DatagramChannel: Read and write the data channel in the network through UDP.
*
* 3, Get channel
* 1.java The getChannel() method is provided for classes that support channels
* Local IO:
* FileInputStream/FileOutputStream
* RandomAccessFile
*
* Network IO:
* Socket
* ServerSocket
* DatagramSocket
*
* 2.NiO in JDK 1.7 2. The static method open() is provided for each channel
* 3.NiO in JDK 1.7 newByteChannel() of Files tool class of 2
*
* 4, Data transmission between channels
* transferFrom()
* transferTo()
*
* 5, Scatter and gather
* Scattering Reads: scatter the data in the channel into multiple buffers
* Gathering Writes: aggregates data from multiple buffers into channels
*
* 6, Character set: Charset
* Encoding: String - character array
* Decoding: character array - string
*/
public class TestChannel {
//Use channel to copy files (indirect buffer)
@Test
public void test1(){
long start=System.currentTimeMillis();
FileInputStream fis=null;
FileOutputStream fos=null;
FileChannel inChannel=null;
FileChannel outChannel=null;
try{
fis=new FileInputStream("d:/1.avi");
fos=new FileOutputStream("d:/2.avi");
//1. Access
inChannel=fis.getChannel();
outChannel=fos.getChannel();
//2. Allocate a buffer of the specified size
ByteBuffer buf=ByteBuffer.allocate(1024);
//3. Store data in buffer in channel
while(inChannel.read(buf)!=-1){
buf.flip();//Switch the mode of reading data
//4. Write the data in the buffer to the channel
outChannel.write(buf);
buf.clear();//Empty buffer
}
}catch(IOException e){
e.printStackTrace();
}finally{
if(outChannel!=null){
try {
outChannel.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(inChannel!=null){
try {
inChannel.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(fos!=null){
try {
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(fis!=null){
try {
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
long end=System.currentTimeMillis();
System.out.println("Time consuming:"+(end-start));//Time consuming: 1094
}
//Use direct buffer to copy files (memory mapped files)
@Test
public void test2() {
long start=System.currentTimeMillis();
FileChannel inChannel=null;
FileChannel outChannel=null;
try {
inChannel = FileChannel.open(Paths.get("d:/1.avi"), StandardOpenOption.READ);
outChannel=FileChannel.open(Paths.get("d:/2.avi"), StandardOpenOption.WRITE,StandardOpenOption.READ,StandardOpenOption.CREATE);
//Memory mapping file
MappedByteBuffer inMappedBuf=inChannel.map(MapMode.READ_ONLY, 0, inChannel.size());
MappedByteBuffer outMappedBuf=outChannel.map(MapMode.READ_WRITE, 0, inChannel.size());
//Directly read and write data to the buffer
byte[] dst=new byte[inMappedBuf.limit()];
inMappedBuf.get(dst);
outMappedBuf.put(dst);
} catch (IOException e) {
e.printStackTrace();
}finally{
if(outChannel!=null){
try {
outChannel.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(inChannel!=null){
try {
inChannel.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
long end=System.currentTimeMillis();
System.out.println("The time spent is:"+(end-start));//Time spent: 200
}
//Data transfer between channels (direct buffer)
@Test
public void test3(){
long start=System.currentTimeMillis();
FileChannel inChannel=null;
FileChannel outChannel=null;
try {
inChannel = FileChannel.open(Paths.get("d:/1.avi"), StandardOpenOption.READ);
outChannel=FileChannel.open(Paths.get("d:/2.avi"), StandardOpenOption.WRITE,StandardOpenOption.READ,StandardOpenOption.CREATE);
inChannel.transferTo(0, inChannel.size(), outChannel);
outChannel.transferFrom(inChannel, 0, inChannel.size());
} catch (IOException e) {
e.printStackTrace();
}finally{
if(outChannel!=null){
try {
outChannel.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(inChannel!=null){
try {
inChannel.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
long end=System.currentTimeMillis();
System.out.println("The time spent is:"+(end-start));//Time spent: 147
}
//Dispersion and aggregation
@Test
public void test4(){
RandomAccessFile raf1=null;
FileChannel channel1=null;
RandomAccessFile raf2=null;
FileChannel channel2=null;
try {
raf1=new RandomAccessFile("1.txt","rw");
//1. Access
channel1=raf1.getChannel();
//2. Allocate a buffer of the specified size
ByteBuffer buf1=ByteBuffer.allocate(100);
ByteBuffer buf2=ByteBuffer.allocate(1024);
//3. Distributed reading
ByteBuffer[] bufs={buf1,buf2};
channel1.read(bufs);
for(ByteBuffer byteBuffer : bufs){
byteBuffer.flip();
}
System.out.println(new String(bufs[0].array(),0,bufs[0].limit()));
System.out.println("--------------------");
System.out.println(new String(bufs[1].array(),0,bufs[1].limit()));
//4. Aggregate write
raf2=new RandomAccessFile("2.txt", "rw");
channel2=raf2.getChannel();
channel2.write(bufs);
}catch (IOException e) {
e.printStackTrace();
}finally{
if(channel2!=null){
try {
channel2.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(channel1!=null){
try {
channel1.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(raf2!=null){
try {
raf2.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(raf1!=null){
try {
raf1.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
//Output supported character sets
@Test
public void test5(){
Map<String,Charset> map=Charset.availableCharsets();
Set<Entry<String,Charset>> set=map.entrySet();
for(Entry<String,Charset> entry:set){
System.out.println(entry.getKey()+"="+entry.getValue());
}
}
//character set
@Test
public void test6(){
Charset cs1=Charset.forName("GBK");
//Get encoder
CharsetEncoder ce=cs1.newEncoder();
//Get decoder
CharsetDecoder cd=cs1.newDecoder();
CharBuffer cBuf=CharBuffer.allocate(1024);
cBuf.put("Lala, ha ha");
cBuf.flip();
//code
ByteBuffer bBuf=null;
try {
bBuf = ce.encode(cBuf);
} catch (CharacterCodingException e) {
e.printStackTrace();
}
for(int i=0;i<12;i++){
System.out.println(bBuf.get());//-64-78-64-78-71-2-7-2-80-55-80-55
}
//decode
bBuf.flip();
CharBuffer cBuf2=null;
try {
cBuf2 = cd.decode(bBuf);
} catch (CharacterCodingException e) {
e.printStackTrace();
}
System.out.println(cBuf2.toString());
}
}
3. Selector
The Selector is responsible for monitoring the IO status of the channel, which can also be called a multiplexer. It is one of the core components of Java NIO. It is used to check whether the state of one or more NiO channels is readable and writable. In this way, multiple channels can be managed by a single thread, that is, multiple network links can be managed.
Sample code (you can understand it, it's encapsulated, and you don't need to write it yourself):
/*
* 1, Three cores of network communication using NIO:
*
* 1,Channel: responsible for connection
* java.nio.channels.Channel Interface:
* |--SelectableChannel
* |--SocketChannel
* |--ServerSocketChannel
* |--DatagramChannel
*
* |--Pipe.SinkChannel
* |--Pipe.SourceChannel
*
* 2.Buffer: responsible for data access
*
* 3.Selector: it is the multiplexer of SelectableChannel. Used to monitor the IO status of SelectableChannel
*/
public class TestBlockingNIO {//Useless Selector, blocking
//client
@Test
public void client() throws IOException{
SocketChannel sChannel=SocketChannel.open(new InetSocketAddress("127.0.0.1",9898));
FileChannel inChannel=FileChannel.open(Paths.get("1.jpg"), StandardOpenOption.READ);
ByteBuffer buf=ByteBuffer.allocate(1024);
while(inChannel.read(buf)!=-1){
buf.flip();
sChannel.write(buf);
buf.clear();
}
sChannel.shutdownOutput();//Closing the transmission channel indicates that the transmission is completed
//Receive feedback from the server
int len=0;
while((len=sChannel.read(buf))!=-1){
buf.flip();
System.out.println(new String(buf.array(),0,len));
buf.clear();
}
inChannel.close();
sChannel.close();
}
//Server
@Test
public void server() throws IOException{
ServerSocketChannel ssChannel=ServerSocketChannel.open();
FileChannel outChannel=FileChannel.open(Paths.get("2.jpg"), StandardOpenOption.WRITE,StandardOpenOption.CREATE);
ssChannel.bind(new InetSocketAddress(9898));
SocketChannel sChannel=ssChannel.accept();
ByteBuffer buf=ByteBuffer.allocate(1024);
while(sChannel.read(buf)!=-1){
buf.flip();
outChannel.write(buf);
buf.clear();
}
//Send feedback to client
buf.put("The server successfully received the data".getBytes());
buf.flip();//Read mode
sChannel.write(buf);
sChannel.close();
outChannel.close();
ssChannel.close();
}
}
SelectionKey
SelectionKey: indicates the registration relationship between channel and selector. Each time a channel is registered with the selector, an event is selected (selection key). The selection key contains two sets of operations represented as integer values. Each bit of the operation set represents a type of selectable operation supported by the channel of the key.
For example, when calling register(Selector sel, int ops) to register the channel with the selector, the listening event of the selector on the channel needs to be specified through the second parameter ops.
ops represents the event type that can be monitored (represented by four constants of SelectionKey):
Read: selectionkey OP_ READ (1)
Write: selectionkey OP_ WRITE (4)
Connection: selectionkey OP_ CONNECT (8)
Receive: selectionkey OP_ ACCEPT (16)
If you listen to more than one event during registration, you can use the "bit or" operator to connect.
Sample code (just understand):
public class TestNonBlockingNIO {
//client
@Test
public void client()throws IOException{
//1. Access
SocketChannel sChannel=SocketChannel.open(new InetSocketAddress("127.0.0.1", 9898));
//2. Switch to non blocking mode
sChannel.configureBlocking(false);
//3. Allocate a buffer of the specified size
ByteBuffer buf=ByteBuffer.allocate(1024);
//4. Send data to the server
Scanner scan=new Scanner(System.in);
while(scan.hasNext()){
String str=scan.next();
buf.put((new Date().toString()+"\n"+str).getBytes());
buf.flip();
sChannel.write(buf);
buf.clear();
}
//5. Close the channel
sChannel.close();
}
//Server
@Test
public void server() throws IOException{
//1. Access
ServerSocketChannel ssChannel=ServerSocketChannel.open();
//2. Switch to non blocking mode
ssChannel.configureBlocking(false);
//3. Binding connection
ssChannel.bind(new InetSocketAddress(9898));
//4. Get selector
Selector selector=Selector.open();
//5. Register the channel on the selector and specify "listen for receive events"
ssChannel.register(selector,SelectionKey.OP_ACCEPT);
//6. Get the "ready" event on the polling selector
while(selector.select()>0){
//7. Obtain all registered "selection keys (ready listening events)" in the current selector
Iterator<SelectionKey> it=selector.selectedKeys().iterator();
while(it.hasNext()){
//8. Get ready events
SelectionKey sk=it.next();
//9. Determine when it is ready
if(sk.isAcceptable()){
//10. If "receive ready", obtain the client connection
SocketChannel sChannel=ssChannel.accept();
//11. Switch to non blocking mode
sChannel.configureBlocking(false);
//12. Register the channel on the selector
sChannel.register(selector, SelectionKey.OP_READ);
}else if(sk.isReadable()){
//13. Get the channel of "read ready" status on the current selector
SocketChannel sChannel=(SocketChannel)sk.channel();
//14. Read data
ByteBuffer buf=ByteBuffer.allocate(1024);
int len=0;
while((len=sChannel.read(buf))>0){
buf.flip();
System.out.println(new String(buf.array(),0,len));
buf.clear();
}
}
//15. Deselectionkey
it.remove();
}
}
}
}