Checkpoint location analysis under spark structured streaming, connection with Grafana monitoring and submission of Kafka Lag monitoring

1, Introduction to spark structured streaming checkpoint location Structured Streaming was introduced in spark version 2.0 in 2016. It is an extensible and fault-tolerant stream processing engine based on Spark SQL Engine. Compared with traditional Spark Streaming, due to the reuse of Spark SQL Engine, the code writing method is the same as bat ...

Posted by ShashidharNP on Mon, 22 Nov 2021 09:44:12 +0100