spark-structured-streaming

Credit Card Fraudulent Detection with Random Forest

This Spark Java project serves as a demonstration of Gradle Spark configuration, specifically focusing on utilizing the MemoryStream class as the streaming source.

This project presents a distributable solution based on Spark Java, aiming to connect start and end session events together in a stateful manner. The project utilizes `flatMapGroupWithState`functionality which is a powerful feature for stateful stream processing in Spark. It enables you to maintain and update the state across batches.

Repository ini merupakan Project End-to-End pengembangan pipeline data streaming yang bertujuan untuk melakukan deteksi serangan DDoS. (Pengerjaan 19 January 2026 - Sekarang)

In this solution, the issue of creating a table with case-sensitive columns (in the scenario where the table doesn't exist or when writing the table in overwrite mode) in Oracle has been addressed by developing a custom Oracle dialect and registering it.

MapReduce Job Development, RDDs Programming, Medical Data Management, Sales Analysis, And Efficient Data Integration For Big Data Analysis. Spark: Big Data Processing, SQOOP Integration, And Spark Structured Streaming For Real-Time Data.

Real-Time Data Pipeline: Postgres CDC → Kafka → Spark Streaming → MySQL → Live Dashboard

Distributed event processing with Apache Spark Structured Streaming: Kafka ingestion, stateful deduplication, watermark-based window aggregation, and reproducible Parquet/Kafka outputs.

Real-time analysis of hospital incident data using Apache Spark Streaming to track incidents by service and identify the top years with the most incidents.

Projects related to Big Data technologies