SmartCSV: A Parallel CSV Processing Library for Spring Boot

Introduction

SmartCSV is a Spring Boot library designed to simplify the handling of CSV file uploads and processing. It allows developers to:

• Upload CSV Files Easily: Streamline CSV file uploads via REST endpoints.
• Validate and Process Data in Parallel: Improve performance by concurrently validating and processing records.
• Define Custom Validations: Specify custom validation rules for CSV headers and data fields.
• Flexible Error Handling: Choose how to handle validation errors—skip records, stop processing, or collect and return all errors.
• Annotation-Based Configuration: Integrate seamlessly into existing projects with minimal effort.

Features

• Annotation-Based Setup: Use @SmartCsvProcessor and @SmartCsvField annotations to configure processing.
• Custom Validations: Implement your own validation logic for headers and records.
• Parallel Processing: Utilize multi-threading to process large datasets efficiently.
• Flexible Error Handling: Decide whether to skip invalid records, halt processing on errors, or collect errors for reporting.
• Detailed Logging: Monitor processing steps and errors through comprehensive SLF4J logging.

Installation

Add the following dependency to your project's pom.xml or build.gradle file: This project uses JitPack to host the library. To add the dependency to your project, follow the instructions below: For Maven (pom.xml):

<repositories>
<repository>
<id>jitpack.io</id>
<url>https://jitpack.io</url>
</repository>
</repositories>

<dependency>
<groupId>com.github.saadzarook</groupId>
<artifactId>smartcsv</artifactId>
<version>v1.0.0</version>
</dependency>

For Gradle (build.gradle):

repositories {
mavenCentral()
maven { url 'https://jitpack.io' }
}
dependencies {
implementation 'com.github.saadzarook:smartcsv:v1.0.0'
}

Usage

1. Annotate your processing class with @SmartCsvProcessor

@SmartCsvProcessor(
model = User.class,
validationStrategy = "collect", // Options: skip, stop, collectheaderValidator = CustomHeaderValidator.class// Optional
)
publicvoidprocessUser(Useruser) {
// Your processing logic hereuserService.save(user);
}

2. Define Your Model Class

publicclassUser {
@SmartCsvField(header = "Name", required = true)
privateStringname;
@SmartCsvField(header = "Email", required = true, validation = "[^@ ]+@[^@ ]+\\.[^@ ]+")
privateStringemail;
// Getters and setters
}

3. Implement a Custom Header Validator (Optional)

publicclassCustomHeaderValidatorimplementsHeaderValidator {
@OverridepublicbooleanvalidateHeaders(String[] headers) {
returnArrays.asList(headers).contains("Name") && Arrays.asList(headers).contains("Email");
}
}

4. Handle CSV File Uploads via REST Endpoint

@RestController@RequestMapping("/users")
publicclassUserController {
@AutowiredprivateCsvProcessingServicecsvProcessingService;
@PostMapping("/upload")
publicResponseEntity<?> uploadCsv(@RequestParam("file") MultipartFilefile) {
try {
// Get the method annotated with @SmartCsvProcessorMethodprocessUserMethod = this.getClass().getMethod("processUser", User.class);
// Process the CSV fileList<RecordError> errors = csvProcessingService.processCsv(file, processUserMethod, this);
if (!errors.isEmpty()) {
// Return errors to the clientreturnResponseEntity.badRequest().body(errors);
}
returnResponseEntity.ok("CSV file processed successfully");
} catch (Exceptione) {
// Handle exceptionsreturnResponseEntity.status(500).body("Internal Server Error");
}
}
@SmartCsvProcessor(
model = User.class,
validationStrategy = "collect",
headerValidator = CustomHeaderValidator.class
)
publicvoidprocessUser(Useruser) {
// Processing logicuserService.save(user);
}
}

Error Handling Strategies

Specify how the library should handle validation errors using the validationStrategy parameter:

• skip: Skip invalid records and continue processing.
• stop: Stop processing when an error is encountered.
• collect: Collect all errors and return them after processing.

Contribution

Contributions are welcome! Feel free to open issues or submit pull requests to help improve this library.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

For questions or support, please contact Saad Zarook at saadzarook@gmail.com