Ticker

6/random/ticker-posts

MaFiA: Manipulate Files Asynchronously - Asynchronous File Manipulation Service Spring Boot Application

murat öksüzer Ocak 08, 2022

Manipulate Files Asynchronously (MaFiA): Asynchronous file manipulation service, designed for microservice architectured projects, is now avaliable on GitHub here!

Most of the big web peojects requires file manipulation. Some images needed to be scaled, some documents needed to be converted from docx to pdf, pdf to image, excel (xlsx) to pdf reports, or between any format. Some part of a file needed to be extracted somewhere else, etc... While implementing these tasks, you, developers, need to implement some kind of management codes too. You need to keep track of which files are processed, which ones are waiting. You need to be sure that all files are processed. You need asynchronous processing in order to gain time and non-blocking computing.

At this point, MaFiA is here to execute! Developers will only develop manipulation logic they need and apart from that, every management requirement will be ready on this tiny microservice: MaFiA.

 

Mafia: Asynchronous File Manipulation
Here is the ready features in this Spring Boot application code base:

  • Provide simple RESTFul API.
  • Create file manipulation job by uploading a file with manipulation parameters and get a UUID for the process. For example, upload a big resolution image.
  • Determine which manipulation is needed to apply to the uploaded file in parameters. i.e. say "scale_image".
  • Give job specific parameters. i.e. give scale factor as 0.3.
  • Query the job status using generated UUID.
  • Download output file using generated UUID.
  • At configuration file (application.properties) define whatever executor pool size you want. You can increase the executor thread pool size in order to increase number of files to manipulate simultaneously.
  • Define your own manipulation logic and integrate easily into application.

Test Demos on Youtube here:  



How Can I Define and Integrate My Own File Manipulation Logic?

You only need to

  • Create a manipulation class that extends AbstractManipulation
  • Implement the applyManipulation() method
  • Register this new class to ManipulationStoreService in the InitManipulationStoreService class

package com.smartycoder.mafia.store;


import java.nio.file.Path;

import java.util.Map;


import org.apache.commons.io.FileUtils;


import com.smartycoder.mafia.manipulation.AbstractManipulation;


public class JustCopyManipulation extends AbstractManipulation {


    @Override

    public void applyManipulation(Path uploadedFilePath, Path outputFilePath,

            Map<String, String> manipulationParameters) throws Throwable {


        FileUtils.copyFile(uploadedFilePath.toFile(), outputFilePath.toFile());

    }

}


Here is the InitManipulationStoreService

@Component

@Order(1)

public class InitManipulationStoreService implements ApplicationListener<ApplicationReadyEvent>{


    @Autowired

    private ManipulationStoreService manipulationStoreService;

   

    @Override

    public void onApplicationEvent(ApplicationReadyEvent event) {

       

        manipulationStoreService.registerManipulation("justcopy", JustCopyManipulation.class);

        manipulationStoreService.registerManipulation("scale_image", ScaleImageManipulation.class);

    }


}


How Can I Test It?

Here is an example MaFiA usage for a single image file. In this example, we use Apache HttpClient to consume MaFiA REST API, send HTTP requests and receive responses.

import java.io.BufferedInputStream;

import java.io.BufferedOutputStream;

import java.io.File;

import java.io.FileOutputStream;

import java.io.IOException;


import org.apache.commons.io.IOUtils;

import org.apache.http.HttpEntity;

import org.apache.http.HttpResponse;

import org.apache.http.client.ClientProtocolException;

import org.apache.http.client.HttpClient;

import org.apache.http.client.methods.HttpGet;

import org.apache.http.client.methods.HttpPost;

import org.apache.http.entity.ContentType;

import org.apache.http.entity.mime.MultipartEntityBuilder;

import org.apache.http.entity.mime.content.FileBody;

import org.apache.http.entity.mime.content.StringBody;

import org.apache.http.impl.client.HttpClientBuilder;

import org.json.JSONObject;


public class MAFIAClient {


    public static void main(String[] args) throws ClientProtocolException, IOException, InterruptedException {


        File file = new File("image1.jpg");

        HttpPost httpPost = new HttpPost("http://localhost:8080/job-management/mafia-jobs");


        JSONObject manParms = new JSONObject();

        manParms.put("scale", "0.3");


        JSONObject jsonObject = new JSONObject();

        jsonObject.put("fileType", "JPEG");

        jsonObject.put("manipulationName", "scale_image");

        jsonObject.put("manipulationParameters", manParms);


        StringBody userBody = new StringBody(jsonObject.toString(), ContentType.APPLICATION_JSON);

        FileBody fileBody = new FileBody(file, ContentType.DEFAULT_BINARY);


        MultipartEntityBuilder entityBuilder = MultipartEntityBuilder.create();

        entityBuilder.addPart("mafiaJob", userBody);

        entityBuilder.addPart("file", fileBody);

        HttpEntity entity = entityBuilder.build();

        httpPost.setEntity(entity);


        HttpClient client = HttpClientBuilder.create().build();


        HttpResponse response = client.execute(httpPost);

        HttpEntity responseEntity = response.getEntity();


        // print response

        String responseString = IOUtils.toString(responseEntity.getContent(), "utf8");

        System.out.println(responseString);


        JSONObject responseJson = new JSONObject(responseString);

        String responseUuid = (String) responseJson.get("uuid");


        while (true) {


            HttpGet httpGet = new HttpGet("http://localhost:8080/job-management/mafia-jobs/status/" + responseUuid);


            HttpEntity statusResponse = client.execute(httpGet).getEntity();


            String statusResponseString = IOUtils.toString(statusResponse.getContent(), "utf8");


            System.out.println(statusResponseString);


            JSONObject statusResponseJson = new JSONObject(statusResponseString);


            if ("FINISHED".equals(statusResponseJson.get("status"))) {

                break;

            }


            Thread.sleep(500);

        }


        HttpGet httpGet = new HttpGet("http://localhost:8080/job-management/mafia-jobs/output/" + responseUuid);


        HttpEntity outputResponse = client.execute(httpGet).getEntity();


        BufferedInputStream bis = new BufferedInputStream(outputResponse.getContent());

        String filePath = "scaledImae.png";

        BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(new File(filePath)));

        int inByte;

        while ((inByte = bis.read()) != -1) 

            bos.write(inByte);

        bis.close();

        bos.close();

       

        System.out.println("Download finished");


    }


}



Which Technologies Are Used in MaFiA?

MaFiA is a Spring Boot application. For database, it uses PostgreSql. There is no security implementation here because every project has its own different security approach. One developer can easily integrate security logic into MaFiA. It is open source and freely available on GitHub here.

Similarly if you want, database technology or file storage method can be cutomized easily for your own project technology stack.

For 100 files manipulation and 1000 files multi threaded file manipulation client examples watch this demo on Youtube:




Yorum Gönder

1 Yorumlar