Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

A Tiny Functional CRUD for Minio (S3) in a Vaadin App

DZone's Guide to

A Tiny Functional CRUD for Minio (S3) in a Vaadin App

Want to bring S3 storage to a Vaadin app? Read on to learn how to build CRUD functionality using minio via Docker and Java.

· Java Zone ·
Free Resource

Verify, standardize, and correct the Big 4 + more– name, email, phone and global addresses – try our Data Quality APIs now at Melissa Developer Portal!

This demo app will show how to create blob storage based on minio storage and uses the Vaadin Framework to present the blobs. The connection between Vaadin and Minio is implemented in a functional style.

For the impatient:

The corresponding GitHub repo is here https://github.com/Java-Publications/vaadin-027-S3-storage-minio. To start and play with this demo, you have to go via command line to the folder _data and start the S3 storage with Docker-Compose up After this is done, you can go to the URL http://localhost:9999 to log into your fresh S3 storage. The login details are minio/minio123. To start the Vaadin app, make a mvn clean install followed by starting the main method of the class JumpstartUI

Image title


S3 Storage: Minio

The project we are using here is on GitHub: https://github.com/minio/minio. With this, we can start building document storage for our Vaadin Apps.

From the minio page itself:

It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.

But first, I will show how to start with the S3 Storage itself. For this example, we have to install Docker and Docker-Compose. The Docker installation itself is out of the scope of this article. After you have done this, we could start polling the image with the docker command.

docker pull minio/minio

Now we can start using this. For the first examples, I would create a container without external volumes. Data will be lost after the container is stopped and deleted. For the first steps, this is perfect!

docker run -p 9000:9000 --name minio minio/minio server /data

But if you want/need persistent volumes, up to 16 volumes per minio node can be used.

docker run -p 9000:9000 --name minio \
  -v /mnt/data1:/data1 \
  -v /mnt/data2:/data2 \
  -v /mnt/data3:/data3 \
  -v /mnt/data4:/data4 \
  -v /mnt/data5:/data5 \
  -v /mnt/data6:/data6 \
  -v /mnt/data7:/data7 \
  -v /mnt/data8:/data8 \
  minio/minio server /data1 /data2 /data3 /data4 /data5 /data6 /data7 /data8


After the first start (in this example, I just used the first version), you will get the information about the node via the logs.

Drive Capacity: 50 GiB Free, 60 GiB Total

Endpoint:  http://172.17.0.2:9000  http://127.0.0.1:9000
AccessKey: 1TEQLU3S6N19ID4A32QJ 
SecretKey: KQamn/OWyGZPnuGq+1ZNYgRZqJLeiAJ06bJwNmJ9 

Browser Access:
   http://172.17.0.2:9000  http://127.0.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://172.17.0.2:9000 1TEQLU3S6N19ID4A32QJ KQamn/OWyGZPnuGq+1ZNYgRZqJLeiAJ06bJwNmJ9

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide


The following are important:

  • Endpoint: http://172.17.0.2:9000 http://127.0.0.1:9000
  • AccessKey: 1TEQLU3S6N19ID4A32QJ
  • SecretKey: KQamn/OWyGZPnuGq+1ZNYgRZqJLeiAJ06bJwNmJ9

With this, you can start using the node that was just created. Use a web browser and access the following URL: http://127.0.0.1:9000

Image title

For the first login, use the AccessKey and SecretKey that were created by the minio server itself.

Image title

After the first login, there will be a more or less empty screen. Now we have all preparations done time to start with the Java-side as well.

Image title

S3 Storage: Minio-Java SDK

The Java SDK is provided at Maven central.

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>3.0.12</version>
</dependency>


The SDK will give you a class called MinioClient. With this, all commands are operated on S3-Storage. To initialize the client, you need the information’s from the log of your minio-container.

      final MinioClient minioClient = new MinioClient(
                                       "https://localhost:9000", 
                                       "1TEQLU3S6N19ID4A32QJ", 
                                       "KQamn/OWyGZPnuGq+1ZNYgRZqJLeiAJ06bJwNmJ9");


Now we can play around with the API itself. Check the class MinioBasicTest in the project for more code examples.

S3 Storage: Minio With Docker Compose

Minio itself can be started in a distributed version. To show this, I created a Docker-Compose file under _data/ that will create a four node cluster of minio. The Access- and Sec-Key is set to minio/minio123. You will find this information inside the Docker-Compose file. For production, you MUST change this. The minio-nodes themselves are not publishing ports by themselves. To connect, we will install nginx in front of them.

 minio1:
  image: minio/minio
  container_name: minio1
  hostname: minio1
  volumes:
   - data1:/data
  environment:
   MINIO_ACCESS_KEY: minio
   MINIO_SECRET_KEY: minio123
  command: server http://minio1/data http://minio2/data http://minio3/data http://minio4/data

 minio2:
  image: minio/minio
  container_name: minio2
  hostname: minio2
  volumes:
   - data2:/data
  environment:
   MINIO_ACCESS_KEY: minio
   MINIO_SECRET_KEY: minio123
  command: server http://minio1/data http://minio2/data http://minio3/data http://minio4/data

 minio3:
  image: minio/minio
  container_name: minio3
  hostname: minio3
  volumes:
   - data3:/data
  environment:
   MINIO_ACCESS_KEY: minio
   MINIO_SECRET_KEY: minio123
  command: server http://minio1/data http://minio2/data http://minio3/data http://minio4/data

 minio4:
  image: minio/minio
  container_name: minio4
  hostname: minio4
  volumes:
   - data4:/data
  environment:
   MINIO_ACCESS_KEY: minio
   MINIO_SECRET_KEY: minio123
  command: server http://minio1/data http://minio2/data http://minio3/data http://minio4/data

## By default this config uses default local driver,
## For custom volumes replace with volume driver configuration.
volumes:
  data1:
  data2:
  data3:
  data4:


In front of this cluster, there is a proxy server with easy load balancing based on nginx. No failover or other production-related configuration is done here in this example.

services:
 web:
  image: nginx
  container_name: proxy
  hostname: proxy
  volumes:
   - ./_nginx/proxy.conf:/etc/nginx/proxy.conf:ro
   - ./_nginx/nginx.conf:/etc/nginx/nginx.conf:ro
  ports:
   - "9999:80"
  environment:
   - NGINX_HOST=proxy
   - NGINX_PORT=80
  command: [nginx, '-g', 'daemon off;']


The nginx itself will know the minio nodes, export the port 80 to the local port 9999. The connection to the minio-nodes is done over nginx only.

events {
  worker_connections  4096;  ## Default: 1024
}

http {
include    /etc/nginx/proxy.conf;

upstream minio_servers {
    server minio1:9000;
    server minio2:9000;
    server minio3:9000;
    server minio4:9000;
}
server {
    listen 80;
    server_name proxy;

    location / {
        proxy_set_header Host $http_host;
        proxy_pass       http://minio_servers;
        proxy_redirect     off;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
    }
}
}


The Vaadin App

Now we are ready to start with the Vaadin app itself. This version is based on Vaadin 8. The functionality is quite easy — after providing the credentials for the S3 storage, the app will randomly load a picture every few seconds from the storage and present it on the screen.

Image title

Vaadin Ramp Up

The ramp up of the Vaadin app is done in the class CoreUIService. Here you will see that this example is using an Undertow core to provide the Vaadin app. 

If you want to read more about the Nano-Vaadin approach, check this git repo:  https://github.com/vaadin-developer/nano-vaadin
public class CoreUIService {

  @FunctionalInterface
  public static interface ComponentSupplier extends Supplier<Component> { }

  @PreserveOnRefresh
  @Push
  public static class MyUI extends UI implements HasLogger {
    public static final String COMPONENT_SUPPLIER_TO_USE = "COMPONENT_SUPPLIER_TO_USE";
    @Override
    protected void init(VaadinRequest request) {
      final String className = System.getProperty(COMPONENT_SUPPLIER_TO_USE);
      logger().info("class to load : " + className);
      ((CheckedSupplier<Class<?>>) () -> forName(className))
          .get() //TODO make it fault tolerant
          .flatMap((CheckedFunction<Class<?>, Object>) Class::newInstance)
          .flatMap((CheckedFunction<Object, ComponentSupplier>) ComponentSupplier.class::cast)
          .flatMap((CheckedFunction<ComponentSupplier, Component>) Supplier::get)
          .ifPresentOrElse(this::setContent,
                           failed -> logger().warning(failed)
          );
    }
  }

  @WebServlet("/*")
  @VaadinServletConfiguration(productionMode = false, ui = MyUI.class)
  public static class CoreServlet extends VaadinServlet {
    //customize Servlet if needed
  }

  public static void main(String[] args) throws ServletException {
    new CoreUIService().startup();
  }

  public void startup() throws ServletException {
    DeploymentInfo servletBuilder
        = Servlets.deployment()
                  .setClassLoader(CoreUIService.class.getClassLoader())
                  .setContextPath("/")
                  .setDeploymentName("ROOT.war")
                  .setDefaultEncoding("UTF-8")
                  .addServlets(
                      servlet(
                          CoreServlet.class.getSimpleName(),
                          CoreServlet.class
                      ).addMapping("/*")
                      .setAsyncSupported(true)
                  );

    final DeploymentManager manager = Servlets
        .defaultContainer()
        .addDeployment(servletBuilder);
    manager.deploy();
    PathHandler path = path(redirect("/"))
        .addPrefixPath("/", manager.start());
    Undertow.builder()
            .addHttpListener(8080, "0.0.0.0")
            .setHandler(path)
            .build()
            .start();
  }
}


To connect to the UI class, the System-Property public static final String COMPONENTSUPPLIERTOUSE = “COMPONENTSUPPLIERTOUSE”; must be set with the class name that will provide the Supplier<Component>. This Supplier is used to create the ContentRoot element.

public class JumpstartUI extends CoreUIService implements HasLogger {

  static {
    setProperty(COMPONENT_SUPPLIER_TO_USE, MySupplier.class.getName());
  }

  public static class MySupplier implements CoreUIService.ComponentSupplier {
    @Override
    public Component get() {
      return new DashboardComponent().postConstruct();
    }
  }
}


The Basic UI is implemented inside the class DashboardComponent.

  private final TextField accessPoint = new TextField("Access point");
  private final TextField accessKey   = new TextField("accessKey");
  private final TextField secKey      = new TextField("secKey");
  private final TextField bucketName  = new TextField("bucketName");
  private final Button    connect     = new Button("connect");

  private final FormLayout layout = new FormLayout(accessPoint,
                                                   accessKey,
                                                   secKey,
                                                   bucketName,
                                                   connect
  );

  private Image  image      = new Image();
  private Layout mainLayout = new VerticalLayout(layout, image);


  public DashboardComponent() {
    setCompositionRoot(mainLayout);
  }


For every info element we need, the corresponding UI element is created. For this, a TextField is used. After the button is pressed, the connection to the S3 storage will be created, based on the input values from the TextFields.

Check the class MinioClientFunctions. Here you will find a functional wrapper around the MinioClient class. Why? Well, the MinioClient is throwing a lot of exceptions. That will lead to ugly code if you want to avoid a global try-catch-block or nested try-catch-blocks.

Additionally I am using the tiny frp lib that you can find on github https://github.com/functional-reactive/functional-reactive-lib

Creating a MinioClient Instance

  static CheckedFunction<Coordinates, MinioClient> client() {
    return (coord) -> new MinioClient(coord.endpoint(),
                                      coord.accessKey(),
                                      coord.secretKey()
    );
  }


Create a Bucket if it Does Not Already Exist

  static CheckedBiFunction<MinioClient, String, MinioClient> bucket() {
    return (minioClient, bucketName) -> {
      if (!minioClient.bucketExists(bucketName)) {
        minioClient.makeBucket(bucketName);
      }
      return minioClient;
    };
  }


Put Object in Bucket

  static CheckedBiFunction<MinioClient, Blob, MinioClient> putObj() {
    return (minioClient, obj) -> {
      minioClient.putObject(obj.getT1(),
                            obj.getT2(),
                            obj.getT3(),
                            obj.getT4()
      );
      return minioClient;
    };
  }


Get Object From Bucket

  static CheckedBiFunction<MinioClient, BlobCoordinates, InputStream> getObj() {
    return (minioClient, obj) -> minioClient
        .getObject(
            obj.bucketName(),
            obj.objectName()
        );
  }


From S3 to Vaadin

With this, we can finally connect the Vaadin app to the S3 storage.

The access info from the TextFields will be provided and wrapped into a Coordinates instance with a Supplier<Coordinates>.

  private Supplier<Coordinates> access() {
    return () -> new Coordinates(accessPoint.getValue(),
                                 accessKey.getValue(),
                                 secKey.getValue()
    );
  }


The connection itself will be established after the button is pressed. It will create a registration that could be used from the timer service to publish the next random image.

public class ImageTimerService {
  private ImageTimerService() {
  }

  //TODO - JVM static
  private static final Set<ImagePushListener> REGISTRY = newKeySet();

  private static final Timer TIMER = new Timer(true);

  public interface ImagePushListener {
    void updateImage(final String imageID);
  }


  public static Registration register(ImagePushListener imagePushListener) {

    REGISTRY.add(imagePushListener);

    return () -> {
      REGISTRY.remove(imagePushListener);
      Logger.getAnonymousLogger().info("removed registration");
    };
  }

  public static void updateImages() {
    // not nice coupled
    REGISTRY.forEach(e -> e.updateImage(nextImageName().get()));
  }

  static {
    TIMER.scheduleAtFixedRate(
        new TimerTask() {
          @Override
          public void run() {
            ImageTimerService.updateImages();
          }
        },
        5_000,
        2_000
    );
  }
}


With every new imageID that is provided by the Timer-Service, a new Minio-Client will be created. If this is done, the blob will be loaded from the S3 storage. This blob must be wrapped inside a StreamRessource that will be the next content of the Image instance. Vaadin-Push will publish this change to the browser.

        client()
            .apply(access().get())
            .ifPresentOrElse(
                minioClient -> imageStream()
                    .apply(minioClient, new BlobCoordinates(DEFAULT_BUCKET_NAME, imageID))
                    .ifFailed(failed -> logger().warning(failed))
                    .map(bytes -> new StreamResource(
                        (StreamSource) () -> bytes,
                        imageID + "." + nanoTime()
                    ))
                    .ifPresentOrElse(
                        ok -> image.getUI()
                                   .access(() -> image.setSource(ok)),
                        failed -> {
                          logger().warning(failed);
                          image.getUI()
                               .access(() -> image.setSource(
                                   asStreamSource()
                                       .apply(failedImageAsInputStream().apply(imageID),
                                              imageID
                                       ))
                               );
                        }
                    ),
                failed -> logger().warning(failed)
            )


The important piece of code here is the part that will connect from the outside to the UI-Thread. For this, you need the current UI instance. This will provide the method access(..). The method needs a Runnable as an argument that will be executed from the UI-Thread. This method is provided as blocking and non-blocking versions.

image.getUI().access(() -> image.setSource(ok))


Now all parts are available to build an app against S3 Storage based on Vaadin.

If you want to know more about Vaadin, you can find more about it at https://vaadin.com.

If you have any questions, ping me via Twitter https://twitter.com/SvenRuppert or mail sven.ruppert (x) vaadin.com

Happy coding!

Developers! Quickly and easily gain access to the tools and information you need! Explore, test and combine our data quality APIs at Melissa Developer Portal – home to tools that save time and boost revenue. Our APIs verify, standardize, and correct the Big 4 + more – name, email, phone and global addresses – to ensure accurate delivery, prevent blacklisting and identify risks in real-time.

Topics:
vaadin ,minio ,s3 server open source ,java ,crud ,tutorial

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}