Upload and Download Files to S3 Using Maven

Throughout the years I’ve seen many teams using Maven in many different ways. Maven can be used for many CI/CD tasks instead of using extra pipeline code, or it can be used to prepare the development environment before running some tests.

Generally, it is a convenient tool, widely used among Java teams, and will continue to do so since there is a huge ecosystem around it.

The CloudStorage Maven plugin helps you with using various cloud buckets as a private Maven repository. Recently, CloudStorageMaven for S3 got a huge upgrade, and you can use it in order to download or upload files from S3, by using it as a plugin.

The plugin assumes that your environment is configured properly to access the S3 resources needed.
This can be achieved individually through  aws configure :

aws configure

Other ways are through environmental variables or by using the appropriate iam role.

Suppose you want to download some certain files from a path in S3.

<build>
        <plugins>
            <plugin>
                <groupId>com.gkatzioura.maven.cloud</groupId>
                <artifactId>s3-storage-wagon</artifactId>
                <version>1.6</version>
                <executions>
                    <execution>
                        <id>download-one</id>
                        <phase>package</phase>
                        <goals>
                            <goal>s3-download</goal>
                        </goals>
                        <configuration>
                            <bucket>your-bucket</bucket>
                            <downloadPath>/local/download/path</downloadPath>
                            <keys>1.txt,2.txt,directory/3.txt</keys>
                        </configuration>
                    </execution>
                <executions>
            <plugin>
        <plugins>
</build>

The files 1.txt, 2.txt, directory/3.txt ,  once the execution is finished, will reside in the local directory specified (/local/download/path).

Be aware that the file discovery on S3 is done with a prefix, so if you have file 1.txt and 1.txt.jpg both files will be downloaded.

You can also download only one file to one file that you specified locally, as long as it is one-to-one.

<execution>
    <id>download-prefix</id>
    <phase>package</phase>
    <goals>
        <goal>s3-download</goal>
    </goals>
    <configuration>
        <bucket>your-bucket</bucket>
        <downloadPath>/path/to/local/your-file.txt</downloadPath>
        <keys>a-key-to-download.txt</keys>
    </configuration>
</execution>

Apparently files with a prefix that contain directories (they are fakes ones on s3) will downloaded to the directory specified in the form of directories and sub directories

<execution>
    <id>download-prefix</id>
    <phase>package</phase>
    <goals>
        <goal>s3-download</goal>
    </goals>
    <configuration>
        <bucket>your-bucket</bucket>
        <downloadPath>/path/to/local/</downloadPath>
        <keys>s3-prefix</keys>
    </configuration>
</execution>

The next part is about uploading files to s3.

Uploading one file

<execution>
    <id>upload-one</id>
    <phase>package</phase>
    <goals>
        <goal>s3-upload</goal>
    </goals>
    <configuration>
        <bucket>your-bucket</bucket>
        <path>/path/to/local/your-file.txt</path>
        <key>key-to-download.txt</key>
    </configuration>
</execution>

Upload a directory

<execution>
    <id>upload-one</id>
    <phase>package</phase>
    <goals>
        <goal>s3-upload</goal>
    </goals>
    <configuration>
        <bucket>your-bucket</bucket>
        <path>/path/to/local/directory</path>
        <key>prefix</key>
    </configuration>
</execution>

Upload to the root of the bucket.

<execution>
    <id>upload-multiples-files-no-key</id>
    <phase>package</phase>
    <goals>
        <goal>s3-upload</goal>
    </goals>
    <configuration>
        <bucket>your-bucket</bucket>
        <path>/path/to/local/directory</path>
    </configuration>
</execution>

That's it. You can now download and upload files to S3 during the Maven phase of your choice!