Resolve Encoding Issues of Resource Files in Java Projects

In Java projects, resource files play a crucial role in storing and managing application data, such as localization strings, configuration settings, and other static content. However, working with resource files can sometimes lead to encoding issues, which can cause problems with text display and processing.  

In the first place, let us take a look at the definition of encoding. It refers to the process of representing characters in a specific format using bytes. Java uses Unicode as its character set, which supports a wide range of characters from various languages and scripts. 

If you experience an encoding issue within your Java project, you might see the following Java exception.

java.nio.charset.MalformedInputException: Input length = 1

MalformedInputException exceptions show up if an input byte sequence is not legal for given charset or an input character sequence is not a legal sixteen-bit Unicode sequence, according to the definition of the Oracle JavaDoc regarding Java 8. For years, this kind of exception is mentioned in online comments of different communities such as StackOverflow.  In principle, we can define three causes.

Causes of Encoding Issues May Be

Garbled or Incorrectly Displayed Text: When a resource file is not encoded correctly, the text it contains may appear garbled or incorrectly displayed. This issue often manifests as a series of strange characters or question marks instead of the expected text. Dealing with resource files, especially those containing non-ASCII characters, encoding issues may arise if the chosen encoding format is not compatible.

Let us take a quick look at the following sample: Assume we want to read external resources (files) within our Java-based Maven project. The project has specified a character encoding scheme UTF-8. To specify the character encoding scheme, we set the following in the POM:

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

Another way to set the default (file) encoding for Java is to use an environment variable:

JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF-8"

In this case, we would experience an MalformedInputException exception. One way to resolve the problem, open the resource in a text editor as Notepad++ and save the file again with the code format UTF-8.

By the way, special care has to be taken if you are filtering properties files. If your filtered properties files include non-ascii characters and your project.build.sourceEncoding is set to anything other than ISO-8859-1, you might be affected by MalformedInputException exceptions. 

When properties files are used as ResourceBundles, the encoding required differs between versions of Java. Up to and including Java 8, these files are required to use ISO-8859-1 encoding.

Starting with Java 9, the preferred encoding is UTF-8 for property resource bundles. It might work with ISO-8859-1, but as you can see in the Internationalization Enhancements in JDK 9 documentation, you should consider converting your property resource bundles into UTF-8 encoding. To define the encoding format, check out the following sample: 

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-resources-plugin</artifactId>
        <version>3.3.1</version>
        <configuration>
          ...
          <propertiesEncoding>ISO-8859-1</propertiesEncoding>
          ...
        </configuration>
      </plugin>
    </plugins>
    ...
  </build>

Another way to handle the exception we want to specify the files that we want to exclude, including the file with the wrong code format. For instance, the POM might look like:

<resources>
      <resource>
        <directory>[your directory]</directory>
        <excludes>
          <exclude>[non-resource file #1]</exclude>
          <exclude>[non-resource file #2]</exclude>
          <exclude>[non-resource file #3]</exclude>
          ...
          <exclude>[non-resource file #n]</exclude>
        </excludes>
      </resource>
      ...
    </resources>

Reading or Writing Issues: Incorrect encoding can also lead to problems when reading or writing to resource files. Reading a file with the wrong encoding can result in data corruption or loss, while writing to a file with incompatible encoding may produce unexpected results or render the file unusable.

Let us check out a sample. In this sample, we have a program in Java that reads through a directory's text-based files. The line of code would look like the following:

BufferedReader reader = Files.newBufferedReader(file,Charset.forName("UTF-8"));

This line of code would create a MalformedInputException exception. To avoid the exception, we rewrite the line of code as follows:

new BufferedReader(new InputStreamReader(new FileInputStream("a.txt"),"utf-8"));

The first line uses CharsetDecoder default action. The default action formalformed-input and unmappable-character errors is to report them, while the second line uses the REPLACE action. Another solution could be changing the charset to ISO-8859-1.

Compatibility with External Systems: If your Java project interacts with external systems or APIs that have specific encoding requirements, incorrect encoding in resource files can cause compatibility issues. Data sent or received from these systems may be misinterpreted, leading to communication failures or incorrect processing of information. Let us check some examples regarding a Jenkins server:  The exception occurs when the following situation occurs:

• Jenkins Primary system is set to accept UTF-8 characters.
• Jenkins Build Agent is set to return the ANSI character set.
• When Snyk tries to return a UTF-8 character from the build agent to the primary system, it fails to convert to UTF-8 and dies with the MalformedInputException.

As a solution, set the environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8 and restart the Jenkins agent process.

new BufferedReader(new InputStreamReader(new FileInputStream("a.txt"),"utf-8"));

Solution Strategies

• Specify the Correct Encoding: Ensure that you specify the correct encoding when reading or writing resource files. Use UTF-8 as the default encoding in most cases, as it supports a wide range of characters and is widely compatible. However, if you are working with legacy systems or have specific requirements, consult the relevant documentation to determine the appropriate encoding.
• Configure the Build System: If your resource files are part of a build system, such as Maven or Gradle, make sure to configure the encoding settings correctly. Specify the desired encoding in the build configuration file (e.g., pom.xml for Maven), ensuring that it aligns with the encoding used in your resource files.
• Verify and Convert Existing Files: Inspect your existing resource files to ensure that they are encoded correctly. Use tools like native2ascii or iconv to convert files from one encoding to another if necessary. Be cautious when converting files, as incorrect usage can lead to further issues. Always make a backup before performing any conversion.
• Use Encoding-Aware Libraries: When working with resource files, utilize encoding-aware libraries to read and write data. Libraries such as Apache Commons IO provide convenient methods for handling encoding issues, allowing you to specify the desired encoding explicitly.
• Test and Validate: Regularly test and validate your resource files across different platforms and environments to ensure proper encoding compatibility. Verify that the text is displayed correctly and the files can be read and written without any issues.

Conclusion

Correctly managing encoding issues in resource files is crucial for Java projects, particularly when dealing with multilingual applications or systems with specific encoding requirements. By understanding the common encoding issues and implementing the solutions mentioned above, you can ensure that your resource files are accurately encoded, leading to seamless text display, proper data processing, and improved compatibility with external systems.