Writing and Reading Files Using Slices in Go

While tackling some programming tasks recently, I realized that no methods allow me to get slices of lines and save processed lines on disk. Why do I need such a package, and why does reading the whole text as a string or []byte from the os package not meet my requirements? 

The reason is quite simple: I have to process template source code line by line, ignoring commentaries. Therefore, it is easier to process single-line commentaries, especially if they start from the start of a line. And also, such line-by-line processing simplifies multi-line commentary processing. Of course, I could split a string into a slice of strings and use that slice, but I would like to have some reusable package once and use it efficiently whenever needed. 

Key Methods in gfu Package

The gfu package provides three primary methods for file manipulation with slices of strings:

1. ReadAllLines that reads file content and returns a tuple []string,error.
2. WriteAllLines to write a slice of strings.
3. AppendAllLines to add a slice of strings to the end of an existing file.

The New Package That Meets All Requirements

So after I decided what methods such a package should have, I wrote gfu (gfu stands for Go File Utils) package and would like to share them; see the GitHub repo. 

The gfu package could be used for solving the following tasks:

• Represent file content as a slice of lines ([]string); this representation could be used for language (not only programming but a natural, too) syntax analysis or some processing (ReadAllLines function).
• Create or replace file content with the required number of lines with specialized lines ending (CR, LF, or CRLF) using the WriteAllLines function.
• Add a set of lines to the end of the existing file using AppendAllLines function.

ReadAllText Method

This method is used for file reading as a slice of line; under the hood, it does the following:

1. Returns tuple ([]string, error) of result with auto-detect line ending (CR, LF, or CRLF).
2. Removes line-ending symbols from slice items.
3. Removes empty lines if omitEmpty argument is set to true.

Example:

lines, err := gfu.ReadAllLines("myFile.txt", true)

In the above example, if myFile.txt contains the following lines: Line1\n \n \n Line2\n, the lines slice will contain []string{"Line1", "Line2"}. If we pass false as the second parameter, the result will be []string{"Line1", "", "", "Line2"}. One important thing to note is that all line separators are removed from the slice of strings.

WriteAllText Method

This method is used for writing []string as a file content; under the hood, it does the following:

1. Inserts the line separator symbol defined in the third argument of WriteAllLines function.
2. Truncates a file if it exists; if not, a file will be created.

Example:

lines := []string{
    "{",
    "    \"id\": 1,",
    "    \"name\": \"Michael Ushakov\"",
    "}",
}
file := "write_all_lines_test.txt"
err := gfu.WriteAllLines(file, lines, "\n")

In the above example, if write_all_lines_test.txt does not exist, it will be created with the following lines:

{
    "id": 1,
    "name": "Michael Ushakov"
}

If we would like to set the Windows file ending, then we should pass \r\n as a latter parameter.

AppendAllText Method

WriteAllLines overwrites file content, but what should we do if we need to add some lines portion to an existing file? We should use the AppendAllLines function that, by signature, is identical to WriteAllLines:

lines := []string{
    "{",
    "    \"id\": 1,",
    "    \"name\": \"Michael Ushakov\"",
    "}",
}
file := "append_all_lines_test.txt"
err := gfu.WriteAllLines(file, lines, "\n")

additionalLines := []string{
    "{",
    "    \"id\": 2,",
    "    \"name\": \"Alex Petrov\"",
    "}",
}

err := gfu.AppendAllLines(file, lines, "\n")

In the above example, we create a new file with the name append_all_lines_test.txt with content equal to the previous (WriteAllLines) example, and after that, we're appending a new JSON object to the end of a file. Therefore, the result content is: 

{
    "id": 1,
    "name": "Michael Ushakov"
}
{
    "id": 2,
    "name": "Alex Petrov"
}

Conclusion

All these functions are quite convenient and combined in a small package. Tests have also been written for each function so we can consider them reliable. In conclusion, I could also say that my hypothesis was right, and I successfully solved my tasks with source code processing and generating source code from templates. One more thing that should be noted is the reading method (ReadAllLines) reads the entire file in one operation. Therefore, it would be inefficient to read a text file of hundreds of megabytes and more in memory, but we are solving tasks step by step, and early optimization is evil.