How To Scan GCP Storage Files for Threats Using Go

As enterprise cloud storage solutions steadily gain momentum across global markets, the anti-virus and malware security policies deployed to protect these pay-per-scale services become more and more robust. Naturally, the taller the castle wall becomes, the higher the siege ladder is built to climb it, but the perpetual battle between threat actors and cloud security practitioners appears to march us ever closer toward a safer digital future.

That said, the catastrophic damage from successful cloud storage attacks cannot be understated, and the evolution of this attack vector should never be underestimated. Public, private, and hosted cloud storage services are all extremely worthwhile targets for threat actors, playing host to valuable data that moves frequently between user devices, and a variety of potent attack vectors are regularly explored to penetrate networks through these locations. Threat actors can leverage well-known techniques like phishing to goad users into saving and storing malicious content in a public location, and they can disguise malware very effectively in direct client upload scenarios through built-in file features like Office macros or PDF password-protection measures.  

If in-storage threat detection policies don’t perform scans quickly enough, or if they aren’t configured to flag disguised file threats, malicious files can be rapidly distributed to unsuspecting users who, upon opening them, unwittingly initiate sudden and unpredictable attacks. It’s critical to always view cloud storage security as a multi-policy effort, with more than one integrated security solution in place to increase the likelihood that file upload threats are detected rapidly after reaching a storage instance.

Google Cloud Platform (GCP) Storage, which occupies roughly 10-13% of the global cloud storage market share compared to its competitors, is one of many popular and fast-growing solutions which benefit from the integration of external threat detection policies. Much like its competitors, GCP makes it easy for developers to integrate additional security measures alongside its built-in security architecture. In the remainder of this article, I’ll demonstrate one free-to-use API solution which can be quickly integrated to incorporate a wide range of threat detection policies against file uploads in any given GCP bucket.

Demonstration

The cloud storage scanning API provided below is designed to integrate with a Google Cloud Storage Bucket and scan files using the basic information available in any GCP admin account. This solution is equipped to reference files against a frequently updated list of virus and malware signatures (currently totaling more than 17 million signatures, including trojans, ransomware, and spyware), and it’s also capable of detecting a variety of increasingly common hidden threat types through in-depth content verification. These hidden content threat types include executables, invalid files, scripts, password-protected files, macros, XML files (including XXE threats), Insecure Deserialization objects, and HTML input, and each one of these threats can be blocked or allowed by setting custom policies (using Boolean values in the API request body).  

For storage use cases involving direct user file uploads to GCP buckets, you can additionally set custom restrictions against unwanted file types by supplying a comma-separated list of accepted file extensions in the API request. When this policy is set, each file upload will be verified against your list of accepted file extensions.

To integrate this API with your GCP bucket, you’ll need to set the following information in your requests:

1. Bucket name: The name of your bucket in GCP storage
2. Object name: The name of the objects or files in GCP storage
3. JSON credential file: The Service Account credential for Google Cloud, stored in a JSON file

Files that contain virus or malware signatures and/or violate the custom threat policies set in the API request body will receive a CleanResult: False response from the underlying security service, with a sub-second typical response time. This CleanResult Boolean can be used, for example, to separate clean and infected files into additional GCP buckets for eventual deletion or file quarantine/threat analysis.

Your API request can be structured using a variety of common programming languages, and in this article, I’ll provide Go code examples for your convenience.

To structure your request, use the below ready-to-run code examples and input the relevant GCP bucket information in the labeled headers. To authorize your request, copy a free-tier API key (these allow 800 API calls per month and no additional commitment) into the API Key header.

package main

import (
     "fmt"
     "bytes"
     "mime/multipart"
     "os"
     "path/filepath"
     "io"
     "net/http"
     "io/ioutil"
)

func main() {

     url := "https://api.cloudmersive.com/virus/scan/cloud-storage/gcp-storage/single/advanced"
     method := "POST"

     payload := &bytes.Buffer{}
     writer := multipart.NewWriter(payload)
     file, errFile1 := os.Open("/path/to/file")
     defer file.Close()
     part1,
         errFile1 := writer.CreateFormFile("jsonCredentialFile",filepath.Base("/path/to/file"))
     _, errFile1 = io.Copy(part1, file)
     if errFile1 != nil {
          fmt.Println(errFile1)
          return
     }
     err := writer.Close()
     if err != nil {
          fmt.Println(err)
          return
     }


     client := &http.Client {
     }
     req, err := http.NewRequest(method, url, payload)

     if err != nil {
          fmt.Println(err)
          return
     }
     req.Header.Add("bucketName", "<string>")
     req.Header.Add("objectName", "<string>")
     req.Header.Add("allowExecutables", "<boolean>")
     req.Header.Add("allowInvalidFiles", "<boolean>")
     req.Header.Add("allowScripts", "<boolean>")
     req.Header.Add("allowPasswordProtectedFiles", "<boolean>")
     req.Header.Add("allowMacros", "<boolean>")
     req.Header.Add("allowXmlExternalEntities", "<boolean>")
     req.Header.Add("restrictFileTypes", "<string>")
     req.Header.Add("Content-Type", "multipart/form-data")
     req.Header.Add("Apikey", "YOUR-API-KEY-HERE")

     req.Header.Set("Content-Type", writer.FormDataContentType())
     res, err := client.Do(req)
     if err != nil {
          fmt.Println(err)
          return
     }
     defer res.Body.Close()

     body, err := ioutil.ReadAll(res.Body)
     if err != nil {
          fmt.Println(err)
          return
     }
     fmt.Println(string(body))
}

I’d recommend setting all the custom threat detection policies to “False” to get the most out of this solution – it can make a huge difference in improving your GCP bucket’s threat profile.