{{announcement.body}}
{{announcement.title}}

Cleaning MuleSoft CloudHub Resources: A DevOps Approach

DZone 's Guide to

Cleaning MuleSoft CloudHub Resources: A DevOps Approach

Let's look at how embracing DevOps principle to manage CloudHub resources eases the operational overhead.

· Integration Zone ·
Free Resource

Introduction

MuleSoft provides excellent business agility to companies by connecting applications, data, and devices, both on-premises and in the cloud with an API-led approach. Together, APIs and DevOps deliver greater business value than what they can deliver individually.

DevOps is the combination of cultural philosophies, practices, and tools that increases an organization's ability to deliver applications and services at high velocity: evolving and improving products over the time.

A common first step is to establish continuous integration processes, taking advantage of tools like Jenkins to automate repeatable aspects. The next step is to produce software artifacts efficiently, creating a complete pipeline that takes code from concept to production in a repeatable and secure fashion. The below write-up is one such example where a repeatable operational activity on the platform can be automated using the DevOps principles (we have used Jenkins here).

Problem

In any staged environment, the DEV and SIT should ideally be used by the developers to perform rapid development, unit tests, integration tests to ensure connectivity is fine with other end systems. But this generally does not happen in real practice and sometimes the consumers start integrating with the APIs available in DEV/SIT for their testing. However, this is supposed to happen in higher environments like QA or UAT, which ideally integrates with other business systems that are expected to contain production-like data.

Having APIs running on DEV/SIT for a long time brings an additional overhead of maintaining them which is more effort and challenging when this must be managed by a central Platform Support Team. Also, with the increase in the number of APIs getting developed and deployed to DEV/SIT, the cloud resources like the vCores get locked up unnecessarily, which otherwise could be allocated to the inflight APIs that are in development phase.

Solution

A simple solution is to adopt a strategy where the consumers should be allowed to use the APIs in QA for their testing and not from DEV/SIT (this should be ideal in our view). The DEV/SIT should be a floating environment, as in the APIs should be removed post they are moved to QA (or Production) and are stable there with no more recent changes anticipated. Remember that the CI/CD process for API build and deployment should be in place in case if the APIs are to be made available in DEV/SIT anytime on short notice. Only inflight APIs (which are in development phase) should be allocated the resources in DEV/SIT for longer. The challenge is in managing this piece of work that involves auditing, cleaning up those APIs to deallocate vCores, informing the concern development team that their APIs are being removed from the environments, etc. Thanks to the Anypoint Platform APIs and Jenkins, we have created a scripted pipeline logic that orchestrates the platform APIs to seamlessly perform this task. We run this script from Jenkins every night to ensure that the vCores are released out of those APIs that meet our criteria.

Pipeline Code

pipeline {
   agent any

   stages { 

/**
*Initialize pipeline variables:
*prodAPIList - Array that will contain APIs deployed in Production
*appsDeletedFromDEV - Array that will contain APIs deleted from DEV
*appsDeletedFromSIT - Array that will contain APIs deleted from SIT
*appsToIgnore - Configure the names of APIs that shouldn't be part of this logic. Used to specify those critical APIs that should never be deleted.
*devEnvironmentID - ARM Environment ID for DEV
*sitEnvironmentID - ARM Environment ID for SIT
*prodEnvironmentID - ARM Environment ID for PROD
*/

stage('Initialize') {
            steps {
                script {
                    prodAPIList = [] 
                    appsDeletedFromDEV = [] 
                    appsDeletedFromSIT = [] 
                    appsToIgnore = ["XXXX"] 
                    devEnvironmentID = "XXXX" 
                    sitEnvironmentID = "XXXX" 
                    prodEnvironmentID = "XXXX" 
                }
            }
        }

/**
*Invoke the Login API to generate the access_token.
*Note: It is recommended to inject the platform credentials through Jenkins Credentials Plugin
*/

        stage('Login to ARM') {
            steps {
                script {
                    def loginContents = httpRequest consoleLogResponseBody: true, contentType: 'APPLICATION_JSON', httpMode: 'POST', ignoreSslErrors: true, requestBody: '{"username":"XXXX","password":"XXXX"}', responseHandle: 'NONE', timeout: 30000, url: 'https://anypoint.mulesoft.com/accounts/login', validResponseCodes: '200'
                    authToken = 'Bearer ' + new groovy.json.JsonSlurper().parseText(loginContents.getContent()).access_token;
                }
            }
        }

/**
*Invoke Platform API to fetch applications deployed in PROD environment.
*Filter those applications that are not updated for more than 15 days.
*/

        stage('Fetch Production Applications') {
            steps {
                script {
                    def apiResponse = httpRequest(customHeaders: [[name: 'Authorization', value: authToken], [name: 'X-ANYPNT-ENV-ID', value: prodEnvironmentID]], ignoreSslErrors: true, responseHandle: 'STRING', timeout: 30000, url: 'https://anypoint.mulesoft.com/cloudhub/api/applications', validResponseCodes: '200')
                    def parseResponse = new groovy.json.JsonSlurper().parseText(apiResponse.getContent())
                    parseResponse.each {
                        int daysBetween = calculateDaysBetween(it.lastUpdateTime)
                        if(daysBetween > 15){
                            prodAPIList << it.domain
                        }
                    }
                }
            }
        }

/**
*Invoke Platform API to fetch application deployed in DEV environment.
*For every applications fetched:
*- Determine if the application is not updated since last 15 days
*- Determine if the application is part of the list which shouldn't be deleted ever
*- Delete the application otherwise
*/

        stage('Clean DEV vCores) {
            steps {
                script {

                    def jsonParser= new groovy.json.JsonSlurper()
                    for(item in prodAPIList){

def devAPIName = "dev"+item //The application deployed in DEV environment is always prefixed with "dev"
                        def apiResponse = httpRequest(customHeaders: [[maskValue: false, name: 'Authorization', value: authToken], [maskValue: false, name: 'X-ANYPNT-ENV-ID', value: devEnvironmentID]], ignoreSslErrors: true, responseHandle: 'STRING', url: 'https://anypoint.mulesoft.com/cloudhub/api/applications/'+devAPIName, validResponseCodes: '200,404')

if(apiResponse.toString()=="Status: 404" || item in appsToIgnore) {
                            continue
                        } else {
                            def appLastUpdateTime = jsonParser.parseText(apiResponse.getContent()).lastUpdateTime
                            int daysBetween = calculateDaysBetween(appLastUpdateTime)
                            if(daysBetween > 15){
                                def deleteAPIResponse = httpRequest(contentType: 'APPLICATION_JSON', httpMode: 'DELETE', customHeaders: [[maskValue: false, name: 'Authorization', value: authToken], [maskValue: false, name: 'X-ANYPNT-ENV-ID', value: devEnvironmentID]],  ignoreSslErrors: true, responseHandle: 'STRING', url: 'https://anypoint.mulesoft.com/cloudhub/api/applications/'+devAPIName, validResponseCodes: '204,200')
                                appsDeletedFromDEV << devAPIName
                            }
                        }
                    }
                }
            }
        }

/**
*Invoke Platform API to fetch application deployed in SIT environment.
*For every applications fetched:
*- Determine if the application is not updated since last 15 days
*- Determine if the application is part of the list which shouldn't be deleted ever
*- Delete the application otherwise
*/

        stage('Clean SIT vCores') {
            steps {
                script {
                    def jsonParser= new groovy.json.JsonSlurper()

                    for(item in prodAPIList){

                        def sitAPIName = "sit"+item //The application deployed in SIT environment is always prefixed with "sit"
                        def apiResponse = httpRequest(customHeaders: [[maskValue: false, name: 'Authorization', value: authToken], [maskValue: false, name: 'X-ANYPNT-ENV-ID', value: sitEnvironmentID]], ignoreSslErrors: true, responseHandle: 'STRING', url: 'https://anypoint.mulesoft.com/cloudhub/api/applications/'+sitAPIName, validResponseCodes: '200,404')

if(apiResponse.toString()=="Status: 404" || item in appsToIgnore) {
                            continue
                        } else {
                            def appLastUpdateTime = jsonParser.parseText(apiResponse.getContent()).lastUpdateTime
                            int daysBetween = calculateDaysBetween(appLastUpdateTime)
                            if(daysBetween > 15){
                                def deleteAPIResponse = httpRequest(contentType: 'APPLICATION_JSON', httpMode: 'DELETE', customHeaders: [[maskValue: false, name: 'Authorization', value: authToken], [maskValue: false, name: 'X-ANYPNT-ENV-ID', value: 'sitEnvironmentID']],  ignoreSslErrors: true, responseHandle: 'STRING', url: 'https://anypoint.mulesoft.com/cloudhub/api/applications/'+sitAPIName, validResponseCodes: '204,200')
                                appsDeletedFromSIT << sitAPIName
                            }
                        }
                    }
                }
            }
        }
    }
}

/**
*Function that calculates the days between the supplied lastTimestamp and current time.
*/
public int calculateDaysBetween(long lastTimestamp) {
    def currentDatetime= System.currentTimeMillis()
    long diff = currentDatetime - lastTimestamp
    int diffDays = diff / (24 * 60 * 60 * 1000)
    return diffDays
}

Rules

There may be a question on how should we decide that an API can be removed from DEV/SIT on the fly. Well, we applied below set of rules (as you may infer from the above pipeline) to determine that. Note that these rules can be altered depending on your constraints and might require you to update the values in the above logic aptly

  • The API should be deployed and running in Production for more than 15 days.
  • There should be a QA (or UAT) environment where the same API is running. This instance will be used by the consumers to perform testing any time. (Note that this is an assumption and is not a check implemented in the above logic as our pipeline that performs the build and deployment is sure to take the API through QA without fail. So any API available in Production should be available in QA)
  • The API should be available in DEV/SIT and is not updated for more than 15 days

Note: The value of 15 days is something that worked for us but may have to be analyzed before being used elsewhere.

Conclusion

This solution shows how easy it is to embrace and adopt the culture of automation to manage environments and resources more aptly. DevOps processes and automation not only enables us to deliver projects quickly but also helps us to operate the environments and APIs more efficiently.

Topics:
mulesoft ,jenkins pipeline ,devops ,cloud resource management ,ipaas ,integration

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}