Push CloudHub logs to Amazon S3

Logging is a powerful aid for understanding and debugging programs run-time behaviour. Logs capture and persist essential data and make it available for analysis at any point in time. When apps are deployed on CloudHub, logging is taken care of by Mulesoft itself. Application logs are kept for 30 days or up to 100 MB whichever comes first. Further, if the application is deleted, logs are also gone. Often, there is a requirement to keep the logs for a longer time for auditing or archiving purposes.

This can be achieved either by using external logging tools like Splunk or if there is no external logging system, can be pushed to Amazon S3 (an external storage system).

This article would develop an ad hoc application to fetch logs for any application (passed as a query parameter) from CloudHub and push to Amazon S3.

Prerequisites

1. Amazon S3 connector: Download/Add it from Exchange.
2. Application deployed to CloudHub: One is enough.
3. AWS account: Sign Up to AWS or login
4. AWS Access and Secret Key: To create them, login to AWS account, click username > Credentials -> Access Keys -> Create new Access key and download the file

Without further ado, let’s start.

1. Configure the source for flow to trigger the application. Create a new variable to save the application name received in the query parameter.
2. To fetch logs from CloudHub, we first need to sign in to Cloudhub using Access Management API. For that, we need to pass Mulesoft Anypoint username and password in the payload when calling the API.

output application/JSON

{
    "username": "anypoint-username",
    "password": "anypoint-password"
}

3. Configure the HTTP Requester to hit https://anypoint.mulesoft.com/accounts/login and get the access token.  Save the response in a variable using the target variable.

4. Next, add HTTP Requester and hit https://anypoint.mulesoft.com/accounts/api/me to get the organization ID. Pass the access token received in the previous step in the Authorization header.

Here, we are saving the organization ID in orgPayload variable.

5. To get the environment Id of the environment in which our application is deployed, send a GET request to https://anypoint.mulesoft.com/accounts/api/organizations/{orgId}/environments

Pass query parameter name to get an ID for the Sandbox environment only where our application is deployed.

Create a variable to save environment ID using the following expression:- vars.envPayload.data.id[0]

6.Send a request to https://anypoint.mulesoft.com/cloudhub/api/v2/applications/{app-name}/deployment to get application deployments. Add X-ANYPNT-ENV-ID header to pass environment id.

The process to fetch logs is complete now. We would need to send the logs to Amazon S3.

1. Add a new Amazon S3 configuration.

2. Drag and Drop Create Object operation from the palette and provide the bucket name and file name (called key here).

Here’s how to push CloudHub logs to Amazon S3; this can help when AWS subscription is already there and setting up an external logging system is not feasible or required.