AWS EC2 Monitoring Extension

Use Case

Captures statistics for EC2 instances from Amazon CloudWatch and displays them in the AppDynamics Metric Browser.

Prerequisites

1. Please give the following permissions to the account being used to with the extension.

cloudwatch:ListMetrics
cloudwatch:GetMetricStatistics
ec2:describeinstances

2. Before the extension is installed, the prerequisites mentioned here need to be met. Please do not proceed with the extension installation if the specified prerequisites are not met.

3. Download and install Apache Maven which is configured with Java 8 to build the extension artifact from source. You can check the java version used in maven using command mvn -v or mvn --version. If your maven is using some other java version then please download java 8 for your platform and set JAVA_HOME parameter before starting maven.

4. The extension needs to be able to connect to AWS Cloudwatch in order to collect and send metrics. To do this, you will have to either establish a remote connection in between the extension and the product, or have an agent on the same machine running the product in order for the extension to collect and send the metrics.

Installation

1. Clone the "aws-ec2-monitoring-extension" repo using git clone <repoUrl> command.
2. Run 'mvn clean install' from aws-ec2-monitoring-extension
3. Copy and unzip AWSEC2Monitor-<version>.zip from 'target' directory into <machine_agent_dir>/monitors/
4. Edit config.yml file in AWSEC2Monitor and provide the required configuration (see Configuration section)
5. Restart the Machine Agent.

Please place the extension in the "monitors" directory of your Machine Agent installation directory. Do not place the extension in the "extensions" directory of your Machine Agent installation directory.

Configuration

In order to use the extension, you need to update the config.yml file that is present in the extension folder. The following is an explanation of the configurable fields that are present in the config.yml file.

config.yml

Note: Please avoid using tab (\t) when editing yaml files. You may want to validate the yaml file using a yaml validator.

Below is an example config for monitoring multiple accounts and regions:

accounts:
  - awsAccessKey: "XXXXXXXX1"
    awsSecretKey: "XXXXXXXXXX1"
    displayAccountName: "TestAccount_1"
    regions: ["us-east-1","us-west-1","us-west-2"]

  - awsAccessKey: "XXXXXXXX2"
    awsSecretKey: "XXXXXXXXXX2"
    displayAccountName: "TestAccount_2"
    regions: ["eu-central-1","eu-west-1"]

credentialsDecryptionConfig:
    enableDecryption: "false"
    decryptionKey:
    
proxyConfig:
    host:
    port:
    username:
    password:
    
ec2InstanceNameConfig:
    useNameInMetrics: "true"
    tagKey: "Name"

#Filter the instances by tag name. tagValue is optional. if tagValue is not provided, we will fetch all the instances with the tagName.
tags:
  - name: "Name"
    #value: ["tag1", "tag2", "tag3"]

# Global metrics config for all accounts
metricsConfig:

    # By default, all metrics retrieved from cloudwatch are 'Average' values.	
    # This option allows you to override the metric type. 
    #
    # Allowed statTypes are: ave, max, min, sum, samplecount
    #
    # Note: Irrespective of the metric type, value will still be reported as
    # Observed value to the Controller
    includeMetrics:
       - name: "CPUUtilization"
         alias: "CPUUtilization"
         statType: "ave"
         aggregationType: "OBSERVATION"
         timeRollUpType: "CURRENT"
         clusterRollUpType: "COLLECTIVE"
         delta: false
         multiplier: 1
       - name: "NetworkOut"
       - name: "NetworkIn"

    metricsTimeRange:
      startTimeInMinsBeforeNow: 10
      endTimeInMinsBeforeNow: 0

    # Rate limit ( per second ) for GetMetricStatistics, default value is 400. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.html
    getMetricStatisticsRateLimit: 400

    # 
    # The max number of retry attempts for failed retryable requests
    # (ex: 5xx error responses from a service) or throttling errors
    #
    maxErrorRetrySize: 0


#Allowed values are Basic and Detailed. Refer https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html for more information
# Basic will fire CloudWatch API calls every 5 minutes
# Detailed will fire CloudWatch API calls every 1 minutes
cloudWatchMonitoring: "Basic"

#If you want any other interval ( other than the mentioned values in the above configuration ) configure it here, if not leave it empty. This value is in minutes
cloudWatchMonitoringInterval: 0

concurrencyConfig:
  noOfAccountThreads: 3
  noOfRegionThreadsPerAccount: 3
  noOfMetricThreadsPerRegion: 3
  #Thread timeout in seconds
  threadTimeOut: 30

regionEndPoints:
  ap-southeast-1: monitoring.ap-southeast-1.amazonaws.com
  ap-southeast-2: monitoring.ap-southeast-2.amazonaws.com
  ap-northeast-1: monitoring.ap-northeast-1.amazonaws.com
  eu-central-1: monitoring.eu-central-1.amazonaws.com
  eu-west-1: monitoring.eu-west-1.amazonaws.com
  us-east-1: monitoring.us-east-1.amazonaws.com
  us-west-1: monitoring.us-west-1.amazonaws.com
  us-west-2: monitoring.us-west-2.amazonaws.com
  sa-east-1: monitoring.sa-east-1.amazonaws.com

#prefix used to show up metrics in AppDynamics. This will create this metric in all the tiers, under this path
#metricPrefix: "Custom Metrics|Amazon EC2|"

#This will create it in specific Tier/Component. Make sure to replace <COMPONENT_ID> with the appropriate one from your environment.
#To find the <COMPONENT_ID> in your environment, please follow the screenshot https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-troubleshoot-missing-custom-metrics-or-extensions/ta-p/28695
metricPrefix: "Server|Component:<COMPONENT_ID>|Custom Metrics|Amazon EC2|"

Configuration Steps

1. Configure the "COMPONENT_ID" under which the metrics need to be reported. This can be done by changing the value of <COMPONENT_ID> in
   metricPrefix: "Server|Component:<COMPONENT_ID>|Custom Metrics|Amazon EC2|".

   For example,

   metricPrefix: "Server|Component:100|Custom Metrics|Amazon EC2|"
   

More details around metric prefix can be found here.

2. Configure "awsAccessKey" and "awsSecretKey". If you are running this extension inside an EC2 instance which has IAM profile configured then you don't have to configure these values, extension will use IAM profile to authenticate.
3. Configure "regions". Extension collects metrics from all the regions configured here.
4. If you want to encrypt the "awsAccessKey" and "awsSecretKey" then follow the "Credentials Encryption" section and provide the encrypted values in "awsAccessKey" and "awsSecretKey". Configure "enableDecryption" of "credentialsDecryptionConfig" to true and provide the encryption key in "encryptionKey"

Metrics

Typical metric path: Application Infrastructure Performance|<Tier>|Custom Metrics|Amazon EC2|<Account Name>|<Region>|Instance|<instance id or name> followed by the metrics defined in the link below:

• EC2 Metrics

Credentials Encryption

Please visit this page to get detailed instructions on password encryption. The steps in this document will guide you through the whole process.

Once you encrypt the access key and secret key add the excrypted values to the awsAccessKey and awsSecretKey respectively. Based on the "enableDecryption" in "credentialsDecryptionConfig" section extension will decide if it has to decrypt the keys or not. Provide the encryption key used to encrypt in the "encryptionKey" of "credentialsDecryptionConfig".

Extensions Workbench

Workbench is an inbuilt feature provided with each extension in order to assist you to fine tune the extension setup before you actually deploy it on the controller. Please review the following document on How to use the Extensions WorkBench

Troubleshooting

Please follow the steps listed in this troubleshooting-document in order to troubleshoot your issue. These are a set of common issues that customers might have faced during the installation of the extension.

Contributing

Always feel free to fork and contribute any changes directly here on GitHub.

Version

Note: While extensions are maintained and supported by customers under the open-source licensing model, they interact with agents and Controllers that are subject to AppDynamics’ maintenance and support policy. Some extensions have been tested with AppDynamics 4.5.13+ artifacts, but you are strongly recommended against using versions that are no longer supported.