Skip to content

Cloudwatch max batchSize was increased from 20 to 1000 #3376

Closed

Description

@peterfouquet0001

With AWS SDK 2.17.249 the PutMetricData request supports a batch size of 1000 records.
The batch size in mircometer CloudWatchConfig is only 20.

Current CloudWatchConfig
int MAX_BATCH_SIZE = 20;

Change in AWS SDK 2.17.249 ()
"input":{"shape":"PutMetricDataInput"},
"documentation":"

Publishes metric data points to Amazon CloudWatch. CloudWatch associates the data points with the specified metric. If the specified metric does not exist, CloudWatch creates the metric. When CloudWatch creates a metric, it can take up to fifteen minutes for the metric to appear in calls to <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_ListMetrics.html\">ListMetrics.

You can publish either individual data points in the Value field, or arrays of values and the number of times each value occurred during the period by using the Values and Counts fields in the MetricDatum structure. Using the Values and Counts method enables you to publish up to 150 values per metric with one PutMetricData request, and supports retrieving percentile statistics on this data.

Each PutMetricData request is limited to 1 MB in size for HTTP POST requests. You can send a payload compressed by gzip. Each request is also limited to no more than 1000 different metrics.

Activity

  1. shakuzen commented on Aug 29, 2022

    @shakuzen
    Member

    Do you know if there is a way we can programmatically check the max batch size with the AWS SDK?

  2. added this to the milestone on Aug 29, 2022
  3. peterfouquet0001 commented on Aug 29, 2022

    @peterfouquet0001
    Author

    I don't know how to check these value from the aws sdk at runtime.

  4. shakuzen commented on Aug 30, 2022

    @shakuzen
    Member

    It looks like this commit has the related changes. There are other changes to the constraints we should update. I wonder if the AWS SDK does any validation or if validation is only done server-side. If it is done server-side, then there's less issue with us updating limits in the Micrometer CloudWatch module, but if the SDK client does validation, then users will have issues if they are using an older version of the client that doesn't have the linked changes.

  5. shakuzen commented on Aug 30, 2022

    @shakuzen
    Member

    My testing leads me to believe there isn't validation client-side, so we should be okay to update this without a concern of SDK version mismatch.

  6. changed the title [-]Cloudwatch PutMetricData supports greater batchSize since AWS SDK 2.17.249[/-] [+]Cloudwatch max batchSize was increased from 20 to 1000[/+] on Aug 30, 2022
  7. modified the milestones: , 1.10.0-M5 on Aug 30, 2022
  8. added a commit that references this issue on Aug 30, 2022
    2aba568
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions