What this usually means
At its core, 'missing custom metrics' means the data you sent to CloudWatch either never arrived, arrived with a problem, or arrived but is stored in a way you aren't looking. The CloudWatch API PutMetricData is asynchronous and eventually consistent—it can accept data silently even if the metric won't appear. Common underlying causes: namespace mismatch (you send to 'MyApp' but look under 'CustomNamespace'), timestamp too old (more than 14 days in the past or 2 hours in the future), storage resolution defaults to 1-minute but you query with a different aggregation period, or the IAM role doesn't have cloudwatch:PutMetricData. For cross-account scenarios, the destination account must have the appropriate CloudWatch cross-account access configured.
The first ten minutes — establish facts before touching code.
- 1Run aws cloudwatch list-metrics --namespace YOUR_NAMESPACE --region us-east-1 to verify the metric exists. If empty, the data didn't arrive or namespace is wrong.
- 2Check the exact PutMetricData API call (e.g., from CloudTrail or application logs). Confirm the metric name, namespace, and dimensions match what you're querying.
- 3Verify the timestamp in the metric data: it must be within the last 14 days and not in the future beyond 2 hours. Use aws cloudwatch get-metric-statistics with a wide time range.
- 4If using a CloudWatch agent on EC2, check /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log for errors like 'Failed to put metrics' or 'Unable to assume role'.
- 5In the CloudWatch console, check the 'Custom Namespaces' section specifically, not just 'All metrics'. Your custom namespace may not appear in the default browse list.
- 6For Lambda: verify the IAM execution role has cloudwatch:PutMetricData. Add it if missing and redeploy.
- 7Use the CloudWatch API directly: aws cloudwatch put-metric-data --namespace TestNS --metric-data '[{"MetricName":"TestMetric","Value":1,"Timestamp":"2025-01-01T00:00:00Z"}]' and then query immediately with a 5-minute delay.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchCloudTrail event history for PutMetricData API calls to confirm they are being made
- searchCloudWatch Logs for the application that sends metrics (e.g., /var/log/myapp/metrics.log)
- search/var/log/amazon/amazon-cloudwatch-agent.log on EC2 instances running the CloudWatch agent
- searchIAM policy for the role/user sending metrics: ensure cloudwatch:PutMetricData is allowed
- searchCloudWatch console → Metrics → Custom Namespaces (browse all custom namespaces explicitly)
- searchAWS CloudWatch GetMetricStatistics API with explicit parameters to bypass eventual consistency
- searchCross-account monitoring: source account CloudWatch metrics → 'Cross-account metrics' section if using monitoring account setup
Practical causes, not theory. These are the things you will actually find.
- warningNamespace mismatch: sending to 'MyApp' but querying 'Custom/MyApp' or vice versa
- warningTimestamp out of range: metric timestamp is more than 2 hours in the future or 14 days in the past
- warningDimensions mismatch: metric was sent with 'Environment=prod' but query filters include 'Environment=dev'
- warningStorage resolution confusion: data sent at 1-minute resolution but query uses 60-second period with 'SUMS' aggregation—works, but 'AVERAGE' might show nothing if single point
- warningIAM permissions: the principal sending the metric lacks cloudwatch:PutMetricData on the namespace
- warningCloudWatch agent configuration: the agent is configured but not sending metrics due to incorrect config file or network issues
- warningCross-account: destination account does not have the necessary CloudWatch cross-account subscription or the source account is not sharing metrics properly
Concrete fix directions. Pick the one that matches your root cause.
- buildStandardize namespace: use a consistent namespace across all code and configuration, e.g., 'Custom/AppName' and stick to it
- buildFix timestamp generation: ensure the application uses the correct system time (NTP synced) and sends timestamps in ISO 8601 UTC format
- buildAlign dimensions: define a fixed set of dimensions for each metric in your application; log the exact dimensions sent
- buildGrant proper IAM permissions: attach a policy with 'cloudwatch:PutMetricData' to the IAM role/user that sends metrics
- buildAdjust storage resolution: if using high-resolution metrics (1-second), ensure the PutMetricData call includes 'StorageResolution=1'
- buildFor CloudWatch agent: run sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status to verify agent health; restart if needed
- buildCross-account: in the source account, create a metric stream to a monitoring account, or use CloudWatch cross-account observability
A fix you cannot prove is a guess. Close the loop.
- verifiedRun aws cloudwatch list-metrics --namespace YOUR_NAMESPACE and confirm the metric name appears
- verifiedUse aws cloudwatch get-metric-statistics --namespace YOUR_NAMESPACE --metric-name YourMetric --statistics Sum --period 60 --start-time <2 hours ago> --end-time <now>; expect a non-empty Datapoints array
- verifiedCheck the CloudWatch console graph for the metric: set the time range to the last 1 hour and verify data points appear
- verifiedIf using an alarm, set it to 'Treat missing data as missing' and then observe the alarm state transitions after sending new data
- verifiedReview CloudTrail PutMetricData events to confirm the exact API call succeeded
- verifiedFor agent-based metrics, tail the agent log and look for 'Successfully put metrics' messages
Things that make this bug worse or harder to find.
- warningDo not spam metrics with the same name but different dimensions—it can lead to throttling and data loss
- warningDo not rely on the CloudWatch console immediately: it has a 15-minute delay for custom metrics
- warningDo not forget that metrics are stored by namespace, metric name, and dimensions—if you change any of these, it's a new metric
- warningDo not set timestamps to the local time without UTC conversion—CloudWatch expects UTC
- warningDo not assume cross-account metrics appear automatically; you must set up the monitoring account access
- warningDo not ignore the CloudWatch agent log errors; they often indicate misconfiguration or permission issues
The Case of the Disappearing Latency Metric
Timeline
- 09:15Deployed a new latency tracking feature that sends custom metrics to CloudWatch via boto3 PutMetricData.
- 09:30Checked CloudWatch console; no custom metrics appeared under the 'AppLatency' namespace.
- 09:35Ran aws cloudwatch list-metrics --namespace AppLatency --region us-west-2; returned empty.
- 09:40Checked application logs: no errors from boto3, but log shows metric calls with timestamp in local time (PST).
- 09:45Reviewed IAM role attached to the EC2 instance; it has cloudwatch:PutMetricData but only for 'MyApp' namespace, not 'AppLatency'.
- 09:50Updated IAM policy to allow cloudwatch:PutMetricData for 'AppLatency'.
- 09:55Also fixed timestamp to be UTC in the application code.
- 10:10Redeployed and resent metrics. After 10 minutes, metrics appeared in console.
I deployed a new feature that measures API latency and sends it as a custom metric to CloudWatch. After deployment, I eagerly opened the CloudWatch console to see the data, but nothing showed up under my namespace 'AppLatency'. I waited 15 minutes—still nothing. My first thought was that the PutMetricData call was failing silently.
I checked the application logs and saw no errors from boto3, but I noticed the timestamp in the log was in local time (PST) with no timezone indicator. I also ran the aws cloudwatch list-metrics command for my namespace and got an empty response. That confirmed the data wasn't reaching CloudWatch. Then I inspected the IAM role attached to the EC2 instance and found that it only allowed PutMetricData for the namespace 'MyApp', not 'AppLatency'.
I fixed the IAM policy to include 'AppLatency' and updated the application to send timestamps in UTC. After redeploying, I waited 10 minutes and the metrics finally appeared. The root cause was a combination of a restrictive IAM policy and a non-UTC timestamp that was likely out of the acceptable window.
Root cause
IAM role lacked permission for the custom namespace 'AppLatency', and the metric timestamp was in local time (PST) which fell outside the allowed 14-day window forward/backward when interpreted as UTC.
The fix
Updated IAM policy to allow cloudwatch:PutMetricData for the 'AppLatency' namespace. Also changed the application to send timestamps in ISO 8601 UTC format.
The lesson
Always verify IAM permissions for custom namespaces and ensure timestamps are in UTC. Also, don't rely solely on the console—use the API to check for metrics.
CloudWatch is an eventually consistent system for custom metrics. After a PutMetricData call, the metric may not be visible for up to 15 minutes. This is by design: data is aggregated across multiple storage nodes. If you query immediately, you may see nothing. Always wait at least 5-10 minutes before declaring a metric missing.
Additionally, the storage resolution determines how data is aggregated. If you send high-resolution metrics (StorageResolution=1) but query with a period of 60 seconds using 'Average', you will get a single datapoint per minute. If you send at 1-minute resolution and query with a period of 60 seconds using 'Sum', you get the same. But if you query with a period of 300 seconds, you get a sum of 5 datapoints. Inconsistent query parameters can lead to perceived missing data.
The IAM action cloudwatch:PutMetricData can be scoped to specific namespaces using a condition key (cloudwatch:namespace). If your policy only allows 'MyApp', any PutMetricData calls to other namespaces will be silently denied (no error returned by default). Always check the IAM policy for the exact namespace you are using.
Also, if you are using a CloudWatch agent, the agent uses the instance's IAM role. Ensure the role has the necessary permissions. For cross-account, the source account must have a policy that allows PutMetricData to the destination account, and the destination account must have a CloudWatch metric stream or cross-account observability configured.
AWS CloudWatch expects timestamps in UTC. If you send a timestamp in local time without the 'Z' suffix or without converting to UTC, the timestamp may be interpreted as a different time. For example, a timestamp of '2025-03-20T10:00:00' (local PST) is interpreted as UTC, which is actually 10:00 UTC but your local time is 10:00 PST = 18:00 UTC. This can push the timestamp outside the allowed range (14 days back, 2 hours forward) or cause it to appear in the wrong time bucket.
To avoid this, always generate timestamps in UTC using libraries like datetime.utcnow() in Python or new Date().toISOString() in Node.js. Also, ensure your system clock is synchronized with NTP to avoid clock drift.
If you use the CloudWatch agent to collect custom metrics from EC2, the agent configuration file (usually at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json) must define the metrics collection. Common mistakes: incorrect JSON syntax, wrong collection interval, or missing metrics_declaration section. Validate the config with: sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status -m ec2
Also check the agent logs at /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log. Look for lines containing 'Error' or 'Unable to put metrics'. If the agent cannot assume the IAM role (e.g., due to network issues), it will silently drop metrics.
For cross-account monitoring, you need to set up a monitoring account and a source account. The source account must have a metric stream that sends metrics to the monitoring account, or use CloudWatch cross-account observability (enabled via CloudWatch console). If not configured, the metrics will only appear in the source account. Check the CloudWatch console under 'Cross-account metrics' to verify the source account is listed.
Also, IAM permissions on the monitoring account must allow ListMetrics and GetMetricStatistics from the source account. If you see metrics in the source account but not the monitoring account, the metric stream is likely misconfigured.
Frequently asked questions
How long does it take for custom CloudWatch metrics to appear?
Custom metrics are eventually consistent and typically appear within 15 minutes, but can take up to 30 minutes in some cases. If you don't see them after 30 minutes, there is likely a configuration issue.
Can I send metrics to a namespace that doesn't exist yet?
Yes, CloudWatch automatically creates a new namespace when you send a metric to it for the first time. However, ensure you have IAM permission for that namespace (or for all namespaces).
What is the maximum number of metrics I can send per PutMetricData call?
The maximum is 20 metric data objects per call. If you need to send more, batch them into multiple calls. Also, each metric data object can contain up to 10 dimensions.
My metric shows in list-metrics but get-metric-statistics returns empty. Why?
This usually means the metric has no data points in the requested time range. Check your timestamp range. Also, if you specified a statistic other than 'Sum' or 'SampleCount', and you have only one datapoint, they may be the same but 'Average' might be undefined. Use 'Sum' to confirm data exists.
Do I need to enable detailed monitoring for custom metrics?
No, detailed monitoring is only for AWS services like EC2. Custom metrics are always sent with the resolution you specify (1-minute by default, or high-resolution if you set StorageResolution=1).