In one sentence
SNS Topic sending too many notifications to HTTPs subscriber.
More detailed
I have enabled events in one of my S3 buckets, and I have subscribed one of my applications to its SNS Topic
via HTTPS
. Everything was working fine until today, when my app started taking too long to respond.
After restarting the EC2 instance and noticing that the problem was still there, I SSH'd to the EC2 instance and realized that I was connected almost instantly, commands were being executed with no noticeable delays. So I knew the problem was not with the instance network.
I went to check the apache logs and that's when I noticed hundreds (perhaps thousands) of POST requests per second. See a snippet of the logs below.
Is there a way to see whether it is S3 that is sending all these notifications events to SNS?
Also, could this be that S3 is receiving that many uploads per second, is that even possible?
Lastly, what can I do to stop this other than simply unsubscribing from the topic?
Apache logs:
72.21.217.87 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.133 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 6404 "-" "Amazon Simple Notification Service Agent"
72.21.217.141 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.139 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.75 - - [02/Mar/2017:15:33:51 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.182 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.137 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.157 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.161 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.139 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.156 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 6404 "-" "Amazon Simple Notification Service Agent"
72.21.217.70 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.135 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.152 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.113 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.83 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.158 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.129 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.75 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.156 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.64 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.134 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.130 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.86 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.167 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.135 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.169 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.76 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.75 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 6404 "-" "Amazon Simple Notification Service Agent"
72.21.217.79 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.180 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.75 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.80 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.74 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 6404 "-" "Amazon Simple Notification Service Agent"
72.21.217.137 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.155 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.87 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.152 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.130 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.161 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
72.21.217.74 - - [02/Mar/2017:15:33:52 -0500] "POST /notifications HTTP/1.1" 500 619 "-" "Amazon Simple Notification Service Agent"
[Update]
After looking closer at the logs, I see that my app is responding with 500
. At first I thought it was Apache2 since it was receiving so many requests. I guess SNS continues to re-deliver messages (indefinitely?) when it doesn't receive a success message.
[Update 2]
I was able to get what notification was being sent, and it's a "ObjectRemoved:Delete" for a file that was deleted. The file no longer exists in the bucket.
I subscribed via email as suggested by @EEAA for less than 15 seconds, and have received 252 emails and counting :[ for the same notification.
[Update 3]
Checked the delivery policy on the SNS Topic, and confirmed that retries is set to 3!
Note: As of now, I've received almost 900 notifications from S3. I've also noticed that the x-amz-request-id
and x-amz-id-2
is always different, does this mean something?
No comments:
Post a Comment