403 Error when retrieving photos
We are seeing a 403 error when retrieving photos for Charlottesville Area Association of Realtors MLS. Could you please help us understand why we are seeing this error?
Here is the error we are seeing:
<html><head><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <title>ERROR: The request could not be satisfied</title> </head><body>
The request could not be satisfied.
Request blocked. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
Generated by cloudfront (CloudFront) Request ID: qm_2xzMUSzPvPCKg1DLPkYlyIGuj4s7KVyEqYYCibr_ZCzwkPPhRzw==</body></html>
Hello! I can definitely look into this for you! Could you please provide the user name you're acessing RETS with?
Hi Sadie! Username: Redfin
So I was able to test your account, and I utilized a getObject for a recent ID from your queries: 620386, and was able to pull the photo. Is there a particular listing you're working with?
The issue that we were (and may still be) having is that we are still able to pull the photos from our dev and test environments, but our production attempts are being blocked and we are getting 403 errors back.
We suspect that some automatic process might have been triggered due to how fast/frequently we were querying for photos, and our attempts to pull them may have been labelled as a potential Denial of Service attack. It appears that the block may have been cleared automatically, as we are now able to pull photos from our production environment again.
We will follow up if we get blocked
Ah okay, that makes sense. Let us know! We can also always verify you're not blocked by IP as well, so if you run into it again - just reach out!
Hi Sadie! We are blocked again on a 403 error
Would you be able to supply your IP address so we can verify for you?
In addition I was reviewing your account - I recommend doing these requests in offsets, so not so many queries are being consistently requested to the server at once - this can definitely cause a flag.
Are you also utilizing location=1? I strongly suggest this method if you are requesting photos consistently.
You can also find more information starting at around page 10 for how to utilize offset and location most effectively, found here
For the IP address, we operate a distribbuted load system, so the IPs our production importers use can be within the following range:
Also, RE: “I recommend doing these requests in offsets” We will once we get past the initial data download. The high traffic they are seeing recently is our attempt to complete the initial all data download. For listings, everything went fine, but we started getting blocked on photos almost right away… Once we complete the initial bulk download, our photo downloads will decrease dramatically to only updated listings photos.
RE: “Are you also utilizing location=1? I strongly suggest this method if you are requesting photos consistently.” Yes, and we get the URLs back just fine, but when we go to extract the photo from the URL is when we are getting the 403 response.
Have you looked at the answer on this question:
Image URL Download Rate Limits
It looks like we are running into this issue with ALL Paragon sources. It’s blocking photo imports across many different sources. Will you please look into why this is?
Is it possible to up the rate limit temporarily?
The CDN image limiting is by IP address across all environments.
Would there be a way you can add delays between image pulls in your application to keep from exceeding the 5000 in 5 minute limit?
Or maybe use multiple IP addresses for your pulls.
Thanks so much! We'd like to avoid delaying our attempts, if possible. Is it possible to raise the rate limit on your end until we are able to complete the backfill?
Any updates? We are still running into this issue. We have also hit an issue where we are not able to split into multiple IP addresses, as we have other data sources who whitelist IP addresses and can only accept a single IP address (rather than accepting a range of addresses).
Hello - we are still seeing this issue. Any assistance from your end would be helpful!
We have made some changes on our side for the IP address range you listed above.
Hopefully it will be of some help.
Is it working any better now since the changes?
Thanks! Yes, it is now working better.
What changes were made? We just worry that this could happen again down the road.