403 Error when retrieving photos

asked 2021-12-30 18:22:54 -0500

updated 2021-12-31 11:16:45 -0500

Hello!

We are seeing a 403 error when retrieving photos for Charlottesville Area Association of Realtors MLS. Could you please help us understand why we are seeing this error?

Here is the error we are seeing:

<html><head><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <title>ERROR: The request could not be satisfied</title> </head><body>

403 ERROR

The request could not be satisfied.


Request blocked. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

Generated by cloudfront (CloudFront)
Request ID: qm_2xzMUSzPvPCKg1DLPkYlyIGuj4s7KVyEqYYCibr_ZCzwkPPhRzw==
</body></html>

Username: REDFIN

edit retag flag offensive close merge delete

Comments

Hello! I can definitely look into this for you! Could you please provide the user name you're acessing RETS with?

Sadie gravatar imageSadie ( 2021-12-31 09:11:29 -0500 )edit

Hi Sadie! Username: Redfin

Dion James gravatar imageDion James ( 2021-12-31 10:36:03 -0500 )edit

Thank you!

So I was able to test your account, and I utilized a getObject for a recent ID from your queries: 620386, and was able to pull the photo. Is there a particular listing you're working with?

Sadie gravatar imageSadie ( 2021-12-31 11:17:39 -0500 )edit

The issue that we were (and may still be) having is that we are still able to pull the photos from our dev and test environments, but our production attempts are being blocked and we are getting 403 errors back.

We suspect that some automatic process might have been triggered due to how fast/frequently we were querying for photos, and our attempts to pull them may have been labelled as a potential Denial of Service attack. It appears that the block may have been cleared automatically, as we are now able to pull photos from our production environment again.

We will follow up if we get blocked

Dion James gravatar imageDion James ( 2021-12-31 11:30:54 -0500 )edit

Ah okay, that makes sense. Let us know! We can also always verify you're not blocked by IP as well, so if you run into it again - just reach out!

Thank you,

Sadie gravatar imageSadie ( 2021-12-31 11:43:31 -0500 )edit

Hi Sadie! We are blocked again on a 403 error

Dion James gravatar imageDion James ( 2021-12-31 13:13:26 -0500 )edit

Would you be able to supply your IP address so we can verify for you?

In addition I was reviewing your account - I recommend doing these requests in offsets, so not so many queries are being consistently requested to the server at once - this can definitely cause a flag.

Are you also utilizing location=1? I strongly suggest this method if you are requesting photos consistently.

You can also find more information starting at around page 10 for how to utilize offset and location most effectively, found here

Sadie gravatar imageSadie ( 2021-12-31 14:29:14 -0500 )edit

For the IP address, we operate a distribbuted load system, so the IPs our production importers use can be within the following range:

216.182.80.128/28

Also, RE: “I recommend doing these requests in offsets” We will once we get past the initial data download. The high traffic they are seeing recently is our attempt to complete the initial all data download. For listings, everything went fine, but we started getting blocked on photos almost right away… Once we complete the initial bulk download, our photo downloads will decrease dramatically to only updated listings photos.

Dion James gravatar imageDion James ( 2021-12-31 15:53:02 -0500 )edit

RE: “Are you also utilizing location=1? I strongly suggest this method if you are requesting photos consistently.” Yes, and we get the URLs back just fine, but when we go to extract the photo from the URL is when we are getting the 403 response.

Dion James gravatar imageDion James ( 2021-12-31 15:53:11 -0500 )edit

Have you looked at the answer on this question:
Image URL Download Rate Limits

bwolven gravatar imagebwolven ( 2022-01-05 08:10:10 -0500 )edit

Hello!

It looks like we are running into this issue with ALL Paragon sources. It’s blocking photo imports across many different sources. Will you please look into why this is?

bcar

bridgemls

caar

carmls

cmcmls

cmlssc

gaar

galmls

gbrmls

ggmls

gprmls

hcmls

hrar

hvcrmls

mmls

neren

nnrmls

rammls

rase

realsource

rebgv

sasjc

sdmls

sfarmlsnm

sjsrmls

spmls

spokanemls

gmlsfl

cmsmls

TMLS

tsmls

wvmls

Dion James gravatar imageDion James ( 2022-01-06 14:07:15 -0500 )edit

Is it possible to up the rate limit temporarily?

Dion James gravatar imageDion James ( 2022-01-06 16:07:24 -0500 )edit

The CDN image limiting is by IP address across all environments.
Would there be a way you can add delays between image pulls in your application to keep from exceeding the 5000 in 5 minute limit?
Or maybe use multiple IP addresses for your pulls.

bwolven gravatar imagebwolven ( 2022-01-07 10:45:57 -0500 )edit

Thanks so much! We'd like to avoid delaying our attempts, if possible. Is it possible to raise the rate limit on your end until we are able to complete the backfill?

Dion James gravatar imageDion James ( 2022-01-11 13:04:25 -0500 )edit

Any updates? We are still running into this issue. We have also hit an issue where we are not able to split into multiple IP addresses, as we have other data sources who whitelist IP addresses and can only accept a single IP address (rather than accepting a range of addresses).

Dion James gravatar imageDion James ( 2022-01-12 17:13:53 -0500 )edit

Hello - we are still seeing this issue. Any assistance from your end would be helpful!

Dion James gravatar imageDion James ( 2022-01-14 19:27:25 -0500 )edit

We have made some changes on our side for the IP address range you listed above.
Hopefully it will be of some help.

bwolven gravatar imagebwolven ( 2022-01-18 17:37:55 -0500 )edit

Is it working any better now since the changes?

bwolven gravatar imagebwolven ( 2022-01-19 15:14:43 -0500 )edit

Thanks! Yes, it is now working better.

What changes were made? We just worry that this could happen again down the road.

Dion James gravatar imageDion James ( 2022-01-24 13:22:08 -0500 )edit