Ask Your Question

Slow Keyset Fetch in JACKSONMLS

asked 2020-11-09 14:29:49 -0600

In our daily reconciliation process to purge/delete old listings, we are observing that the data feed takes several hours (> over 3 hours) to pull a full keyset from the RETS system, especially in the largest property class (residential). We are specifying a pagination size of 5000 listings per fetch. However, we observed that the feed only returns 500 at a time. This max size was also confirmed in our email followup with the MLS.

Our team actually observed and posted on a similar issue as the one mentioned here. We are following best practices as outlined in this post: using Offset and Limit in our pagination, selecting only fields in the InKeyIndex list (L_StatusCatID, L_UpdateDate, and L_ListingID), etc. Can you confirm that these fields are truly in the InKeyIndex? And recommend how the response time can be improved? We'd also like to know if the pagination size can be increased, as the current limit of 500 records means a little over > 3000 round trips to fetch all listings from the residential class. Conversely, in our other Paragon MLS feeds, we are able to fetch/paginate full keysets in a very timely manner.

Our RETS url is:

edit retag flag offensive close merge delete


Can you provide me with the account name (NO PASSWORDS).

bwolven gravatar imagebwolven ( 2020-11-09 18:11:55 -0600 )edit

Yes, it's: REALSCOUTrets


ChristineBeta gravatar imageChristineBeta ( 2020-11-09 19:19:07 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2020-11-10 08:26:50 -0600

The InKeyIndex functionality is only enabled when Limit=NONE.
I would suggest trying to do the pull of those three fields with it set to NONE.
You may need to increase your transaction timeout value to ensure it completes.
When I pulled all for Property:RE_1 it took around 3 minutes to download all 1,609,707 records.
Using query ((L_UpdateDate=2020-11-02T22:00:00Z-2020-11-09T21:00:00Z)|(L_Last_Photo_updt=2020-11-02T22:00:00Z-2020-11-09T21:00:00Z)) it took around 3 seconds to return 8,286 records.

edit flag offensive delete link more


Thanks for clarifying. You're right, I just double-checked the documention re: the use of InKeyIndex + Limit=None. But to clarify, we would like to use pagination if at all possible for the jacksonmls feed (as opposed to fetching all at once), as this is how we process other Paragon MLSs. Other feeds are able to return the entire keysets within 20 - 30 minutes. Since the Limit + Offset is needed to paginate, can you suggest if the 500 max limit can be increased? OR what else can be done to query/paginate the data set in a timely manner?

ChristineBeta gravatar imageChristineBeta ( 2020-11-10 11:56:39 -0600 )edit

Hi Bruce, just bumping this in case in got missed... any thoughts to my last comment? We'd prefer to use pagination if at all possible. Thanks.

ChristineBeta gravatar imageChristineBeta ( 2020-11-11 15:23:22 -0600 )edit

Checking into it.

bwolven gravatar imagebwolven ( 2020-11-11 15:44:55 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-11-09 14:29:49 -0600

Seen: 22 times

Last updated: Nov 10