First time here? Check out the FAQ!

Keyset Fetch Performance?

  • retag add tags

Once a day, we're fetching a full RETS keyset for the purposes of detecting and pruning records that have been deleted. It looks like it takes almost 3 hours to pull a full keyset from the system, even when selecting only the listing key, status field, and modification timestamp field.

This is more or less the case for all MLSs in Michigan, but we're generally able to grab the equivalent in other (even of this size) markets in a few seconds or minutes, rarely ~30 minutes. Is there something wrong with our query, or is this in line with the servers' usual performance? We're issuing a (L_Type_=|0,1,2,3,6,7),(L_ListingID=0+) DMQL2 query since we only include a few property types on our site. While we're pulling this keyset, we temporarily pause our incremental data pulls, and so service is degraded for our customers during this operation.

Account at

mtb8332's avatar
asked 2020-03-27 13:42:03 -0500
bwolven's avatar
updated 2020-03-30 14:01:21 -0500
edit flag offensive 0 remove flag close merge delete


Are you saying that you have single transactions taking 3 hours to pull?
Or that your process takes that long to complete?

bwolven's avatar bwolven (2020-03-30 14:18:16 -0500) edit

Also please change the login URL you are using to this:

bwolven's avatar bwolven (2020-03-30 14:20:25 -0500) edit

specifically that the RETS query (and pagination) is taking 3 hours. Our process completes in a few minutes once it has the keyset.

I'll change the URL, thanks.

mtb8332's avatar mtb8332 (2020-03-31 15:30:46 -0500) edit

The L_Type_ you were returning isn't normally an InKeyIndex field.
I changed it so you can see if it does any better next time you pull the keys.

bwolven's avatar bwolven (2020-03-31 16:17:59 -0500) edit

Is this working any better for you now?

bwolven's avatar bwolven (2020-04-01 11:18:26 -0500) edit

Hi, it's doing a bit better now for MF_5 and LD_3; RE_1 is still taking a while to pull the whole keyset. Do you think it'll be much faster if we remove the type filter?

mtb8332's avatar mtb8332 (2020-04-01 16:47:13 -0500) edit

Probably not because it would be pulling more records if you aren't limiting the types.

bwolven's avatar bwolven (2020-04-01 16:49:17 -0500) edit

Got it. The RE_1 query finished in about 20 minutes, making the sum total about 30 across the classes we import. That's good enough for now, thanks.

mtb8332's avatar mtb8332 (2020-04-01 19:48:19 -0500) edit
add a comment see more comments

1 Answer


Paragon added an index on L_Type_ which helped a good deal.

mtb8332's avatar
answered 2020-04-01 19:48:55 -0500
edit flag offensive 0 remove flag delete link


add a comment see more comments

Your Answer

Login/Signup to Answer