Ask Your Question

Keyset Fetch Performance?

asked 2020-03-27 13:42:03 -0600

updated 2020-03-30 14:01:21 -0600

Once a day, we're fetching a full RETS keyset for the purposes of detecting and pruning records that have been deleted. It looks like it takes almost 3 hours to pull a full keyset from the system, even when selecting only the listing key, status field, and modification timestamp field.

This is more or less the case for all MLSs in Michigan, but we're generally able to grab the equivalent in other (even of this size) markets in a few seconds or minutes, rarely ~30 minutes. Is there something wrong with our query, or is this in line with the servers' usual performance? We're issuing a (L_Type_=|0,1,2,3,6,7),(L_ListingID=0+) DMQL2 query since we only include a few property types on our site. While we're pulling this keyset, we temporarily pause our incremental data pulls, and so service is degraded for our customers during this operation.

Account at

edit retag flag offensive close merge delete


Are you saying that you have single transactions taking 3 hours to pull?
Or that your process takes that long to complete?

bwolven gravatar imagebwolven ( 2020-03-30 14:18:16 -0600 )edit

Also please change the login URL you are using to this:

bwolven gravatar imagebwolven ( 2020-03-30 14:20:25 -0600 )edit

specifically that the RETS query (and pagination) is taking 3 hours. Our process completes in a few minutes once it has the keyset.

I'll change the URL, thanks.

mtb8332 gravatar imagemtb8332 ( 2020-03-31 15:30:46 -0600 )edit

The L_Type_ you were returning isn't normally an InKeyIndex field.
I changed it so you can see if it does any better next time you pull the keys.

bwolven gravatar imagebwolven ( 2020-03-31 16:17:59 -0600 )edit

Is this working any better for you now?

bwolven gravatar imagebwolven ( 2020-04-01 11:18:26 -0600 )edit

Hi, it's doing a bit better now for MF_5 and LD_3; RE_1 is still taking a while to pull the whole keyset. Do you think it'll be much faster if we remove the type filter?

mtb8332 gravatar imagemtb8332 ( 2020-04-01 16:47:13 -0600 )edit

Probably not because it would be pulling more records if you aren't limiting the types.

bwolven gravatar imagebwolven ( 2020-04-01 16:49:17 -0600 )edit

Got it. The RE_1 query finished in about 20 minutes, making the sum total about 30 across the classes we import. That's good enough for now, thanks.

mtb8332 gravatar imagemtb8332 ( 2020-04-01 19:48:19 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2020-04-01 19:48:55 -0600

Paragon added an index on L_Type_ which helped a good deal.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-03-27 13:42:03 -0600

Seen: 320 times

Last updated: Apr 01 '20