Keyset Fetch Performance?
Once a day, we're fetching a full RETS keyset for the purposes of detecting and pruning records that have been deleted. It looks like it takes almost 3 hours to pull a full keyset from the system, even when selecting only the listing key, status field, and modification timestamp field.
This is more or less the case for all MLSs in Michigan, but we're generally able to grab the equivalent in other (even of this size) markets in a few seconds or minutes, rarely ~30 minutes. Is there something wrong with our query, or is this in line with the servers' usual performance? We're issuing a (L_Type_=|0,1,2,3,6,7),(L_ListingID=0+) DMQL2 query since we only include a few property types on our site. While we're pulling this keyset, we temporarily pause our incremental data pulls, and so service is degraded for our customers during this operation.
Account at http://mimls.rets.fnismls.com/rets/fn....
Comments
Are you saying that you have single transactions taking 3 hours to pull?
Or that your process takes that long to complete?
Also please change the login URL you are using to this:
"http://mimls-rets.paragonrels.com/rets/fnisrets.aspx/MIMLS/login?rets-version=rets/1.7.2"
specifically that the RETS query (and pagination) is taking 3 hours. Our process completes in a few minutes once it has the keyset.
I'll change the URL, thanks.
The L_Type_ you were returning isn't normally an InKeyIndex field.
I changed it so you can see if it does any better next time you pull the keys.
Is this working any better for you now?
Hi, it's doing a bit better now for MF_5 and LD_3; RE_1 is still taking a while to pull the whole keyset. Do you think it'll be much faster if we remove the type filter?
Probably not because it would be pulling more records if you aren't limiting the types.
Got it. The RE_1 query finished in about 20 minutes, making the sum total about 30 across the classes we import. That's good enough for now, thanks.