History DMQL queries are timeing out
http://neren.rets.paragonrels.com/ret... rus0808
History DMQL queries are timing out when the query results are large.
The RE1 History query below sent today 18-Oct-2018 09:26:10 EDT.
Watching the results live showed the RETS server pasuing sending the results at the end of last line in the middle of the final ending tag data tag "<da" then approximately 10 minutes and 05 seconds later continued sending where it left off at "ta><RETS-STATUS ReplyCode="0" ReplyText="Operation Successful" />".
However by that time our client timed out raising an error which aborts our download job.
Note we typically have a LIMIT of 1000, but as shown in this query we lowered it to 100 with the same result.
[Select] => L_Class,L_DisplayId,L_HistoryId,L_Last_Photo_updt,L_ListingID,L_Sequence,L_Status,L_StatusDate,L_UpdateDate
[Query] => (LM_CommentDate=2018-10-18T00:00:00Z+)
[Resource] => History
[Class] => RE_1
[QueryType] => DMQL2
[Count] => 1
[Format] => COMPACT-DECODED
[Limit] => 100
[Offset] => 1
[RestrictedIndicator] =>
[StandardNames] => 0
Comments
What is your HTML timeout value set to?
Hi Bruce,
What HTML timout are you referring to?
This is a back office script using php with phprets v1 with no explicit HTML timeout is set.
Seems like the source issue is the RETS server pauses for 10 minutes when the query is for a large result set.
I'm not aware of any of our other bnfs or other accounts where the RETS server pauses in the middle of sending results from a query for 10+ minutes.
I'm looking into the issues.
OK, thanks
Tony. Can you test NEREN history queries again and see if they are performing any better?
Can you also try it with your 1000 batch size again instead of the 100 you are using now?
As of this post, from today's logs: All queries times are EST using batch of 100, following this response will change to 1000.
Just changed batch to 1,000 Queries run hourly next run at 1700 EST
11/8/2018 16:02:28 Elapsed Time 15:44 (batch 100)
Is that 15 minutes?
And for your whole process to run?
That 15 minutes is for the whole job to run, which consists of many DMQL queries.
So this shows a difference where the job was running for 45 - 50 minutes +/- when it was slow.
Today at a batch job of 100 it's must faster at 15 minutes.
To morrow I will share with you the totals with batch 1,000.
If you would like a copy of the log to see all the DMQL queries let me know and I will send by email
I can see the queries in our logs.
Unless you have overhead on your side in your batch process I show:
Login: 2018-11-08 16:59:58.357 to logout: 2018-11-08 17:03:54.763 for your last urn.
The list below shows 3 jobs beginning at 1700.
Only the last job starting at 17:01:54 was set to use batch 1,00. This is where the history issue has been occurring.
As you can see job starting at 17:01:54 with elapsed time 7:45 is a huge improvement over all the other DMQL History queries at +/50 - 54 minutes.