Ask Your Question
0

History DMQL queries are timeing out

asked 2018-10-18 09:18:27 -0600

Tony gravatar image

http://neren.rets.paragonrels.com/ret... rus0808

History DMQL queries are timing out when the query results are large.

The RE1 History query below sent today 18-Oct-2018 09:26:10 EDT.

Watching the results live showed the RETS server pasuing sending the results at the end of last line in the middle of the final ending tag data tag "<da" then approximately 10 minutes and 05 seconds later continued sending where it left off at "ta><RETS-STATUS ReplyCode="0" ReplyText="Operation Successful" />".

However by that time our client timed out raising an error which aborts our download job.

Note we typically have a LIMIT of 1000, but as shown in this query we lowered it to 100 with the same result.

[Select] => L_Class,L_DisplayId,L_HistoryId,L_Last_Photo_updt,L_ListingID,L_Sequence,L_Status,L_StatusDate,L_UpdateDate
[Query] => (LM_CommentDate=2018-10-18T00:00:00Z+)
[Resource] => History
[Class] => RE_1
[QueryType] => DMQL2
[Count] => 1
[Format] => COMPACT-DECODED
[Limit] => 100
[Offset] => 1
[RestrictedIndicator] => 
[StandardNames] => 0
edit retag flag offensive close merge delete

Comments

What is your HTML timeout value set to?

bwolven gravatar imagebwolven ( 2018-10-18 11:23:10 -0600 )edit

Hi Bruce,

What HTML timout are you referring to?

This is a back office script using php with phprets v1 with no explicit HTML timeout is set.

Seems like the source issue is the RETS server pauses for 10 minutes when the query is for a large result set.

I'm not aware of any of our other bnfs or other accounts where the RETS server pauses in the middle of sending results from a query for 10+ minutes.

Tony gravatar imageTony ( 2018-10-18 11:49:22 -0600 )edit

I'm looking into the issues.

bwolven gravatar imagebwolven ( 2018-10-18 18:22:12 -0600 )edit

OK, thanks

Tony gravatar imageTony ( 2018-10-18 18:59:38 -0600 )edit

Tony. Can you test NEREN history queries again and see if they are performing any better?

bwolven gravatar imagebwolven ( 2018-11-08 14:07:13 -0600 )edit

Can you also try it with your 1000 batch size again instead of the 100 you are using now?

bwolven gravatar imagebwolven ( 2018-11-08 14:11:33 -0600 )edit

As of this post, from today's logs: All queries times are EST using batch of 100, following this response will change to 1000.

  • 11/8/2018 15:02 Elapsed Time 22:45
  • 11/8/2018 14:02 Elapsed Time 46:24
  • 11/8/2018 13:01 Elapsed Time 45:59
  • 11/8/2018 12:01 Elapsed Time 51:49
  • 11/8/2018 11:01 Elapsed Time 45:40
  • 11/8/2018 10:03 Elapsed Time 45:39
  • 11/8/2018 09:06 Elapsed Time 44:46
  • 11/8/2018 07:05 Elapsed Time 54:56
  • 11/8/2018 04:08 Elapsed Time 53:17
Tony gravatar imageTony ( 2018-11-08 15:34:02 -0600 )edit

Just changed batch to 1,000 Queries run hourly next run at 1700 EST

Tony gravatar imageTony ( 2018-11-08 15:37:48 -0600 )edit

11/8/2018 16:02:28 Elapsed Time 15:44 (batch 100)

Tony gravatar imageTony ( 2018-11-08 15:53:34 -0600 )edit

Is that 15 minutes?
And for your whole process to run?

bwolven gravatar imagebwolven ( 2018-11-08 15:56:12 -0600 )edit

That 15 minutes is for the whole job to run, which consists of many DMQL queries.

So this shows a difference where the job was running for 45 - 50 minutes +/- when it was slow.

Today at a batch job of 100 it's must faster at 15 minutes.

To morrow I will share with you the totals with batch 1,000.

If you would like a copy of the log to see all the DMQL queries let me know and I will send by email

Tony gravatar imageTony ( 2018-11-08 16:39:37 -0600 )edit

I can see the queries in our logs.
Unless you have overhead on your side in your batch process I show:
Login: 2018-11-08 16:59:58.357 to logout: 2018-11-08 17:03:54.763 for your last urn.

bwolven gravatar imagebwolven ( 2018-11-08 16:47:49 -0600 )edit

The list below shows 3 jobs beginning at 1700.

Only the last job starting at 17:01:54 was set to use batch 1,00. This is where the history issue has been occurring.

As you can see job starting at 17:01:54 with elapsed time 7:45 is a huge improvement over all the other DMQL History queries at +/50 - 54 minutes.

  • 11/8/18 17:01:54 Elapsed time 07:45 <--- this is the job for history
  • 11/8/18 17:01:45 Elapsed time 00:08
  • 11/8/18 17:00:00 Elapsed time 01:44
Tony gravatar imageTony ( 2018-11-08 17:07:30 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-11-15 09:17:33 -0600

So is everything working correctly now?

edit flag offensive delete link more

Comments

Let me check with NEREN and I will update you.

Tony gravatar imageTony ( 2018-11-15 09:47:09 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-10-18 09:18:27 -0600

Seen: 69 times

Last updated: Nov 15