|
Optimized an agent’s runtime from >30 minutes to less than 30 seconds
Julian Buss, May 25th, 2010 17:10:52
Tags: Development
I had that agent in one application... the app contains about 300.000 documents, and some of them needs to be checked multiple times a day for events (for example, if they are expired). The agent doing this check needed over 30 minutes to run, and due to agent manager's configuration was always terminated after 30 minutes.
Pretty bad performance, isn't it? Well... not anymore, I'm down to less than 30 SECONDS now, only by re-engineering the logic and one view. There were several performance killers in that agent, and all of them based on one very common problem: using time/date functions in a view or using the NotesDatabase.search() instead of a ftsearch(). Every Notes dev should know that using time/date functions in a view, either in the select or column formula, is a very bad idea. Granted, there are times where using that technique cannot be avoided, or where this does not cause severe performance problems. But in general, it's a bad idea and if possible, code should be changed so that there is no time/date function used in a view. In my case, the agent did the following: 1.) Using a NotesDatabase.search() with time/date functions three times (!) to search for a subset of documents, only to remove one item from them. 2.) Refreshing a view which uses multiple time/date functions, then iterating over all documents of that view. The three NotesDatabase.search() calls alone took 20 minutes or so. Boy, what a bad architecture. Oh, and yes, I have to take all the shame on myself... I wrote that agent some years ago myself. In that time, the application didn't contained many documents, and I was not aware that these techniques could create such a performance problem :-) So, I managed to remove all NotesDatabase.search() calls and all time/date functions from the view. Now, the agent just iterates over all documents in that view. For every documents some time/date checks are performed and then, when the conditions are met, the event is fired. The view contains many more documents now, and the agent iterates over many more documents - but still that's WAY faster than before. Remember: iterating over lots of documents in a view is much faster than using time/date functions in that view.
Comments (5) | Permanent Link
1)
Optimized an agent’s runtime from >30 minutes to less than 30 seconds
When you iterate through the view, are you getting each document and evaluating the date, or are you using view entries? The latter is much faster, but you need to exposed the values you want to look at in the view. 2)
Optimized an agent’s runtime from >30 minutes to less than 30 seconds
yes, I am aware of that, But since I have to do work with the backend documents anyway, I read the date from the documents, not from the viewentries. Or to put it another way: I'm very satisfied with the result as it is now, there is no need to invest time for further improvements. 3)
Optimized an agent’s runtime from >30 minutes to less than 30 seconds
Julian, are you saying that you're iterating over 300,000 documents in a view, and actually accessing a summary item from the NotesViewEntry.Document handle, and it's running in 30 seconds? Because that's 10,000 docs a second, or 100ns to process each row. That seems extremely fast, even for a well-optimized process. 4)
Optimized an agent’s runtime from >30 minutes to less than 30 seconds
I have reached also good performance by putting all viewEntry data in a list (if you have to do some fast lookup) or an array. Creating an "ad hoc" class was a good idea. 5)
Optimized an agent’s runtime from >30 minutes to less than 30 seconds
Nathan, yes, I just found the holy grail of LotusScript programming that enables me to iterate through 300.000 documents in 30 seconds... but I won't tell you how I did it :-)))) . . . . . . . Just kidding. No, I don't iterate over 300.000 documents, the database contains over 300.000 documents all together, and that caused the former db.search() calls to be very slow. Now the view I iterate over contains only some thousand documents. And I do some work with each document, therefore 30 seconds is ok. |
|
|
Julian Buss/YouAtNotes Blog News and stories of a Lotus Notes developer, consultant and entrepreneur. |
|
