We have run into an interesting scaling problem with Domino on Windows Server 2003. We have 6 Domino mail servers with about 850 users on each, about 5000 users in total. These systems are configured as 3 Windows servers, each supporting two Domino partitions. With a corporate policy of unlimited mail file size this means about 6 TB of data total, or about 2 TB per Windows server. Due to the high volume of open mail files, Windows can't quite cope, and occasionally corrupts mail files. When you see the message "Insufficient system resources..." in the Domino log, you know database corruption is not far away! The problem is caused by the way Notes uses the Windows Page pool to cache files (See IBM technote 1093511 and Microsoft document Q312362), Of course you can disable OS caching altogether, but, as IBM says, this will seriously impact server performance.
Initially we were alerted to the problem last December when massive data corruption occurred on all mail servers in the space of a week or so. That first occurrence was shortly after we completed our server consolidation project, putting two Domino partitions on one Windows server. After this cache problem happens, Domino finds some mail files with thousands of corrupt documents. It then removes those documents (i.e. deletes them without leaving a deletion stub). Missing documents are replaced from the clustered mail server in the next scheduled replication, and life continues.
Since this is a known limitation on Windows, server operations teams patched the Windows NetApp drivers. They also tweaked the servers to increase the size of the Page pool cache. That certainly reduced the problem, but in the first 3 months of this year we had another 3 minor occurrences, seemingly at random. There certainly seems little correlation between server load and the problem happening. In fact, it is possible to go for quite a while and not noticing the problem. Unless it happens to the CEO's mail file, which it did on Monday.
We are tackling the problem on a number of fronts. Short term we are adding a new (clustered) server, and moving about 100 user off each Domino server to this new server. That will reduce the number of open files on each Windows box by about 200. In Q3 we are upgrading to Domino 7.0.2, and we will upgrade Windows to the 64 bit version at the same time, which increases the size of the page pool cache.
I am also curious to see how well Linux fares under a real production load, so long term we are setting up a production mail server on Red Hat Linux. Fortunately Domino supports application level clustering, which means we can leave the current pairs of clustered Windows servers in place, and bring up a Linux server as a 3rd member of the cluster. We will then migrate users over to it slowly (can you tell I am risk adverse?!!). Finally we will restrict access to the current main production server, and run Domino on Linux with a full production load. This is a long term project, and I doubt we will get it complete this year. But it promises to be very interesting comparison to watch.
Wednesday, March 21, 2007
Tuesday, February 13, 2007
25 Hours Days?
Working with the teams responsible for the (US) change in Daylight Savings I came across this: a 25 hour day occurs once every year when changing from daylight saving time back to normal time. Time keeping systems that limit you to 24 hours per day are going to break if you work 25 hours on that day! Just thought you might want to know...
Monday, February 5, 2007
Lotusphere: Worst Practices "Pre" video
... and here is the promised video, posted on You Tube. (Editing it was MUCH more work that I expected, mainly because it was done on a slow laptop.)
Thursday, January 25, 2007
Some feedback on Lotusphere 2007
I am sitting on a plane returning from Lotusphere 2007. Thought about live blogging, but decided that it's better to pay attention during the presentations. Besides a laptop gets very heavy by the end of the day. For the first time in years, I can say that IBM finally "gets" Notes. What I mean is that Notes now finally has a decent roadmap for the future.
Because of my work, I was looking to see the plans for the future of Notes. I well remember the collective groan that went up at the opening session of Lotusphere a few years ago when Lotus introduced the "dual lane" approach. So I was very interested in where Notes is going. My real interest is "How effectively can I use this tool to solve business problems that other tools haven't been able to solve.
With that in mind, I attended a designer session. It was mildly interesting until last 15 minutes when Maureen Leyland arrived and demoded where the Designer client is going; based on Eclipse, the Designer now has an open architecture. Same for the Notes client, also based on Eclipse. Finally, I saw IBM with a decent, logical, plausible future for Notes.
With these new directions, I think Microsoft finally has a reason to look over their shoulder. When IBM and Microsoft battle it out, I score as a customer. Remember how long IE6 was Microsoft's browser? Years. Only when Firefox appeared and started taking market share did Microsoft start improving IE. I see the same thing happening with the Apple / Windows race. These races are ones that we, as customers, do not want any vendor to win! As long as these companies are competing we score.
Much to my surprise, I bumped in Gary Devendorf at the Dolphin on Wednesday. Gary used to be "Mr LotusScript" until IBM laid him off. A few months later he was working at Microsoft, helping them integrate their products with Notes. It was very interesting to hear his comments on the culture, and on how people relate at Microsoft, compared to IBM. Personally, I think IBM rather shot themselves in the foot letting someone like Gary go.
This year my favorite presentation was "Worst Practices" by Bill Bucan & Paul Mooney. When you put an Irishman and a Scot of doubtful sobriety on the stage together you will have an interesting banter. I hope to post a video on this soon...
Because of my work, I was looking to see the plans for the future of Notes. I well remember the collective groan that went up at the opening session of Lotusphere a few years ago when Lotus introduced the "dual lane" approach. So I was very interested in where Notes is going. My real interest is "How effectively can I use this tool to solve business problems that other tools haven't been able to solve.
With that in mind, I attended a designer session. It was mildly interesting until last 15 minutes when Maureen Leyland arrived and demoded where the Designer client is going; based on Eclipse, the Designer now has an open architecture. Same for the Notes client, also based on Eclipse. Finally, I saw IBM with a decent, logical, plausible future for Notes.
With these new directions, I think Microsoft finally has a reason to look over their shoulder. When IBM and Microsoft battle it out, I score as a customer. Remember how long IE6 was Microsoft's browser? Years. Only when Firefox appeared and started taking market share did Microsoft start improving IE. I see the same thing happening with the Apple / Windows race. These races are ones that we, as customers, do not want any vendor to win! As long as these companies are competing we score.
Much to my surprise, I bumped in Gary Devendorf at the Dolphin on Wednesday. Gary used to be "Mr LotusScript" until IBM laid him off. A few months later he was working at Microsoft, helping them integrate their products with Notes. It was very interesting to hear his comments on the culture, and on how people relate at Microsoft, compared to IBM. Personally, I think IBM rather shot themselves in the foot letting someone like Gary go.
This year my favorite presentation was "Worst Practices" by Bill Bucan & Paul Mooney. When you put an Irishman and a Scot of doubtful sobriety on the stage together you will have an interesting banter. I hope to post a video on this soon...
Thursday, June 1, 2006
Reusability with Standards
In February this year I did a presentation for Teamstudio on Notes Best Practices. Part of that presentation covered standards, and I want to extend an idea on standards by making the following claim: "In the corporate IT world a standard amounts to an agreed upon solution to a recurring problem."
In the software world there is a lot of talk about reusing code. Well, this same thought can be extended (reused?) to go beyond just code. When you create a standard (rule, procedure etc.), you effectively create a solution to a recurring problem. Each time you implement or use that standard, you are taking all the thought & effort that went into creating the standard the first time and reusing it. Let me supply some examples...
* You develop a standard configuration for a Notes client install (or any other software package). Now you company's help desk has just one client install to work with. Every time new help desk people come on board they only have to learn how to support that one client install. Compare this to the situation where there are multiple client installs, each one slightly different.
* You create a standard for your Notes development: all projects must go into under the "Dev\Projects\ProjectName folder, where "ProjectName" is the name of the project. Thus all project files will be grouped in one place. Now, every time a new project is started there is no need to decide where to put the project: just follow the standard. If an old project suddenly needs work, you know exactly where to find it. Compare this to the situation where developers can put projects anywhere they want on a development server, and chaotic mess that results from this approach.
Now, I have often had people say to me "But with all these rules, standards, procedures etc. you are taking away our freedom to be creative". But reality is quite different: standards actually provide greater freedom to be creative. Why? Because standards solve the mundane, recurring problems and leave you free to be creative on the more interesting problems. Do you really want to solve the same problems again and again, or would you rather focus on new problems?
To sum up, creating standards to solve recurring problems and you have reused the effort of solving those problems the first time. While there is a cost to manage standards, the reusability component can lead to significant cost savings. Another benefit is that a standards driven environment is much simpler to manage (i.e. complexity is reduced), and tends to be significantly more reliable. Finally, standards free people from mundane problems, allowing them to focus on more interesting problems. And you though reusability applied only to code...?
In the software world there is a lot of talk about reusing code. Well, this same thought can be extended (reused?) to go beyond just code. When you create a standard (rule, procedure etc.), you effectively create a solution to a recurring problem. Each time you implement or use that standard, you are taking all the thought & effort that went into creating the standard the first time and reusing it. Let me supply some examples...
* You develop a standard configuration for a Notes client install (or any other software package). Now you company's help desk has just one client install to work with. Every time new help desk people come on board they only have to learn how to support that one client install. Compare this to the situation where there are multiple client installs, each one slightly different.
* You create a standard for your Notes development: all projects must go into under the "Dev\Projects\ProjectName folder, where "ProjectName" is the name of the project. Thus all project files will be grouped in one place. Now, every time a new project is started there is no need to decide where to put the project: just follow the standard. If an old project suddenly needs work, you know exactly where to find it. Compare this to the situation where developers can put projects anywhere they want on a development server, and chaotic mess that results from this approach.
Now, I have often had people say to me "But with all these rules, standards, procedures etc. you are taking away our freedom to be creative". But reality is quite different: standards actually provide greater freedom to be creative. Why? Because standards solve the mundane, recurring problems and leave you free to be creative on the more interesting problems. Do you really want to solve the same problems again and again, or would you rather focus on new problems?
To sum up, creating standards to solve recurring problems and you have reused the effort of solving those problems the first time. While there is a cost to manage standards, the reusability component can lead to significant cost savings. Another benefit is that a standards driven environment is much simpler to manage (i.e. complexity is reduced), and tends to be significantly more reliable. Finally, standards free people from mundane problems, allowing them to focus on more interesting problems. And you though reusability applied only to code...?
Thursday, April 20, 2006
More on Corporate Directories
I have worked at a number of companies, and each time this entailed meeting lots of new people. I don't know about you, but sometime I can only remember the first or last name, and sometimes only the nickname e.g. Bob in the corner office. When I need to send an email or look up a phone number, the corporate directories often aren't very helpful.
There is a simple enhancement that can be added to corporate directories to significantly improve their usability, and that is listing people more than once, once by each name. Usually when this is proposed, I get a horrified look, but once people see it and use it, they really appreciate it.
As an example consider Robert Smith, also known as Bob. List him 3 times in the corporate directory, and the people can find him by first, last or nickname. In a small company this makes little difference, but once you are over a 1000 people, it really does make a difference. Suddenly the corporate directory becomes significantly more useful.
Note how Bob Smith appears 3 times, once for each name. That makes it really easy to find him.
This is but one example of a small improvement that adds to the "Continuous Improvement" bottom line.
There is a simple enhancement that can be added to corporate directories to significantly improve their usability, and that is listing people more than once, once by each name. Usually when this is proposed, I get a horrified look, but once people see it and use it, they really appreciate it.
As an example consider Robert Smith, also known as Bob. List him 3 times in the corporate directory, and the people can find him by first, last or nickname. In a small company this makes little difference, but once you are over a 1000 people, it really does make a difference. Suddenly the corporate directory becomes significantly more useful.
Note how Bob Smith appears 3 times, once for each name. That makes it really easy to find him.
This is but one example of a small improvement that adds to the "Continuous Improvement" bottom line.
Wednesday, April 19, 2006
Drill Down Views in Notes
In The View Nov/Dec 2001 (page15) Damien Katz wrote a short article on a technique for "faking" response documents. (Also see his very interesting articles on the R6 formula engine rewrite and the Formula Language's dirty secret.) This "fake response docs" technique can be used to create drill down lists in Notes. A very useful example of this is a hierarchical corporate directory, showing who reports to whom, and how many direct and indirect reports people have. Use this to drill down from the CEO to the janitor, and see the complete chain of command. Typically this information is updated automatically from the HR system every day, and is a LOT easier to manage and use that the endless out of date Visio diagrams that float around organizations. Recently I posted a screen shot of this type of directory, and Scott Giesbrecht asked me to describe how to do it - so here are the details.
Response documents have $Ref field which contains the UNID of the parent document. If a view's properties are set to "Show response documents in a hierarchy", and $Ref field exists on document "A" containing the UNID of parent document "B", then the view index makes document "A" a response of document "B".
Damien's technique is to use a FakeRef field with the UNID of the intended parent, and then use the view selection formula to set (i.e. fake) $Ref on all documents to the value in the FakeRef field - just for that view. This causes the view to be built with those documents as responses to their parents.
In the sample code posted at the end of this article, each person document has a FakeRef field called "ManagerUNID" that contains the UNID of their manager. (See Notes 1 & 2 below) The only way I have got this to work is to use the LotusScript "MakeResponse" method of the document - see the code fragment below, which is part of a loop processing all documents. The parent doc is "docManager", and the person document is "doc'. This code runs in an agent that updates the values of the ManagerUNID fields after updated data has been manually entered or imported from the HR system. (See Note 3 below)
Call doc.MakeResponse ( docManager ) 'Makes the person doc a response to the manager doc.
Set item = doc.GetFirstItem ( "$Ref" )
Call doc.ReplaceItemValue ( "ManagerUNID", item ) 'Creates the FakeRef field
Call doc.RemoveItem ( "$Ref" ) 'Removes $Ref so the person doc is no longer a response.
Call doc.Save ( True, True )
Now that evey document has it's parent defined, we are ready to create the drill down view (the Org Chart view in the sample code). To do this we fool the view into thinking that a document has a $Ref field by adding a line to the view's selection formula:
DEFAULT $Ref := ManagerUNID ; 'ManagerUNID is the FakeRef field in the sample code
And, in Damien's words, "This is all it takes". Remember that Notes limits you to 32 levels of response documents, but this is more than adequate for corporate hierarchies.
When you open the sample code, go to People by... Org Chart. Then expand all. You can experiment by adding people to the hierarchy. After adding people run the "Manual Org Chart refresh" agent (Under the View button).
Click link to download file
CorporateDirectory.zip
Note 1
For a document to appear as a response in a view, the FakeRef field (ManagerUNID in this example) MUST contain the hex / binary value of the UNID, and NOT the ASCII version (i.e. @text(@documentUnid) will not work.). The original article in The View was a little unclear here. If a field contains the hex / binary value of a UNID and that field shows on a form, then the contents of the field will appears as a doclink to the parent document. To see this in action, put the $Ref field on a response document form. Then open any response document created with that form in the Notes client. Instead of the UNID of the parent doc showing, you see a doclink to that parent document.
Note 2
You are not limited to just one FakeRef field - you can have several. Each view that uses fake response documents would select the appropriate FakeRef field. This is how one document can have different parents in different views.
Note 3
Heaven is when the HR system exports it's data with canonical names. This makes it trivial to update people's managers, and add the ManagerUNID field. Unfortunately this seldom seems to be the case, so you must find some way of linking manager names from the HR system to the canonical names used by Notes. The next best is when you have unique employee ID numbers, but some HR departments treat these like social security numbers and do their best to hide them. If that happens, you need a routine that somehow uniquely identifies people so that the update agent can put the correct name in the manager field. For more thoughts on the problems of names, see When is a name not a name?.
Response documents have $Ref field which contains the UNID of the parent document. If a view's properties are set to "Show response documents in a hierarchy", and $Ref field exists on document "A" containing the UNID of parent document "B", then the view index makes document "A" a response of document "B".
Damien's technique is to use a FakeRef field with the UNID of the intended parent, and then use the view selection formula to set (i.e. fake) $Ref on all documents to the value in the FakeRef field - just for that view. This causes the view to be built with those documents as responses to their parents.
In the sample code posted at the end of this article, each person document has a FakeRef field called "ManagerUNID" that contains the UNID of their manager. (See Notes 1 & 2 below) The only way I have got this to work is to use the LotusScript "MakeResponse" method of the document - see the code fragment below, which is part of a loop processing all documents. The parent doc is "docManager", and the person document is "doc'. This code runs in an agent that updates the values of the ManagerUNID fields after updated data has been manually entered or imported from the HR system. (See Note 3 below)
Call doc.MakeResponse ( docManager ) 'Makes the person doc a response to the manager doc.
Set item = doc.GetFirstItem ( "$Ref" )
Call doc.ReplaceItemValue ( "ManagerUNID", item ) 'Creates the FakeRef field
Call doc.RemoveItem ( "$Ref" ) 'Removes $Ref so the person doc is no longer a response.
Call doc.Save ( True, True )
Now that evey document has it's parent defined, we are ready to create the drill down view (the Org Chart view in the sample code). To do this we fool the view into thinking that a document has a $Ref field by adding a line to the view's selection formula:
DEFAULT $Ref := ManagerUNID ; 'ManagerUNID is the FakeRef field in the sample code
And, in Damien's words, "This is all it takes". Remember that Notes limits you to 32 levels of response documents, but this is more than adequate for corporate hierarchies.
When you open the sample code, go to People by... Org Chart. Then expand all. You can experiment by adding people to the hierarchy. After adding people run the "Manual Org Chart refresh" agent (Under the View button).
Click link to download file
CorporateDirectory.zip
Note 1
For a document to appear as a response in a view, the FakeRef field (ManagerUNID in this example) MUST contain the hex / binary value of the UNID, and NOT the ASCII version (i.e. @text(@documentUnid) will not work.). The original article in The View was a little unclear here. If a field contains the hex / binary value of a UNID and that field shows on a form, then the contents of the field will appears as a doclink to the parent document. To see this in action, put the $Ref field on a response document form. Then open any response document created with that form in the Notes client. Instead of the UNID of the parent doc showing, you see a doclink to that parent document.
Note 2
You are not limited to just one FakeRef field - you can have several. Each view that uses fake response documents would select the appropriate FakeRef field. This is how one document can have different parents in different views.
Note 3
Heaven is when the HR system exports it's data with canonical names. This makes it trivial to update people's managers, and add the ManagerUNID field. Unfortunately this seldom seems to be the case, so you must find some way of linking manager names from the HR system to the canonical names used by Notes. The next best is when you have unique employee ID numbers, but some HR departments treat these like social security numbers and do their best to hide them. If that happens, you need a routine that somehow uniquely identifies people so that the update agent can put the correct name in the manager field. For more thoughts on the problems of names, see When is a name not a name?.
Subscribe to:
Posts (Atom)