First, a disclaimer: I do not use Lotus Notes Archiving. Instead, I use another method to invisibly move my finished work (emails, calendar, and tasks) to my archive. However, I serve many Lotus Notes users that use, want to use, or are forced to use Lotus Notes archiving. It is from these clients that I have learned a great deal about the relationship between archiving and lost productivity. This blog post is written for these people and it will consider the impact of poorly configured archiving and what you can do about it.

I received an email from a customer today asking me a question about email archiving:
"My company requires that I archive my mail for anything that is greater than 30 days old. Many of my lists and contexts have disappeared as a result. NOT GOOD. HELP!"

This kind of problem gives Lotus Notes a bad name

Before I describe the problem and a work-around, I'd like to share another more tragic story: Many years ago, I was hired by a large organization to deliver one of my productivity seminars. This organization had over 180,000 Notes users  and they wanted to learn how to use Lotus Notes more effectively; they wanted me to teach their people how to really get things done with Notes.

If you are familiar with David Allen's Getting Things Done (GTD) methodology or if you have follow my blogs for any period of time, you know the  importance of having a single "trusted system" in which to store your actionable information. I was delivering an in-house seminar to about 300 people from one organization and I was talking about the benefits of Lotus Notes and how well Notes works as my "trusted" system when a senior manager from the audience interrupted me:
"Sir, what you describe is fine, but I would never use Lotus Notes as my "trusted system" because I can't trust that Notes won't lose my important information!"

After the audience stopped applauding in agreement, I asked him why he felt that he couldn't trust Lotus Notes and he explained that his calendar and task information would automatically disappear after 90 days, convincing them that Lotus Notes is an untrustworthy (and generally despised) application.

The wind blew out of my sails; I had to agree with him. If my tasks and calendar items disappeared from my system,  I wouldn't trust Lotus Notes either. Fortunately, it doesn't have to be this way.

Apparently, the way the administrators at this organization had configured archiving, ALL of documents in the user's mail file were being archived after 90 days after last modification.

20091027-Archiving_in_Vanila_Notes_8.51_ Default_Settings.jpg
This is a common approach and they used the standard archiving process provided by Lotus. The problem with this approach is that a user's mail file contains much more than just e-mail. A Lotus Notes mail file may contain:
  • E-Mail messages, responses, and drafts, as well as all email filed in folders
  • Calendar entries, including appointments, invitations, events, and recurring entries
  • Tasks, individual and delegated, as well as task tracking information
When the system administrator blindly sets up a rule to archive all documents older than 90 days, much more than email will be lost. Unless the archiving rules exclude calendar and tasks, the user will find calendar entries as well as projects and actions missing from the mail file. No wonder users at so many companies tell me they don't trust Lotus Notes. I wouldn't trust it either!

Fortunately, there's an easy solution

It's important that any archiving policy specifically exclude calendar and tasks or that it intelligently archive only those calendar entries and tasks that no longer have current meaning or action for the user. (Remember, it's ALL about the end-user!)

20091027-Archiving_in_Vanila_Notes_8.51_ My_Settings.jpg

For my clients, I usually recommend the following archiving guidelines:
  • E-Mail: 390 Days AFTER last modification
  • Calendar: 390 Days AFTER the calendar date has passed
  • Tasks: Completed items that are older than 390 days only
I offer my clients these recommendations for archiving based on my own extensive experience consulting with highly productive individuals in organizations large and small. For e-mail, it is unproductive to have to hunt for email in an archive. And, if I don't see an email where I filed it I may not realize that it is still available elsewhere. If I haven't used it in a year, I probably do not need it - at least I do not need it immediately available to me. For Calendar, I have to be able to trust that what I put on my calendar will stay there. Personally, I never allow automatic archiving of my calendar, but for organizations that must do so, I request 390 days. This allows sufficient time after a calendar entry to see the calendar items (e.g. during a weekly or annual review). Finally, for tasks, I personally prefer to manage these myself, but for clients with archiving policies, I insist that tasks not be archived until 390 days after their completion date. Incomplete tasks should never be archived.

Unfortunately, vanilla Lotus Notes does not allow this level of granularity over what is archived. For this reason, I recommend that archiving be set to 390 days after last modification. This is not great - future projects and calendar entries can still be lost - but it is much better than what I see at many organizations we serve.

For our customers that use eProductivity, we created a special Archiving view that accomplishes all of the recommended archiving objectives (and more) with ease:

20091027-Archiving_ eProductivity_View.jpg

So, whose problem is it?

At the end of the day, it is the system administrator's responsibility to ensure that the archiving policy will not cause the users unnecessary data loss and cause them to distrust their tools (Lotus Notes) or the IT team. In cases where the users are expected to create their own archiving rules, I recommend that training be provided so that users can safely set up a set of conservative archiving rules without worrying about losing important information along the way. I would much prefer have a user that is upset because an archiving rule did not archive enough (it has never happened to me) than being upset because an aggressive archive rule took away too much; that does happen - a lot.

This represents an opportunity for Lotus to step up and provide an improved set of default archiving rules that won't hurt users or their information. While it's ultimately the administrators problem, when users think that Lotus Notes is the problem it's now IBM's problem. As I've said and written many times before, I think Lotus should create default archiving sets that will keep end-users and their system administrators from hurting themselves. An archiving view like this is not hard to set up. I know - we did it for our customers and any competent system administrator with a copy of Lotus Designer (now free!) can do the same.

How to recover from an over aggressive archiving rule?

The first thing to do is to determine where the archiving is happening. Is it running on the server, or is it a local rule? is it occurring within the Notes mail file or is it being done by a third-party utility or compliance tool? Once you know how the archiving is happening, the next step is to stop it. Once stopped, two steps remain: restore the lost data, and set up appropriate archiving rules that balance the needs of the user with those of the organization. In most cases, restoring the data can be done by opening the archive database and Copying & Pasting information from the archive back into the Notes Mail file. When restoring repeating calendar entries, it's important to get all of them. It's always best to discuss this with your administrators as they may have an easier way; many have an automated tool to assist with this.

Parting thoughts

Remember that the purpose of technology is to enable tools for productive knowledge work. IT and the policies they impose exist for the purpose of enabling the users they serve to work productively. When this happens, tools become personal, people think highly of them, and they use them to get things done.

What are your  thoughts on archiving and productivity?

If you are an end-user how do you archive? Does it work for you? If you are a Notes Administrator, how do you keep mailbox files sizes down while keeping your users productive and happy?

Discussion/Comments (9):

Lars Olufsen ( 10/28/2009 4:42:20 AM
Thoughts on email archiving (Admins: Here’s how to keep your users from hating you or tell you Lotus Notes sucks)

I think it is really poor customer care to ask end-users to handle archiving themselves.

They should be able to use their system the way they themselves feel is optimal, even if that includes storing year-old data.

Typically, data stays in Notes, because the users have no better place to put it, or the process to do so stinks. Putting it on them to "archive", "clean up" or similar, just makes that process even worse. Bad business processes hurt productivity.

If you can't improve the process or implement a better tool for handling your knowledge, tasks etc, then you must allow your users to use the systems they know will work, and not cripple that choice.

One thing that can be done to reduce space and improve performance, is unload attached files using products like Lotus' own "DAOS" which is included in the Domino server or IBM's Commonstore product, both handling the unloading of files without user interaction, and also still allowing access to the files directly from the original Notes-document, making it possible for the end-user to work seemingly without changes to business processes and definitely without having to spend time manually archiving content.

The time they save, the lack of helpdesk calls due to archiving errors and the unhindered productivity should outweigh the cost of disc-space.

Keith Brooks ( 10/28/2009 6:28:27 AM
Thoughts on email archiving (Admins: Here’s how to keep your users from hating you or tell you Lotus Notes sucks)

Sadly the issue of archiving is not the admins fault at all, usually.

It's corporate policy, as dictated by legal teams or executives, rarely by IT.

Neither of which care about the people using the software.

In older times, yes disk space premiums may have necessitated it and in some cases those guidelines never changed. Just like having 100MB email box quotas.

Leaving archiving up to the users is a help desk nightmare as invariably they will lose their data or permanently do something which the admin must deal with as an emergency.

IF archiving was easier to understand, manage and use so any person, not just IT people or heavy Notes users could do it, well that would be another story.

By the way, IT's lack of communication to users of their systems cause more issues than the system.

Richard Schwartz ( 10/28/2009 9:02:48 AM
Thoughts on email archiving (Admins: Here’s how to keep your users from hating you or tell you Lotus Notes sucks)

There is no reason at all why corporate policy should ever impact user-visible archiving. If archiving is visible to users at all, it is because of an IT decision to make it so. Either they decided to implement corporate policy on the cheap; or they decided that (in addition to whatever corporate-mandates they are fulfilling) they also want to use archiving as a way to save storage and improve on-line mail performance.

There are plenty of tools designed specifically for compliance archiving which operate entirely in the background, with no impact on user's mail file at all. Certainly, leaving compliance requirements in the hands of the users is a disaster waiting to happen. The City of Boston is currently dealing with the fall-out from incompetent IT doing exactly that, though the public reporting of the issue mostly prefers to blame the users.

Bill Malchisky (): 10/28/2009 9:31:25 PM
Thoughts on email archiving (Admins: Here’s how to keep your users from hating you or tell you Lotus Notes sucks)

In this situation Eric, remind those users that it's not Notes that is untrustworthy, but how it is configured. Untrustworthy software is almost impossible to fix; configuration issues can be managed.

I also fault the IT Team here, because although they may be under orders/policy from Management to implement Notes in this manner, there needs to be an effective communication campaign to train and inform the users of the data cut-off and where it can be located after 90 days.

Too many companies lack customer vision and the IT Teams think that since they know what will occur, the change is fine...meanwhile the end users light-up the Help Desk with complaints and issues that are easily manageable with a few simple e-mails and a bulletin or two. I see this lack of communication far too often, and well, the IT Teams are apathetic to changing in some cases.

In the end, Notes gets the bad rap for ineffective configuration, training, communication, or incompetent admins. For this case, Notes did not delete their data, just moved it to another location, and your upset users just need to know where to find it. That should rebuild trust and the reputation.

Richard is correct in that corporate policy should not impact users up-front. Plenty of great products exist to handle this.

Regulated firms are more strict with data handling requirements established by legal, and the 90-days is appropriate for some firms.

So remember, with archive after 90 days, doesn't mean disappear, just relocated. A big difference. Notes is just doing what it is told to do; it's the configuration not the product!

Stephan H. Wissel ( 11/1/2009 8:27:00 AM
Thoughts on email archiving (Admins: Here’s how to keep your users from hating you or tell you Lotus Notes sucks)

There are a lot of "should" both in the article and the comments. While I largely concur, there are 2 sides to the issue: one is how to change corporate policies/politics (good luck on that one), the other (for ICA as an ISV) how to live with that reality [of contra productive archival rules] and make eProductivity a trusted tool. There are several approaches to that (which all are flawed when mobile devices or web access come into play, but that's a different story for a different day). One is to run eProductivity "outside" the mail file and just pull the emails in. Another is to include the archive in the list of things to show (a bit more trickier, but achievable with R8 capabilities).

Stephan H. Wissel ( 11/1/2009 8:29:35 AM
Thoughts on email archiving (Admins: Here’s how to keep your users from hating you or tell you Lotus Notes sucks)

the third one (sneaky): touch all active documents, so they won't fall into "not modified in x days category" You could read the x number from the archive profile to optimize the write events.

Jake ( 11/4/2009 8:02:01 AM
Mobile users have smaller quotas

I am a very mobile user, and I replicate to my local mail file all the time, very rarely accessing the server replica.

My quote is hard set to 200meg. 390 days is not an option.

What I want, is an archiving strategy that keeps me under my mail file quota, but that shows all the archived documents in a combined view with my active documents.

David O (): 5/2/2013 9:38:39 AM
Thoughts on email archiving (Admins: Here’s how to keep your users from hating you or tell you Lotus Notes sucks)

This doesn't address the massive amounts of database corruption that seems to take place on the .nsf files, especially after Notes upgrades. RRV Bucket Error, for example.

Ian O (): 5/16/2014 9:43:46 AM
Thoughts on email archiving (Admins: Here’s how to keep your users from hating you or tell you Lotus Notes sucks)

I know this is an extremley old thread, but we have just started to implement Archiving in Notes as all our users are hoarders. A large chunk of our mail files are 20GB + in logical size (the largest is 45GB). We have implemented DAOS and shrunk these to a few GB each.

Our Archive plan is 1st Jan archive all emails with a last modified date more than 2 years and perform this once a year only.

Discussion for this entry is now closed.