Authoring custom expiration policies and actions in SharePoint 2007

Posted on 10/27/2008 @ 1:38 AM in #SharePoint by | Feedback | 7674 views

First of all, what the heck am I talking about?

This is one of the things I will be covering at SharePoint connections this November where I am presenting 3 talks on ECM in SharePoint.

One of the fairly common things you will end up doing in an ECM project is to implement custom RM policies. RM = Records Management. Basically to make things simple, based on certain characteristics of any content, it is classified as a distinct type of record. Depending upon the record type, you may want to apply different kinds of policies and actions.

  • A policy may be .. this record will expire, if it's column "Collectable" value is "Yes", and it hasn't been touched for the past 1 month.
  • An action could be ... if this record expires, delete it from Tier 1 storage, and move it to cheaper Tier 2, or 3 storage. (and thus tiered storage is totally implementable in SharePoint, don't let anyone tell you any different!!)

So there you have it. And this is very easily implementable in SharePoint, and deployable as a feature. Lemme explain!

Craft up a solution, that contains a WebApplication scoped feature. Here is my feature.xml -->

   1: <Feature xmlns="http://schemas.microsoft.com/sharepoint/"
   2:          Id="54428A7B-8F06-4932-ACF1-017D20FF26F4"
   3:          Title="Winsmarts Expiration Policy"
   4:          Scope="WebApplication"
   5:          Description="Winsmarts Expiration Policy"
   6:            ReceiverAssembly="Winsmarts.RMPolicies, Version=1.0.0.0, Culture=neutral,PublicKeyToken=44050b881185bf1b"
   7:          ReceiverClass="Winsmarts.RMPolicies.RMPoliciesReceiver"
   8:          ImageUrl="wslogo.gif"
   9:          />

Such a feature has to be web application level, and frankly that makes sense, because your RM policies should be scoped to everywhere the content may appear. This is very good architecture, since you have a singular place to maintain your record definitions and rules, rules that include policies and actions.

As you can see, I have a feature receiver. Keeping things brief, here is the feature receiver's FeatureActivated event -

   1: public override void FeatureActivated(SPFeatureReceiverProperties properties)
   2: {
   3:     string xmlManifest;
   4:     TextReader policyStream =
   5:         new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream(
   6:             "Winsmarts.RMPolicies.policy.xml"));
   7:     xmlManifest = policyStream.ReadToEnd();
   8:     PolicyResource.ValidateManifest(xmlManifest);
   9:     PolicyResourceCollection.Add(xmlManifest);
  10: }

 

What I'm doing here is that I am adding a policy.xml definition to my Web Application. The PolicyResource and PolicyResourceCollection appear in the Microsoft.Office.RecordsManagement.PolicyFeatures, and Microsoft.Office.RecordsManagement.InformationPolicy namespaces.

So, what is policy.xml? It's an XML file that has it's build action set to "Embedded Resource", so it can be retrieved as above. Here is what it contains -->

   1: <PolicyResource
   2:     xmlns="urn:schemas-microsoft-com:office:server:policy"
   3:     id = "CustomExpiration.CustomExpirationFormula"
   4:     featureId="Microsoft.Office.RecordsManagement.PolicyFeatures.Expiration"
   5:     type = "DateCalculator">
   6:     <Name>Winsmarts Filtered Expiration</Name>
   7:     <Description>Items Expire based on a filtered critereon</Description>
   8:     <AssemblyName>Winsmarts.RMPolicies, Version=1.0.0.0, Culture=neutral,PublicKeyToken=44050b881185bf1b</AssemblyName>
   9:     <ClassName>Winsmarts.RMPolicies.FilteredExpiration</ClassName>
  10: </PolicyResource>

Finally, you are at a point where the underpinnings are done, and now you can implement the Expiration policy or Custom Action.. here's a sample Expiration policy -->

   1: public class FilteredExpiration : IExpirationFormula
   2: {
   3:     #region IExpirationFormula Members
   4:  
   5:     public DateTime? ComputeExpireDate(SPListItem item, System.Xml.XmlNode parametersData)
   6:     {
   7:         if (item["Collectable"].ToString().Equals("Yes"))
   8:         {
   9:             return DateTime.Now;
  10:         }
  11:         else
  12:             return null;
  13:     }
  14:  
  15:     #endregion
  16: }

Nice huh? Basically, the Item needs to contain a column called "Collectable" and then if it's value is "Yes", delete it immediately. You can make this logic as involved as you want. But I'm a simple guy, so I'll keep it simple.

Okay good! Now, package up the above feature in a solution, install & deploy the solution and activate the feature on your port 80 site.

Debugging the above

Well, your code is basically done! But the problem is, the information management policy is a job definition defined under "PolicyConfigService" service, which is a farm level service. All these jobs are configured to run once a day. That seems like a good idea for production, but it's a sucky idea for development, because you can't wait a firkin' day to debug.

So, author up a quick console app to fix the schedule as shown below:

   1: static void Main(string[] args)
   2: {
   3:     foreach (SPJobDefinition j in 
   4:         SPFarm.Local.Services[new Guid("675d6a05-c321-414b-bd6e-ee936540c719")].JobDefinitions)
   5:     {
   6:         j.Schedule = SPSchedule.FromString("every 1 minutes between 0 and 59");
   7:         j.Update();
   8:     }
   9:     Console.Read();
  10: }

Yeah I know what you're thinking! Why didn't I use stsadm?  Well heck I wud've, but stsadm's commands don't let me target the above job .. if I'm wrong, please leave a comment below. There are some jobs that are targetable via stsadm OOTB though.

So anyway, once you run the above, and you check out the jobs in central admin, they should look like as shown below:

As you can see, all the jobs are now set to run every minute. NEAT! Now lets take it for a spin.

1. Author a policy based on the above.

Such a policy can be created at a Site Collection Level, List Level, or Content Type. For this article, I will create one at the site collection level. Go to Site Settings --> Site Collection Administration --> Site Collection Policies, and create a new policy called "CollectMe" with Expiration details as shown below:

2. Go to your favorite list, and add a column called "Collectable" with 2 values - "Yes" and "No" (default is No). Then assign the policy as follows:

Go to List Settings, and under "Permissions and Management" click on "Information Management Policy Settings". In there, choose a Site Collection Policy that you defined earlier as shown below:

3. Create a new list item to test it with.

Leave Collectable to "No". Go for a long walk. Come back in an hour .. what do you see .. the item is still there. Now, change the "Collectable" value to "Yes". Wait a minute. What do you see? The item is deleted. WOOHOO!!

NEAT! Now for some FAQs -

FAQs

  • Why is the above useful?
    • Well, the above is RM 101.
  • Why should my company care?
    • I'm guessing the email your CEO wrote carries more weight than the joke you sent me this morning. Well, they need to be treated differently then, the above will let you distinguish the two.
  • Is the above scalable?
    • Totally! You have to be careful though that your schedule shouldn't be too short to cause re-entrancy issues. SP will get confused if you don't care for re-entrancy yourself.
  • Give me a typical usage scenario
    • The above can be used very easily to implement a tiered storage model in SharePoint.
    • The above can also be used to move content based on size to external storage such as DFS
  • Can I write other kinds of custom policies and actions?
    • Heck yeah man!
  • Sahil, why are you so cool?
    • I don't know .. I just am.
  • Can the above be used for formal records management?
    • Totally! But you'd have to implement all those standards.
  • How does the above compare with ECM products such as Documentum?
    • Well, DCTM comes with many of the above built in, SP means you can write your own in .NET. And as you can see, it wasn't that hard to write. Where it gets hairy is formal records management, but I think if enough demand existed, 3rd party vendors could easily fill that gap. Obviously enough demand hasn't cropped up yet (large orgs don't move that quick).
  • Anything else?
    • Yeah, I gotta go sleep. Goodnight!

Sound off but keep it civil:

Older comments..


On 10/27/2008 6:48:29 PM Brian H. Madsen said ..
Hey Sahil!!!!

nice post and thanks for the indepth explanation and breakdown of how to achieve this.

been wringing my own neck for a bit on how to extend on the DMS capabilities within MOSS at the moment - i think i can extend this to include not just records management but also site based DMS policies...hmmm

Thanks for putting me in the "right" direction - will give this a go.


On 12/11/2008 10:50:53 AM Steve McDonnell said ..
Wow, nice post. Wish I had stumbled across this about a month ago as it would have saved me quite a lot of time and effort.

Thanks!


On 12/11/2008 1:39:04 PM Sahil Malik said ..
LOL, well, why didn't you? ;)


On 12/25/2008 11:03:22 PM vejay said ..
Hi sahil, I have a document library with “Item Scheduling” enabled. I need to send an e-mail to the document owner twice. 30 days before the expiration date “Publishing End Date” for the first time and if no action has been taken send another e-mail 15 days before the expiration date.


I thought the approach you described might work and went ahead and created a custom formula that calculates the date the e-mail has to be sent but its executed only once (when document has been uploaded). Please let me know if this is a right approach or suggest an alternate one


On 1/27/2009 12:11:38 PM umkclag0 said ..
We have made a custom expiration similar to the above, works wonderfully except on office 2007 documents .xlsx, docx, pptx, etc. Any thoughts or ideas would be helpful.


On 1/29/2009 1:30:40 PM umkclag0 said ..
To elaborate a bit. Our custom expiration is based on a Date Closed field and a document status. When a document is closed and a "date closed" is provided, the event handler passes the record to a customexpiration.dll. It looks up the retention months in a table and adds those to the date closed to get an expiration date. Works beautifully on all kinds of documents (txt, pdf, msg, etc, etc) but throws an error on docx, xlsx and pptx files. This may have something to do with office2007 dates. Any help is most appreciated.


On 2/12/2009 7:47:49 AM SP said ..
i am getting an error when i am running the console application which u hav specified can u just tel me how to proceed . I am getting exception at new Guid("675d6a05-c321-414b-bd6e-ee936540c719")].JobDefinitions can u help me


On 9/14/2009 8:45:04 AM pryank said ..
I followed above mentioned steps but getting following error in event log on running Expiration policy.


Uploaded a new item and set expiry date(custom field). Run the Expiry policy manually by pressing "process Expired items now". Expiration policy job runs. and in event log I get following error:

Event Type: Error


Event Source: Office SharePoint Server


Event Category: Information Policy Management


Event ID: 3935


Date: 14/09/2009


Time: 12:37:18


User: N/A


Computer: SDI-DEV-MOSS-CL


Description:


The custom IMPolicies.customDocumentExpiryPolicy expiration action is no longer valid or available.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


Any idea whats going wrong. I don't see any help any where for event id 3935.

To give you brief summary of what I have done:

I had a smiliar requirement where Sharepoint should delete item from doc library after expiry date(Custom field). But before deleting it should send an email to user 30 days before expiry. I added custom logic in ComputeExpireDate() method to check expiry date and days left in expiry. If days left is less than=30 I use SPUtility.Sendemail to send email to content owner and return "Expiry Date" value as required by ComputeExpireDate() method.


I also added simple logic in OnExpiration() method which recyles the item if expiry date is today' date. I deployed these as custom policy resources for Expiration Policy using a webapp scoped feature.


I created a site collection level policy and configured "Custom retention formula-Check Document Expiry " based on ComputeExpireDate() method and "custom delete method-DeleteExpiredDocument " method to delete doc on expiry. I attached this policy to custom content type(Custom doc library content type).


Uploaded a new item and set expiry date(custom field). Manually run the Expiry policy by pressing "process Expired items now". Expiration policy job runs successfully BUT in event log I get following error :(

Event Type: Error


Event Source: Office SharePoint Server


Event Category: Information Policy Management


Event ID: 3935


Date: 14/09/2009


Time: 12:37:18


User: N/A


Computer: DEV-MOSS


Description:


The custom IMPolicies.customDocumentExpiryPolicy expiration action is no longer valid or available.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Any one ever faced this irritating behaviour. I googled for this error and few people have faced same issue like me! But no one has posted any resolution.


On 3/29/2010 12:04:06 PM Ladan Zadeh said ..
Hi,


Where did you get the Guid in the code: SPFarm.Local.Services[new Guid("675d6a05-c321-414b-bd6e-ee936540c719")].JobDefinitions?


On 3/29/2010 5:01:02 PM Sahil Malik said ..
Ladan, you can either pick it from querystrings in CA, or through the objectmodel.


On 5/25/2010 12:01:30 AM Ajay said ..
I ended up liking whatever you write. You rock


On 9/7/2010 9:17:17 AM Manonamission said ..
Hi could you please post some detail on how you package and deployed this. Was it a wsp, and how was it deployed. It's a very cool post BTW.


On 10/26/2010 12:04:40 PM Manonamission said ..
Hi any views on how you might debug these things?


On 4/25/2011 12:00:45 AM harish said ..
Hi guys,


I hv set an policy on expiration date which uses one of the list column 'XYZ',This wil trigger an workflow.


Now my requirement is to trigger the workflow again if in case I change the XYZ column date on changed date.