• Crafter Studio- An authoring and publishing solution that includes user-friendly features for marketers and content editors, including WYSIWYG authoring, in-context preview, workflow, integrated third-party analytics and multi-channel publishing.
• Crafter Engine- A high performance content delivery system that provides developers with the foundation for building high performance web and mobile applications. In turn, this enables business users, through Crafter Studio, to create engaging, personalized experiences for end users.

To learn more about Crafter Software check out their recent press release or visit their site at www.craftersoftware.com.

Also, mark your calendar for May 17th. Crafter is hosting an, “Introduction to Crafter” webinar. During this one hour event, Russ Danner, VP Products of Crafter Software, will provide a short summary of Crafter’s features and benefits, followed by a demo of Crafter’s key capabilities.

You can see more tips and tricks in our video library!

You can see more tips and tricks in our video library!

Challenge
Located in Toledo, Ohio, Davis College, a private, two-year college, must adhere to state and federal regulations regarding archiving of student records and transcripts. Eight years ago, Davis College implemented Digitech Systems PaperVision and PaperFlow, document capture and document management technology, to help digitize and electronically store student records.

Using one dedicated computer and a high-volume scanner, staff scanned and indexed paper documents into TIFF files. Overtime, the college expanded the solution to include other document centric processes, such as time sheets and enrollments. With only one user license, sharing the dedicated computer across departments was a challenge. The system could not easily scale without a significant financial investment by the college.

Since upgrading the system was cost prohibitive, Davis College researched other content management solutions that would allow them to reuse their existing high-volume scanners and multifunctional printers (MFPs — scan, copy and print in one device). Key requirements included: ease of use; Web access to documents; unlimited user license; easy to scale; metadata support; and affordability.

Solution
Alfresco was selected to replace the PaperVision system. Alfresco’s per server licensing model provided the school with unlimited user access and data storage, rich document management functionality and greater flexibility. In addition, it was priced significantly lower than PaperVision’s basic five-user subscription and could scale to meet future archiving demands.

Development and testing on Alfresco took Davis College seven weeks. The college worked with Alfresco Platinum and Certified partner Blue Fish Development Group to provide support through the implementation and with basic customization such as changing the default date/time format.

To simplify the scanning and indexing process, the college implemented the Blue Fish Document Indexing Module for Alfresco Share. With the Document Indexing Module, the college scans documents into Alfresco using their existing high speed scanner or departmental MFPs processing the documents in easy-to-use indexing queues. The college can now store and index PDF, GIF, JPEG, Word, Excel, and most other office and image file formats. Once the documents are scanned and indexed, they are moved to one of seven Alfresco Share sites for easy user access and search.

Davis College worked with Blue Fish to help migrate over 30,000 documents to Alfresco. Blue Fish programmed a CSV import routine to allow the college to move the document indexes and reclassify the image metadata that was exported from previous software.

Davis College created rules on how indexed data should be represented or formatted (such as dates and social security numbers). This ensures metadata consistency, data accuracy and improves document search. Since this was not available in the previous system, Alfresco identified numerous content indexing errors during the migration. The college is now verifying the information on any highlighted documents with the original paper documentation. Once complete, the school will begin disposing of paper documents in order to free up physical storage space.

Today, Davis College has added several users as indexers that can scan documents or images from the closest scanner and transfer the documents via a mapped network drive or an FTP client to an Alfresco Share site. The system will automatically convert any TIFF files to PDF. Before storing the document in Alfresco, a custom Java script runs in the background creating a new folder based on the current date and renames the file based on the date and time – down to the millisecond.

Users can log into an Alfresco Share site from any browser seamlessly with single sign-on and LDAP authentication. The user opens the Blue Fish Document Indexing Module for Alfresco Share to classify the document with the appropriate metadata. Once complete, the system automatically moves the document to the selected document library where college personnel can access and search for archived records.

“We have multiple records management needs throughout the college and require a scalable system that makes it easy to scan, index and digitally store information from any department or location across the campus. Not only is Alfresco more affordable, it provides us the flexibility our users and departments require. Information is securely available for users to search from any desktop and we can validate information for accuracy. This is a huge improvement over our previous solutions and we will look for ways to continue expanding our Alfresco implementation,” said Aaron Cowell, Network Administrator at Davis College.

Conclusion

• Archived student records are now searchable and accessible to administrative staff directly from their desktop. No longer do users have to travel to a separate building or dedicated computer to scan or search archived transcripts saving time and resources.
• The new system validates that information is entered correctly ensuring data accuracy and consistency.
• LDAP integration guarantees that information is only available based on a user’s security permissions.
• Users can scan, upload, and indes them from virtually anywhere on campus.
• Davis College can expand document archiving to other departments without incurring additional costs.
• The current Alfresco repository stores over 20GB of content.

What’s Next
The college has expanded the system to include the digital archiving of historical information from its Heritage Center which houses documents, images and news clippings about the college from the past 154 years. The center has one dedicated person who is scanning documents into Alfresco and another team that reviews, indexes and catalogs the information. Soon a portion of the information will be available internally for searching.

“With the Blue Fish Document Indexing Module for Alfresco Share, it is easy to turn Alfresco into an imaging system. Alfresco provides the scalable repository, while the Document Indexing Module enables you to scan from any MFP directly into Alfresco, creating indexing queues. Clients like Davis College can classify documents with appropriate metadata,” said Jeff Philipp, CEO of Blue Fish Development Group.

Check it out now!

You can see more tips and tricks in our video library!

This time around, I’ll be focusing on how you can simplify document management, achieve a paperless office, and streamline your business processes. The main topics I’m going to cover are:

• General Document Management – Alfresco works out of the box with your email and office applications, and has some very clever ways to automate routine document processing tasks. I’ve been implementing document management systems for almost 20 years, and I can tell you that it’s the easiest to use system I’ve ever seen for this type of thing.
• Scanning and Archiving Paper Documents – I’ll show you how you can use the scanner or multi-function printer you already have to turn paper files into electronic images.
• Workflow and Business Process Automation – Alfresco can help you automate your business processes with it’s powerful workflow engine. From approving invoices to revising documents to automating complex business rules, I’ll show you the full breadth of Alfresco’s workflow capabilities.

I’ve designed a special workflow that really shows off Alfresco’s bells and whistles, and I’ll be showing it for the first time at this lunch and learn.

So if you are curious about what Alfresco does or how it might be able to benefit your organization, now’s your chance to learn more about it. I’ll be giving a full demonstration of the software and answering all your questions.

All events are from 11:30-1:30, and lunch will be provided.

• Boston, MA – Tuesday, April 26
• Orlando, FL – Wednesday, April 27
• Miami, FL – Thursday, April 28
• Austin, TX – Tuesday, May 3
• Houston, TX – Wednesday, May 4
• Dallas, TX – Thursday, May 5
• San Antonio, TX – Tuesday, May 10
• Nashville, TN – Wednesday, May 11
• Phoenix, AZ – Thursday, May 12

Social Content Management is a term coined by Gartner Group, and adopted by Alfresco. It’s a brand new concept, and the industry hasn’t yet settled on a standard definition. Alfresco uses it to mean managing the content created by social software. That’s a platform-oriented view, and it makes sense coming from a platform vendor like Alfresco.

But when I think about Social Content Management, I think about the user. Folks who know me well know I’m always thinking about the user. I’m constantly asking questions like “what are the users trying to accomplish when they use this application?” or “what’s the best user experience to help them achieve their goals?” At heart, I’m much more of an applications guy than a platform guy.

So I got to thinking about some of the social networking applications I use regularly, and wondering how we could apply those concepts and features to the realm of document management and collaboration. What I came up with is a list of features I’d like to see added to Alfresco that I think would make it more social and improve users’ ability to collaborate. To me, that’s what Social Content Management should be.

Following Users in Alfresco (Inspired by Facebook)

At the heart of any social network are our friends – the people we are connected to. We follow them to find out what’s going on in their personal lives.

But in a business setting, it’s a bit different. We don’t want to follow our colleagues’ personal lives; we want to follow their business lives.

You may want to know when a team member has completed the report he was working on. Or a manager might want to keep tabs on what her employees are doing. There are subject matter experts that you would follow to learn more about your company’s products. People in Sales might want to see what’s cooking over in Marketing, and vice-versa.

The first feature I want to see is the ability to follow my colleagues in Alfresco. The documents they produce, the discussions they participate in, the comments they make on other people’s documents – this is the information I want to see when I follow someone’s business life, and all of this information is stored in Alfresco.

To make it easier to maintain, I want to be able to follow groups as well as users so that I don’t have to constantly be managing my “friend” list as people join and leave the company or move into different departments.

Kicking it up a notch, I also I also want to be able to follow (or subscribe to) a document, folder, discussion thread, site, etc. I want Alfresco to notify me when someone edits a document I’m following or adds a document to a folder I’m following.

To make this concept work, I’ll need an activity feed that’s personalized for me based on the users and groups and documents and folders I’m following. And since Alfresco added status updates in 3.4, it should include status updates for the users I follow.

To keep my activity feed from getting out of hand, I should be able to fine-tune the types of activities that appear in my feed (see below), and to keep the database from growing out of control, administrators should be able to adjust how long feed items are kept before they are deleted.

Configurable Activity Feed Dashlet (Inspired by TweetDeck)

I’d like to see a configurable activity feed dashlet, so that I can filter out activity types I’m not interested in. This way, I could add the dashlet to my dashboard multiple times with different settings. Maybe I would have one dashlet showing me the activities that are related to documents I’ve created, another one that shows activities related to users I follow, and a third that is showing subjects I’m interested in (see below).

The inspiration for this idea comes from TweetDeck, the Twitter client I use on my Macintosh. Using TweetDeck, I’ve created multiple Twitter feeds that let me watch certain subjects or groups of users, and it’s a thousand times easier for me to make sense of the zillion tweets that zoom past me everyday. I’ve got some feeds that I watch closely and others that I’m more casual about.

In large sites, Alfresco’s current activity feed can contains way too much information, so being able to filter out the activities that aren’t interesting to me will help me cut through the noise created by a zillion activities.

Email Digests (Inspired by LinkedIn)

Activity feeds are great for users that spend a lot of time in Alfresco, since they see their feed every time they look at their dashboard. Users that have embraced Facebook in their personal lives are comfortable scanning a feed to stay on top of things.

But the activity feed isn’t going to work for everyone. Some users don’t spend much time in Alfresco, and others will be resistant to constantly checking a giant activity feed. Many of these users still live in an email-oriented world, so for them, I want email digests that recap their activity feeds or notify them when documents they are following have been modified. This is similar to what I get from LinkedIn, a social networking site that I don’t visit very often. LinkedIn sends me emails from time to time that keep me updated and let me easily re-engage.

If I had my way, the email settings would be configured as part of the Activity Feed dashlet. This way, I could elect to have some of my feeds email me a daily digest and other feeds email me in real-time when the activity happens.

According to my friends at Alfresco, a basic version of this feature is coming in the next release or two, but it will likely only be a daily digest of a single global feed.

Include Users and Subjects in Comments (Inspired by Twitter)

The current version of Alfresco lets users comment on documents, folders, blog entries, and links. This is nice, but there are a couple of improvements I’d like to see.

First, the way that the comment is displayed could be improved. Today, when a user comments on a document, a message is added to the activity feed saying, “[User] commented on [Document] in site [Site].”

I have to click the link to view the document details page so that I can see the comment. I don’t like this, because it takes a while to load the document details page, and that just wastes time. Also, it breaks my train of thought. The activity feed is great for quickly staying up to date on what’s happening in Alfresco. But when I click a link in the feed, I’m taken out of that context to an entirely different page, and when I return to the dashboard, I struggle to find where I was in the feed.

So, the next enhancement I want to see is for the text of the comment to be included in my activity feed.

I also want to be able to direct my comments to specific users, similar to how Twitter works. I want to be able to include @username in a comment and have the comment be highlighted in that user’s feed. This would make the discussion around documents much richer, since I could say something like “I think this report needs more detail. @johndoe, what do you think?” This tells John Doe that I’m specifically looking for a response from him.

This notation, used widely on Twitter, will be easy for users to pick up, and if we had configurable dashboards, I’d create a feed just of the comments that referenced my user name. I’d set this feed up to email me in real time as the comments were created, so that I could respond quickly.

Building on the idea of using Twitter syntax, I also want to use hashtag notation in Alfresco comments. For the uninitiated, hashtags (placing ‘#’ in front of a short word or phrase) is a simple way of tagging a tweet so that others interested in that subject will see it. For example, #alfresco is used on Twitter to indicate that the tweet is about Alfresco, and I have a search set up on Twitter that shows me all the tweets tagged with #alfresco, whether I follow that user or not.

Allowing hashtags in Alfresco comments would let users tie their comments to a particular project or subject, and other users could search for the hashtag or create a feed of comments related to that hashtag. It would provide another dimension to staying on top of what’s going on, and would be particularly useful in a large site with lots of users and lots of comments.

What social features do you want to see in Alfresco? Continue the conversation in the comments.

Let’s say that you also have some sites that you use to collaborate with external customers, partners, or vendors. You’ve opened up a hole in your firewall so that these external users can access Alfresco Share, and you’ve been careful to invite those users only to the specific sites that they should have access to.

You may think everything is secure, but no matter how careful you are, when you are letting external users into your Alfresco system, it’s possible for your internal users to accidentally share sensitive confidential information with those external users.

Here’s how it can happen.

1) Creating Public Sites

When you create a new site in Share, the default visibility is Public, which means that every user in the repository can participate in the site as a consumer, even if they haven’t officially joined the site.

Documents in public sites are easily returned in search results, and external users can download them and read them without your knowledge. Public sites are a bad idea if you have any external users in your system.

2) Sharing Documents with Everyone

Users in Share have the ability to change permissions on documents they create. They can share the document with other users, other groups, or with a special group called “EVERYONE”.

It’s not uncommon for a user to share a document with EVERYONE, assuming that it means everyone in the company, and not realizing that it includes every user in the repository. Most casual users don’t even realize that the system is being accessed by external users. But when a document is shared with EVERYONE, it’s available to both internal and external users, and this can allow sensitive information to fall into the wrong hands.

3) Inviting External Users to Internal Sites

Managers of sites have permission to invite other users to join their site. They can invite users that already exist in the repository, or they can invite external users that don’t yet have a user id. Alfresco Share makes it really simple to invite external users, and this can be a problem for some companies.

Let’s say a site manager invites an external user, but the user does not yet have an login ID for Alfresco. No problem – Alfresco will automatically create a user account, give that account access to the site, and send the user an email with their username, password, and a link to join the site.

While this behavior might be just what you want for an external community site, most IT departments would be horrified to learn that random Alfresco users can give strangers access to the corporate document repository without anyone knowing about it. To make matters worse, the current site manager can give the external user manager-level access, and now the external user can invite even more external users.

4) Putting External Users in Internal Groups

Like most enterprise applications, Alfresco has the ability to create groups of users that make managing permissions much easier. In many companies, documents are shared with groups of users, rather than individuals. You might want to give the Marketing group access to the sales team’s documents, for example.

Lets say that several months later, the marketing team brings on a contractor from an external advertising agency. It only makes sense to put that contractor in the Marketing group so she can get access to the marketing documents. But it’s often the case that the left hand doesn’t remember what the right hand has done, and unbeknownst to everyone, this external contractor now has access to all the proposals and sensitive sales documents that you probably wouldn’t want her to see.

The Blue Fish Solution

The way we’ve solved this problem for our clients is to introduce the concept of external users, external groups, and public folders. In our solution, we’ve written a job that executes every 30 seconds, identifies external users, and restricts them to accessing only those documents under a public folder. The nice thing about this approach is that it doesn’t require any customizations to Share or any serious modifications to the Alfresco server.

Here’s how it works.

When our job runs, it identifies all the external users. Any user without a company email address is considered an external user.

The first thing that the job does is to make sure that no external users are members of your internal groups. This prevents problem #4 above. We consider most groups in Alfresco to be internal groups, so if we find an external user in an Alfresco group, we will remove that user from the group.

Our job then finds any sites we haven’t seen before. If we encounter a new site, we create a Public folder in that site and set up a new site role called External User. This role appears in all the normal Share dialogs, so you can invite users to your site and give them the External User role. The External User role has no access to any documents except those in the Public folder.

Our job then starts to examine every existing site, looking at the site roles. If we find an external user with a standard site role (Consumer, Collaborator, Contributor, or Manager), we will remove the external user from that role and assign them the External User role. This solves problem #3 above.

The last thing our job does is to look at all the existing documents and folders. If we find that an external user has access to a document or folder outside of the Public folder, we remove that user from the permissions. So even if you explicitly grant an external user access to an internal document, our job will revoke that access 30 seconds later. If an internal user wants to share something with an external user, she has to move it or copy it into the Public folder.

If our job sees any documents or folders that give access to EVERYONE, we revoke that access since it would be visible to external users. In our system, you can’t give a document access to EVERYONE. This solves problem #1 and problem #2 above.

Whenever our job does takes some action, it emails the Alfresco administrators so they are aware of what’s going on.

Every once in a while, Alex Bilstein, our user interface design guru, writes up an internal report to keep the rest of the Blue Fish team up to speed on the rapid advancements that are taking place in the world of web development, particularly as it relates to front-end development in the web browser tier.

We thought it might be helpful to others out there, so we’re sharing it outside of Blue Fish for the first time. Enjoy!

Download Alex’s Client Side Report: January 2011 (pdf).

Watch Painless Document Scanning and Indexing with Alfresco.

There are tons of wiki pages, blogs, articles, code samples, and other resources on the internet that contain valuable information about Alfresco, but wading through it all can be a daunting chore. In fact, there’s so much information out there, it’s hard to know where to start.

My colleague, Nicki DuBose, has written the article I’ve been waiting for! It’s a list of best places to go to learn about Alfresco. Nicki and the rest of the Blue Fish team picked out their most helpful articles, web casts, blog posts, wiki pages and compiled them into the ultimate Alfresco resource index on the internet.

Check out Where to Go to Learn About Alfresco.

Getting Started: How Alfresco Works

• Terminology Guide – Skim the Alfresco terminology guide before diving into details of the product.
• Aflresco Documentation Resource Center – These “Getting Started Tutorials” are great for first-time Alfresco users: “Alfresco Explorer Document Management” and “Getting Started with Alfresco Share Collaboration”. This site is organized like online help, so navigate the tree on the left to find these tutorials.
• Alfresco in an Hour – This webcast will show you many of the features available in Alfresco.
• Replacing your Shared Drive with Alfresco – This webcast walks you through the “new to Enterprise Content Management” scenario. Take a look if you are new to ECM.
• BOOK: Professional Alfresco: Practical Solutions for Enterprise Content Management – Many will find the first 5-8 chapters useful in understanding the Alfresco landscape.
• Try Alfresco for Yourself – This is the best way to learn about how Alfresco works is to play with it.
  • The Cloud Trial is a simple way to take a look at Alfresco Share without installing the product.
  • You can Download Alfresco’s Community Edition free of charge and install it on your own server.
  • You can try the Enterprise edition of Alfresco free for 30 days.

Administering and Configuring Alfresco

• Data Dictionary – To understand the existing object model, you should review the existing Alfresco Data Dictionary. This link also gives you a step-by-step guide to defining a new model definition with new content types, aspects, and associations.
• BOOK: Professional Alfresco: Practical Solutions for Enterprise Content Management – Chapters 14-18 will guide you through many of the most common customizations.
• Administering an Alfresco ECM Production Environment – In addition to information about installing and administering Alfresco, this document contains a big section on configuration that is worth reviewing. (Available only with an enterprise license.)
• Alfresco Documentation for Community Edition – Installation and configuration guides for the Alfresco Community Edition.
• Alfresco Wiki – The Alfresco Wiki is full of great information, but it can be hard to know where to start. The content on the Wiki is always in flux, so pay close attention to the version/release information in your quest for knowledge.

Alfresco Development and Customizations

• BOOK: Professional Alfresco: Practical Solutions for Enterprise Content Management – This book is useful for eveything from understanding the Alfresco landscape to implementing your own customizations.
• BOOK: Alfresco Developer Guide – This book is full of step-by-step instructions on how to implement many of the most common configurations and customizations. The book will help you do everything from setting up your development environment to packaging and deploying your newly developed customizations.
• Alfresco Forums – A decent, free resource for getting your questions answered. If you are browsing the forums, in search of a particular roadblock or issue, pay particular attention to the date on the forum threads. Alfresco has undergone many changes in the years since it was created, so some of the information on both the forums and the Alfresco Wiki may be deprecated or obsolete in the most recent releases.
• Alfresco Issues – Search for existing issues in Alfresco or add new ones.
• Alfresco Developer Tools – The developer tools is a sample code archive. Make sure the code is still applicable to your Alfresco release.
• Alfresco API Reference – The API reference is a great resource when you’re writing custom Web Scripts.

Keeping Up (Blogs and Tweets)

• ECM Architect – Jeff Pott’s blog is a great way to keep up with some of the changes in Alfresco and Enterprise Content Management.
• Peter Monks: Alfresco from the trenches – As head of the Alfresco North American consulting group, Peter Monks blogs about everything ECM.
• Alfresco Blogs – A listing of lots of other Alfresco Blogs on the web
• If you use twitter, here’s what to follow:
  • @mikeytrafton – Blue Fish founder and Chief Architect – retweets the most interesting Alfresco tweets from around the tweetosphere
  • @Alfresco – Alfresco’s company twitter account, posts Alfresco related news and retweets most Alfresco employees
  • @pmonks – Alfresco consulting guru
  • @johnnewton – Alfresco’s co-Founder and CTO
  • #Alfresco – tweets about Alfresco from the entire Alfresco community

General Resources

Books

• BOOK: Professional Alfresco: Practical Solutions for Enterprise Content Management – Written by Alfresco CTO and engineering staff
• BOOK: Alfresco Developer Guide – Jeff Potts’ Book
• BOOK: Alfresco 3 Web Services
• BOOK: Alfresco 3 Enterprise Content Management Implementation

Downloads

• Community Edition – The open-source aspect of Alfresco means it is free to download, customize and even use in a production environment.
• Enterprise Edition – The Alfresco 3 Enterpise Edition is available on a per CPU license basis.

Websites

• Alfresco.com
• Aflresco Documentation Resource Center
• Aflresco Forge Community
• Alfresco Forums
• Alfresco Network (Requires Enterprise License)
• Alfresco SlideShare
• Alfresco Webcasts
• Alfresco Wiki
• Alfresco Youtube

Technical

Alfresco Technical Resources

• Alfresco Web Scripts
• Web Scripts Framework
• Spring Surf Platform – Developers Guide
• Alfresco Wiki

Code Samples

• Dr. Q’s Workshop – Hello World Dashlet
• Blue Fish – Alfresco Source Code Organization
• Example Dashlet Source Code

Third-Party Resources

• Apache Lucene
• FreeMarker
• Java Server Faces
• JSMin – The JavaScript Minifier
• Spring
• Spring Surf

We’ll be hosting these events in 6 cities across 3 states:

• Austin, TX – Tuesday, January 18 – Register
  
• Houston, TX – Wednesday, January 19 – Register
  
• Dallas, TX – Thursday, January 20 – Register
  
• Boston, MA – Tuesday, January 25 – Register
  
• Orlando, FL – Wednesday, January 26 – Register
  
• Miami, FL - Thursday, January 27- Register

I’m really excited to share this information and help you plan your 2011 ECM initiatives. Space is limited, so please register early to secure your place. I hope to see you there.

We’re trying to get the word out to as many people as possible, so this time, we’ll be hosting these events in 7 cities across 3 states:

• Houston, TX – Tuesday, October 5 – Register
• Dallas, TX – Wednesday, October 6 – Register
• Austin, TX – Thursday, October 7 – Register
• San Antonio, TX – Friday, October 8 – Register
• Atlanta, GA – Tuesday, October 12 – Register
• Tampa, FL – Wednesday, October 13 – Register
• Jacksonville, FL – Thursday, October 14 – Register

This round of lunch and learns will be focused on Web Content Management (WCM) and Records Management. I’ll be presenting at each of them, and I’m hoping to be able to show you some of the cool new WCM features that are coming down the road.

Space is limited, so please register early to secure your place. I hope to see you there.

Custom fonts are all those cool looking fonts that you see on posters, billboards, and other print advertising. Most computers don’t have these fonts installed, so designers buy them from font “foundries” to use in their designs. Web site designers are at a disadvantage, because in order to use a font on a web site, it needs to be installed on the web visitor’s computer, and most computers have only a handful of standard fonts installed. This is why most web sites use the same two or three fonts over and over again – those are the fonts that are installed on every computer out there.

Over the years, crafty designers and web developers have been coming up with clever ways for web sites to use custom fonts even if they are not installed on their visitors’ computers. The techniques have evolved over time, and now there are some pretty good ways of doing this.

Alex has written three articles that will get you up to speed on what you need to know to use custom fonts on your web sites.

• What to Know When Considering Custom Fonts for Your Web Site
• @font-face: The Easiest Way to Use Custom Fonts on Your Web Site
• Web Font Realities – A Case Study in Implementing Custom Fonts on a Modern Web Site

Introduction

Recent advances in browser technology and font creation tools mean that web fonts can be used today with very good browser support. Unfortunately, web font licensing can complicate the issue. In this article, we’ll discuss the technical challenges and legal process we followed when implementing web fonts on the Blue Fish Development Group web site.

Web Font LIcensing

The Blue Fish corporate identity includes the Trade Gothic font from Linotype GmbH. We’d like to use that font on our web site for various headers, links, and other callouts in order to reinforce our brand throughout the site. Let’s take a look at what an article title looks like with Arial (top) compared to Trade Gothic (bottom):

Figure 1:
A comparison of font faces showing Arial (top) and Trade Gothic Bold Condensed No. 20 (bottom).

That’s a pretty striking difference! So, how can we display our preferred font for our article titles? As we described in our “@font-face: The Easiest Way to Use Custom Fonts on Your Web Site” article, implementing @font-face is our recommended best practice for implementing custom fonts on modern web sites. But that doesn’t necessarily mean that we could just run off and follow our own best practice – first we had to find out if Linotype allows web font linking (the general term for using @font-face) as part of their license agreement.

We were in luck – according to the Linotype web site, they do allow web font linking! Unfortunately, their web font embedding license page states they only permit the EOT font format, and there is no information on pricing. We had to contact them directly for more information.

After some communication with a Linotype representative, we learned that Trade Gothic can be licensed for EOT web linking, but not TTF, OTF, or WOFF linking due to the lack of encryption in those formats. That’s a start, but EOT is only supported by Internet Explorer, which is about 50% of our traffic – how will we display our font to everyone else?

Fortunately, our Linotype contacts told us that they also allow the use of sIFR and Cufon, even though it isn’t explicitly stated on their web site. While we would prefer to use @font-face for all browsers, this is an acceptable method which will allow us to use our preferred font.

With the allowed formats understood, we then needed to purchase a license. Linotype’s US licensor is Monotype Imaging (fonts.com), and we contacted a customer service representative. The first thing they asked is how we’ll be using the font. Is it only for displaying text on the web site (requiring an EOT license), or will it be used in an online application that allows users to generate their own text (which requires a Web Server License. In our case, we only need an EOT license.

A sales representative then contacted us with a bit of surprising news – Monotype had just launched a new hosted web font service. They will continue to offer their traditional web font license but their preferred model is the new hosted service. There will be an initial free beta period with approximately 2,000 fonts available, followed later in the year by a commercial service featuring their entire library of over 7,000 fonts. While we are currently hosting the fonts ourselves, we have decided to purchase their hosted license in the future once it moves out of beta (until then, we’ll keep using the method described in this article). In both cases the license fee is determined by the number of unique visitors per month and the number of fonts licensed. However, not only is the hosted service cheaper, but it includes a content delivery network which means better performance for end users.

With the licensing issue resolved, we moved on to the technical implementation. In our case, that means using @font-face to serve EOT fonts to Internet Explorer users with a Cufon fallback for all other browsers.

Generating the EOT Font

Monotype doesn’t provide EOT files, so we needed to generate those ourself. In order to do that, we used Font Squirrel’s @font-face generator and our standard desktop OTF font. Here are the settings we used: selected EOT, Add Hinting, Remove Kerning, subset our fonts to Basic Latin (because our site is English only), choose the Mo’ Bulletproofer CSS code, and agreed to the EULA statement. This is what it looked like on the Font Squirrel web site:

Figure 2:
Font Squirrel’s @font-face Kit Generator

The generator provided us with the following three files:

• EOT font file
• CSS sample file
• HTML sample file

The resulting EOT file is only 20KB as opposed to the original 29KB OTF. The generated CSS file has two rules, but we only needed the first one for EOT:

        
@font-face {
	font-family: 'trade-gothic-bold-condensed';
	src: url('tradegothicltstdbdcn20.eot');
}

    

This is the only CSS rule needed to support Internet Explorer, which according to our web site analytics is about 50% of our traffic. We used Cufon to support the other half of our visitors.

Generating the Cufon Font

As I mentioned earlier, Linotype allows the use of sIFR and Cufon under the EOT license. These are similar technologies, and each has their own advantages. But ultimately we decided to use Cufon due to its use of standards based technologies rather than relying on the Flash plugin. The main advantage this provides is that our fonts will work on Apple’s iPhone and iPad.

You may have noticed that the Font Squirrel @font-face generator offers Cufon as one of the format options. However, the official Cufon generator sometimes provides better compression, and it also allows domain binding as an extra security measure. Once again using our original OTF Trade Gothic Font, we went to the official Cufon generator and uploaded our font. We chose a name for the font (the same font-family name we are using for EOT), agreed with the EULA, subset the font to Basic Latin, and limited usage to *.bluefishgroup.com. The rest of the options we left as defaults. This is what it looked like on the Cufon web site:

Figure 3:
Cufon font upload and naming

Figure 4:
Cufon font settings

Figure 5:
Cufon security

The resulting Javascript font file is only 16KB. After we had generated the Cufont Javascript font file, we downloaded the minified cufon-yui.js library file as well.

Putting it all Together

At this point, we had an EOT font file, a Javascript font file, and the Cufon Javascript library file. We needed to upload those three files to our server and include them in our site header. Next, we added the CSS rule above to our existing CSS file.

When Internet Explorer reads this rule, it will download the EOT file and use it on our site. All other browsers will ignore it because they do not support EOT. Ideally, you should place this CSS in a separate style sheet that is only served to Internet Explorer.

Finally, we needed to enable Cufon on our site. We had already completed step 1 and 2 in their usage guide (generating the font and downloading the library). The last thing we had to do was add the Cufon Javascript call to our base Javascript file.

        
Cufon.replace('.title');

    

This tells Cufon to use the Trade Gothic font for any element on the page with the class ‘.title’. You may have noticed there is a step 4 in the Cufon Usage guide – ‘Making Internet Explorer behave’. We don’t have to worry about that because we’re serving EOT to IE.

We were not quite done yet. The above code will replace all of our ‘.title’ elements with Cufon elements, but we only want Cufon to target browsers other than IE. So we had to add some conditional compilation to our Javascript to make sure that only non-IE browsers get that code:

        
/*@cc_on
    @if (@_jscript == false)
        */initCufon();/*
    @end 
@*/

    

I won’t go into the details here, but conditional compilation is a fully supported IE mechanism to target different versions of the IE JScript engine. This particular snippet calls the Cufon method for all browsers except for IE.

At this point, our web site could serve Trade Gothic article headers to all of our users, but we were not quite ready to go live. Next, we optimized our Javascript (including the Cufon generated Trade Gothic Javascript font) by using Gzip on the server. This can reduce the file size by 25%-50%, which translates to a better user experience.

Finally, we needed to make sure we were using the right font stack. In order to reduce the impact of any flash of unstyled text, we chose a stack with fonts of similar width and kerning. In the case of Trade Gothic Bold Condensed No. 20, we choose the following stack:

        
.title {
	font-family: trade-gothic-bold-condensed, "Arial Narrow", Impact, Sans-Serif;
}

    

Which results in the following font cascade:

Figure 6:
Font stack cascade from Trade Gothic to Sans-Serif

Conclusion

Ideally, we would use @font-face for all browsers. But because of our restricted web font license, a little creative fall back was used. With the help of a little conditional compilation in our Javascript, IE users are served EOT font files using @font-face, and all other browsers are served Cufon fonts.

Hopefully, one day browser vendors, font foundries, and web designers can all agree on a single format and implementation for web typography. Until then, we’ll continue to use a combination of techniques to ensure we deliver the best experience possible to all of our users.

Introduction

Web designers want to create beautiful web sites that stand out from the crowd, and in order to do so these designs often use non-standard fonts or typefaces. However, since most computers probably don’t have these custom fonts installed, they will often display the designer’s web site with unexpected results. For example, notice the typography in the design below:

Figure 1:
Web site brizk.com with generic fonts on the left, and custom fonts using @font-face on the right. Notice that not only do the fonts change in visual appearance, but the layout is affected as well.

See the difference? When attempting to use custom typography on the web, designers have typically been left with two options – creating images that contain the text on the web site, or changing the design to use one of the boring standard fonts that are supported on every computer.

However, recent advances in browser technology and emerging font formats are affording new opportunities for typographic control. Unfortunately, these advances are not without their issues, because different browsers support different font formats, and some font foundries restrict the ways in which their fonts can be used on the web.

This article will help you understand the two most important aspects of using a custom font on your web site:

1. When, why, and how to use the latest web font technologies, and
2. How to navigate the emerging web font licensing landscape.

Technical Options for Using Custom Fonts on the Web

In the past, most content on the web was static. Designers and developers created their pages locally and then uploaded them to a server. If there was a need for custom typography, the designer would create the text as an image using an application such as Photoshop. Although not ideal (it required a graphic designer to make any text changes), it worked well enough for the committed web team.

However, with the introduction of web content management systems and user generated content, this strategy became difficult at best. No longer could images be manually created to render text with custom fonts, because the graphic designer no longer made all of the updates to the web site.

To address this lack of font flexibility, a number of workaround techniques have been developed that allow custom fonts to be used without creating a separate image for every piece of text. Let’s take a look at the three of the most common techniques from recent years.

Dynamic Image Rendering

There are server side techniques that can be used to convert plain text into image text on the fly. This is essentially the same thing as having a designer create the image text for you, but it is instead done automatically by software. The advantage to this technique is that it is cross browser compatible and works even in very old browser versions as the end result is simply a standard image. However, the images cause pages to render more slowly, and there are often technical implementation challenges. There are third party solutions that will do the processing for you, but in the end you’re still left with an image that is less desirable in respect to SEO, accessibility, and usability.

Today and even in the near future, rendering text as an image still provides the most control and browser consistency. Graphic design programs like Adobe Photoshop allow a degree of control over kerning, tracking, scaling, and other properties that browsers are simply incapable of via CSS. If it has to be pixel perfect, this is still the only way to go. However, this technique doesn’t scale very well for user generated content.

sIFR

sIFR has probably been the most popular font replacement technique in recent years. It is a Javascript and Flash solution that dynamically replaces your plain text with a Flash object.

Here’s how it works. You identify headlines or other text where you want to use a custom font by assigning a specific CSS class. After the page renders in a browser, sIFR will parse the page and replace the rendered text with a small flash file that displays that text in your custom font. This usually happens so quickly that the web page visitor doesn’t even notice.

With sIFR, you can achieve very satisfactory results with a good degree of typographic control. However, it comes with a performance cost, has some limitations in regard to styling via CSS, is a bit cumbersome to set up, and requires the Flash plugin (which means it won’t work on the iPhone or iPad). Despite these shortcomings, sIFR is quite popular and offers broad browser support due to the ubiquity of the Flash plugin.

Cufon

A relatively recent technique, Cufon is similar in concept to sIFR but relies on web standards as opposed to a proprietary plugin. Using the appropriate combination of Javascript, Canvas, SVG, and VML, Cufon replaces your plain text with HTML vector elements in your desired font. It works in all modern browsers including mobile devices. The results are quite nice, and the Cufon web site has a generator that creates all the necessary files for you.

The down side of Cufon is that the text is not easily selectable, some common CSS styling is unavailable (I’m looking at you, text-decoration), and there are some accessibility issues.

@font-face

@font-face is a CSS rule that allows you to publish fonts on your server and then serve them up to browsers just as you would any other font already installed on the client operating system. This technique is also called font linking, because the CSS links to a font file on your server that is downloaded and used by the browser.

The @font-face CSS rule has been around for over 10 years (although only recently as a web standard), but it has seen little use because it only worked in Internet Explorer. However, recent versions of Firefox, Chrome, Safari, and Opera all support @font-face in one way or another.

While all modern browsers support font linking, they differ in what font formats they support. Fortunately, it is now possible to support virtually all browsers with a couple of font formats and some simple CSS tricks that we will show you later in this article. While there are limitations to the amount of typographic control one has when using @font-face, the benefits far outweigh the limitations.

Let’s take a look at the different font formats currently in use.

Embedded Open Type (EOT)

Embedded Open Type has been around since the late 1990′s, but because it was difficult to work with and only worked in Internet Explorer, it never gained widespread use. Today, however, EOT files are much easier to create and can be used alongside other formats, so EOT has become much more relevant. While it still only works in IE, EOT is stable, compressed, and because it is encrypted, it is the font linking format favored by most font foundries.

True Type Fonts (TTF) and Open Type Fonts (OTF)

True Type Fonts and Open Type Fonts are probably what you have installed on your desktop computer. Recent versions of Firefox, Safari, Chrome, and Opera all support @font-face linking to TTF and OTF fonts. These font formats are very easy to set up and work well. However, these are considered raw font formats and they lack compression which means longer download times. Perhaps more importantly, they are not encrypted. Most font foundries don’t allow their fonts to be linked with these formats, since unencrypted font formats make font piracy easier.

Web Open Font Format (WOFF)

WOFF, or Web Open Font Format, is a recent format that is an open standard very similar to TTF and OTF. WOFF improves on TTF and OTF by employing compression which helps reduce download times. It also features a special PrivateData entry in the font header that allows a foundry to insert copyright and licensing information. While this gives the foundry more control over WOFF than they have with TTF or OTF, it doesn’t actually encrypt the font. It’s important to note that many foundries have publicly announced support for the WOFF format, but that doesn’t mean they actually license it for @font-face linking. WOFF has recently been submitted to the W3C as a formal standard, so we may be seeing broader support in the future.

SVG

Some browsers support SVG, or Scalable Vector Graphics, for rendering fonts. The results often look nice, but there are accessibility, usability, and security concerns. This format should only be used as a last resort or in very controlled situations.

Browser Support

@font-face is supported in recent versions of all major browsers in at least one of the aforementioned formats. As long as your font license allows web linking to these formats, you can reliably support 90%-95% of all browsers today. And that number is only going to rise as users upgrade their browsers. If it’s important that you reach the remaining 5%-10% of your audience today, you can use a fallback technique and serve those users sIFR or Cufon.

IE version 4+	Firefox version 3.5+	Chrome version 4+	Safari version 3.1+	Opera version 10+
Embedded Open Type (.eot)	True Type (.ttf), Open Type (.otf),Web Open Font Format (.woff)	True Type (.ttf), Open Type (.otf)	True Type (.ttf), Open Type (.otf)	True Type (.ttf), Open Type (.otf)

@font-face Implementation

Read our companion article @font-face: The Easiest Way to Use Custom Fonts on Your Web Site for a step-by-step description of how to implement @font-face on your web site.

Fluid Font Stack

Now that we know about @font-face let’s discuss using a fluid font stack. If you’ve worked with CSS, then you’ve probably seen the typical font stack:

        
.title {
	font-family: arial, helvetica, sans-serif;
}

    

Look familiar? The browser is simply going to step through this list of fonts and use the first one that it finds installed on the user’s machine. If we want to use a custom font, our CSS might look something like this:

        
@font-face {
	font-family: 'museo-slab';
	src: url('museoslab.ttf') format('truetype');
}

.title {
	font-family: museo-slab, arial, helvetica, sans-serif;
}

    

In this case, the @font-face rule defines a font-family which we’ve named ‘museo-slab’ which in turn uses the font on our server named ‘museo-slab.ttf’. When the browser reads the class with the font-family property, it knows to download the ‘museo-slab.ttf’ font from the server instead of skipping over it. While the font is being downloaded, some browsers will move on to the next font and use that until the download is complete. This means that it’s more important than ever to use fonts with similar width and kerning so that the shift in text isn’t so harsh. Consider the following image showing a line of text at the same size and weight, but using different fonts, resulting in different sentence lengths:

Figure 2:
Using different font faces can result in different text widths.

When using @font-face, rather than choosing a stack of similarly styled fonts, you may wish to choose a stack of similar width fonts. These fonts should adhere to the following order: custom web font, web-safe font, core font, and finally a generic font. Read our companion article “Web Font Realities – A Case Study in Implementing Custom Fonts on a Modern Web Site” for one example of choosing a fluid font stack in conjunction with @font-face.

Licensing Issues When Using Custom Fonts on the Web

Now that you understand @font-face and how to use it, let’s talk about licensing. The first thing to understand is that desktop licensing is different from web licensing. A typical desktop license allows the font to be installed on a fixed number of computers, and it can be used on multiple projects for a one time license payment.

Most web licenses only allow a font to be used on a single web domain, and are typically licensed through a subscription payment with tiered pricing based on how much traffic comes to your web site. This can be further complicated depending upon whether you intend to use the font to simply display content, or if you are going to use the font in some type of web application that will allow end users to use the font in some way (for example, a custom t-shirt design application). Font foundries have different licenses for each of these uses.

Web licensing is really the issue that is preventing @font-face from wider implementation today. In order to use a font across all browsers, the foundry’s license must allow linking to unencrypted font files (TTF, OTF, WOFF). Most foundries simply do not allow this because unauthorized end users could download and use those fonts on their desktop. Before using @font-face, be sure to check the End User License Agreement (EULA) at the font foundry’s web site to determine what formats of web linking they allow, if any. You may need to contact a sales rep as web licensing is typically more complicated than desktop licensing, and there are some foundries that do not allow web linking in any format.

A great source of free fonts that do allow all formats of @font-face linking is Font Squirrel. It’s important to understand that just because a font is free to download and use on your desktop, that doesn’t necessarily mean that it’s free to use on the web. Font Squirrel has done the hard work of locating quality free fonts that do allow web linking. Read our companion article @font-face: The Easiest Way to Use Custom Fonts on Your Web Site for a detailed explanation of how to implement @font-face on your site.

Fonts as a Service

If web licensing sounds too complicated, or your font’s EULA doesn’t allow font linking, you should investigate font service bureaus. Service bureaus handle all the licensing issues for you, host the fonts, and provide you with some simple CSS and Javascript that enable you to use @font-face as already described. Some foundries act as their own service bureau, and some bureaus are independent and offer fonts from numerous foundries. Browser support is usually very good, reaching about 90%-95% of all users. Font service bureaus can simplify your setup process, but there is typically a subscription payment model based on bandwidth, and you will be relying on a third party service to serve your fonts.

Here is a short list of foundries and font service bureaus:

• Google – Small collection of open source fonts
• Typekit – Numerous mid-size and small foundries
• Kernest – Numerous mid-size and small foundries
• Ascender Web Type – Monotype, Microsoft, Ascender
• Typotheque Typotheque fonts
• Fontdeck Numerous mid-size and small foundries
• Fonts.com Web Fonts – Monotype, Linotype, ITC, more

When and Where to Use Web Typography

Headers, menus, and callouts are the most common use cases for custom typography and typically the safest to implement. These elements are usually more predictable in their content and behavior. For example, it’s unlikely that you will need special symbol glyphs in a navigation menu. However, it’s important to pay extra attention to any elements that may have line wraps. Some browsers will render the content in a default font, and then render the custom font once it has finished downloading. If the default font is wider than the custom font, it may force a line wrap that the custom font does not need. Depending on the browser, this may result in a reflow of the text or an undesired line wrap. This is sometimes called the FoUT, or Flash of Unstyled Text.

Using custom fonts for body text should probably be avoided in most cases. Body fonts are typically rendered at a smaller size than headers and menus. This means your font choices are limited because not all fonts read well at small sizes. Additionally, body copy is less predictable in terms of content – it may require special symbols, styles, or languages all of which would contribute to a very large font subset (see Internationalization below). In most cases it’s better to use a web safe font or core OS font.

Internationalization

Internationalization is another aspect to user generated content where @font-face can really shine. Rather than creating different images for each language (in a menu for example), @font-face renders the text using your custom font in the specified language. The one caveat here is file size. In order to support languages other than English, you have to subset your font to include these other languages. This can drastically increase the size of your font file and consequently the user’s download time. If you are providing content in multiple languages, you’ll probably want to create a font file subset for each language and only serve up the requested language font file. Fortunately the process of subsetting is easy with a tool like Font Squirrel’s @font-face Generator.

Conclusion

The future of web typography is @font-face, and with the right font (and license) selection you can begin using it today. But just like any new technology, be sure to only use it when appropriate. Headers, menus, and callouts are the most predictable content elements. Make sure to take download times into consideration, subset your fonts appropriately, and test thoroughly. Even though a new standard may emerge in the future that is supported by designers, browser vendors, and foundries, it will most likely simply be a new format that works with @font-face and plays nicely alongside the formats of today.

Introduction

Creative typography has long been unavailable to most designers on the web. We’re stuck with the same old boring fonts installed on all machines: Arial, Times New Roman, Georgia, etc. However, recent advances in browser technology and typographic tools are affording us new opportunities. Font license permitting, with very little effort, you can now choose from hundreds of fonts and still support 90%-95% of today’s browsers with very little effort. This article will tell you how.

Choosing the Right Font

If you have the authority to select the font that is used in your project, then your first step is the most important – choosing a font with the appropriate license. This can be a complicated issue, so if you wish to follow the path of least resistance, then head straight to Font Squirrel and choose one of the fonts offered on their site. All are free for commercial use in print or on the web. While there are other sources of free fonts on the web, some are of questionable quality. Also keep in mind that free for desktop use doesn’t necessarily mean free for web use. Font Squirrel has done the hard work of sourcing fonts from all over the web that are not only free for web use, they are of high quality as well.

Figure 1:
Font Squirrel is a great source of free fonts and the @font-face Generator

If you can’t find the right font at Font Squirrel, you’re not completely out of luck. Some foundries offer commercial licenses for web fonts, but be sure to carefully check the End User License Agreement (EULA). The license needs to allow something called unencrypted (or raw) font linking (.ttf, .otf, .woff) in order to follow the steps in this article.

Generating the Required Files

There are many different ways to download the necessary files and code from Font Squirrel, but I’m going to show you the easiest and safest method that will work with 90%-95% of all browsers. Once you’ve selected a font at Font Squirrel, download it to your desktop. Font Squirrel offers prepackaged @font-face Kits, but in this case we’re just going to download the TTF or OTF file because the @font-face Kit is overkill for our needs. You can simply browse all the fonts until you find one you like, then download the .ttf or .otf file.

Figure 2:
Font Squirrel font download vs. Kit download

After downloading the TTF or OTF file, visit their @font-face Generator web tool. Here we’ll upload our font, select some options, and download some generated files. So, after uploading your font (in our example below, “Daniel Regular”), here are the selections we want to make:

Figure 3:
Font Squirrel @font-face Generator

Implementing @font-face

Click on the “Download your Kit” button, and Font Squirrel will produce five files – three font files, a CSS stylesheet, and a demo HTML file. Upload the font files (.eot, .ttf, and .woff) to your server – these are the font files you’re going to link to. The HTML and CSS files are simply demo files to help you style your markup appropriately. The CSS file has two rules – let’s take a look at them:

        
    @font-face {
    	font-family: 'DanielRegular';
    	src: url(//:) format('no404'), url('daniel.woff') format('woff'), url('daniel.ttf') format('truetype');
    }
    
    @font-face {
    	font-family: 'DanielRegular';
    	src: url('daniel.eot');
    }
    
    

The first rule is for all browsers except Internet Explorer, and the second rule is only for Internet Explorer. Go ahead and add the first rule to your main stylesheet. Then, create a second stylesheet just for IE and add the second rule there. In your document head, include your primary stylesheet and then your IE stylesheet using conditional comments. For example:

        
        <link rel="stylesheet" type="text/css" href="style.css" media="screen" />
        <!--[if IE]><link rel="stylesheet" type="text/css" href="style-ie.css" media="screen" /><![endif]-->
    
    

So what’s going on here? The separate stylesheets and conditional comments ensure that each browser is served the proper font format. The first property inside the @font-face rule is where we define the name of the font. We can call it anything we want, and in this case we’re going to call it ‘DanielRegular’.

The second line is where we define the location of the font file. In the case of IE, this is the EOT font.

For all other browsers, the second line is a little more interesting. It has three comma delimited value pairs. The first value pair is actually a bit of misdirection for Internet Explorer. Normally, IE will attempt to download the TTF and WOFF files which it can’t actually use. That creates an unnecessary HTTP request which we would like to avoid. The syntax of the first value pair will cause IE to bail out and simply use the EOT file already defined, and other browsers will simply ignore it and continue on. The second value pair is the WOFF font file which will be used if the browser supports it (currently only Firefox 3.6). The third value pair is the True Type Font which will be used by Safari, Chrome, Opera, and Firefox 3.5.

After adding the @font-face rules to your CSS, you’re ready to go. Now when you’re defining your font-family elsewhere in your CSS, you can use ‘DanielRegular’ like this:

        
    .title {
    	font-family: DanielRegular, Arial, Sans-Serif;
    }
    
    

That’s it! This method supports Internet Explorer 4+, Firefox 3.5+, Safari 3.1+, Chrome 4+, and Opera 10+. That’s about 90%-95% of all browsers, and that number is only going to go up. If closing that last 10% gap is important to you, read our companion article Web Font Realities – A Case Study in Implementing Custom Fonts on a Modern Web Site.

Performance Tuning and Browser Quirks

You’ve got everything you need to begin using web fonts on your site at this point. However, there is some additional performance tuning that you can perform, and there are some dangers you should be aware of.

Because your custom font is not pre-installed on the end user’s machine, it must be downloaded from your server to the user’s browser which may cause a FOUT, or Flash Of Unstyled Text. The FOUT occurs while the fonts are being downloaded for the first time. Depending on the browser, the user may see either a blank space or a default font until the download is complete. It is therefore important to minimize the download time in order to eliminate or at least reduce the duration of the FOUT.

File Size

The first thing to consider is file size. A typical font face is around 50KB of data. Add in bold and oblique faces and you’ve got 150KB of data the user must download. So make sure that each font you use will bring enough value to the user to justify the additional download time. Once you’ve decided on your font faces, the following techniques can help reduce their file size.

Subsetting

A typical font face may have hundreds of glyphs, or characters. In addition to the standard latin alphabet, there are numbers, symbols, accented characters, foreign languages, etc. Subsetting allows you to select only a portion of the total glyphs to include in your font file. The subsetting process is made easy by the Font Squirrel @font-face generator. By default, the generator will use the Mac Roman subset which is more than adequate for most use cases. However, if your web site is only going to display English letters and numbers you should select the Basic Latin subset, cutting the number of glyphs in half and reducing the file size of the font. In our example above, our font was reduced from 53KB to 37KB just by subsetting to Basic Latin.

Gzip

In addition to subsetting you can configure your server to deliver the fonts using Gzip compression (WOFF and EOT are already compressed, so no need to Gzip those) with a future expiry date. Gzip will further reduce the file size and the expiry date will ensure that the user’s browser will cache the fonts after the initial download. Gzip can give you a 25%-50% savings in file size, so it’s definitely worth the effort.

Internet Explorer Quirks

There are a few special precautions you need to take for Internet Explorer. First, make sure that all the fonts linked in your CSS are actually being used on your site. Internet Explorer will download all files linked from your CSS file whether they are actually used or not.

Second, make sure that your @font-face rule is defined before any <script> tag is encountered. There is a bug in Internet Explorer that will cause @font-face to fail if the <script> tag is encountered first. CSS before Javascript is a good practice in general.

Targeted Downloads

As with any other file that must be downloaded by the user (CSS, Javascript, images), target your font downloads appropriately. If the font is used site wide, it makes sense to include the @font-face in your global CSS. However, if the font is only used by a few pages on the site, then it makes sense to create a specific CSS file just for those pages. That way the font will only be downloaded by users that visit those specific pages.

URI Encoding

I also recommend that you do not URI encode your fonts. It’s possible to use Base64 encoding to store your font files in CSS, but this can actually increase the size of the font. Additionally, because URI encoded fonts are generally stored in a single CSS file the browser will download all the different formats whether it can use them or not.

Local SRC

You may have noticed that there are three styles of CSS in the Font Squirrel @font-face generator. I recommend the Mo’ Bulletproof method because it avoids using a local source reference. Using a local source can theoretically improve performance because the browser will look for the font on the end user’s machine, but it’s been reported that Opera can crash in certain situations (my own testing has confirmed this). It’s also possible that a user might have a corrupt font which could potentially cause problems. Ultimately, it’s simply safer to deliver the font from your own server over which you have complete control.

Testing

Finally, it’s very important to test. @font-face is an emerging technology with behavioral and rendering variations between browsers. Make sure you test all the major browsers on both Mac and Windows platforms and continue testing as new browser versions are released.

Conclusion

Thanks to modern browser support for the @font-face rule and tools like Font Squirrel’s @font-face Generator, creative typography on the web is here today. If you follow the steps outlined above, you’ll be able to serve up custom fonts to the majority of your users today in a performant manner. If you’d like to read more about Web Typography in general or find out what you can do when your font license isn’t quite so liberal, read our companion articles “What to Know When Considering Custom Fonts for Your Web Site” and “Web Font Realities – A Case Study in Implementing Custom Fonts on a Modern Web Site”.

What makes Intranets so tricky is that they have to serve several masters. They often have to:

• Provide easy access to relevant information when an employee wants it (bottom-up)
• Communicate important information to employees whether they want it or not (top-down)
• Help teams collaborate amongst themselves in a secure manner (within groups)

These competing priorities can make it hard for a single Intranet website to meet all of your company’s needs. What makes it even worse is that since everyone in the company uses the Intranet, everyone in the company has an opinion on what it should look like (even if they don’t really know much about how to design these types of things).

Luckily, my clients have taught me a few tricks over the years about how to deal with these issues. So this webinar discusses some neat ideas for how Alfresco Share can be used as a corporate Intranet, creating a network of Intranet sites that work together to serve different stakeholder needs at different levels of the company.

Download now!

We’ll be hosting these events in 5 cities across 3 states:

• Austin, TX – Tuesday, July 13 – Register
• Houston, TX – Wednesday, July 14 – Register
• San Antonio, TX – Thursday, July 15 – Register
• St. Louis, MO – Tuesday, July 20 – Register
• Orlando, FL – Tuesday, July 27 – Register

This round of lunch and learns will provide an overview of Alfresco and describe how companies like yours are using Alfresco to address their ECM challenges at a fraction of the cost of traditional solutions. I’ll be at each and every one of them, and I’m planning to give a live demo, so wish me luck that the demo gods favor me on those days.

Space is limited, so please register early to secure your place. I hope to see you there.

You can download a trial of Alfresco Enterprise here.

The first webinar is Extreme Migration Makeover, and it was presented at EMC World by Bas Ursem with Solvay Pharmaceuticals. It’s Wednesday June 9th at 10:00 CDT (Chicago Time). If you are planning a content migration project, this webinar may be a must-see. It may be particularly important to you if…

• you are worried about system down time, or
• you are under the gun to do your migration on a short time line, or if
• you are afraid that the migration may be a lot harder than it appears on the surface.

And if that’s not exciting enough, another friend of ours, Nils Schlote with Roche, will be presenting a related topic, on Thursday 17 June at 10:00 CDT. Roche has a roadmap with a myriad of new systems, upgrades and consolidations so they have devised a global migration strategy that will enable project teams to address their unique migration challenges in a consistent, repeatable manner.

This presentation may be particularly interesting to you if…

• your company seems to always be doing migrations, or
• you are worried about a barrage of upgrades and consolidations coming down the pipe, or if
• you are struggling to get your arms around a global strategy.

You can watch more webinars here.

The room was full of people from life sciences companies, and they seemed to hang on every word. EMC World doesn’t have very many presentations like this on real-world case studies, and from what I saw, there is incredible demand for them. Attendees really want to hear and talk to companies that have been through these types of projects before. We are all hoping EMC gets the message and includes more case study presentations in the future.

Bas is a great speaker. He was able to turn a detailed presentation about migrating regulated pharmaceutical content into a fun and interesting session. At the end of the session, Bas rewarded folks that asked questions by giving them “goodies” that he had brought over from The Netherlands (one was a toy bus that fit the theme of the presentation, and the other was a bottle opener in the shape of a Dutch wooden shoe).

If you weren’t able to make it to EMC World, or If you were there but missed his session, you can catch an “encore presentation” of Bas’ session in June. Bas will be repeating his presentation on a webinar that you can stream over the Internet. If you are interested in seeing it, or you want to share it with a colleague, you can click here to register for Solvay Pharmaceuticals: Extreme Migration Makeover! on June 9th.

In no particular order, here are the things that stood out for me:

One Box Search Improved

The default search behavior will switch from “OR” syntax to “AND” syntax when you search for more than one word. Imagine that you search for “bear” and get 10,000 results, so you decide to be more specific by searching for “black bear”. In the FAST search engine, you might get back 100,000 results (all the documents containing either “black” or “bear”). In DSS, you would get back fewer results, since DSS returns only the documents that contain both “black” and “bear”. This is how Google and other internet search engines work, so it’s more intuitive for your users.

Optimized for Documentum Use Cases

Because EMC now has control over the internals of their search engine, they are able to optimize it for some of the key Documentum use cases. First, they optimized the ability to exclude results that the user doesn’t have permission to see. Queries that used to take minutes in FAST now take seconds in DSS. Second, they optimized the “folder descend” queries that search inside deep folder structures. In FAST, a query that took 44 seconds now takes less than 1 second in DSS.

Native Support for Facets

Faceted search refinement is something I’ve been excited about for about 5 years. For the uninitiated, this technology allows users to quickly refine large search result sets by drilling down on pre-defined filters (or facets). This approach is used by Amazon, Home Depot, and lots of other online retailers.

For example, if you search for “drug interaction” and get back 10,000 results, you can quickly refine that list to, say, only Word documents by drilling down on the file format facet. This might limit the results to 1,000 documents. Then you might refine on the document type facet, showing only the reports. That might limit the results to 100 documents. Refine by date modified to selecting only the documents created in the past week, and it will be easy to find the document you are looking for in the small number of documents remaining.

DSS supports facets out of the box, so Webtop, CenterStage, and other Documentum products can have much improved search interfaces.

No Additional Cost

Because DSS is part of the Content Server, it is available at no additional cost to existing Documentum customers with a current maintenance contract.

You Must Plan on Upgrading Next Year (2011)

Because Microsoft is no longer supporting FAST as an OEM product, EMC has some pretty aggressive timelines for replacing FAST with DSS. Support for FAST expires at the end of 2011, so if you want to be on a supported product, you’ll have to install DSS before then.

This might mean that you have to upgrade Documentum, since DSS will only work on D6.5 SP2 and later. So be sure to plan your budgets next year for the hardware and personnel necessary to perform the upgrade.

The good news is that the upgrade should be a lot less painful than the last time they replaced Documentum’s full-text engine. DSS can be installed alongside FAST (on different servers, but as part of the same Documentum environment). While it is indexing all of your content, you can keep using FAST. When you are ready, you can cut over from FAST to DSS pretty painlessly.

So what is xCP does and why ECM feel it’s necessary? xCP is trying to solve two problems:

• It’s hard to upgrade Documentum solutions because there are so many customizations that need to be migrated to the new version
• There is a class of important ECM challenges (managing what EMC calls “cases”) that are not being well served with traditional ECM solutions

What is a Case?

A case is the term EMC is using to describe the collection of information (structured and unstructured) that goes through a business process in order to reach a business decision. It might include multiple documents, some data from an external system, and some data captured from participants in the process. All this data has to come together in order for some important decision to be made.

Case Management is the creation, routing, collaboration on, and monitoring of these case materials. It includes the orchestration of various interactions that are needed to make an optimal business decision. If a case is a file folder full of documents, case management is the set of processes that makes sure the folder goes to the right people and gets filled up with the right documents.

For example, imagine that you are a mortgage company and someone applies for a loan. They fill out a form and give it to a loan administrator. Based on the type of loan being requested and the amount, the load administrator requests a credit report, tax documents and other financial information. Once all of that has been gathered, the loan administrator passes the file over to the loan approver with the ability to approve a loan of that amount. This person looks over all the information, requests some additional documents (maybe a survey or an appraisal) and then approves or denies the loan. There are lots of other steps in the process – you get the picture.

Collecting all these documents and forwarding them to the right people so that the best decisions can be made is what xCP is trying to automate.

Case-related processes have some unique attributes that make them challenging to automate:

• They are non-deterministic – You can’t predict ahead of time the exact process that a case will follow




• They are driven by human decision making – the next step in the process is often decided by the person performing the current step
• They are also driven by content status – cases can’t proceed to the next step in the process until all the necessary information has been provided, and often the next step is determined by some attribute of the content (or metadata about the case itself)




Case management fits in a gray area between ad-hoc collaboration where pre-defined workflows just aren’t effective and transactional document management with strict, linear processes like scanning invoices. When you have a process with lots of variation and lots of pieces of information coming together, you probably have a case management problem.

How Does xCP Help Companies Manage Cases?

EMC’s vision is to make it easy for companies to design and deploy custom case management solutions that are tailored for their specific business processes. With xCP,
the idea is that you can easily define a case (with all its metadata and associated content), design forms that are need to capture structured data, and model the business processes that the case goes through.

Once you have completed these design and modeling activities, xCP will automatically assemble a custom application with work queues, custom forms, work flows, and every thing else needed for your users to create and manage cases. In effect, you are building a custom application without writing any code.

And this is where the other key goal of xCP comes in – making it much easier to upgrade Documentum without breaking your custom application.

Preventing Upgrade Headaches Using a Declarative Model

One of the common complaints about Documentum is that when you customize it to meet your needs, it’s likely that your customizations will not work when the next version of the product comes along. This problem isn’t unique to Documentum – it’s common to all enterprise software applications.

This is a problem I know a lot about. Blue Fish builds custom applications on top of Documentum all the time (it’s our sweet spot), and while we have developed best practices to insulate our customizations from changes in the platform, sometimes it simply can’t be avoided. If you write an application in WDK and a couple of years later, EMC gets rid of WDK, you are pretty much screwed.

EMC’s approach to eliminating customization-related upgrade hassles is to eliminate the customizations themselves. They do this in xCP with what’s called a declarative model. WIth a declarative model, you express what you want the system to do, not how it should go about doing it.

Here’s how it works. Let’s say you need a form to capture the loan application we described earlier. With xCP, you use Forms Builder to drag the various fields onto a form and click various checkboxes to specify which fields are mandatory and such. Forms Builder saves this layout in an internal format that xCP interprets at runtime when it’s time to display the form.

A year or two later, imagine that EMC completely changes the architecture of xCP, migrating it from a WDK application to a rich internet application like CenterStage. You shouldn’t have to redefine your form, because the new version of xCP will be able to read that same form layout file, and even though the form will look completely different in the new version of xCP, it will have all the same fields in the same layout.

By the way, this example isn’t that far-fetched. EMC is in the process of doing this very thing! When xCP 2.0 is released next year, it will be completely re-implemented using a rich internet application toolkit like ExtJS. But John promises that you won’t have to change any of your forms definitions, process definitions, data models, or any of the other xCP components that define your case management solution.

It’s important to note that this approach means you can’t customize xCP. If there isn’t a form control that you need, you’ll be out of luck. If the xCP designers didn’t anticipate your use case, you’re probably out of luck. If you don’t like the look and feel of the UI or the performance of the system, you’re out of luck. And while that might be scary for a crotchety old developer like me, for most of EMC’s customers, it’s a tradeoff they will gladly accept if it means fewer upgrade headaches.

He also dropped a real bombshell about a potential change in the architecture.

Service Orientedness

Documentum Foundation Services (DFS) is Documentum’s web services API. It’s been around for a few years, and EMC is promoting as the best way to interact with the Content Server. Developers are encouraged to use DFS instead of Documentum Foundation Classes (DFC), which they now consider a low-level API. You use DFC to create services, and you use DFS to create applications.

EMC is taking this advice to heart. Centerstage and Media Workspace are two recent Documentum products that are built on DFS (as opposed to Webtop, which is built using DFC).

There are 30 Core Services in DFS, but not all the functionality in Documentum is exposed through DFS. They only add the services that they need (or for which there is popular demand).

DFS is a SOAP-based web service, which is suitable for strongly typed languages such as Java and C#, but is apparently not so good for dynamic languages like Ruby or JavaScript. Developers that use dynamic languages prefer RESTful web services, which EMC does not currently support.

Victor is a fan of RESTful services, and he was looking for some support from the attendees that he could use to convince his managers to invest in adding support for REST. Victor asked how many people would really use RESTful services if he were to build them, and in a room of over 200 people, only TEN raised their hands. Victor was crestfallen.

This raises a question. Why don’t Documentum developers care about REST when other developers in other fields are so passionate about it? My (completely speculative) opinion is that the day of the Documentum developer is mostly behind us. Most people in the room were Documentum implementers, not developers. They are not tasked with building interesting new applications that mash-up data from different sources while integrating the result into some internal portal or other kind of web application. They don’t spend all day every day developing, so they don’t value using scripting languages like JavaScript to decrease development time.

Contrast this with Alfresco’s developer community, which is as likely to be building a content-enabled web application as they are to simply implement the technology as a document repository for their office documents. As a result, Alfresco’s SOAP API has being placed on the back burner while their RESTful services get constant attention.

CenterStage – An Example of Using DFS

Victor cited Centerstage as the shining example of how EMC wants to build apps from now on.

He noted that one of the big limitations of WDK is that it does not separate presentation logic from application logic. This is a problem because presentation technologies evolve very quickly, changing every couple of years. Keeping up with these changes would mean re-writing the entire application constantly, but ignoring these changes makes your application seem outdated and clunky.

With Centerstage, the presentation logic lives in the browser, while the application logic lives on the server. EMC uses a JavaScript toolkit to make requests to the DFS services which return their results as XML files. The browser then renders the user interface by parsing the XML file as a data file. This is different from WDK which generates the UI on the server, renders HTML files and the browser simply displays the HTML file.

One of the benefits of this approach is that it allows the presentation tier to use a mix of technologies, such as Ajax and Flex. Centerstage is mostly a JavaScript application, but it has a few Flex components for doing things like viewing thumbnails.

Because DFS is a SOAP-based web service, and those are hard to use in applications like this, EMC has written their own middle tier in the browser. This allows the browser to use direct web remoting (DWR) to communicate with DFS.

Victor told us about a couple of interesting technology decisions they made. When Centerstage began development 3 years ago, ExtJS was the JavaScript toolkit they chose. But if they were to make the decision today, they would probably use JWt (Java Web Toolkit) GWT (Google Web Toolkit).

The other interesting decision they made was to use Mozilla XUL to define the layout of their user interface. Then they translate the XUL XML file into ExtJS in the browser via XSLT.

Enterprise Readiness

A big focus area for Documentum 6.6 was to improve what EMC calls “Enterprise Readiness”, which I took to mean “making it easier for big giant enterprises to run and support Documentum.”

One way they have done this is to improve Documentum’s ability to ingest very large volumes of data. Version 6.5 is already very good at this, and with 6.6, they are making it even better.

In version 6.5, Documentum added batching (importing multiple documents in a single batch), scoping (batching related documents together), partitioning (storing related documents together), and lightweight sysobjects (storing fewer attributes when the complete set of metadata is not needed). Together, they allow Documentum to ingest 5 times as many documents in an hour than in previous versions.

In version 6.6, Documentum has improved the batching and partitioning and has added some additional improvements:

• True null support – reduces the storage required for attributes with no value
• Use of small int (2 bytes) and tiny int (1 byte) where possible to save database storage
• Storing boolean values more efficiently to save storage space
• Storing ID attributes more efficiently in MS SQL Server

Documentum also added some DQL Enhancements that improve sub-selects and joins (they added support of left outer joins).

I’m excited about the new support in DQL for pagination, or returning only a certain subset of the rows that result from the query. This is useful in web applications when you have “pages” of results that you want to the user to navigate through.

In the past, you had two options, neither of them good. You could keep the database cursor open and wait for the user to request the next page of results (this is expensive and you never know if the user is going to request another page or not). Or you could rerun the entire query and manually discard the rows that you’ve already returned. Pagination lets you rerun the query each time while specifying row number to start with. DQL will only return the small result set you want to show to the user.

Documentum Search Services

Documentum is releasing a new search engine called Documentum Search Services (DSS). This will replace the FAST search engine. Apparently, FAST makes it much too difficult to support a really large number of documents (100 million+).

DSS stores the metadata in xDB (EMC’s XML database) and the full text index in Lucene. This allows it to issue queries across both metadata and full text very quickly. One advantage is that since xDB contains all the ACL metadata, it can weed out any results that the user does not have permission to see. With FAST, each item in the search result had to be checked against the Content Server’s ACL metadata before it was returned to the user. EMC says that this alone makes DSS 10 times faster.

DSS has other “enterprise readiness” features as well.

• It provides low-cost high availability (n+1 servers instead of 2n servers)
• Support for vmware (FAST didn’t support it)
• Support for NAS
• Support for Disaster recovery
• If you are running a distributed architecture, less data will be shipped across the wide area network during indexing because the text extraction takes place at the remote server and only the “raw text” is sent across the WAN to DSS

Other Platform Improvements in 6.6

• Documentum added support for ATMOS, EMC’s cloud storage solution.
• Java Messaging Service (JMS) will failover will to a JMS server on other content server if one is available
• DCTM Messaging Service will now failover (this is mostly used in distributed content architectures)
• 6.6 adds Kerberos support at the content server, DFC, and DFS layers
• In response to requests from IT shops that don’t want to support Java on their Windows desktops, EMC is releasing UCF.net, a pure Microsoft version of Documentum’s content transfer solution

Complexity Reduction

The big talk around Momentum is about xCP, Documentum’s xCelerated Composition Platform. Victor says that everything about xCP is about reducing complexity so that it’s easier to create and deploy Documentum-powered content applications. He called it the “Visual Basic of ECM.”

xCP uses a model driven (declarative) approach to reduce the need to write code. You “declare” what you want the application to do (via configuration) and xCP will assemble the application that you have described.

xCP 1.5 is based on WDK, but it’s being shelved for a new version, xCP 2.0 that will be based on a rich internet application model similar to Centerstage. Victor hinted that the architectural patterns will be similar to Centerstage, but the implementation technologies are likely to be different in xCP 2.0.

Victor called this a “very ambitious project”. xCP 2.0 is targeted for the end of next year (2011).

Documentum Rejecting the Relational Database?

The final item I’ll leave you with is the most exciting from an architecture perspective. Victor told us that they have been thinking about ways to use xDB as the metadata storage mechanism for Documentum, replacing the relational database that they use today.

This is 3-5 years away, if it happens at all, but it’s an incredibly intriguing idea. Eliminating the database would dramatically reduce the complexity and overhead required to operate a large, distributed Documentum environment, and while Victor said that he personally thinks it’s the right thing to do, there is a lot of thinking and testing to be done first. He has a small team working on it now, and Victor told the audience that while he’s making no promises, he hopes to be standing at an EMC World 2-3 years from now revealing it as an upcoming architectural change.

New Name for Content Management and Archiving

The division within EMC that owns Documentum has changed their name from Content Management and Archiving to “Information Intelligence Group”. Not sure what this is about yet – I guess I’ll find out over the course of the conference.

Momentum is Small This Year

Momentum is the name of the old Documentum user group conference, and it’s also the name of EMC World’s “conference within a conference” that is focused on Documentum and the related content management technologies.

There are only 16 breakout sessions on Documentum today, and most are delivered by EMC product managers. Only three sessions today are being presented by customers.

The “Momentum Zone” at the solutions pavilion is tiny. There are probably less than 20 booths total, and only a handful of them are names I recognize from previous conferences. Blue Fish, Technology Services Group, and many other Documentum-focused partners that have had booths in the past elected not to exhibit this year.

Missed The One Presentation I Wanted to See the Most

I’m totally bummed out that I couldn’t get into John McCormack’s presentation on the Documentum roadmap. I guess I’m going to have to download the slides when they are available, but it’s not the same.

Private Cloud

From my perspective, EMC World is a conference about Documentum. Of course, that’s only one percent of what’s going on here, but it’s the one percent that I care about. But I listened to Joe Tucci’s keynote anyway to here his vision for the future of EMC.

EMC is pushing something they call the “Private Cloud”, EMC’s vision for operating a cloud model within your own data center. The idea is that with vmWare and other EMC technologies, you’ll be able to provision new computing services in your own datacenter in a manner similar to the way that Amazon, Google, Facebook, and other massive online companies do. Instead of having infrastructure dedicated to an application the way we do today, you’ll have a slew of generic computing resources (CPU, memory, bandwidth, and storage) that can seamlessly provision themselves as demand for them increases and then release resources into a common pool as demand falls off.

CenterStage Update

I attended Matt Coblenz’s presentation on CenterStage, EMC’s replacement for eroom. I had heard the CenterStage pitch in past years, and was excited to learn more about what exciting changes have been introduced over the past year. It turns out, not much is new.
CenterStage was a big topic at last year’s EMC World, but it wasn’t officially released until about six months ago. As Matt presented some of his thoughts about the future, one of the audience members put him on the spot.

“Your product is nearly two years late, how can I trust your roadmap and vision?” the audience member asked. Matt handled it as well as he could, saying, “You won’t until I earn your trust again. We haven’t done a good job of meeting commitments in the past, but we are trying harder.”

So the bottom line is that nothing much is new to report about CenterStage.

Introduction

Recent releases of the EMC Documentum platform have offered web content authors alternatives to creating content via the traditional Web Publisher template editor. Despite the alternatives, we have found that the tried and true method of creating content via the forms based templates is still the most widely adopted (and really, most powerful). As part of the Web Publisher product, EMC provides a variety of content creation “widgets” that web developers assemble into these custom forms (called Rules Files) that content authors then use for content entry. While each of these widgets is a valuable tool in the art of template design, there is one widget that stands heads and shoulders above the rest. The content widget, colloquially referred to as the “Rich Text Editor”, allows content authors to contribute rich text (that is, text that contains images, links, and styling) to their web page. This provides for flexibility and diversity in content that would be impossible to achieve with the other widgets alone.

This article will give a behind the scenes tour of the Web Publisher Rich Text Editor and will get you started with configuring and customizing it to meet the needs of your business. It assumes that you have a working knowledge of Web Publisher and the content creation process using the template editor. If you are new to Web Publisher, I’d suggest reading this primer on Web Publisher Template Basics first to give you the appropriate context.

What’s What

Unlike the other template editor widgets, the Web Publisher Rich Text Editor is not actually an EMC product at all, but is rather the embedded product called EditLive! developed by Ephox. Before diving any deeper, it is important to familiarize yourself with the key files that make up the Web Publisher template editor (including the Rich Text Editor):

• %appserver%/wp/webcomponent/library/contentxfer/swingTe.jar
  

  The swingTe.jar contains the implementation classes for all of the Web Publisher widgets, including the EditLiveWidget which embodies the Web Publisher Rich Text Editor. The EditLiveWidget customizes the EditLive! component provided by Ephox to allow for Documentum specific repository interactions (for instance, browsing for links in the repository and importing images into the repository).

• %appserver%/wp/webcomponent/library/contentxfer/editlivejava.jar
  

  The editlivejava.jar contains the implementation classes for the EditLive! directly as it is provided by Ephox.

• %appserver%/wp/wp/editors/contenteditor/resources/EditorUI.properties
  

  The EditorUI.properties provides some configuration settings that drive the look and feel for the entire Web Publisher Template Editor.

• %appserver%/wp/wp/editors/contenteditor/resources/editliveConfig.xml
  

  The editliveConfig.xml is the configuration file for the EditLive! editor itself.

• %appserver%/wp/wp/library/ceedit/ceedit.jsp
  

  The ceedit.jsp generates the Web Publisher Editor applet.

Configuration

As mentioned above, the editliveConfig.xml is the configuration file for the EditLive! editor – it drives all of the underlying configuration settings for the Rich Text Editor. Many of the settings in this file are actually overwritten by settings specified in the Rules File definition of the editor instance. For instance, you can specify in your rules file whether or not the “HTML” code tab is available (along with the default “Design” tab) with the parameters htmlTextEdit="Y" or htmlTextEdit="N".

Some settings, however, are only configurable via this file. Here are some of the most important configuration settings to be aware of in a Web Publisher implementation:

Specifying Styles

One of the biggest downsides to the forms based editing is that as users modify the content, what they are modifying does not look exactly like it will appear on the website (users have to WebView or Preview the content to see their modifications in context). This is not that large of a concern when users are interacting with other widgets that collect specific pieces of information that the user knows are going to be formatted correctly by the site. The Rich Text Editor, however, grants quite a bit more flexibility to the content author in how they author their content – and hence also requires from them more responsibility to format their content appropriately.

Wouldn’t it be nice for content authors to get a better sense of how the content they are entering in the Rich Text Editor will look on the final web page directly as they are authoring it? The EditLive! widget actually allows for two mechanisms for including style information to drive the way the content is rendered in the Rich Text Editor itself – Importing a Stylesheet or Adding Inline Styles.

Importing a Stylesheet

Much as you would import a stylesheet for use on your rendered web page, the Ephox editor supports importing a stylesheet for use in the Rich Text Editor. To do this, in the <head> of the editliveConfig.xml, simply add the stylesheet as a <link> element:

<link href="http://www.myserver.com/includes/css/base.css" rel="stylesheet" type="text/css"/>

If you are developing your site from scratch (or have the ability to undergo a refactoring of your site CSS), modularizing your CSS files such that the relevant styles for the body content of your page can be included in the Rich Text Editor could provide a greatly enhanced authoring experience without any extra work. If you are not so lucky, however, your best bet would be to create an “editor environment” stylesheet specifically designed for the authoring experience that replicates the styles for your actual site CSS.

Inline Styles

Just as in your web page you always have the option to provide inline styles instead of relying on included CSS, so it is with the Rich Text Editor. Inline styles can be especially useful when you just want to target a few styles to include in the content authoring experience. For instance, let’s say your authors are contributing content to a fixed width site. The content that they are authoring always fits into a main content area that is 700px wide. Wouldn’t it be nice if the author could see where their content was going to wrap as they typed instead of having to preview first? This is actually very easy to do – simply add this fixed body width as an inline style to the Rich Text Edtior by adding a <style> block to the head of the editliveConfig.xml:

        
<style type="text/css">
	body {width:700px;}
</style>
					
    

Figure 1:
Web Publisher Rich Text Editor – Body Width Style Applied

Modifying Copy/Paste Behavior

Anyone performing a Web Content migration from a legacy system into Web Publisher will most likely find themselves spending quite a bit of time copy/pasting existing content into the Rich Text Editor. There are 2 configuration settings that drive the behavior of how text is pasted into the editor – one that controls pastes from Microsoft Word, and one from existing websites.

The behavior when pasting in content from Word is governed by the configuration setting:

<wordImport styleOption="[specify value here]"/>

And behavior when pasting in HTML content from an existing web page is governed by the configuration setting:

<htmlImport styleOption="[specify value here]"/>

The following values are supported:

• user_prompt – Users will be prompted as to whether they wish to import the existing styles as inline or embedded styles or strip them from the imported text.
• merge_embedded_styles – This setting will result in EditLive! importing styles from the selected content with the styles stored in the <head> of the document and class attributes will be applied where relevant.
• merge_inline_style – This setting will result in EditLive! importing styles from the selected content with style information applied inline in place of class attributes.
• clean – The clean setting will result in EditLive! stripping the styles from the imported text and only structural HTML elements will be imported, this includes <b>, <p> etc.
• plain_text – The plain_text setting will strip out all style information from the imported text.

Modifying the Spell Checker

One helpful feature of the Rich Text Editor is the built in “spell checker”. Web Publisher installs with a set of dictionaries provided by Ephox. A copy of the spell checker’s dictionary/word list is maintained on the server and is downloaded to the client machine on its first use of EditLive! and it is cached there. This ensures responsiveness and does not require content to be sent to the server for spell checking.

Adding New Locale Definitions

When a content author modifies a piece of content, the correct dictionary is automatically used, based upon the locale of the document being modified. The list of which dictionary to use for which locale is specified in the editliveConfig.xml. If you add a new locale to Web Publisher, there is a very good chance you will need to update the EditLive! configuration to know about it as well. In an OOB Web Publisher installation, the following locales are pre-defined:

        
<dictionary>
   <!--default dictionary to use when object has no locale-->
   <default>/wp/editors/contenteditor/dictionary/en_us_4_0.jar</default>
   <!-- English US-->
   <en_US>/wp/editors/contenteditor/dictionary/en_us_4_0.jar</en_US>
   <!-- English UK -->
   <en_GB>/wp/editors/contenteditor/dictionary/en_br_4_0.jar</en_GB>
   <!-- Danish -->
   <da_DK>/wp/editors/contenteditor/dictionary/da_4_0.jar</da_DK>
   <!-- Dutch -->
   <nl_NL>/wp/editors/contenteditor/dictionary/du_4_0.jar</nl_NL>
   <!-- Finnish -->
   <fi_FI>/wp/editors/contenteditor/dictionary/fi_4_0.jar</fi_FI>
   <!--French - European and Canadian -->
   <fr_CA>/wp/editors/contenteditor/dictionary/fr_4_0.jar</fr_CA>
   <fr_FR>/wp/editors/contenteditor/dictionary/fr_4_0.jar</fr_FR>
   <!-- German -->
   <de_DE>/wp/editors/contenteditor/dictionary/ge_4_0.jar</de_DE>
   <!-- Italian -->
   <it_IT>/wp/editors/contenteditor/dictionary/it_4_0.jar</it_IT>
   <!-- Norwegian -->
   <no_NO>/wp/editors/contenteditor/dictionary/no_4_0.jar</no_NO>
   <!--Portuguese (Iberian) -->
   <pt_PT>/wp/editors/contenteditor/dictionary/po_4_0.jar</pt_PT>
   <!--Spanish - European, Mexican and South American-->
   <es_ES>/wp/editors/contenteditor/dictionary/sp_4_0.jar</es_ES>
   <!--Swedish-->
   <sv_SE>/wp/editors/contenteditor/dictionary/sw_4_0.jar</sv_SE>
</dictionary>
					
    

To add a new locale, simply add the relevant element to the editliveConfig.xml and specify the desired dictionary to use.

Customizing a Dictionary

While the dictionary provided OOB is very good, nearly every organization has its own set of common words – perhaps the name of the organization, a product name, or other industry-specific terms that will show up as “misspelled” in the Rich Text Editor. Individual users of course always have the option to update the local copy of their dictionary via the “Add” feature of the spell checker (similar to the spell checker “Add” feature in Microsoft Word, this will update the cached dictionary on the user’s local machine). Since these are words that are really organizational in nature, however, wouldn’t it be great if you could just push these updates to all of the content authors? Well, it turns out you can! By using an existing dictionary as a starting point, you can create your own custom dictionary for use by your content authors:

1. Unzip the dictionary you want to modify (found in %appserver%/wp/wp/editors/contenteditor/dictionary) to a temporary location. For example, to create your custom dictionary based on the US English dictionary, unzip the file en_us_4_0.jar. You will find in the location where you unzipped this the following files:
   • accent.tlx
   • correct.tlx
   • dictionary2.clx
   • dictionary.tlx
2. In the directory with this unzipped files, using Notepad (or your favorite plain text editor) create a plain text file called customdictionary.tlx.
   • At the top of this file add the line: #LID 24941
   • For each word that you want to add to the dictionary, add the word on its own line in the file, then one “tab space” from the word, add the letter “i”. For instance, if you wanted to add the words WebPublisher and Ehpox to your custom dictionary, your customdictionary.tlx file would look like:

                             
     #LID 24941
     WebPublisher	i
     Ephox	i
     								
                         

3. From within the temporary directory that contains these dictionary files, using the Java jar command, create a new jar file that packages your new file with the four files that came from the OOB dictionary. For instance, if you have java installed in C:Java, you would run the command:

   C:Javabinjar cvf customdictionary.jar .

4. Place your customdictionary.jar file in the same %appserver%/wp/wp/editors/contentedtior/dictionary directory with the OOB dictionaries.
5. Update the appropriate locale(s) in the editliveConfig.xml to use your new custom dictionary.

EditLive! will automatically detect this change in the server-side dictionary and will update the cached version on any client machines the next time they load them.

Customization

Configuration can obviously only take you as far as the product has been designed to take you. Sometimes that just isn’t far enough and you need to customize the existing functionality. Here are two strategies for customizing the Rich Text Editor behavior:

Adding a Custom Toolbar Button

The Rich Text Editor provided with Web Publisher includes a variety of “toolbar buttons” that drive user interaction with the content. For instance, there are toolbar buttons for inserting images, hyperlinks, tables, bulleted lists, and so on. Which toolbars show up in the editor depends not only on the configuration setting in the editliveConfig.xml, but also which have been configured in the rules file for the current template. You are not limited, however, to the toolbar buttons provided in the OOB implementation – the editor actually has built in support for adding custom toolbar buttons.

Let’s take a very simple use case – let’s say that you want your content authors to be able to add a disclaimer (which is really just a simple HTML fragment) anywhere in their content in an ad-hoc manner. Since they do this often, you don’t want them to have to bother typing it out every time. You can add a toolbar button that inserts the disclaimer into the content area for them. All you have to do is:

1. Create a 16x16px toolbar icon – this icon is required for the imageURL attribute of your custom toolbar definition. (Hint: This image needs to be saved in .gif format – if you create a .jpg it won’t work. Don’t shoot the messenger.).
2. Add the toolbar definition to the editliveConfig.xml where you would like it to appear in the toolbar. For instance, if you wanted to add the new toolbar button between the “Insert Symbol” and “Insert Bookmark” functions, you would modify the file to read:

                   
   <toolbarSeparator/>
   	<toolbarButton name="tlbHRule"/>
   	<toolbarButton name="tlbHLink"/>
           <toolbarButton name="tlbSymbol"/>
           <customToolbarButton name="customInsertDisclaimer" text="Insert Disclaimer" imageURL="http://www.myserver.com/images/my16x16icon.gif" action="insertHTMLAtCursor" value="%3Cp%3EThis%20is%20my%20disclaimer%3C%2Fp%3E" />
           <toolbarButton name="tlbBookmark"/>
           <toolbarButton name="tlbImageServer"/>
   <toolbarSeparator/>
   						
               

   Note that the value attribute that you specify for the customToolbarButton (the text that is actually inserted into the content area) must be URL encoded.

3. Include the unique name of your toolbar in your rules file definition. For instance, for this example you would add to the content rule:

   customInsertText=”Y”

Voila! You now have a custom toolbar button added to the Rich Text Editor. When you click on the button, the disclaimer paragraph will be added to the content editor output.

Figure 2:
Web Publisher Rich Text Editor – Custom Toolbar Button

Extending the EditLiveWidget

What if none of the configuration and customization hooks we have described thus far provide the flexibility you need, what if you simply need to add custom business logic to the RichText Editor? For instance, we recently had a client that wanted to add logic that determined which users should have access to the “HTML” (or code view) tab. What are your options then?

At this point you are going to need to step into the world of custom development and extend the EditLiveWidget itself. Here are a couple of pointers for successfully extending the EditLiveWidget:

• As mentioned above, the ceedit.jsp generates the Web Publisher Editor applet. It is in this file that you will need to add your custom .jar’s to make them available to the template editor (just as you would if you were creating your own custom widget from scratch). It is also in this file that many parameters are defined and passed to the applet. You can add your own custom parameters here (populated with whatever logic necessary based upon the user, repository, content, etc) and they will be available in your custom widget.
• You may have created your own custom widgets in the past, and had been able to simply package them in a jar and add them to the ceedit.jsp class definition and had no problems. When extending the EditLiveWidget, however, this won’t be good enough and you will need to sign your applet jar. It appears that EditLive! uses some system resources that cause security exceptions if your jar is not signed properly. In fact, when customizing certain releases we’ve found that signing our own jar was not sufficient and we needed to sign the swingTe.jar and edlitlivejava.jar as well.

Other Tips & Tricks

Hey – I want that feature!

Since EditLive! is a true enterprise product of its own right, it is continually being improved and upgraded. With nearly each new release of Web Publisher a new version of the EditLive! editor is included. While upgrading the Documentum stack to support a new version of Web Publisher can be a large undertaking (particularly if you have a bunch of WDK customizations), it actually is quite straight forward to “upgrade” the template editor functionality. All you need to do is replace the swingTe.jar and the editlivejava.jar from your current Web Publisher installation with those from your “desired” release. Depending on what releases you are upgrading from and to, you may also need to update the <ephoxLicenses> section of the editliveConfig.xml file with the relevant license information from your target release.

        
<ephoxLicenses>
       <license
            accountID="123456789100"
            activationURL="http://www.ephox.com/…"
            domain="DOCUMENTUM"
            expiration="NEVER"
            forceActive="false"
            key="8FFF-FFFF-FFFF-FFFF"
            eqEditor="false"
            licensee="DOCUMENTUM"
            product="EditLive for Java"
            release="6.0"
       />
</ephoxLicenses>
				
    

I tried that, but it didn’t work – HELP!

Well let’s see, we have a WDK application invoking a custom applet that relies on a template specific rules file and then leverages a third party developed application, which reads its own set of configuration data, all to present the user with a swing application – what could go wrong? Obviously there are a lot of moving parts here, and sometimes to figure out what is going on, tracing will become your best friend.

Here are some of the most useful of the Web Publisher tracing options:

• %APPSERVER%/wp/WEB-INF/classes/WcmTraceProp.properties
  • WCMSESSIONENABLEDBYDEFAULT – This value must be set to true to enable any Server or Client side tracing
• %APPSERVER%/wp/WEB-INF/classes/com/documentum/contenteditor/resources/EditorServerTraceProp.properties
  • EDITORSERVER – Traces the Server side of the editor (lookup of the Web Publisher Editor, parser errors, validation & entity resolution, double-byte character encoding exceptions & checksum errors)
  • CONTENTIF – Traces editor operations (export path, export errors, and XML file encoding)

The location of the generated log files will vary by installation, but can be specified in the log4j.properties that is located in your %APPSERVER%/wp/WEB-INF/classes/log4j.properties.

To trace the template editor itself, you will need to not only turn on the tracing flag:

• %APPSERVER%/wp/wp/editors/contenteditor/resources/EditorUI.properties
  • contenteditor.showDebugButton – adds a “Debug” button to the template editor for viewing debug information from the template editor
  • editlive.debug – turns on tracing for the EditLive! This tracing can be viewed in the applet Java console

Resources

The following links and documents were referenced in writing this article, and are a great source for more information on all of the topics above:

• EditLive! Java Developers Guide: http://www.ephox.com/developers/editliveforjava/v40/html/
• EditLive! Configuration File Specification: http://liveworks.ephox.com/documentation/editlive/v60/RefHTML/index.html?configuration_file_element.htm
• EditLive! Hints & Tips: http://liveworks.ephox.com/hints-tips
• Web Publisher Developer’s Guide http://powerlink.emc.com

From time to time, we get requests from our clients to customize this editor. Some want us to add extra toolbar buttons that provide some custom capability. Others want us to make the style of the editor match the style of their web site to give an even better preview of what the content will look like.

Lisa’s article shows how to do all of these things and provides some great tips on how to debug and troubleshoot your customizations. If you are thinking of modifying the Web Publisher Rich Text Editor, this article is a must read.

Read Customizing the Web Publisher Rich Text Editor.

To be useful, most dashlets neet to be configurable by the end user. For example, you might want to select which team’s documents to monitor or which city’s weather to display. Dashlets themselves are pretty easy to create (it only takes a little JavaScript and some XML files), but if you want your dashlet to be configurable, it can get a little tricky.

Josh explains it all in Making Alfresco Share Dashlets Configurable.

Introduction

Note: This article isn’t an exhaustive dashlet development how-to. For an introduction to the platform and technologies used to build dashlets, a great place to start is the Alfresco Surf Platform Developers Guide.

Dashlets are a lot of fun. They expose a tremendous amount of functionality, and they give users and project teams the power to assemble and organize that functionality as they see fit. They can be as simple as the two file Hello World dashlet from Dr. Q’s Workshop. On the other hand, they can also be incredibly complex, with well over 20 files making up a single dashlet.

I know that when I first waded into the deep end of the dashlet pond, I started drowning quite quickly. Back when I was a naive ‘noob’, it took me a while to find my way around and comfortably accomplish some of the tasks I had assumed would be very easy. One of the areas I struggled with was figuring out how to make my dashlets configurable, so that end users could change the way they look or behave. I wanted to make dashlets similar to Alfresco’s Web View dashlet that lets users provide a URL, and then displays the page referenced by that URL within the dashlet.

Figure 1:
Unconfigured instace of the Alfresco Web View dashlet.

The goal of this article is to help you understand how to make your dashlets configurable so that you can focus on the development of new features, and not the plumbing of configuration values.

Dashlets Are Easy, Sometimes

Dashlets can be very simple. For example, a simple Hello World dashlet can be implemented with just two files:

helloworld.get.desc.xml:

        <webscript>
  <shortname>Hello World</shortname>
  <description>Hello World Dashlet</description>
  <family>dashlet</family>
  <url>/components/dashlets/helloworld</url>
</webscript>

    

helloworld.get.html.ftl:

        <div class="dashlet">
  <div class="title">Hello World</div>
  <div class="body">
    <span>Hello World</span>
  </div>
</div>
    

Drop these two files into the shareWEB-INFclassesalfrescosite-webscriptsorgalfrescocomponentsdashlets directory and you’ve got a new dashlet ready to go.

If some dashlets can be so simple, why are others often so hard? Well, in this article I’m going to show you how taking this simple, static Hello World dashlet, and adding configuration capabilities expands these two source files into a total of 16 files. I know, I know; that sounds like a lot. But once you learn your way around you’ll be adding configuration capabilities to all you dashlets.

A Configurable Example

For the purposes of this article, I’ll be using an example dashlet that accepts and displays three configuration values. Not earth shattering functionality, but then very few people would have guessed Twitter was going to change the world when it was first rolled out. Here is a look at the end product in action:

Figure 2:
Unconfigured instace of our example dashlet.

This dashlet allows users to dynamically configure its three displays values. As you can see below, it accepts a title, headline, and story:

Figure 3:
Configuration dialog for our example dashlet.

Once the configuration dialog is saved the dashlet on the dashboard is automatically updated with the new values, and those values are persisted for all future views of that instance of the dashlet.

Figure 4:
Configured instace of our example dashlet.

Anatomy of a Dashlet’s Configuration Process

As you begin experimenting with dashlet development, I highly recommend that you check out the Alfresco Developers Wiki. This site has some tremendously useful Web Scripts framework documentation and examples. One particularly helpful section discusses the framework’s use of the MVC pattern, and how a Web Script’s execution flows through the framework. In that section the following diagram is used to help explain the flow:

Figure 5:
(from http://wiki.alfresco.com/wiki/Image:WebOrientedRestStyleArchitecture.jpg)

Our example dashlet will actually complete this loop three different times as we interact with it. Let’s quickly review each of the three loops.

Main Dashlet Display

The initial loop is completed every time our dashlet is drawn onto a dashboard. In this loop, the steps highlighted in the diagram execute as follow:

1. The dashboard includes the dashlet and makes a call to the Web Script Runtime using the dashlet’s URL.
2. aafb.get.desc.xml – The Web Script Runtime finds the appropriate Web Script as defined in the aafb.get.desc.xml then hands off to the controller for execution.
3. aafb.get.js / aafb.get.config.xml – The JavaScript controller (aafb.get.js) executes, using the configuration values in the config xml (aafb.get.config.xml) as necessary.
4. aafb.get.head.ftl / aafb.get.html.ftl / aafb.get.properties – The view uses values set on the model by the controller and properties from the aafb.get.properties to assemble and render a response with the FreeMarker templates defined in aafb.get.head.ftl and aafb.get.html.ftl.
5. The Web Script Runtime sends the results back using the appropriate protocol.
6. The Web Script client receives the response in the requested format. In this case, that format is HTML.

Configuration Dialog Display

The second loop is completed when users click on the dashlet’s Configure link. In this loop, the steps highlighted in the diagram execute as follow:

1. aafb.js – The onConfigAafbClick method defined in the aafb.js is called when a user click the Config link, and this initiates a request to the Web Script Runtime.
2. config-aafb.get.desc.xml – The Web Script Runtime finds the appropriate Web Script as defined in the config-aafb.get.desc.xml then hands off to the controller for execution.
3. No controller logic is necessary to render the dialog, so control passes straight to the view.
4. config-aafb.get.head.ftl / config-aafb.get.properties – The view uses properties from the config-aafb.get.properties to assemble and render a response using the FreeMarker template defined in config-aafb.get.html.ftl.
5. The Web Script Runtime sends the results back using the appropriate protocol.
6. The Web Script client receives the response in the requested format. In this case, that format is HTML.

Configuration Dialog Processing

The final loop is completed when a user submits the configuration dialog by clicking the OK button. In this loop, the steps highlighted in the diagram execute as follow:

1. The Web Script Runtime is called a third time when a user submits the configuration dialog.
2. config-aafb.post.desc.xml – The Web Script Runtime finds the appropriate Web Script as defined in the config-aafb.post.desc.xml then hands off to the controller for execution.
3. config-aafb.post.json.js – The JavaScript controller (config-aafb.post.json.js) executes.
4. config-aafb.post.json.ftl / config-aafb.post.properties – The view uses values set on the model by the controller and properties from the config-aafb.post.properties to assemble and render a response using the FreeMarker template defined in config-aafb.post.json.ftl.
5. The Web Script Runtime sends the results back using the appropriate protocol.
6. The Web Script client receives the response in the requested format. In this case, that format is JSON.

A Closer Look

First, let’s look at where the 16 files that make up this dashlet live within the context of the Share application. Here is a view of the files that make up our example dashlet deployed into Alfresco Enterprise 3.2r:

Figure 6:
Deployed view of all the files that make up our example dashlet.

If this looks a little foreign to you, then let me direct you over to a great article about Alfresco source code organization written by my colleague Josh Toub.

These 16 files require 9 easy steps to enable our configuration functionality. Let’s dive right in.

Define Defaults

The first thing we need to do is define some default values for our configuration parameters. These default values will be used when the dashlet is first instantiated onto the dashboard. Default values are defined in the aafb.get.config.xml file as follows:

        <aafb>
  <title>Configurable Dashlet Title</title>
  <headline>Configure your headline</headline>
  <story>Tell a story, but make it short!</story>
</aafb>
    

Set Configuration Parameter Values

Next, we need to tell our JavaScript controller how to handle our configuration parameters and where to look for defaults. This is all done in the aafb.get.js file as follows:

        function main()
{
  var title = args.title;
  var headline = args.headline;
  var story = args.story;
  
  // Create XML object to pull values from
  // configuration file
  var conf = new XML(config.script);

  // Use the defaults from the XML configuration file
  // (aafb.get.config.xml) if no values in args array
  if (!title)
  {
    title = conf.title[0].toString();
  }

  if (!headline) 
  {
    headline = conf.headline[0].toString();
  }

  if (!story) 
  {
    story = conf.story[0].toString();
  }
  
  // Set values on the model for use in templates
  model.title = title;
  model.headline = headline;
  model.story = story;
}

main();
    

Layout/Use Configuration Values on Dashlet Display

Now that we’ve got some configuration values defined, we need to display them within our dashlet. We define the dashlet’s layout with a FreeMarker template in the aafb.get.html.ftl file as follows:

        <script type="text/javascript">//< ! [CDATA[
  new Alfresco.Aafb("${args.htmlid}").setOptions(
  {
    "componentId": "${instance.object.id}",
    "titleStr": "${title}",
    "headlineStr": "${headline}",
    "storyStr": "${story}"
  });
  new Alfresco.widget.DashletResizer("${args.htmlid}", "${instance.object.id}");
//] ] ></script>

<div class="dashlet aafb">
  
  <div id="${args.htmlid}-title" class="title">
    ${title}
  </div>

  <div class="toolbar">
    <a id="${args.htmlid}-configAafb-link" class="configure theme-color-1" href="#">${msg("config")}</a>
  </div>
  
  <div class="body scrollableList">
    <h2><span id="${args.htmlid}-headline">${headline}</span></h2>
    <p><span id="${args.htmlid}-story">${story}</span></p>
  </div>

</div>
    

This template uses properties values from the aafb.get.properties file and has access to the scripts included in the aafb.get.head.ftl file.

Create Instance of Custom Object for Managing Configuration

At the top of our aafb.get.html.ftl you probably noticed a script block. The first seven lines within that block create a new instance of a custom object. This custom object is used to manage the values of our configuration parameters and control the behavior of the configuration dialog used to set those values.

When the script block executes on the loading of our dashlet, the constructor and setOptions methods of our custom object are called. Those methods are defined in the aafb.js file. The code snippet below provides the entire contents of the aafb.js so that you can see how these two methods fit into the overall file. We’ll discuss the individual sections of the file a little bit later.

        /**
 * Alfresco.Aafb
 *
 */
(function()
{
  /**
   * YUI Library aliases
   */
  var Dom = YAHOO.util.Dom,
      Event = YAHOO.util.Event;

  /**
   * Alfresco Slingshot aliases
   */
  var $html = Alfresco.util.encodeHTML;

  Alfresco.Aafb = function(htmlId)
  {
    this.name = "Alfresco.Aafb";
    this.id = htmlId;
    this.configDialog = null;

    /**
     * Register this component
     */
    Alfresco.util.ComponentManager.register(this);

    /**
     * Load YUI Components
     */
    Alfresco.util.YUILoaderHelper.require([],this.onComponentsLoaded,this);

    return this;
  };

  YAHOO.extend(Alfresco.Aafb, Alfresco.component.Base,
  {
    /**
     * Object container for initialization options
     *
     * @property options
     * @type object
     */
    options:
    {
      /**
       * The component id
       *
       * @property componentId
       * @type string
       * @default ""
       */
      componentId: "",

      /**
       * The configurable title to display
       *
       * @property titleStr
       * @type string
       * @default ""
       */
      titleStr: "",

      /**
       * The configurable headline to display
       *
       * @property headlineStr
       * @type string
       * @default ""
       */
      headlineStr: "",

      /**
       * The configurable story to display
       *
       * @property storyStr
       * @type string
       * @default ""
       */
      storyStr: ""
    },

    /**
     * Fired by YUI when parent element is available for scripting.
     * Component initialisation, including instantiation of YUI widgets
     * and event listener binding.
     *
     * @method onReady
     */
    onReady: function A_onReady()
    {
      // Add click handler to config Aafb link that will be visible if user
      // is site manager.
      var configFeedLink = Dom.get(this.id + "-configAafb-link");
      if (configFeedLink)
      {
        Event.addListener(configFeedLink,"click",this.onConfigAafbClick,this,true);
      }
    },

    /**
     * Called when the user clicks the config Aafb link.
     * Will open an Aafb config dialog
     *
     * @method onConfigAafbClick
     * @param e The click event
     */
    onConfigAafbClick: function A_onConfigAafbClick(e)
    {
      var actionUrl = Alfresco.constants.URL_SERVICECONTEXT +
       "modules/aafb/config/" + encodeURIComponent(this.options.componentId);

      if (!this.configDialog)
      {
        this.configDialog = new Alfresco.module.SimpleDialog(this.id + "-configDialog").setOptions({
          width: "30em",
          templateUrl: Alfresco.constants.URL_SERVICECONTEXT + "modules/aafb/config",
          actionUrl: actionUrl,
          onSuccess:
          {
            fn: function Aafb_onConfigFeed_callback(response)
            {
              var aafb = response.json;

              // Save values for new config dialog openings
              this.options.titleStr    = (aafb && aafb.title) ? aafb.title : this.options.titleStr;
              this.options.headlineStr = (aafb && aafb.headline) ? aafb.headline : this.options.headlineStr;
              this.options.storyStr    = (aafb && aafb.story) ? aafb.story : this.options.storyStr;

              // Update dashlet body with new values
              Dom.get(this.id + "-title").innerHTML    = aafb ? aafb.title : "";
              Dom.get(this.id + "-headline").innerHTML = aafb ? aafb.headline : "";
              Dom.get(this.id + "-story").innerHTML    = aafb ? aafb.story : "";
            },
            scope: this
          },
          doSetupFormsValidation:
          {
            fn: function Aafb_doSetupForm_callback(form)
            {
              form.addValidation(this.configDialog.id + "-title",
               Alfresco.forms.validation.mandatory, null, "keyup");
              form.addValidation(this.configDialog.id + "-headline",
               Alfresco.forms.validation.mandatory, null, "keyup");
              form.setShowSubmitStateDynamically(true, false);

              Dom.get(this.configDialog.id + "-title").value = this.options.titleStr;
              Dom.get(this.configDialog.id + "-headline").value = this.options.headlineStr;
              Dom.get(this.configDialog.id + "-story").value = this.options.storyStr;
            },
            scope: this
          }
        });
      }
      else
      {
        this.configDialog.setOptions(
        {
          actionUrl: actionUrl
        });
      }
      this.configDialog.show();
    }
  });
})();
    

Layout Configuration Dialog

Next, let’s look at the implementation of the dialog used to set configuration values. Much like the dashlet itself, the configuration dialog is rendered with a FreeMarker template. In our example, the config-aafb.get.html.ftl file is used as follows:

        <div id="${args.htmlid}-configDialog" class="config-aafb">
   <div class="hd">${msg("label.dialogTitle")}</div>

   <div class="bd">
      <form id="${args.htmlid}-form" action="" method="POST">
         <div class="yui-gd">
            <div class="yui-u first"><label for="${args.htmlid}-title">${msg("label.title")}:</label></div>
            <div class="yui-u"><input  id="${args.htmlid}-title" type="text" name="title" value="" maxlength="30" />&nbsp;*</div>
         </div>
         <div class="yui-gd">
            <div class="yui-u first"><label for="${args.htmlid}-headline">${msg("label.headline")}:</label></div>
            <div class="yui-u"><input  id="${args.htmlid}-headline" type="text" name="headline" value="" maxlength="75" />&nbsp;*</div>
         </div>
         <div class="yui-gd">
            <div class="yui-u first"><label for="${args.htmlid}-story">${msg("label.story")}:</label></div>
            <div class="yui-u"><input  id="${args.htmlid}-story" type="text" name="story" value="" maxlength="256" /></div>
         </div>
         <div class="bdft">
            <input type="submit" id="${args.htmlid}-ok" value="${msg("button.ok")}" />
            <input type="button" id="${args.htmlid}-cancel" value="${msg("button.cancel")}" />
         </div>
      </form>
   </div>

</div>
    

Just like our aafb.get.html.ftl, the config-aafb.get.html.ftl uses properties from its own property file, config-aafb.get.properties.

Define Configuration Dialog Validation Behavior

With our dialog in place, let’s discuss how to define the validation behavior for fields on that dialog.

In the doSetupFormsValidation method we set up validation rules that make both the title and headline fields mandatory, and prevent users from submitting the dialog if the rules aren’t satisfied. In this method, we also set the form values to the current values of the configuration parameters so users can see what they’re about to change. This is all done in aafb.js as follows:

        doSetupFormsValidation:
{
  fn: function Aafb_doSetupForm_callback(form)
  {
    form.addValidation(this.configDialog.id + "-title", Alfresco.forms.validation.mandatory, null, "keyup");
    form.addValidation(this.configDialog.id + "-headline", Alfresco.forms.validation.mandatory, null, "keyup");
    form.setShowSubmitStateDynamically(true, false);

    Dom.get(this.configDialog.id + "-title").value = this.options.titleStr;
    Dom.get(this.configDialog.id + "-headline").value = this.options.headlineStr;
    Dom.get(this.configDialog.id + "-story").value = this.options.storyStr;
  },
  scope: this
}
    

Alfresco supplies many other form validation rules as part of its forms service. You can find more documentation on the Alfresco forms service on the Alfresco Wiki.

Submit Configuration Dialog

The configuration dialog is submitted via a POST to a JavaScript controller. The controller is implemented in the config-aafb.post.json.js file. This controller takes the values from the dialog’s form, sticks them in properties that are persisted, and then sets them on the model so the new values can be delivered back to our waiting callback function. You can see how simple this whole process is here:

        var c = sitedata.getComponent(url.templateArgs.componentId);

var title = String(json.get("title"));
var headline = String(json.get("headline"));
var story = String(json.get("story"));

c.properties["title"] = title;
c.properties["headline"] = headline;
c.properties["story"] = story;

model.title = title;
model.headline = headline;
model.story = story;

c.save();
    

Define Configuration Dialog Response

The response from our POST to the JavaScript controller carries our news values back to the aafb.js file in a JSON object. This object is handled by the onSuccess function that we’ll look at in the next section. The JSON response is defined by a FreeMarker template in the config-aafb.post.json.ftl as follows:

        <#escape x as jsonUtils.encodeJSONString(x)>
{
   "title": "${title!''}",
   "headline": "${headline!''}",
   "story": "${story!''}"
}
</#escape>
    

Process Configuration Dialog Response

The final thing we need to do is define what our dashlet should do when new values are successfully submitted and processed through the configuration dialog. In the onSuccess function we setup a callback mechanism that accepts the JSON response and sets some values. First, we set the values of our custom object. Then we update the contents of the dashlet’s markup so that users will automatically see the new values. This is all done in aafb.js as follows:

        onSuccess:
{
  fn: function Aafb_onConfigFeed_callback(response)
  {
    var aafb = response.json;

    // Save values for new config dialog openings
    this.options.titleStr    = (aafb && aafb.title) ? aafb.title : this.options.titleStr;
    this.options.headlineStr = (aafb && aafb.headline) ? aafb.headline : this.options.headlineStr;
    this.options.storyStr    = (aafb && aafb.story) ? aafb.story : this.options.storyStr;

    // Update dashlet body with new values
    Dom.get(this.id + "-title").innerHTML    = aafb ? aafb.title : "";
    Dom.get(this.id + "-headline").innerHTML = aafb ? aafb.headline : "";
    Dom.get(this.id + "-story").innerHTML    = aafb ? aafb.story : "";
  },
  scope: this
},
    

A Few Words of Advice

Now that you’ve seen how a simple dashlet that supports configuration is assembled, let’s take a moment to talk about a few of the areas that could trip you up.

Don’t Lose Your Head

Don’t forget to include the *.get.head.ftl (aafb.get.head.ftl in our example) file in your dashlet. There isn’t much going on in this file, but without it, you won’t have access to your *.js file(s) or any custom styles you might define in a *.css file. If you see something like this in your JavaScript console:

        Alfresco.Aafb is not a constructor
    

It’s likely that the problem is that you’ve left out your *.get.head.ftl file. This would be a great place to start your troubleshooting efforts. Trust me. I speak from experience.

Minified JavaScript

By default, Alfresco is configured to look for minified JavaScript files. JavaScript minification is the process of removing all unnecessary characters from the source to minimize the file size, without changing the JavaScript’s functionality. This helps improve your site’s performance, but it can make development a little trickier.

In your *.get.head.ftl file you’ll see a line like this:

        <@script type="text/javascript" src="${page.url.context}/components/dashlets/aafb.js"></@script>
    

This line makes your *.js file available to the rest of the files in your dashlet. But by default, Alfresco will actually replace the reference to aafb.js with a reference to aafb-min.js. So, if you haven’t created a -min version of your JavaScript, your dashlet won’t work.

One way to address this is by ensuring that the minified version of your JavaScript file(s) is current and available inside the same folder as your base JavaScript file. You could do this by manually copying and renaming your base JavaScript file after each change (ouch!), or through the use of a development tool such as JSMIN to generate JavaScript files that are actually minified.

Alternatively, you could configure Alfresco to not look for the minified JavaScript file during your development process. One simple way to do this is to change the <@script> tag in your *.get.head.ftl file to a <script> tag. This would change the reference in your *.get.head.ftl to look something like:

        <script type="text/javascript" src="${page.url.context}/components/dashlets/aafb.js"></script>
    

Parting Thoughts

There you have it, a straightforward (as straightforward as I can put it anyway) example of a custom dashlet that supports configuration. The number of files required to support configuration can seem intimidating at first, but I hope this article has shown you that once you understand how the files work together, it’s really not all that complex.

So, now that you know how enable configuration in your own custom dashlets, what will you build? To help you get started, I’ve included a collection of documentation, links, and source files in the Resources section. Please share your questions, thoughts and ideas in the comments below. Thanks, and happy configuring!

Resources

The following links and documents were referenced in writing this article, and are a great source for more information on all of the topics above:

• Alfresco Web Scripts
• Web Scripts Framework
• Alfresco Surf Platform – Developers Guide
• Dr. Q’s Workshop – Hello World Dashlet
• Blue Fish – Alfresco Source Code Organization
• Alfresco Wiki
• JSMin – The JavaScript Minifier
• Example Dashlet Source Code

The event will preview Alfresco 3.3, the latest release from the leading open source ECM provider. We’re hosting this two-hour session in five Texas cities, so hopefully you can find one to attend. All sessions run from 11:00 – 1:00 and lunch will be provided.

• Dallas – Tuesday, April 20
• Ft. Worth – Wednesday, April 21
• Austin – Thursday, April 22
• San Antonio – Friday, April 23
• Houston – Tuesday, April 27

Register for all Lunch and Learns on the Alfresco web site.

This session going to be great for CIOs, IT Managers, or anyone else considering Alfresco for an ECM initiative. During the event, we’ll cover the following topics:

• Sneak Preview of Alfresco 3.3 – Includes CMIS 1.0, enhancements to Alfresco Share, and IBM Lotus integration
• Major WCM Enhancements – Demonstration of some exciting new features for editing web content and deploying web sites

Space is limited, so please register early to secure your place.

I hope to see you there.

Why is that?

The answer is that Alfresco is bringing the benefits of Enterprise Content Management to companies that don’t have millions of dollars to spend on ECM solutions. It’s also bringing ECM to smaller business problems that can’t justify big money to solve them.

Alfresco is a disruptive force in the ECM industry, similar to the way that SalesForce.com disrupted traditional CRM vendors like Seibel. Let’s look at how the two compare.

At Blue Fish, we use SalesForce, and we would never in a million years use Seibel. For a sales team of only 2 people, it’s just not worth it. But SalesForce gives us a world-class CRM solution for a price that we can afford. SalesForce isn’t just for small shops, either. Dell uses SalesForce, and they have one of the largest sales teams in the world. That’s true disruption, and it all but put Seibel out of business.

SalesForce did this by using a SaaS business model to dramatically lower the Total Cost of Ownership (TCO) of CRM solutions.

• SalesForce is cheaper for me to acquire (its per user pricing is less than the competition)
• SalesForce is cheaper for me to operate (I have no servers to maintain and no dependencies on databases, etc.)
• SalesForce is cheaper for me to support and maintain (no backups, no servers crashing, and I don’t have to do any upgrades myself)

Now, SalesForce’s customers don’t care about the cloud, SaaS, or any of that business model stuff – they just want it to be cheap. It is, and that changed the world of CRM.

Alfresco is doing something similar in the world of ECM. They are going about it differently, but they are shooting for the same result. Alfresco uses an open source model that lets their customers try before they buy and do so without ever talking to a sales rep. Eventually, companies who rely on Alfresco for their critical business data will want to buy an Enterprise subscription with a support agreement, and that’s how Alfresco makes their money.

But that’s not really disruptive (it’s a clever marketing strategy, but it’s not truly disruptive). What’s disruptive is the price of their Enterprise license – it’s dirt cheap. You can get a fully supported version of Alfresco that can support hundreds of users for less than $20,000 a year. Compare that to the traditional players like EMC and IBM who would charge hundreds of thousands of dollars for a similar solution, and you start to see what I’m talking about.

Alfresco is also cheaper and easier to operate and maintain than traditional ECM solutions. Because of its open source heritage, it can operate on an open source stack, with open source operating systems, databases, and application servers. That reduces the cost of the infrastructure required to run it.

And Alfresco is easier to implement and extend. Because it has come to market recently, it uses modern application development and deployment technologies. Alfresco runs in a Java application server, so you don’t have to learn new deployment or monitoring procedures. Developers extend it using web services and lightweight web scripts, which are easier to learn. And most importantly, you have access to the source code (even for the Enterprise Edition), so it’s a thousand times easier to find a defect and fix it or to follow Alfresco’s patterns when developing your own extensions.

There are places where the big traditional ECM players still make sense. If you want to run your ECM solution on a mainframe, IBM is the only game in town. If you are a pharmaceutical company with regulated content, Documentum is the strongest bet. If you’ve standardized on Microsoft throughout the company, SharePoint might be for you.

But if you are looking for the best value – a full-featured ECM solution that won’t break the bank, Alfresco is the best bet.

As a result, companies that never considered ECM solutions are looking at them for the first time.

• Repository Access from Alfresco Share – The most frequently requested product enhancement is now available. The Document Library in Alfresco Share now allows users to quickly move from working within a project site to full access to the general document management repository.
• Web Content Deployment Options – This release sees Alfresco include the ability to deploy web content to an Alfresco DM repository. This can be used to serve personalized content and can be clustered to achieve maximum performance and scalability for Alfresco-powered websites.
• Enhanced Document Browsing Options – This allows users to browse the document repository using folders, Cloud Tags, Category lists or by Favorites. These new options make it even easier to find the correct content.
• Faster, More Stable Document Conversions – The new OpenOffice transformation subsystem sports the ability to create pools of fully managed OpenOffice services to dramatically improve document conversion speed, reliability and overall system stability.
• New Portal and Portlet Development Options – With this release, Alfresco includes support for portals and portlets, meaning developers can now create new Alfresco-powered portlets and deploy them into a portal with support for single sign-on (SSO) built-in.
• Records Management-ready – Alfresco Enterprise Edition 3.2r is ready to accept the fully certified, DoD 5015.02-certified Records Management module.

I’m most excited about the improvements to Share. We’ve been using Share a lot lately, and it’s becoming my preferred UI for WCM projects as well as DM projects. It’s evolving quickly, and I love it when they release a new version.

You can download a trial version here: http://www2.alfresco.com/e/1234/ry/GIEWE/177864364.

• In-Context Editing – Create and edit content right from your web site (no need to log into a WCM system)
• Sexy New Web Page Forms/Templates – We’re using the fancy new forms service in Share, which makes our templates really pretty and easy to use
• An Easy Way to Preview Changes Before They Go Live – You can put your web site into “preview mode” to see what the site will look like when the new pages are approved
• An Easy Way to Approve Changes – Reviewing and approving changes is a breeze

You can see a recorded version of the webinar here: Making Web Content Easy with Alfresco Share 3.2.

The deal looks like this: FatWire will resell EMC’s digital asset management software, and EMC will resell FatWire’s WCM software.

What’s So Great About FatWire?

FatWire is the largest independent WCM (or as they like to be called, WEM) software vendor. They have over 500 enterprise accounts and 200 employees. Most analysts consider FatWire to be one of the best solutions for WCM/WEM. So they have a lot of credibility behind them.

The FatWire solution is much more holistic than Web Publisher. While Web Publisher is focused only on the creation, approval, and management of web content, FatWire also focuses on delivery and presentation of that content to end users (web site visitors). With FatWire, you get a much more sophisticated solution “out of the box”, while getting the same types of features with a Web Publisher-driven solution would require thousands of hours of development effort from your web team.

Here are some of the features FatWire has that will be new to Web Publisher owners:

• Drag and Drop Authoring – FatWire’s user interface allows web page authors to drag rich page components from a “library” into a web page. For example, you can drag a “slideshow” component into a web page and configure it to cycle through a set of images. FatWire has a bunch of built-in components for things like this, so you don’t have to be a web design expert to create a rich user experience.
• Personalization and Segment Targeting – FatWire has robust targeting and personalization features that can target certain content to certain segments of your visitors. These segments can be defined based on “known” profile information (like a customer’s address from your CRM system) or they can be created based on the visitor’s behavior on the site. So not only can you offer, say, coats and snowshoes to someone that lives in Alaska, you can keep track of the color of other items they’ve looked at on your site and then show them coats and snowshoes in those colors.
• User Generated Content – FatWire has built-in support for user generated content such as ratings, reviews, and comments. And it comes complete with a set of moderation features that allow your content managers to approve the user generated content before it goes live on your site.
• Analytics – FatWire includes rich analytics that help you understand what content is most popular and is driving the highest conversion rates, etc. It can do this not only at the page level, but also at the component level, so if you have a rotating set of product images on your home page, you can see which image is doing the best job of catching your visitor’s attention.
• Edge caching – FatWire includes built-in edge caching servers called “satellite servers” that allow all these fancy targeting rules to work in a high-performance manner.

Integrating with Documentum

FatWire has its own architecture, repository, and technology stack that differ from Documentum’s, so it breaks Documentum’s vision of a single integrated repository for all enterprise content. And since EMC didn’t fully acquire FatWire, there are no plans to replace the FatWire repository with the Documentum repository.

However, FatWire does have an integration with Documentum that will synchronize content from Documentum into FatWire. When you make a change to the content in Documentum, it will automatically update in FatWire (and on your web site), or optionally launch a FatWire workflow to be approved before it goes live. The main use case I see for this is the ability to embed into your FatWire web pages the digital assets (videos, etc.) that are stored in Documentum’s Digital Asset Management solution.

Later this year, there will be an integration that goes the other way, allowing FatWire content to be replicated into Documentum as well.

What is going to happen to Web Publisher?

Peggy Ringhausen, an old friend of mine and the WCM product manager at EMC, says that the new FatWire solution is the wave of the future and that they will be retiring Web Publisher, Page Builder, and the other products in the WCM product line. No date has been announced for this, but it probably won’t happen for at least three years.

EMC says they have some licensing deals that allow existing Web Publisher owners to get their hands on the new FatWire solution, but you shouldn’t expect it to be a free upgrade.

It will be interesting to see how existing Web Publisher owners react to this news. On the one hand, their existing investment is in jeopardy, but on the other hand, the replacement seems like a much stronger solution.

When a developer customizes Alfresco to add a new webscript or portlet or modify an existing class, the common practice is to create new source code files in the same directory structure as the out-of-the-box Alfresco source files. This can cause maintenance problems because it’s hard to tell which files have been modified and which ones were original. It makes troubleshooting and upgrading much more difficult, and it’s harder to share your code or reuse it in other solutions.

Josh’s article shows how you can configure Alfresco to look in a special place for your custom code so that you can keep it out of the main Alfresco directory structure. Read Painless Alfresco Development: Improving Supportability through Source Code Organization for more information.

Introduction

Note:
This article assumes that you already know the ins and outs of basic
Spring configuration and that you know how to build basic Alfresco
extensions. If you need an introduction to these concepts, I
recommend you start with the
Alfresco development wiki
.

Alfresco is great. It’s a rock solid content management and
collaboration platform with an exciting roadmap. Although it’s only a
handful of years old, it can go toe to toe with the legacy players in
the field and often come out ahead. It has an immensely flexible
architecture that makes it pretty much as easy as possible to extend
and adapt the base Alfresco offerings to meet specific requirements.
I love having access to the source. I love that you can “try before
you buy.” I love Alfresco’s enormous community that does everything
from supporting end users to evolving core products. I love that
Alfresco is a disruptive player poised to drastically alter the
landscape of the otherwise stodgy world of Enterprise Content
Management.

Ok, so what don’t I love about Alfresco? Well, the commonly
practiced methods for extending Alfresco sometimes force me to
violate certain development best practices, like encapsulating and
isolating custom source code. For many types of extensions,
out-of-the-box Alfresco requires you to distribute source code across
many different directories, and those directories are typically
nested inside of stock Alfresco directories. For example, a simple
custom dashlet might include the following files:

Figure 1:
Conventional source code organization for an Alfresco Share extension.

And these files would get deployed alongside stock Alfresco source
files within the web application like so:

Figure 2:
Conventionally organized Alfresco Share extension files are deployed alongside out-of-the-box files, making it difficult to easily identify what is custom and what is stock.

As you can see, the extension source files live underneath the
alfresco directory right alongside stock Alfresco files and
potentially files from every other extension that might be deployed.
This makes it difficult to quickly identify which files make up the
extension. From a development best practices perspective, it is
simply bad hygiene to pollute Alfresco’s directories with dozens of
custom files and directories.

Thankfully, Alfresco’s flexible Spring-based configuration
architecture makes it easy to structure an Alfresco development
project in such a way that cleanly separates custom files from
everything else. Before we get into the details of these Spring
configuration mechanics, let’s take a look at what I’d like my source
tree to look like.

Preferred Source Code Organization

There are many ways you can organize your source such that it is
isolated from the out-of-the-box distribution. To make my source
trees feel familiar to seasoned Alfresco developers, I like to create
source structures that closely mirror the out-of-the-box Alfresco and
Share web applications. Revisiting the source for our initial
extension example, I would structure it like so:

Figure 3:
My preferred source code organizational structure for Alfresco extensions.

I find that isolating my files like this makes it easier to debug,
package, and distribute my extensions because they readily stand out
from the stock Alfresco files. It also tends to make it easier for
other developers to understand how my extensions work because they
can quickly identify all of the files that comprise each extension.
Here’s what my deployed dashlet source looks like with this
organizational structure:

Figure 4:
With my preferred source code organizational structure, all of the custom files get deployed under a single folder.

Look at that—all of my source files are grouped under a common
folder!

A Tale of Two Web Apps

You may have noticed that I begin my source tree with a folder named
share. This folder maps directly to the Share web application
deployed by Alfresco. Similarly, if my extensions required that files
be deployed to the Alfresco web application, I would organize those
files under a root folder called alfresco.

If you’re wondering why an extension might require that files be
deployed to multiple web applications, just remember how Alfresco is
architected: the Alfresco web application hosts the core repository,
all of the webscripts that provide access to the core repository, and
the Java Server Faces-based Alfresco Explorer web client; and the
Share web application hosts the Spring Surf-based Alfresco Share user
interface and its supporting webscripts.

If you take a look at your application server’s webapp directory after
installing Alfresco, you should see something like this:

Figure 5:
Alfresco is typically deployed as two (or more) web applications. In this example, “alfresco” hosts the Alfresco repository, and “share” hosts the Share UI.

The Share web application interacts with the Alfresco repository by
calling the webscripts published by the Alfresco web application. It
is important to remember this because your Alfresco customizations
may reside in one web app, the other, or both.

Resolving Custom Resources

In order for Alfresco to be able to find the files in my preferred
folder structure, we need to tell Alfresco where to look. Resources
in Alfresco—be they XML configuration files, Freemarker templates,
webscript implementations, and others—are resolved using Spring’s
configuration framework. Furthermore, Alfresco knows to look for a
handful of specific custom Spring configuration files to pick up
custom values that override default values defined in Alfresco’s
default Spring configuration. This means that the particular methods
for how Alfresco resolves resources can be easily changed simply by
adding a few
<bean> entries to one of these custom Spring configuration files. The
specific files that need to be edited are:

• /WEB-INF/classes/alfresco/web-extension/custom-slingshot-application-context.xml
  (for the Share webapp)
• /WEB-INF/classes/alfresco/extension/custom-web-context.xml (for
  the Alfresco webapp)

You should be aware that these files may not yet exist in your Alfresco
implementation, particularly if you have not built any extensions
yet. If necessary, go ahead and create them with the following empty
Spring bean configuration:

            
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 'http://www.springframework.org/dtd/spring-beans.dtd'>
<!-- Top-level imports of XML configuration -->
<beans>

</beans>

        

Now comes the tough part—finding out which Spring beans need to be
overridden to pick up your particular resources. Thankfully, Alfresco
follows a nice naming convention for their Spring beans and has named
the various search paths with names containing the string
“searchpath.” This makes it easy to find all of the default search
path configurations—just use your favorite IDE to search for the
string id='*searchpath' across all xml files in the Alfresco source
tree. In the current 3.2 Community source, this yields 14 distinct
matches for the Share webapp:

• 
webframework.searchpath
• 
webframework.templates.searchpath
• 
webframework.searchpath.chrome
• 
webframework.searchpath.component
• 
webframework.searchpath.componenttype
• 
webframework.searchpath.configuration
• 
webframework.searchpath.contentassociation
• 
webframework.searchpath.page
• 
webframework.searchpath.pageassociation
• 
webframework.searchpath.pagetype
• 
webframework.searchpath.templateinstance
• 
webframework.searchpath.templatetype
• 
webframework.searchpath.theme
• 
webframework.presets.searchpath

And one match for the Alfresco webapp:

• webscripts.searchpath

To override one of these search paths, start by copying and pasting the
appropriate bean definition into your custom configuration
(custom-slingshot-application-context.xml for the Share webapp or
custom-web-context.xml for the Alfresco webapp). For example, let’s
assume that you have created a new Share component whose source files
you would like to keep separate from the out-of-the-box Share source.
You would copy this from slingshot-application-context.xml into
custom-slingshot-application-context.xml:

            
   <bean id="webframework.searchpath.component" class="org.alfresco.web.scripts.SearchPath">
      <property name="searchPath">
         <list>
         	<ref bean="webframework.classpathstore.component.custom" />
            <ref bean="webframework.classpathstore.component" />
         </list>
      </property>
   </bean>

        

Next, create a new bean that knows how to find your resources. Most of the
time, you will be want to simply extend Alfresco’s
webframework.classpathstore bean so that it looks in specific paths
within the classpath. Continuing the above example, let’s create a
bean that looks for component definitions in
bluefishgroup/share-ext/site-data/components:

            
	<bean id="bluefishgroup.components" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>true</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/components</value>
		</property>
	</bean>

        

Finally, update the bean definition you copied to reference your
new bean:

            
   <bean id="webframework.searchpath.component" class="org.alfresco.web.scripts.SearchPath">
      <property name="searchPath">
         <list>
			<ref bean="bluefishgroup.components" />
            <ref bean="webframework.classpathstore.component.custom" />
			<ref bean="webframework.classpathstore.component" />
         </list>
      </property>
   </bean>

        

Now Share will search for our component definitions in
bluefishgroup/share-ext/site-data/components in addition to the
default locations. Wasn’t that easy? To make it even more convenient,
I use a template custom-slingshot-application-context.xml that
overrides all of the default search path beans so that all I need to
do when I start a new Alfresco development project is add the
specific paths. Here is the template configuration for you to use in
your own projects:

            
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 'http://www.springframework.org/dtd/spring-beans.dtd'>

<beans>

	<!-- Chrome search path overrides -->

	<bean id="bluefishgroup.chrome" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/pages/chrome</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.chrome" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.chrome" />
				<ref bean="webframework.remotestore.chrome" />
				<ref bean="webframework.classpathstore.chrome.custom" />
				<ref bean="webframework.classpathstore.chrome" />
			</list>
		</property>
	</bean>

	<!-- Component search path overrides -->

	<bean id="bluefishgroup.components" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/components</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.component" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.components" />
				<ref bean="webframework.remotestore.component" />
				<ref bean="webframework.classpathstore.component.custom" />
				<ref bean="webframework.classpathstore.component" />
			</list>
		</property>
	</bean>

	<!-- Component types search path overrides -->

	<bean id="bluefishgroup.componentTypes" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/component-types</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.componenttype" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.componentTypes" />
				<ref bean="webframework.remotestore.componenttype" />
				<ref bean="webframework.classpathstore.componenttype.custom" />
				<ref bean="webframework.classpathstore.componenttype" />
			</list>
		</property>
	</bean>

	<!-- Configuration search path overrides -->

	<bean id="bluefishgroup.configurations" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/configurations</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.configuration" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.configurations" />
				<ref bean="webframework.remotestore.configuration" />
				<ref bean="webframework.classpathstore.configuration.custom" />
				<ref bean="webframework.classpathstore.configuration" />
			</list>
		</property>
	</bean>

	<!-- Content associations search path overrides -->

	<bean id="bluefishgroup.contentAssociations" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/configurations</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.contentassociation" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.contentAssociations" />
				<ref bean="webframework.remotestore.contentassociation" />
				<ref bean="webframework.classpathstore.contentassociation.custom" />
				<ref bean="webframework.classpathstore.contentassociation" />
			</list>
		</property>
	</bean>

	<!-- Page search path overrides -->

	<bean id="bluefishgroup.pages" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>true</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/pages</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.page" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.pages" />
				<ref bean="webframework.remotestore.page" />
				<ref bean="webframework.classpathstore.page.custom" />
				<ref bean="webframework.classpathstore.page" />
			</list>
		</property>
	</bean>

	<!-- Page association search path overrides -->

	<bean id="bluefishgroup.pageAssociations" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>true</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/pages</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.pageassociation" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.pageAssociations" />
				<ref bean="webframework.remotestore.pageassociation" />
				<ref bean="webframework.classpathstore.pageassociation.custom" />
				<ref bean="webframework.classpathstore.pageassociation" />
			</list>
		</property>
	</bean>

	<!-- Page type search path overrides -->

	<bean id="bluefishgroup.pageTypes" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/pages</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.pagetype" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.pageTypes" />
				<ref bean="webframework.remotestore.pagetype" />
				<ref bean="webframework.classpathstore.pagetype.custom" />
				<ref bean="webframework.classpathstore.pagetype" />
			</list>
		</property>
	</bean>


	<!-- Presets search path overrides -->

	<bean id="bluefishgroup.presets" class="org.alfresco.web.scripts.ClassPathStore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/presets</value>
		</property>
	</bean>

	<bean id="webframework.presets.searchpath" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.presets" />
				<ref bean="webframework.classpathstore.presets.custom" />
				<ref bean="webframework.classpathstore.presets" />
			</list>
		</property>
	</bean>

	<!-- Templates search path overrides -->

	<bean id="bluefishgroup.templates" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/templates</value>
		</property>
	</bean>

	<bean id="webframework.templates.searchpath" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.webscripts" />
				<ref bean="bluefishgroup.templates" />
				<ref bean="webframework.store.templates.custom" />
				<ref bean="webframework.store.webscripts.custom" />
				<ref bean="webframework.store.templates" />
				<ref bean="webframework.store.webscripts" />
				<ref bean="webframework.store.system-templates" />
			</list>
		</property>
	</bean>

	<!-- Template instances search path overrides -->

	<bean id="bluefishgroup.templateInstances" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/template-instances</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.templateinstance" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.templateInstances" />
				<ref bean="webframework.classpathstore.templateinstance.custom" />
				<ref bean="webframework.classpathstore.templateinstance" />
			</list>
		</property>
	</bean>

	<!-- Template types search path overrides -->

	<bean id="bluefishgroup.templateTypes" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-data/template-types</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.templatetype" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.templateTypes" />
				<ref bean="webframework.remotestore.templatetype" />
				<ref bean="webframework.classpathstore.templatetype.custom" />
				<ref bean="webframework.classpathstore.templatetype" />
			</list>
		</property>
	</bean>

	<!-- Template processor search path overrides -->
	<!--
		We need to set these to ensure that the template processors can find
		our web scripts.
     -->

	<bean id="webframework.webscripts.templateprocessor" class="org.alfresco.web.scripts.PresentationTemplateProcessor">
		<property name="searchPath" ref="webframework.searchpath" />
		<property name="updateDelay">
			<value>0</value>
		</property>
		<property name="defaultEncoding">
			<value>UTF-8</value>
		</property>
	</bean>

	<bean id="webframework.templateprocessor" class="org.alfresco.web.scripts.PresentationTemplateProcessor">
		<property name="searchPath" ref="webframework.templates.searchpath" />
		<property name="defaultEncoding">
			<value>UTF-8</value>
		</property>
		<property name="updateDelay">
			<value>0</value>
		</property>
	</bean>

	<!-- Theme search path overrides -->

	<bean id="bluefishgroup.themes" parent="webframework.classpathstore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/themes</value>
		</property>
	</bean>

	<bean id="webframework.searchpath.theme" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.themes" />
				<ref bean="webframework.remotestore.theme" />
				<ref bean="webframework.classpathstore.theme.custom" />
				<ref bean="webframework.classpathstore.theme" />
			</list>
		</property>
	</bean>

	<!-- WebScripts search path overrides -->

	<bean id="bluefishgroup.webscripts" class="org.alfresco.web.scripts.ClassPathStore">
		<property name="mustExist">
			<value>false</value>
		</property>
		<property name="classPath">
			<value>bluefishgroup/site-webscripts</value>
		</property>
	</bean>

	<bean id="webframework.searchpath" class="org.alfresco.web.scripts.SearchPath">
		<property name="searchPath">
			<list>
				<ref bean="bluefishgroup.webscripts" />
				<ref bean="webframework.store.webscripts.custom" />
				<ref bean="webframework.store.webscripts" />
				<ref bean="webscripts.store" />
			</list>
		</property>
	</bean>

</beans>

        

Caveats

Although I recommend taking the above steps to enable flexible
organization of your source, there are a couple things you should
watch out for.

The first caveat is common to most Spring-based applications.
Because you are overriding existing configuration elements (as
opposed to extending), you may run into forward compatibility issues
when you upgrade to a newer version/build of Alfresco. To stay on top
of this, make it a habit to always examine the source of those
configurations you override each time you update your Alfresco
source. If something changes, simply copy the new element into your
custom configuration file and reinsert your custom bean reference(s).

The second caveat is that it is not always best for all resources
to be resolved from the classpath. While this may be the most
straightforward way to deploy resources (because you can easily see
them within your file system) you can run into consistency problems
if you need to maintain these resources across multiple web
applications (as in a clustered environment). In those cases, you
should consider distributing your resources via the Alfresco
repository itself. You can do this by using instances of the
webframework.remotestore bean. Alfresco already does this for a
handful of resources (search the XML files in the Alfresco source for
instances of webframework.remotestore.component for an example), and
it should be straightforward to add repository lookup capability for
other resources by creating new beans in your Spring configuration.

Parting Thoughts

I like sharing my Alfresco development experiences so that others
may be able to build upon them to continuously improve the quality of
contributions to the Alfresco ecosystem. Along those lines, I hope
you’ve found this article helpful, and I encourage you to ask
questions or leave feedback by adding a comment below. Thanks for
reading!

This session is a must for CIOs, Senior IT Managers, Enterprise Architects and anyone evaluating ECM solutions. We will cover the following topics:

• A comparison of Alfresco Community, Alfresco Enterprise, and Traditional ECM
• A review of Alfresco Enterprise 3.2
• Alfresco implementation Best Practices

Space is limited to 25 people, so book your place as soon as possible. Click here to register for the Alfresco lunch and learn session.

The session will last 2 hours, from 11am – 1pm, and of course we’ll provide lunch. Blue Fish is located in Downtown Austin – here’s a map to the Blue Fish office.

Read it at http://www.cmswire.com/cms/enterprise-20/centerstage-how-emc-is-taking-on-microsoft-sharepoint-005310.php.

Here’s a list of the Documentum-related blogs I read regularly.

• Craig’s Musings – I first met Craig Randall at a Documentum conference about a million years ago. At that time, Craig was the king of Documentum’s Desktop Client and guru of their Microsoft technology strategy. Today, he’s one of EMC’s biggest brains and one of their best bloggers. I read everything he writes.
• Ask Johnny! – I think everyone in the Documentum community knows about Johnny Gee. He’s the most prolific member of the community that I know, answering more questions on dm_developer and EMC’s forums and everyone else combined. Johnny is a great guy and has been around for years – he really knows his stuff. Johnny doesn’t blog that much anymore, but his archives are full of great content.
• Word of Pie – Pie is one of the most prolific bloggers in the Documentum blogosphere. His posts range from technical how-to articles to opinion pieces and book reviews. Along with Big Men on Content, he’s one of the few bloggers that really expresses an opinion. I like that.
• Big Men on Content – Run by Lee Dallas and Marko Sillanpaa, this blog covers Documentum and the broader ECM industry as well. I’ve known Lee for over ten years, and I’ve always enjoyed his opinions.
• Paul Warren – Paul is an EMC developer focused (I think) on Documentum Composer. His blog posts are typically technical and very useful.
• Brilliant Leap – This is the blog for Selective Search, a recruiting agency run by Virginia Backaitis, who has been a member of the Documentum community for (I’m guessing here) more than 10 years. I’ve know her for 7 or 8 years at least. This blog is not technical, and it focuses mostly on EMC’s sales and marketing strategies and tactics.
• Observing Content Management – Lee Smith is not a frequent blogger, but when he does, he speaks about his personal experiences with Documentum and his thoughts on the industry.
• Never Talk When You Can Nod – Andrew Chapman is the GM of the SharePoint Technologies Group for EMC’s CMA group. We’ve been friends for several years, I think, and Andrew is really a super great guy. His blog mostly covers topics related to how SharePoint integrates with Documentum.
• Say Scheveningen – This is Don Robertson’s blog full of Documentum technical tips. Really good stuff in here.

1. Scoble had an idea of what Alfresco was, and he sort of inserted that into John Newton’s mouth. I wouldn’t describe Alfresco Share the way that John did, and it was clear to me that he was using Scoble’s language for Scoble’s benefit. I’m not sure that it turned out to be a very compelling or accurate description of Alfresco.
2. It’s clear that Scoble is focused more on Internet and Web 2.0 technologies than he is commercial/enterprise technologies like Alfresco. His questions and comments would have made more sense if Aflresco was a Web site rather than the enterprise software that it is.

At any rate, you get to see John Newton talking about Alfresco, which is always a good thing.

Watch the video here: Alfresco Enterprise Content Management on Scobelizer.

Some of the new capabilities in Alfresco 3.2 are

• Records Management
• Improved Capabilities in Share (Alfresco’s collaboration application), including improved use in Extranet environments
• Email Archiving via the IMAP email protocol
• iPhone Interface for Share
• Improved Capabilities in Deploying Large WCM Solutions

John’s post is chock-full of information about Alfresco 3.2, and I highly recommend taking a look at it.

Founded by John Newton, one of the co-founders of Documentum, Alfresco provides an ECM platform at a price nearly 95% less than a traditional ECM solution. In fact, their standard license is about half of the cost of a traditional ECM vendor’s yearly maintenance fees. Think about that for a minute – for about half of what a company is spending on maintenance fees for their existing ECM solution, they can move to a fully supported Alfresco solution. That’s a very compelling savings.

Why Alfresco?

Alfresco has a Great ROI

With the current economic climate, our clients are looking for ways to reduce the TCO of their ECM infrastructure. Several of our clients are migrating all or part of their content from expensive traditional ECM platforms (Documentum, FileNet, OpenText) to other platforms that they feel have lower TCO. I’ve talked to at least three pharmaceutical companies, for example, that are moving part of their content out of Documentum because they feel that it’s overkill for their needs. The content in question isn’t regulated and isn’t used very frequently, and they just can’t justify the cost of Documentum licenses for it. One of my clients told me, “It’s like keeping your Timex watch in a bank vault. That makes sense for a Rolex, but it’s overkill for a Timex.” These companies are continuing to use Documentum for their most critical content, but they are using other platforms for less critical content. OSHA, NYPD, Morgan Stanley, and Cisco are just some of the companies that have transitioned from one of the “Big Three” ECM vendors to Alfresco.

Alfresco Broadens Our Market

Alfresco is a great choice for smaller organizations or departments within larger companies that can’t justify the expense of a traditional ECM solution, because they don’t need all the horsepower and don’t operate at true enterprise scale. Alfresco has been benchmarked at over 100 Million documents, which is orders of magnitude more than most ECM systems are currently managing. By partnering with Alfresco, Blue Fish can take our brand of ECM services to more companies than we have been able to in the past.

Familiar Technologies

Alfresco is built using technologies with which we are very familiar. It’s a Java-based solution, and many of the technologies that are used as the building blocks of Alfresco (Hibernate, Spring, Freemarker, etc.) are technologies Blue Fish has been using in our solutions for years. Because the Alfresco technology stack is so familiar to our developers, it’s easy to ramp up and adapt our existing products and solutions to Alfresco.

What about Documentum?

We still love Documentum, and we still believe that there is a ton of value in using Documentum as your Enterprise Content Management platform. Especially for those huge organizations with really complex content management processes, Documentum is one of the best games in town. The breath and depth of the Documentum platform is staggering, there are hundreds of third-party add-ons and integrations, and they continue to innovate and acquire new technologies that keep them at the top of the pack. If your IT strategy is to invest in a single ECM technology across your entire organization, Documentum is a very good choice.

But more and more companies are moving away from a single ECM platform and are instead deploying a hybrid model with Documentum handling the more elaborate content management tasks and Alfresco taking care of the less complex needs. Also, when a smaller department needs content management, they often can’t justify the expense of a commercial ECM platform such as Documentum, so they often just make do with a file system. With Alfresco, a company can provide ECM tools to a larger percentage of their knowledge workers than they can with Documentum alone.

We see Documentum and Alfresco as two different types of tools. There are situations where Documentum is going to be the best fit and others where Alfresco is the best fit.

CenterStage was all the rage this year at EMC World, and it looks like it’s eventually going to be Documentum’s primary knowledge worker UI. CenterStage replaces eRoom, and it provides a collaboration environment for teams. CenterStage works like this: You define a “space”, which is like a room in eRoom – it’s a collection of documents, discussion threads, wiki pages, blogs, etc. all related to a single project or a single team. You can invite users to become members of this space, and then they can organize all their information about that project together in one place. The concept of spaces or rooms is not new – SharePoint, Alfresco, and several others use this same paradigm.

CenterStage is pretty slick looking, and I think it will make a big splash in the Documentum community. The search function is a million times better than WebTop’s, and with the addition of Blogs, Wikis, and Discussions it’s much more suited to the way teams share information than WebTop. WebTop and other document management clients are constrained to documents only, and most teams that use WebTop end up sending a ton of emails to each other to share all the information that doesn’t fit nicely into a document. With CenterStage, a lot of the information that goes into emails could easily transition to CenterStage, being stored in Discussion Groups, on Wiki Pages, as Blog Entries, or (in an upcoming release) as Data Tables (kind of like an online spreadsheet). Not only do your end users benefit from improved collaboration and information sharing, but the organization gets all the traditional compliance benefits that Documentum offers, such as centrally managed retention policies, archiving, eDiscovery, etc.

One of the nice features of CenterStage is the ability to scope your searches. When you issue a search, you can tell CenterStage where to search:

• Search within your current space
• Search within all you favorite spaces
• Search all the spaces you have access to
• Search the entire Documentum repostiory (assuming you are using Documentum for standard document management as well)
• Use EMC’s Federated Search Services to search other repositories, such as FileNet, Wikipedia, etc.

Earlier this week, I mistakenly reported that CenterStage was developed using the FLEX interface, but I was wrong. It’s actually a Rich Internet Application built using a JavaScript framework called Ext JS. There are a couple of components that utilize FLEX, but it’s not a FLEX application. Here’s how CenterStage works. When you load CenterStage for the first time, your browser loads up a ton of fancy JavaScript (the Ext framework). Then, whenever you click on a folder or a document, the browser uses JavaScript to make a call to a DFS Web Service. The result is returned as XML, and Ext then renders that XML as HTML. This is notably different from how WDK applications work. In WDK, a JSP page or Servlet makes a call to Documentum, renders the result as HTML, and then passes that HTML down to the browser. In Ext, there’s no JSP or Servlet acting as a middleman – the browser makes the call directly to the web service and then formats the result and writes it into the DOM.

CenterStage will require a new skill set to customize, and there will be fewer customization points than are available in WebTop. The next release of CenterStage has limited extension points, but these will open up over time. Right now, the main customizations are the ability to create new “widgets”, little UI controls that can be included on Wiki Pages within CenterStage. These widgets are developed in JavaScript or Flex and make WebService calls to back-end services.

Today, it seemed like every session that I attended was talking about xDB, EMC’s high-performance XML database. xDB is the product formerly known as X-Hive/DB, acquired by EMC in 2007. For the past couple of years, EMC has been selling xDB as a stand-alone product, but at the same time, they were busy integrating it with several of their existing products. But before we talk about that, I should probably bring you up to speed on what an XML database does.

An XML database is similar to a relational database except that instead of storing structured data in rows and columns, it stores semi-structured data in the form of XML files. xDB allows the use of XQuery to query the database and return chunks of XML, from an entire document (or several documents) to a single element. xDB treats the elements in XML files like the columns of a database.

Documentum has supported XML documents for over 10 years, but xDB brings that support to a whole new level. Take the following scenario, for example. Imagine you have a large XML document, like a product catalog from one of your suppliers. You get the catalog in XML format from your supplier, and you use it to generate pages on your web site that list the products you sell.

This scenario poses a challenge. You need to somehow get the data in this XML file into a form that can be displayed on your web site. Your web site is dynamic, with product catalog that can be sorted and searched, so you need to be able to query for individual products. What you need is a list of products, but all you have right now is a giant XML file with a <product> tag delineating each product.

In the past, to deal with this scenario, you would create an XML Application – a set of rules for processing the XML when it is imported into the repository. Your XML application would do two main things: first, it would “chunk” the XML file into smaller XML files, one for each product. These would all be linked into a virtual document that represented the main XML file. Your XML application would also parse out certain elements of the product (description, price, image URL, etc.) and write the values into metadata attributes on the document object. Now you have a list of product objects in the repository with attribute values that can be searched and displayed in a list. You would then write code on your web site to query the docbase and display your product catalog.

This approach has been used for years, and I used to build applications for the airline industry that used these techniques to manage huge Aircraft Maintenance Manuals. But there are some drawbacks.

First, in order for this to work, you must have a custom object type that contains attributes for all the elements of the XML file that you want to be able to search. That means you have to know which attributes will be displayed on the web site before you start managing your content. That’s often not the case. Second, you have to have a different object type for each type of product, because you’ll receive different XML files from different suppliers – the schemas will be different and the relevant attributes will be different (for one type of product, color may be important while for another product, size is important). Third, if the supplier changes their XML format or adds new properties, you have to update your object model and do it all over again.

With xDB, you don’t have to do any of this planning or object modeling. xDB can handle any arbitrary XML format, so you just import your document when you receive it from the supplier. And it doesn’t need to be chunked into smaller files, because xDB can return XML fragments when you query it. In effect, you can ask xDB to return you a list of all the products of type “Shoe” where size is “11″ and color is “Black”, and xDB will return a collection of XML fragments, one for each product element that matches that criteria. No chunking or attribute population is required, and there’s no database design involved, either, because xDB does not need any up-front knowledge of the XML schemas you plan to use. Just import your documents and xDB does the rest.

It turns out that xDB is very fast, very scalable, and has very high performance. It’s fast to import documents into it, and it’s fast to query it. I saw a demo today where xDB processed thousands of documents a second on some crappy laptop hardware. Some say that xDB is faster and better than a relational database, not only for hierarchical XML data (which it certainly is), but also for normal structured data. I’m beginning to think that those people may be right.

EMC has decided to make xDB a huge part of its technology strategy, and xDB is now being integrated and embedded in several applications.

• As Part of the Content Server – In one of the first major architecture changes to the Content Server since the introduction of the Method Server, Documentum will now contain an embedded instance of xDB where XML documents will be stored. You’ll be able to combine DQL and XQuery to get access from within Documentum to the features I described above. This is a big deal.
• As Part of Enterprise Search Services (ESS) – Documentum is replacing the FAST full-text engine with ESS, and ESS is powered by xDB. You heard me right. For the past several months, they have been adding full-text capabilities to xDB (leveraging Lucene, an open source search engine), and in an upcoming release of Documentum, the embedded full-text engine will be xDB.
• As Part of Dynamic Delivery Services (DDS) – DDS is a new product built on top of Interactive Delivery Services (IDS). It allows you to publish your web content into xDB and then query it from your web site. Intel is using this to run their entire Intel.com web site, and I can see how something like this would add lots of power to our web content management solutions while actually simplifying the solution.

Tomorrow is the last day of the conference, and I’m only planning on attending two sessions before I head off to the airport:

• THURSDAY 8:30 – Customer Experience – Web 2.0 and Personalized Customer Communications Product Overview and Strategies
• THURSDAY 10:00 – CMIS – Changing the World One Application at a Time

Today was WCM/DAM day for me here at EMC World. I attended two WCM-related sessions and spent some time down in the Exhibition Hall with the DAM team. Overall, what I learned is good news for marketing teams.

EMC is targeting marketing teams by investing in two of its solution suites: the Web Content Management suite of products and the Digital Asset Management suite of products. The biggest investment, and the most impressive results, seem to have come on the DAM side with big improvements to Media Workspace, EMC’s next generation DAM client.

Media Workspace was released last year as a simple client for image browsing and annotation. Developed using the FLEX platform, Media Workspace provided a glimpse into EMC’s future direction in rich web-based clients with a Web 2.0 feel. We evaluated Media Workspace earlier this year, and frankly, it left us scratching our heads. The product didn’t seem complete – it had a very limited set of features, and we couldn’t think of a single real-world scenario where Media Workspace was a better choice than Digital Asset Manager.

Well, I’m happy to report that all that has changed. In my opinion, the previous version of Media Workspace was released a bit prematurely, but the new version that I saw today looks a thousand times better.

The new version of Media Workspace is a fairly full-featured DAM client. In addition to supporting images (as the previous version did), this version also supports video. It has support for workflows, collections, thumbnails, transformations, ratings, annotations, and all the expected library services (checkin/checkout). I can see how a marketing team could use the product to handle most of their digital asset management use cases, and I think the product really shines if you are managing video. There’s even a built-in video viewer so that you can preview a video before downloading the high-res rendition. The overall user interface is improved as well – this version seems a little more polished than the previous version, and I was told that this is one of the advantages of using FLEX – it’s easier to update the user interface than it has been with WDK.

So what will become of Digital Asset Manager? It is still required for all the DAM-related administrative use cases, such as setting up transformation rules, and there are probably some power-user use cases where DAM is better than Media Workspace. But it appears that EMC’s vision is that most marketing users will spend their time in Media Workspace.

The Web Content Management products have seen similar improvements. EMC is releasing a new version of the Contributor UI, a simple client aimed at content authors and reviewers. Like Media Workspace, the new Contributor UI has been developed using the FLEX framework and has a completely new user experience and look and feel. This is a brand new application. Pagebuilder has also seen improvements, and there’s a new FLEX-based Web Publisher Editor. The Web Publisher Editor is the form that authors fill out to create a piece of content. It’s previously been a Java Applet, and many people find it difficult to use. The new editor has two interesting new features:

• A template form can now have multiple tabs, so that instead of the super long form that was required in previous versions, developers can group form fields together on tabs, giving a sort of wizard-like authoring experience. One big advantage of this is that if your page is laid out in “blocks”, you can group all the fields for one of those blocks together on their own tab.
• A template form can now be laid out horizontally as well as vertically. This means that you can have fields side by side, saving space over the previous version that forced fields to be stacked on top of each other in an endless list.

You’ll be able to call the new editor from within the new Contributor UI, the standard Web Publisher client, and Pagebuilder. All this will be available in version 6.5 SP3, which is scheduled for Q1 2010.

Other improvements have been made to the WCM Edition’s publishing capabilities. Site Caching Services (SCS) and Site Delivery Services (SDS) have been renamed and enhanced. The new product, called Interactive Delivery Services (IDS), includes an XML datastore in addition to the relational database that SCS has always provided. Based on the X-Hive product acquired in 2007, this XML datastore provides new options for delivering content to your web properties. Another new feature of IDS is the ability to take user-generated content from your web site (such as ratings or comments) and ingest them back into the Documentum repository. This is done by first writing them into IDS which will then write them into Documentum.

Perhaps the most anticipated improvement that comes with IDS is a completely new architecture for delivering content to a cluster of web servers. SDS has given administrators fits for years, and I can already hear people rejoicing that there is a replacement.

Tomorrow, I’m planning on attending these sessions:

• WEDNESDAY 8:00 – Beyond EMC Documentum 6.5 – A Product Roadmap
• WEDNESDAY 9:30 – EMC Documentum CenterStage – A Technical Review
• WEDNESDAY 11:00 – Best Practices for Moving WDK Customizations to EMC Documentum CenterStage and/or Next-Generation Documentum
• WEDNESDAY 2:45 – Smarten Up Your Content with EMC Documentum XML
• WEDNESDAY 4:15 – Best Practices for EMC Documentum xDB and XML Dynamic Delivery Services

Today was the first full day of EMC World 2009. As I mentioned earlier, EMC has indeed grouped the Documentum-related topics and partners together, and I think they have successfully pulled off the conference in a conference. I talked to Whitney Tidmarsh, Chief Marketing Officer of EMC’s Content Management and Archiving Division, and an old friend (I actually trained her on Documentum when she first joined the company 14 years ago). Whitney told me that the “buying affinity” of EMC’s CMA customers was different from that of its hardware customers, and that they finally realized it was best to separate the CMA sessions from the rest of the conference. The best part is that they are calling the conference within a conference “Momentum”, which is the name of the old Documentum User Group conference before they were acquired by EMC.

I have to admit that pulling out the Documentum sessions into a little Momentum conference has made a HUGE difference. Whitney told me that there’s about 800 CMA attendees this year, so that means that only about 11% of the attendees are Documentum focused. You can start to see how we feel a little underappreciated and overwhelmed in the midst of 6,200 hardware focused attendees.

Overall, the sessions I attended today were decent. I didn’t get to attend most of the ones I targeted due to space limitations. All four of the sessions I wanted to attend had crowds flowing out the door, and I only made it inside the room for two of them. For the other two, I attended my second choice sessions, which were fine.

The best session of the day was John McCormick’s presentation on Collaborative Technologies. John is another old friend of mine, and he’s the GM of Knowledge Worker products, which (if I understand correctly) covers WebTop, CenterStage, and My Documentum. John covered each of these in his presentation, and he’s so freaking smart and knowledgeable that it’s a real pleasure to see him present. The biggest discovery for me today was about My Documentum, which is a rebranding and upgrade of some existing Documentum applications.

My Documentum has three components:

• My Documentum Offline – This is the old Offline Client (available as part of WebTop), and it allows users to synchronize content between their hard drive and the repository. If content in the repository is added or updated, it’s automatically downloaded to your hard drive, and if you edit one of those files while on a plane, it will automatically check it in as a new version the next time you connect to the network.
• My Documentum for the Desktop – This is the old Documentum File Share Services, and it allows you to access Documentum as a shared drive from either a Windows or a Mac desktop.
• My Documentum for Outlook – This is a rewrite of the Documentum Client for Outlook. It now operates as a full-featured Documentum client – you can browse folders, import emails and attachments into Documentum, etc.

There’s a huge push around CenterStage, Documentum’s new collaboration application launched last year. This application breaks away from the WDK model and employs a FLEX-based development framework. EMC claims that their customers were asking for a more Web 2.0 experience, and the CMA group responded by developing a new set of applications using the FLEX framework from Adobe. I’m not convinced that this is the best decision, because most Web 2.0 applications are actually HTML and AJAX based, and I’m sort of tired of EMC changing the development environment every few years. I’m also worried about how anyone is ever going to effectively customize it. But that’s probably a point of view best elaborated on in a future post.

CenterStage is breaking some new ground for Documentum, and probably the biggest thing from my perspective is the addition of faceted search refinement. Blue Fish has been adding Faceted Search Refinement to our Documentum solutions for over 4 years, and I think it makes a huge improvement in usability. You can learn more about what Faceted Search Refinement is by watching a webinar that we recorded with Endeca last year.

The other big push is for SourceOne, EMC’s new Archiving and Records Management suite. I hope to learn more about that later in the week.

Tomorrow, I plan on attending the following sessions. We’ll see if I get into any of them.

• TUESDAY 8:00 – What’s New with Web Content Management and the Product Vision of Web Experience Management
• TUESDAY 11:00 – Configure, Don’t Code! and Other Ways the EMC Documentum Platform is Making it Easier to Build Solutions
• TUESDAY 2:45 – Federated, Virtual, Universal…Can You Have it All?
• TUESDAY 4:15 – Customizing EMC Documentum CenterStage 1.0

I arrived in Orlando last night for the kickoff of EMC World, the big EMC User’s Conference. EMC tells us that 7,000 people will be attending the conference this year, which is mostly focused on storage technology but also has a healthy amount of attention paid to enterprise content management. This year, I’m told that there will be a content management “conference in a conference” which amounts to grouping all the content managment sessions together in the same part of the convention center and grouping all the content management vendors together in the same part of the exhibitor’s hall. The portion of EMC’s customers that are Documentum-focused have been complaining for the past few years that they get lost in the massive hardware focus that is EMC’s primary business, so I’ll let you know if the conference in a conference works well this year.

When I checked in last night, I was immediately struck by the demographics of the attendees. They are probably 90%-95% male (more skewed than other IT conferences I’ve been to), aged 25-45, and without a lot of racial diversity. What is it about computer storage that attracts white dudes in their 30s?

Last night’s event was a concert by the Gin Blossoms. This is a band I know well but haven’t seen play live before. When I was in college, my band aspired to be just like the Gin Blossoms, and their album New Miserable Experience was in heavy rotation in my dorm room. I’m a sucker for a great pop-rock song with catchy melodies and harmony vocals, and that’s the Gin Blossom’s bread and butter. It was nice to see them play live, but you could tell that their hearts weren’t in it. Playing for a convention full of IT geeks that are primarily there for the free buffet is a long way from the club full of rabid fans they are used to. The lead singer tried to get the crowd engaged, but the energy was way off. In a normal gig, even if you are playing to only 100 people, it’s 100 people that stood in line to see you, and the energy of the room reflects that. Last night, there were 2,000 – 3,000 people at the event, but they were just there for something to do, and it showed in the vibe of the crowd.

I’m about to head out to the morning sessions. I’m planning to attend the following, so if you read this and want to find me, you’ll know where to look. My Monday is going to be all about understanding EMC’s current “pitch”. I’m pretty up to date on the technologies below, but I’m always interested in how EMC is packaging them and presenting them to the market.

• MONDAY 8:30 – Fundamentals: Content Management and Archiving Product Overview
• MONDAY 11:30 – What’s New with the EMC Documentum Platform
• MONDAY 3:00 – EMC Documentum CenterStage: Bringing Enterprise Content Management to Everyone
• MONDAY 4:30 – EMC Documentum 6.5 Architecture Overview

Learn more by reading Using Migration Workbench for More than Migrations.

Check out Web Publisher Technical Architecture: A Primer.

Introduction

With the release of Migration Workbench™, Blue Fish has changed the way we perform content migrations projects. Key new capabilities such as true incremental re-processing of changes, dynamic cutover, rules-based transformations, and live-to-live migrations have dramatically reduced the time and cost of even the most complex migrations.

But our clients aren’t stopping there. These new capabilities are being used to solve a variety of content management challenges across the enterprise.

Use Cases

Bulk Archiving

There is a common need for owners of Content Management Systems (CMS) to archive documents. This need may arise from the need to comply with records management policies, or as part of a larger consolidation project. Mergers and divestitures may also require the bulk archiving of content. Some types of CMS perform better when object counts are kept low, and so archiving older versions of documents can help to improve system performance.

Generally this task is performed manually by system administrators, and often using very blunt tools. A common practice is to use a tool that specifies the documents to be removed by query. The administrator then exports the documents to an archive system and deletes them from the source system.

In order to archive a particular piece of content and not another similar object, you need to be able to identify it with a specific query. This does not provide the level of granularity often needed to select all the documents, and only the documents, to be archived. If the archive system is on another platform, an entirely new approach must also be found to load and verify the archived data. The result is often a large amount of custom programming requiring a high level of expertise to operate and maintain.

Applying Migration Workbench to this task, even very complex archiving requirements can be quickly configured as migration rules. For example, you may wish to archive documents that meet the following characteristics:

• located in a specific folder,
• older than a certain date,
• inactive for greater than a year,
• having version numbers less than the current major version,
• only moving the original document to the archive, and
• deleting all pdf renditions from the source.

Using Migration Workbench, writing these requirements as migration rules is easy! First, list the folder as the batch criteria for documents that are candidates for archive. Next, write simple rules to identify 1) documents created since the cutoff date, 2) active documents, 3) documents inactive for less than a year, and 4) current documents. Indicate that these documents are not to be migrated. The remaining documents are those to be archived. The last step is to write a rule to only copy over the original document and not renditions. Migration Workbench takes care of the rest.

Because migration and cutover are two separate activities in Migration Workbench, the user can also preview the documents to be archived prior to making the move permanent. Migration Workbench provides reports that detail exactly which documents are in what state. And using the incremental migration approach, mistakes can be quickly and easily rectified by simply changing the rules and rerunning the tool.

Figure 1:
Using Migration Workbench to Perform System Archiving

Continuous Archiving

Many users want a way to move old or obsolete documents from their CMS to an archive on an ongoing basis. The criteria for moving documents into archive may be based on metadata values such as document age or document status, or it may be based on a user actively selecting to archive the document. This need is very similar to bulk archiving, but is a continual archiving process rather than a big bang event.

As with bulk archiving, Migration Workbench can be configured with migration rules to correctly identify, move, and cut over content to the archive. However, you can also take advantage of the incremental processing capabilities of Migration Workbench to create a continual archiving process. By setting Migration Workbench to run on regular intervals, it effectively monitors the CMS for changes to content or metadata that qualifies that content for archiving. System administrators can easily see and audit the results of this process using standard Migration Workbench reporting.

Publishing Content for Broad Consumption

Many organizations maintain separate content management systems for the authoring and publishing of content. The two systems are frequently built on two different repositories, or even on two different content management platforms. As an example, one recent Blue Fish client wanted a one-way publishing solution to push finished documents from their authoring system, hosted on Documentum, to a read-only system used for distributing training documents and SOPs, hosted on their portal platform, Sharepoint. The authoring system contained all the rich content creation, editing, and workflow of Documentum, but was only accessible to a small subset of content authors. The read-only system was hosted on Sharepoint, reducing the need for additional Documentum licensing, as well as providing security for the authoring system. Because it is cross-platform capable, Migration Workbench was used to automatically publish content from the authoring system to the broader use publishing system.

This case is similar to the case of continually archiving documents, but instead of permanently moving objects over and removing them from the authoring system, individual versions are moved to the publishing system based on status without deleting them from them the source. For example, suppose a content author’s workflow has three states: Draft, Published, and Retired. Initially a document has version 1.0 in Published state and 2.0 in Draft state. Rules in Migration Workbench could be written to select only documents in the Published state. On the initial run, Migration Workbench would pick up version 1.0 and push it from the authoring system to the publishing system. Later, when a user publishes version 2.0, which changes version 1.0 to the Retired state, Migration Workbench will pick-up that incremental change, delete version 1.0 from the publishing system, and push over version 2.0 in its place.

Figure 2:
Using Migration Workbench in a Publishing Role

Content Receipt and Tracking

Many organizations want to provide a way for their partners and vendors to contribute to business processes without giving them direct access to critical internal content management systems. For example, a firm may need a way to receive receipts processed by an external payments management firm, or import lab data collected by a contract laboratory. Migration Workbench can be used to receive this content, validate the metadata, and provide a confirmation that the content was properly processed.

The first step in providing a solution like this is to set up a content ‘drop zone’, such as an ftp site or a document upload web page. Partners can use this to drop off content files and provide the associated metadata for that content in the form of XML or CSV files. Using an upload web page allows for the creation of a database-backed web form, further simplifying the partner’s upload process.

Migration Workbench is then run on a periodic basis to monitor that drop zone for new or changed content. Validation rules are used to check that mandatory data fields are populated and all data is within expected ranges. Migration Workbench keeps a record of what content was imported correctly as well as what content failed the validation rules and was placed in a holding state. Standard reports can be used to easily identify content validation issues so the partner can take corrective action. The next time Migration Workbench executes, incremental processing will identify the changes and re-process the content. Once the content is successfully received and imported to the CMS, a dynamic cutover strategy can be configured in Migration Workbench to delete the content from the drop zone and send the partner a receipt confirmation.

Figure 3:
Using Migration Workbench to Import Documents from External Vendors

Receiving Documents from Scanning Systems

Many large organizations wrestle with processing extremely large volumes of documents created by scanning systems. Common examples of these solutions include scanning of insurance claims, invoices, and trade confirmations. The scanning system typically generates an image file of the document and places that file into a file share. The scanning system’s OCR software then parses metadata values from the body of the document and creates a metadata record in a database, XML, CSV, or similar data file.

Similar to the case of content receipt and tracking, Migration Workbench can be configured to recurse through the file share where the scanned images are located, identify new or changed content, and look up the metadata that corresponds to that piece of content. Transformation rules easily map that content and its metadata to the CMS’s schema, and dynamic cutover rules can be put in place to remove these from the file share once they are imported into the CMS.

Loading Test Systems with Production Data

Many well intended system releases and updates are immediately plagued by defects and issues for the simple reason that they were tested with data that was not representative of real world production data. Content management applications, in particular, are likely to encounter data inconsistencies and errors that simply cannot be anticipated with artificially fabricated test content and data.

With Migration Workbench, application developers have the ability to easily migrate select subsets of content and metadata from live production systems back to their test systems without impacting production system availability. As testing progresses, inclusion and exclusion rules can be written and adjusted to change the type and quantity of production documents being used in the test system. Migration Workbench’s incremental functionality also allows for the test data to easily be kept up to date with content in the production system, if needed.

Figure 4:
Using Migration Workbench to Populate Test Data

Assembling Document Bundles for Submittal

For companies in regulated industries such as pharmaceuticals, there can often be thousands of documents associated with a single submittal to a regulatory agency. Using Migration Workbench’s rules-based document selection capabilities, it is easy to configure individual export batches to collect all documents and metadata associated with a single regulatory submission. This could also include the selection of appropriate versions and renditions for assembly.

Another similar use case is the publishing of engineering drawings. Engineering organizations often create thousands of engineering drawings as work product, and face significant liability if the wrong versions were to be issued to their clients. There is a substantial need to be able to accurately bundle the correct documents in various configurations (site drawings, structural, electrical, plumbing, etc.) and ensure the correct versions are consistently used.

The rules-based approach used by Migration Workbench gives you the power and flexibility to precisely define the documents and data to be packaged. Additionally, incremental processing means that you can easily respond to follow-up requests for documents that have changed since the last submittal. Re-running the export will update the submittal with the just documents that have changed or are new.

Metadata Quality Analysis and Reporting

“How clean is my data?” or “What errors will I get if I run this migration batch?” are common questions faced by CMS administrators. Migration Workbench can be used to make this analysis easier.

Data validation rules can be put in place early in the migration process, even before transformation and mapping rules are defined. Migration Workbench can be configured to run in a ‘test mode’, where all of the rules are evaluated and reported on, but no content or metadata is actually imported into the target system. This gives the user a preview of what exceptions and issues to expect without incurring the time expense of actually importing content.

Conclusion

From the examples above, it is easy to see that Migration Workbench can be used to solve a variety of content management problems above and beyond just migrations. Capabilities such as true incremental re-processing of changes, dynamic cutover, rules-based transformations, and live-to-live migrations provide users with the flexibility to create innovative solutions for a broad range of CMS challenges.

Overview

When you think of a web content management solution, you might envision a toolset that simplifies web content creation and maintenance for non-technical content authors. EMC Documentum fulfills this web content management vision through a suite of products. A basic Documentum web publishing environment configuration will utilize three Documentum products: Web Publisher, to simplify web content creation and modification, Content Server, to enable content management, and Site Caching Services (SCS), to publish content from a content repository to a website.

If you’re new to Web Publisher, you might want to start by checking out Marc Perlman’s Demo: The Power of Web Publisher to get a feel for how content is created in Web Publisher. Otherwise, a good way to gain an understanding of Documentum web publishing is to follow a piece of content through the system from creation to publication on a website.

1. A content manager starts the process by accessing Web Publisher through a web browser.
2. Within the Web Publisher application, the content manager creates a piece of web content by filling out a form using Web Publisher’s template based authoring tools. The content manager sets any number of attributes, such as title and content author, on the new piece of content.
3. The new piece of content is created in the WIP (work in progress) lifecycle state and stored in the file store of the Documentum repository. Under the covers the content is stored as an XML file and the attributes that describe the content are stored in the relational database (RDBMS) within the repository. When the content manager selects the “webview” option in Web Publisher, the page template is transformed into a format that can be published to a website, such as JSP, ASP, or HTML.
4. Next, Site Caching Services (SCS) performs a publish operation, which copies the transformed web page to the target website machine. Assuming the environment is configured to publish attribute values then the SCS database will also be updated with the content object attributes.
5. The content and its attributes now reside on the WIP website where they can be reviewed for any typos or other errors.
6. After the content manager is satisfied with the content, he or she launches the content on an approval workflow. The content is automatically promoted to the Staging lifecycle state and is forwarded to an editor for review. The editor approves the content which is then promoted to the Active lifecycle state. Both the content and its corresponding attributes are then published to the production web server(s).

Figure 1:
Documentum WCM Environment Configuration

Documentum Web Publishing Environment Components

A typical Documentum web publishing environment requires three Documentum products: Web Publisher, Content Server, and Site Caching Services. In addition to these three products from Documentum, a custom website is required to complete a WCM environment. Below I’ve detailed some of the key elements within each of the required components.

Web Publisher

Web Publisher is a web application built on the EMC Documentum Web Development Kit (WDK) framework. The WDK toolkit is a Java based library of reusable components and controls that enable content management functions for Documentum through a web application. Web Publisher is an extension to the WDK framework that simplifies the creation and maintenance of web content using form based content editing. Web Publisher is installed in three parts: the web application, the DocApp, and the server files. The Web Publisher web application and WDK are installed in an application server such as Apache Tomcat, BEA WebLogic, or IBM WebSphere. For a complete list of supported application servers and versions, refer to the Web Publisher Release Notes for your version.

Web Publisher Contributor

Web Publisher Contributor is not a separate product or installation from Web Publisher. Contributor is another user interface option for the user to select at login time. Web Publisher Contributor is aimed at the casual content author. Contributor provides a simplified interface for the casual contributor by focusing on the tasks that they perform on a regular basis. For more information on the Contributor interface, read “Introduction to Web Publisher Contributor”.

Documentum Administrator

Documentum Administrator (DA) is a web application that provides access to administrative functions within Documentum. DA gives system administrators access to monitor, configure and maintain Documentum servers, repositories, and federations. As part of the Web Publisher installation, DA will also be updated with additional jobs for configuring and monitoring Web Publisher and site publishing functionality. DA access is required to complete the configuration of a Documentum web publishing environment.

Content Server

Documentum Content Server is the component responsible for managing the content. Content Server enables functions such as document versioning, workflow, and content security for content stored within a given repository. The Content Server handles all access to the content repository. Content Server is a required component in any Web Publisher solution.

Repository

A content repository (or docbase) stores the tables, indexes and content files used by Documentum Content Server. The repository is essentially a relational database to store content attribute values (or metadata) and a file store to house the content files.

In addition to the Web Publisher web application components and controls mentioned above, Web Publisher also requires a DocApp and a set of Web Publisher server files installed on the Content Server.

Web Publisher DocApp

The Web Publisher DocApp provides the additional workflows and lifecycle scripts for web content management, as well as a number of objects used to manage permissions, groups, and roles. The DocApp must be installed into any repository that will utilize the Web Publisher application. The DocApp is installed using Application Installer or DAR Installer, depending on the Web Publisher version.

Web Publisher Server Files

The Web Publisher server files are made up of two jars files that contain Java methods for changing permissions and executing the additional jobs and methods required for web content management. These files are installed on the Content Server to run in the separate Java Method Server application installed with Documentum Content Server.

Java Method Server

The Java Method Server is an application included by default with the Content Server installation. The Method Server is installed into an application server; either Apache Tomcat for DCTM version 5, BEA WebLogic for DCTM version 6, or JBOSS for DCTM version 6.5. The method server is a web application that allows custom code to execute on the server, independent of the client. The Method Server is often used to execute long-running processes or to execute processes that require super user privileges.

Site Caching Services

Site Caching Services is used to publish content from a Documentum repository to a website. SCS can also be configured to publish document attribute values (or metadata) to an external database for consumption by a website application. The published attribute values can be useful when creating dynamic web pages. For instance, the SCS (or webcache) database might be queried for all news items published with a press date in the last 30 days to create a dynamic news listing page. Additionally, SCS can be configured to write relation objects to an external database. The published relation object information can be useful when building a web page composed of related page snippets.

SCS is installed in two parts on different machines; the “source” is installed on the content server and the “target” on the web server host machine. SCS uses the source and target for communication between the content server and the web server machine(s). SCS can manage multiple websites for a single repository. For instance, a typical Web Publisher environment will support 3 distinct websites: WIP (Work In Progress), Staging, and Active. SCS can be configured to publish all content in the WIP lifecycle state to the WIP website, content in the Staging state to the Staging site, and so forth.

Published Website

Web Publisher is NOT a website presentation engine. In other words, you must create a custom web application framework to support dynamic web pages on your website. Unless the website being maintained and published from Web Publisher is a simple, static HTML site then a presentation engine is required to render the website. Using one of these presentation engines or web application frameworks requires custom development to handle things like database access, website security, and dynamic page rendering. To leverage all of the best features of Web Publisher, a presentation engine should also support querying the published SCS database for content attribute values and related objects. Here at Blue Fish, we’ve developed and refined a website component library to enable website security, display navigation enabled by Navigation Manager, and render many of the most common dynamic pages.

Optional Products and Add-Ons

There are a number of optional add-ons offered by EMC Documentum for simplifying content authoring, creating and deploying portlets, and managing replication of content to multiple web servers, among other things.

Page Builder

Web Publisher Page Builder is a browser based editor that enables web page development while viewing the web page as it will appear on the actual website. Page Builder also provides a tool for building additional content templates. The key benefit of Page Builder is that it allows the user to manage layout in addition to content. The content author will edit a page as it will appear on the end website, unlike the form based editing in the base Web Publisher product. Page Builder is simply another DocApp installed in a Web Publisher repository. Look out for an upcoming article about Page Builder in the near future.

Portlet Builder

Documentum Portlet Builder is an optional product add-on that is used to create and deploy portlets to a portal server. Portlet Builder provides additional templates and wizards for creating portlets, as well as a portal server target for identifying the portal servers to which portlets can be published. Like Page Builder, Portlet Builder is also a DAR (or DocApp) installed in a Web Publisher repository. A separate portal server is required to make use of the portlets published from Web Publisher.

Site Deployment Services

Site Deployment Services (SDS) is a Documentum product that facilitates the replication of content to multiple websites and databases. SDS is often used to keep multiple web servers in sync and refreshed with the latest content published from SCS. Site Caching Services can be configured to work with SDS, such that a publish operation from SCS to target server A will trigger the replication of content to targets B and C. Using the same example, if the replication of content from Target A to B is successful, but replication to Target C fails, then the entire transaction can be rolled back. This ensures a consistent website experience no matter which target site a visitor sees.

SDS is not dependent on the repository in any way. SDS is made up of a series of Tuners installed on each of the host machines. SDS uses Tuners to subscribe to various Channels through which content is replicated. Whenever a channel has changed (i.e. content has been added or removed), the Transmitter is responsible for distributing content and updates to each of the other Tuners subscribed to the updated Channel. The diagram below provides a simplified view of the communications between the various components. To learn more about installing and configuring SDS read “Site Deployment Services Rapid Install, Config and Troubleshooting”.

Figure 2:
Simplified illustration of SDS content replication

Conclusion

The EMC Documentum product suite is capable of delivering a robust and comprehensive web publishing environment, but understanding all of the components is key to configuring a strong solution to meet the needs of your business. I hope this article has provided a basic understanding of those components required by any Documentum web publishing environment and an idea of when to use some of the optional components and add-ons.

References

• Web Publisher Administration Guide
• Web Publisher User Guide
• Site Caching Services User Guide
• Site Deployemnt Services Administrator’s Guide

Read Tips For Making Your Website Section 508 Compliant.

Jason’s article is called Introduction to the WDK Combo Container, and it’s worth a read if you’re going to be doing any serious WDK development.

Introduction

Over the years Blue Fish has been asked to build Web Content Management solutions for a wide range of clients. With any WCM solution, our goal is to enable non-technical users to create and maintain a highly usable website. Section 508 of the Rehabilitation Act of 1973 specifically mandates that government and government funded websites be accessible for handicapped individuals, but many of our clients are simply interested making their site usable by peoples with vision disabilities.

Many of the good design decisions and development habits that we follow can help make your website accessible to the visually impaired – even if the law does not mandate your website to be Section 508 compliant. In general, these practices result in solutions that are more useable for persons with and without disabilities. So even though your website might not be required by federal law to meet the standards laid out in Section 508, you might just find that implementing some of the ideas that follow will result in a better website.

To be clear, I’m not describing a definitive “How To” with this article. Nor are we trying to explain a solution for every single facet of Section 508 of the Rehabilitation Act of 1973. There are plenty of resources available on the web that details the minutiae of Section 508. By leveraging some of these tips on your Web Publisher based WCM solution, I hope you’ll enhance the accessibility and usability of your website for all users regardless of health and disabilities.

A Little Section 508 Primer

Most companies and organizations that wish to make their website accessible for the visually impaired or disabled follow the technical standards defined by Section 508 of the Rehabilitation Act of 1973. Section 508 codifies those technical requirements and all Federal agencies (as well as private institutions that receive funding from or under contract with a government agency) must meet during the development, procurement, use and maintenance of information technology solutions. The standards defined in Section 508 are intended to give disabled users – whether they are government employees or the public – comparable access to the information available to all other users which is why so many websites are designed to meet the standards regardless of legal obligation.

With regards to websites and other web based solutions, Section 508 provides sixteen provisions that must be met for a solution to be accessible to disabled peoples. These sixteen provisions are laid out in Sub-part B, 1194.22 of Section 508 and are widely detailed on various web resources, so listing them in this article would be redundant at best. The ultimate goal of the sixteen provisions – and the following best practices – is to ensure accessibility for visually impaired persons using assistive technologies such as screen readers by providing contextual text labels and descriptions for graphics, navigation, and other web page elements.

Although originally added in 1986 as an amendment to the Rehabilitation Act of 1973, it was not until 1998 that the current implementation and enforcement of Section 508 came to fruition. It is a common misunderstanding to think Section 508 is part of the Americans with Disabilities Act of 1990; Section 504 of the Rehabilitation Act of 1973 was the basis for the ADA, but that is the only connection. Similarly, the World Wide Web Consortium’s Web Content Accessibility Guidelines (WCAG) are related to – but not part of – Section 508 compliance. The WCAG are voluntary and unenforceable whereas Section 508 provides a course of action against violators of the law. In the case of the WCAG, there is indeed significant overlap between the guidelines and the sixteen provisions outlined by Section 508. In some instances, adhering to just one set of rules could quite possibly result in complete compliance with the other set, but the only way to ensure 100% compliance with Section 508 of the Rehabilitation Act of 1973 is to explicitly check against all sixteen points.

Design For Usability, Achieve Accessibility

Section 508 of the Rehabilitation Act of 1973 is in place to guarantee that disabled users of a given website can access the same information as all other users. There’s no guarantee that disabled users (or anyone else, for that matter) will be able to actually use the website efficiently and effectively.

When I first started designing and building WCM solutions, I used to spend a lot of time at the end of projects checking that the resulting website was accessible. Over time, I realized that if the website was designed for usability, the accessibility often came for free. A few simple tips to remember when designing for usability will also help accessibility include:

• Don’t rely on color as a differentiating factor -

  At this point, many of us are familiar with color coded alerts and stoplights. It’s intuitive to many that red means “danger” or “stop” and green means “safe” or “go”. Unfortunately, for a disabled user of your website – for example, someone with color blindness – differentiating red, yellow, and green icons might not be possible.

  For example, in Figure 1, the availability of Apple’s iPhone is indicated on a per store basis solely by the color of the stop light.

  

  
  Figure 1:
  Using Color As The Sole Indicator

  While the availability indicated in this image may have been intuitive to you and me, it’s not even necessarily readable to an individual with color blindness. Rather than just relying on green vs. red, Figure 2 shows a more accessible and useable representation of the same information from another website.

  

  
  Figure 2:
  Using Color and Other Indicators

  Figure 2 is a much better usability and accessibility decision; There is no mistaking that there are two states – in stock vs. out of stock – because the icons are not only two different colors, they are two different shapes.

• Use contextual and descriptive text for links and buttons –

  How many times have you read through a long block of text on a website and, at the bottom, been presented with a link that merely says “Read More”? If you’ve been reading a lot about ACME corporation, the real question is “Read More What.” By changing that link text to say “Read More About ACME’s Solution”, you’ve already made the page significantly more user friendly.

  By using descriptive text for links and buttons, as described in the simple example above, your website moves closer to being Section 508 compliant. A disabled user who is visiting the website with a text reader isn’t going to have the visual clue that the “Read More” link is directly below a huge block of text about ACME’s Solution. What happens if there are multiple “Read More” links on the same page? Changing the links to say “Read More About ACME’s Solution” and “Read More About Working For ACME Corporation” is inherently more usable and accessible for all users.

  Bonus

  Not only does using contextual and descriptive text for links and buttons make your site more usable, it will likely help with your search engine ranking.

• Use text, not images, in page titles and navigational elements –

  Once upon a time, in order to provide compelling page titles and navigational cues to the user, these elements were often stored as images, with different images being presented on mouse roll-over, or when the node was active. With the advent of CSS, background images, and other more advanced styling techniques for web pages, using images for page titles and navigational elements is now all but obsolete, and when employed it presents usability and accessibility issues. Not only will a text reader not be able to read and interpret the images for disabled users, maintaining and updating these images becomes a potential nightmare for your website administrators.

  A robust content management system will allow business users to create and maintain the navigational hierarchy without IT department support. Figure 3 shows an example of a page title, bread crumbing, and menus that don’t use images.

  

  
  Figure 3:
  Text Based Page Elements

  Instead of using images for these page elements, simply rendering text over backgrounds looks just as great as using images and makes updates relatively painless for content authors. In this example, the business user is able to perform changes without having to call up the resident Photoshop expert to get a new image ready to go.

Use Templated Authoring To Your Advantage

When you use a content management system with templated authoring – such as EMC Documentum’s Web Publisher – to administer your WCM solution, you inherently gain the power to change one template and apply those changes to all the relevant instances of content. While designing our clients’ Web Publisher based solutions, a few common patterns repeatedly appear in the website markup that can be easily applied to all pages thanks to templated authoring. These little tweaks to your markup will not only result in a more usable site for all visitors, but will help your solution achieve Section 508 compliance.

• Include a link to skip the navigation –

  Imagine your favorite e-commerce site with 10 tabs across the top of the website and an additional 5 or more submenu options that appear every time you mouse over one of those tabs. Each one of those 10 tabs and 5 dropdown options is HTML that will be read by the text reader on every single page load; If you’re not visually impaired, it’s hard to fathom listening to a site reader read back 50 or more navigational elements before you even find out that the product page you’re viewing isn’t the right product.

  While it’s easy enough for most sighted users to focus immediately on the body of the page and skip the global navigation, the same is not true of disabled persons using a site reader. By including a link to the meat of the page at the very top of every page, a visually impaired user can quickly jump over the navigation.

  It’s just common courtesy and good practice to give every visitor to your website a quick path to the heart of every page.

• Include descriptions for all site assets –

  One of the most important provisions of Section 508 of the Rehabilitation Act of 1973 is that all images must have alternate text descriptions. While it may be obvious that regular content images need these descriptions, it’s easy to overlook all those site assets – like spacer images and logos – but they too must have descriptions. During construction of your presentation files, set the alt attribute for the img tag appropriately. For blank images used as spacers, an alt="" is perfectly acceptable for Section 508 compliance.

• Design for device independence –

  It’s easy to forget that not every user is visiting your website with a mouse. Quickly check all your markup files for device specific references (such as onMouseOver) and change them to device independent instances (such as onFocus).

• Design for CSS and JavaScript independence –

  Cascading Style Sheets (CSS) and JavaScript are a wonderful way to make your website look great. Unfortunately, looking great doesn’t help visually impaired visitors using a text reader or users on browsers with limited functionality (Wireless devices, console gaming systems, etc).

  It is important to design your presentation files to render content in a readable usable manner without CSS and JavaScript. By creating your solution so the markup and content renders in a readable format before the application of CSS and JavaScript, you can ensure accessibility for all users.

Use Web Publisher to Ensure Compliance For Images and Tables

In addition to the aforementioned design decisions than can be easily incorporated into your XSL presentation files, there are a few tips and tricks that can help ensure your Web Publisher based WCM solution is Section 508 compliant.

Ensuring Section 508 compliance for author selected images

As mentioned earlier, a prime directive of Section 508 is to ensure all images include descriptive alternative text for the visually impaired. While it’s straight forward enough to hard code alt attributes in the XSL presentation files for site assets, what about images added or selected by content authors via the Web Publisher content editor?

There are two ways that content authors can add images to an instance of content in Web Publisher:

1. Via the Graphic widget
2. Via the Rich Content Area widget

When a content author selects an image using the Graphic Widget (Figure 4), Web Publisher does not allow the author to enter any descriptive text to be use as the alt attribute for the published rendition.

Figure 4:
Image selection widget

A simple straight forward solution is to include another field in your template that allows a content author to enter a description (Figure 5).

Figure 5:
Alternate text description entry

Now your XSL presentation file can render the img tag with both the src attribute from the graphic widget and an alt attribute using the descriptive text. If this description field is optional, Section 508 compliance can still be ensured by generating an alt attribute text from the image name in the XSL processing, but this will only ensure compliance with the letter of the law, not the spirit. To make your site truly accessible, every image should have a descriptive alt attribute.

If a content author adds an image via the Rich Content Area widget, the author can add the alt text description by either right clicking on the image and selecting “Image Properties” or double-clicking on the image. At that point, the content author can enter the descriptive text in the “Alternative Text” field of the “Image Properties” window shown in Figure 6.

Figure 6:
Image Properties for a Rich Content Area image

Ensuring Section 508 compliance for tables

Another major provision of Section 508 is to ensure accessibility of data contained in tables. Similar to images, tables are required to have descriptive identifiers to be Section 508 compliant.

Section 508 has two requirements related to tables:

1. “Row and column headers shall be identified for data tables.”
2. “Markup shall be used to associate data cells and header cells for data tables that have two or more logical levels of row or column headers.”

Since visually impaired visitors are unable to ascertain the contextual clues (such as color highlighting) that might indicate a table has a header row, these two requirements ensure accessibility. In general, tables should be properly formatted and used for data representation, not for general page layout.

Like images added via the Rich Content Area widget, Web Publisher enables content authors to easily identify and mark tables as necessary to be Section 508 compliant. By highlighting the table cells, right clicking, and selecting “Cell Properties”, a content author can quickly add the appropriate tags to make a table Section 508 compliant. The “Cell Properties” window (Figure 7) allows an author to mark rows and/or columns as header cells (important for requirement #1) as well as set the Header ID and the Headers attribute (both important for requirement #2).

Figure 7:
Cell Properties for a Rich Content Area table

By checking the “Header cell” box, the previously selected cells are changed from td elements to th elements. Assistive devices will read this markup and inform a user that these cells are header cells and not data cells.

To link a data cell with a header cell, as required by #2, a content author can set the “Header ID” on the header cell(s) and then set the “headers” on the data cell.

As with rich content area images, there is no way to be certain content authors have properly entered these descriptors for a table. Blue Fish uses processing in the XSL presentation file as well as best practice education during training to help our clients ensure compliance.

Documentum Accessibility Report

Beginning with D6, Documentum has included a new option – the Accessibility Report — for the Rich Content Area widget in Web Publisher. This option, when enabled, allows content authors to generate a variety of simple reports for a rich content area that highlight potential compliance issues.

To run the Accessibility Report, a content author clicks on the accessibility report icon in the rich content area toolbar (Figure 8 ) and is presented with the report window (Figure 9).

Figure 8:
Rich Content Area toolbar

Figure 9:
D6 Accessibility Report

The accessibility report window allows content authors to check against both Section 508 as well as the WCAG standards. Additionally, by checking the “Errors”, “Warning”, and “Manual Checks” boxes, the content author is given different information to consider for the content’s compliance. This new tool combined with the best practices already outlined can help ensure Section 508 compliance for Web Publisher based WCM solutions.

Conclusion

Compliance with Section 508 of the Rehabilitation Act of 1973 is required for all government affiliated or funded websites to ensure accessibility for all visitors. With a little bit of foresight, adoption of some best practices, and good overall usability, compliance for your website will come almost naturally. Regardless of whether or not your website is required by law to be Section 508 compliant, enacting many of these recommendations will help create a solution that is more useable for persons with or without a disability.

Further Reading

• http://www.section508.gov/ – The Road to Accessibility includes a comprehensive resource that details Section 508 guidelines as well as a searchable database for compliant products.
• http://en.wikipedia.org/wiki/Section_508_Amendment_to_the_Rehabilitation_Act_of_1973 – A great summary of the law, its history, and its goals.
• http://www.access-board.gov/508.htm – Another comprehensive resource detailing the law complete with a tutorial on developing accessible software
• http://www.jimthatcher.com/sidebyside.htm – A useful side by side comparison of Section 508 and the Web Content Accessibility Guidelines

Overview

The WDK Combo Container component is heavily utilized throughout Webtop to provide a
consistent and flexible user interface for performing coordinated actions upon multiple repository objects.
It is used by many of the core document actions in Webtop, such as Import, Checkout, Checkin,
Cancel Checkout, Link Here, and Export, as well as other multiselect actions and use cases that are perhaps a bit
more obscure, and it can provide a robust starting point for building custom multiselect components.
The key features provided by the Combo Container are Multiselect Argument Decoding, Value Propagation,
and Auto-Commit.

A solid understanding of the Combo Container framework is often useful and sometimes vital when
attempting customization of that framework or any of the components which depend on it.
If one is planning to build a new custom component based directly on the Combo Container, it is even more
important to have a firm grasp of the framework’s interactions and extension points.

The Combo Container implementation includes several well-defined hooks and event handler methods which
subclasses can utilize to customize its behavior; however, some of these extension points are poorly
defined or even obfuscated, making it difficult to build reliable and robust custom components that
properly utilize the Combo Container.

The information in this article is meant to provide a high level understanding of how the various
Combo Container mechanisms function and interact, and to warn against some of the more insidious pitfalls.
It is not a detailed blueprint for customizing the framework, but rather a head start toward understanding
the overall landscape, so the reader can focus more time on implementing and less time ramping up.

Note:
This article was written based on Webtop 6.0 SP1.

WDK Context

The ID for the base Combo Container component is combocontainer. Its component
definition can be found at /wdk/config/combocontainer_component.xml. The Java
component class is located at com.documentum.web.formext.component.ComboContainer.

The Combo Container component utilizes two different JSP pages, both of which are configured in the
component XML:

• start
  

  /wdk/container/combocontainer.jsp
  

  Displays the current contained component and any headers, footers, etc.

• autocommit
  

  /wdk/container/comboautocommitex.jsp
  

  Displays the progress bar and drives client-side events during auto-commit.

(More on these pages later.)

ComboContainer Type Hierarchy

The ComboContainer component extends the WizardContainer component, and as such
inherits the behavior of WizardContainer, DialogContainer, Container,
Component, Form, and Control.

Figure 1:
ComboContainer Type Hierarchy

Control, Form, and Component provide the core functionality and
framework integration require by all WDK components.

Container provides the core WDK container capabilities, enabling the component to contain
another component.

DialogContainer provides the standard dialog UI, including the OK and Cancel buttons and their
event handlers, as well as the dialog label.

Figure 2:
DialogContainer Component Overview

WizardContainer provides support for multiple contained components and UI for pagination,
including the Next and Previous buttons and their event handlers, as well as the internal state necessary
to keep track of which contained component is currently being displayed.

Figure 3:
WizardContainer Component Overview

ComboContainer adds support for multiple instances of the same contained component,
and it provides a framework to coordinate and automate the behavior of those components, through
value propagation and auto-commit.

Figure 4:
ComboContainer Component Overview

Basic Mechanics

To begin, let me walk you through a short overview of the basic Combo Container mechanics.

The component lifecycle for the Combo Container is similar to that of the container classes it
extends, with the added dimension of multiple, indexed components. Instead of simply containing a single
component with multiple pages, as is the case for the WizardContainer, ComboContainer
contains multiple, indexed instances of the same component, through which the user can move forward
and backward using the Next and Previous buttons.

To accomplish this, the ComboContainer maintains internal state to keep track of the array of contained
components as well as the index of the “current” component. The basic mechanics of the pagination
and containership are encapsulated by the methods described below.

Public Methods

• void onInit(ArgumentList)
  

  Called once when the container is initialized. This method sets up component state, decodes the
  multiselect arguments, and initializes value propagation.

• boolean hasPrevPage() and boolean hasNextPage()
  

  Overrides implementations inherited from WizardContainer in order to support multiple
  indexed instances of the same component and report whether there are any more components before/after
  the current one.

• boolean onPrevPage() and boolean onNextPage()
  

  Overrides implementations inherited from WizardContainer to move the “current
  component” index backward or forward, as appropriate. The onNextPage also includes
  logic for stopping an auto-commit that has failed due to validation or other problems.

• boolean canCommitChanges()
  

  Hard-coded in the ComboContainer to return true, this method can be overridden
  to drive the visibility of the Finish button and to prevent the onCommitChanges method
  from being called as necessary.

• boolean onCommitChanges()
  

  This method encapsulates logic for saving changes across all components, and for launching the
  auto-commit process for any components beyond the current one. This includes launching the
  auto-commit confirmation prompt dialog, and stopping the auto-commit if any of the component
  commits fail due to validation or other problems.

Protected Methods

• int getComponentCount()
  

  Returns the number of contained components, which should match the number of items multiselected
  and passed into the container. Note that this count does NOT correspond to the two JSP pages utilized
  by the container (start and autocommit).

• int getCurrentComponent()
  

  Returns the index of the “current” component — the one that is currently being rendered by the
  container. The contained components are indexed in order, starting with 0.

• void setCurrentComponent(int)
  

  Sets the current component to be the one with the given index. The contained components are indexed
  in order, starting with 0.

• ArrayList getContainedComponents()
  

  Returns an ArrayList containing the contained components, in order.

Multiselect Argument Decoding

The primary use case for the Combo Container is to provide multiselect behavior for a specific contained
component. When a user makes multiple selections in the WDK UI and performs a multiselect action, the
selections are encoded and passed into the single Combo Container component. The
ComboContainer class’s onInit method then decodes these arguments and stores
them to be passed into the appropriate contained component instance upon initialization.

How It Works

The onInit method looks at the argument "componentArgs", and if it begins
with the string "___dmfStoredArgsKey", it calls the retrieveComponentArgs
method to extract the array of ArgumentList objects to use for all contained components.

This feature of the Combo Container framework is fairly straightforward, but if for some reason you
must override the way that multiselect arguments are encoded, you may need to dig into the implementation
of the onInit method.

Public Methods

• void onInit(ArgumentList)
  

  As mentioned above, this lifecycle method decodes the ArgumentList array corresponding
  to the multiple contained component instances, based on the encoded String[] returned
  by the retrieveComponentArgs method.

• String[] retrieveComponentArgs(String strUniqueComponentArgsKey)
  

  Extracts the arguments for the contained components from the Session. This static method is called
  by the onInit method to get the encoded argument data out of the user’s Session.
  This method is static and cannot be overridden. To provide custom decoding, you may need to
  override the onInit and/or setContainedComponentArgs methods. (See below.)

Protected Methods

• ArgumentList[] getContainedComponentArgs()
  

  Returns an array of ArgumentList objects corresponding to the arguments for each of the contained
  components. The size of the array corresponds to the number of items selected — i.e. the number of
  contained components.

• void setContainedComponentArgs(ArgumentList[])
  

  Sets the array of ArgumentList objects described above. This method is called by the ComboContainer
  code to set the argument array, and so can be a useful extension point for intercepting or modifying
  this behavior, or for using your own ArgumentList array instead of the one decoded by the ComboContainer
  onInit and retrieveComponentArgs methods.

Note:
It is important to recognize that the number of component instances created depends entirely on the size
of the ArgumentList array passed into setContainedComponentArgs.
ComboContainer uses this array to drive its instantiation of the contained component instances.

Value Propagation

One of the key features of the Combo Container is value propagation. Value propagation is the mechanism
by which the control values on the previous contained component are used to pre-populate the
control values for the current contained component when it is first initialized, thereby propagating
those values from one component to the next and saving the user the trouble of entering the same
values over and over.

Configuration

Value propagation can be enabled within the component XML by setting the value of the
propagatepreviouspagevalues element to true. Note that this configuration element
only enables or disables value propagation when the user is manually moving through the contained
components. If the user triggers an auto-commit using the Finish button, this configuration is ignored and
value propagation is always used to populate the remaining components.

How It Works

The full mechanics of value propagation are mostly obfuscated, but at its core the mechanism works
as follows. When ComboContainer is first initialized, it registers itself as a control
listener, so that any time a control is initialized by a contained component, it will trigger the
onControlInitialized method of ComboContainer.

When the first contained component is initialized, the controls are initialized but nothing special
happens. When the form is submitted, control state is updated based on the values in the request
using the updateStateFromRequest method. This method keeps track of any controls whose
values have changed (using Control.hasChanged), so they can be propagated to the next
component if necessary. These changed controls are known as the “propagation controls”.

When the second component is initialized, the onControlInitialized hook looks at each control
as it is initialized and decides whether it is a candidate for value propagation. In order for a control
to be such a candidate, the container must have a similar control in its “propagation controls” set described
above. In this case, “similar” means the controls share the same name, type, and index. If the control
is a candidate for value propagation, the value is propagated, assuming propagation is supported for
that type of control. (See below.)

It is important to note that only certain types of controls are supported by value propagation.
Specifically, the ComboContainer.onControlInitialized method has logic to recognize
and propagate the value of any control that has bean methods for getting and setting that value,
as long as those method signatures are of the form T getValue() and
void setValue(T), where T is the value type. The ComboContainer class has
additional logic for dealing specifically with DateTime controls, which don’t conform to
the signatures above, and DocbaseAttribute and DocbaseAttributeValue controls
(and their subclasses), which require special processing.

Fortunately, most controls extend from StringInputControl, which has the necessary bean
methods and therefore meets the criteria for value propagation. Unfortunately, some controls do not
have bean methods conforming to the method signatures described above, and therefore value propagation
is not supported for these controls. This can be problematic, most notably in the case of the
Checkbox and Radio controls. As of Webtop 6.0 SP1, any value propagation that
is required for these types of controls must be implemented as a customization.

Note:
In order for value propagation to work, each control must actually be rendered by the WDK presentation
engine. This requires each component’s JSP page to be triggered, which is fine if the user is manually
paging through the components; however, the situation is not so simple during an auto-commit.

Note:
The “propagation controls” set is stateful for all contained components, which means that if a control is
changed in the first component, then not changed in the second component, that control is still available
for propagation to the third component.

Public Methods

• void onControlInitialized(Form, Control)
  

  Called for each control contained by the container as the control is initialized
  (recursively into the contained components). This is the hook by which the value propagation
  occurs for each contained control as appropriate. This is also where the custom logic for
  handling DateTime, DocbaseAttribute, and DocbaseAttributeValue
  classes can be found. This method provides a useful extension point for adding value propagation
  support for new Control types. The biggest challenge is being certain to use the same logic as the
  ComboContainer class for determining whether to propagate the value or not.

• void updateStateFromRequest()
  

  Standard component lifecycle method, used to update component state based on data in the Request.
  This method calls out to the private method updatePropagationControls in order to
  acquire handles to any controls whose values have changed and therefore need to be tracked for
  purposes of value propagation. This is the “propagation controls” set described above.

• boolean isValueSetByPropagation(Control)
  

  Returns true if the an auto-commit is active and if the given control is one whose
  value can be set based on the previous component using value propagation; in other words, if a similar
  control can be found in the “propagation control” set stored as part of the ComboContainer
  state.

Protected Methods

• void setPropagatePreviousPageValues(boolean)
  

  Sets the boolean flag indicating whether or not the previous component’s values should be propagated
  to the current component when it is initialized. This method is called by the private method
  initPropagatePreviousPageValues (see below) based on the value configured in
  the component XML. This method can be called any time after the onInit method in order to
  override the configured behavior. In addition, this method is called by the auto-commit code
  in order to turn on value propagation regardless of configuration. (More on that later.)

• boolean isPropagatePreviousPageValues()
  

  Returns the value of the flag described above. This method is called from onControlInitialized
  in order to determine whether or not to execute the value propagation logic when a new control is
  initialized.

• void propagateAdditionalProperties(Control, Control)
  

  Propagates configuration elements and other properties from one control to the other. Presumably
  this is necessary in order for certain contained component instances to function properly. This method
  is called from onControlInitialized as well.

Private Methods

Although private methods cannot be overridden, understanding them can provide useful context.

• void updatePropagationControls()
  

  Called from updateStateFromRequest, this method scans all contained controls recursively
  looking for any that have changed, so that these can be added to the internal hash of controls whose
  values can be propagated. This is part of the internal value propagation state management, and cannot
  be overridden, but it is useful to provide context for what goes on “inside the black box”.

• void initDocbaseAttributeControls(Control)
  

  Called from onControlInitialized when a DocbaseAttribute or
  DocbaseAttributeValue control is initialized (assuming value propagation is
  enabled and there is a value to propagate). This internal method propagates additional state
  from one DocbaseAttribute or DocbaseAttributeValue control to another;
  is called at the end of onControlInitialized after the basic value propagation logic
  has occurred.

• void initPropagatePreviousPageValues()
  

  Sets the flag indicating whether value propagation is enabled, based on the
  propagatepreviouspagevalues element in the component configuration.
  This private method is called from onInit to set the flag initially, and it is also
  called the stopAutoCommit method to reset the value propagation flag when
  an auto-commit is stopped. (More on this below.)

Auto-Commit

When performing an action on multiple repository objects using the Combo Container, it is often
the case that the same attributes can be applied to all objects. Manually paging through the
components and entering the same metadata again and again can be time-consuming and error-prone.
Fortunately, the Combo Container has a solution to this problem, in the form of the auto-commit
mechanism.

The auto-commit mechanism is triggered by the Finish button, and it is responsible for driving automated
cycling through the Combo Container’s contained components. This cycling relies on a delicate coordination
of server- and client-side events which ensure that every contained component is properly initialized and
rendered (though not displayed), and validated, before ultimately committing all components.

Note:
It is critical that this automated request cycling occur. Not only is this the means by which the
user can track the progress of the auto-commit (by way of the progress bar), but it is also the only
way to guarantee that the value propagation logic functions properly, which is usually necessary for
the auto-commit to get past validation.

Figure 5:
Combo Container Auto-Commit Interactions

Configuration

None of the auto-commit behavior is configurable as part of the base combocontainer
component XML. As previously described, the only way to disable auto-commit is via the visibility
of the Finish button, which can be driven using the canCommitChanges method or by
setting the visibility of the Finish button directly.

How It Works

When a user clicks on the Finish button, it triggers the onOk event handler method
from DialogContainer.

This method first calls canCommitChanges, which is hard-coded in ComboContainer
to return true. Then it calls onCommitChanges, which calls
hasNextPage to determine whether there are any components beyond the current one.

If the hasNextPage method reports that there are additional components remaining after
the current one, then the current component is validated, and it is found to be valid, the auto-commit
mechanism is triggered.

First, the showConfirmPrompt method is called to ascertain whether the confirmation prompt
should be shown. If no confirmation is needed, the startAutoCommit method is called
immediately. Otherwise, the component nests to the prompt component to display the auto-commit confirmation
prompt, then triggers the onReturn method when it returns, which calls
startAutoCommit or not, depending on the user’s response.

Note:
The confirmation prompt can be disabled by the user through a user preference, in which case the
showConfirmationPrompt method would return false.

The startAutoCommit method overrides any configured value propagation flag, enabling value
propagation for the duration of the auto-commit process. It then switches the ComboContainer component’s
current page from the start page (used to display the contained component) to the
autocommit page, which is used to display the progress bar and to drive the client-side
events required to automate the request cycle. Finally, it sets the client event onPostAutoCommitEvent
to be triggered client-side when the page is rendered in the user’r broswer.

The behavior of onPostAutoCommitEvent client-side event depends on the value of the
JavaScript variable g_strAutoCommitEvent, which is set inside the onRenderEnd
method by appending some JavaScript code to the page. The value of this variable is used to tell
the client-side event which server-side event it must trigger to continue the auto-commit processing
as appropriate. The onRenderEnd method gets the value for this variable from the
private method getAutoCommitEvent.

The getAutoCommitEvent, although private, is critical. It is here that the auto-commit
mechanism’s core logic resides. As the automated cycling proceeds, this method gets called repeatedly
to automatically trigger the multiple onNext events necessary for paging through the
contained components, the onStopAutoCommit event if something goes wrong, and the
onOk event that it ultimately calls once it has cycled through all the contained
components.

Note:
Unfortunately, the getAutoCommitEvent is private and therefore cannot be overridden.
If it becomes necessary to customize the auto-commit interactions, the best approach is usually to
start with the standard component lifecycle methods, where you can sometimes “correct” or “undo”
changes made internally by the private Combo Container code. Use extreme caution when modifying
auto-commit interactions, as it is very easy to break the framework entirely.

As the framework cycles automatically through the contained components, the onNextPage
method is utilized by the onNext event handler in order to move the “current” component
forward. If it encounters any trouble due to validation, the “stoppingAutoCommit” flag is set, which
will ultimately lead the getAutoCommitEvent method to trigger a client-side event to
stop the auto-commit. But if the onNext calls succeed, eventually the framework runs out
of contained components, and at this point (when hasNextPage returns false,
the getAutoCommitEvent method triggers the onOk event handler, which, as
described above, calls onCommitChanges.

At this point, the onCommitChanges method recognizes that there are no more remaining
components to visit, and so it proceeds to call each contained component’s onCommitChanges
method. If anything goes wrong, the component is displayed along with the error; otherwise, the commit
has succeeded and the component returns to the context from which it was originally called!

Public Methods

• void onReturn(Form, Map)
  

  This is the event handler which processes the return from nest to the auto-commit confirmation prompt.

• boolean inAutoCommit()
  

  Reports whether an auto-commit is currently active, which is true if the current container
  page is the autocommit page and whether the internal “stoppingAutoCommit” flag
  has not been set.

• void onRenderEnd()
  

  If an auto-commit is active, this method appends the JavaScript necessary to trigger any
  client-side events.

• void pauseAutoCommit()
  

  This method sets a flag indicating that the auto-commit should be paused. If set, this flag will
  be picked up when onRenderEnd calls getAutoCommitEvent, which will
  interrupt (i.e. pause) the automated request cycling.

• void resumeAutoCommit()
  

  This method resets the auto-commit pause flag so that the auto-commit can resume.

• void onStopAutoCommit(Control, ArgumentList)
  

  This event handler method handles the onStopAutoCommit event, triggered client-side in
  response to a client-side event triggered by onRenderEnd and
  getAutoCommitEvent in response to the “stoppingAutoCommit” flag.
  This method first calls the stopAutoCommit method to update internal
  container state, then it calls the stoppingAutoCommit method to clear the
  “stoppingAutoCommit” flag.

Protected Methods

• boolean showConfirmPrompt()
  

  Checks the user preference to determine whether the auto-commit confirmation prompt should be displayed.
  Checked by onCommitChanges prior to launching an auto-commit.

• void inhibitConfirmPrompt()
  

  Sets the user preference to prevent the auto-commit confirmation prompt from being displayed. This is
  called by onReturn if the user has chosen to hide the confirmation prompt in the future.

• void startAutoCommit()
  

  This method launches the auto-commit process. It is called from onReturn
  if the confirmation prompt was shown, or directly from onCommitChanges if it wasn’t.

• void stopAutoCommit()
  

  Called by the onStopAutoCommit event handler described above, this method actually
  updates the internal auto-commit mechanism state to prevent the auto-commit from continuing, as
  is necessary in the case of validation or other errors. It sets the container page back to the
  start page, so that the contained component can be displayed instead of the
  progress bar, at which point auto-commit cycling ceases.

• void stoppingAutoCommit(boolean)
  

  Sets the stoppingAutoCommit flag.

Private Methods

Although private methods cannot be overridden, understanding them can provide useful context.

• String getAutoCommitEvent
  

  Determines which client-side event should be triggered, based on the current auto-commit flags
  and container state.

Tips and Pitfalls

Trace Logging

Some of the Combo Container behavior can be observed by turning on the trace logging for
com.documentum.web.formext.Trace.COMBOCONTAINER using either TraceProp.properties
or tracing.jsp. The trace logging is minimal, but it’s still worth enabling.

Finish Button Visibility

The best way to drive the visibility of the Finish button is through the use of the canCommitChanges
method, which is hard-coded to return true in ComboContainer but can be overridden. It is
also possible to acquire the Finish button control and set its visibility directly, but this must be done
after updateControls is called, since it will reset the visibility based on the canCommitChanges
method.

Custom Value Propagation

As described earlier, the value propagation mechanisms found in the Combo Container framework only
work for certain controls — those that have getValue/setValue bean methods, and DateTime.
In order to propagate state not stored in such controls, it may be necessary to override some
of the value propagation and component lifecycle methods described above.

The onNextPage method can provide a useful place to identify controls whose values were not
propagated during an auto-commit and handle them appropriately. However, when implementing your own value
propagation that works during manual pagination as well as auto-commit, it is usually necessary to
override one or more of the lifecycle methods, such as updateStateFromRequest,
onControlInitialized, or onInit, in order to track the new component types
and propagate values from the previous component to the current one.

Remember that value propagation can be tested by enabling it explicitly in the Combo Container component’s
configuration. This makes it possible to observe and verify the value propagation behavior by manually
stepping through the components, rather than having to use auto-commit to trigger value propagation, which
can much more difficult to trace.

Note:
Use caution when overriding value propagation methods and behavior! Use logging judiciously in your
custom code to help trace event handlers and state changes.

Multi-Page Components

Think twice before using a multi-page component as your contained component.

The ComboContainer.onNextPage method is responsible for moving from one page of a component
to the next. This method inherits some behavior from WizardContainer.onNextPage, which
actually calls into the onNextPage method of the contained component. Ditto for onPrevPage.
It therefore appears to be feasible to customize the contained component to contain multiple pages itself —
multiple tabs, for example — and to use the onNextPage and onPrevPage hooks
to drive value propagation and auto-commit behavior.

This approach is certainly possible, but there are many hazards, due to the way the Combo Container
functions. The auto-commit methods drive an active cycling of client-side and server-side activity. This request
cycling is coordinated by the auto-commit state and methods, and it ensures full initialization of every
contained component page. Proper cycling depends on the way the pagination logic interprets return values
from methods such as hasNextPage, onNextPage, canCommitChanges, and the like.

Successful integration often requires careful engineering of the onNextPage method to enable proper
cycling through component pages without the auto-commit mechanism stalling. In rare cases, careful
customization of the auto-commit methods or manipulation of internal auto-commit state may be
necessary as well.

In some cases, it may even be necessary to override the auto-commit state changes in order to “fool” the
auto-commit logic into continuing to the next page or component as needed. Caution, testing, logging,
and a decompiler can be your friends in these situations.

Note:
If multi-page logic is necessary in your contained component, it should extend WizardContainer
if at all possible. Extensive testing is recommended regardless, as the auto-commit mechanism may not
play nicely with certain customizations.

Summary

The WDK Combo Container framework provides a flexible, extensible mechanism for driving multiple
instances of the same component in response to a multiselect action. Although customization can be
dangerous, with the right context and careful planning, the framework can be adapted to a wide array
of situations.

Jamie Balling, one of the Technical Leads here at Blue Fish, developed an application that automatically detects paused workflows and automatically resumes them, freeing up the support team to focus on real fires. He just wrote a great article that describes the program, complete with sample code.

Learn more by reading Resilient Documentum Workflow Integration with Automated Recovery.

Overview

In my last article, I discussed how to monitor services and integration points for a complex Documentum based application. In this article, I turn my attention to strategies we employed to make our workflow integration more resilient to problems that might be encountered. This article will have a more technical focus.

In our experience, portions of the Documentum workflows subsystem – namely, the Java Method Server – can sometimes unexpectedly become unresponsive. The monitoring solution described in the previous article works well at catching this problem and routing it to the support team to give it a “kick.” Unfortunately, any documents which are submitted during the method server outage can become stuck in paused workflows which can prevent users from seeing their tasks. To recover from situations like this, we have implemented a “workflow recovery poller” that finds and restarts affected workflows.

Documentum Workflow Objects Primer

Before getting into the details of our solution, we need to provide some background by covering the key Documentum workflow objects and their relationships. You can refer to the Documentum Object Reference for additional information.

Process (dm_process)

A workflow is really a just a process and Documentum has a “process” object which models the steps. This object is really a template and you can think of each workflow as a specific instance following the steps of that template. Note that these processes can be created and managed with EMC’s Documentum products such as Workflow Manager and Business Process Manager.

Activity (dm_activity)

There is an “activity” object for each step of the process. The process object has a repeating attribute (r_act_name) which lists the names of each activity in the template. There are also related repeating attributes which describe other attributes of each activity.

Workflow (dm_workflow)

The “workflow” object captures the state of a document (or group of documents) going through the steps of a process template. This object has an overall status which is captured using the field “r_runtime_state”. The valid values for runtime state are:


DF_WF_STATE_UNKNOWN = -1

In addition to this overall state, each individual activity in the process template has a state. These are captured in the repeating attribute “r_act_state”. The valid values for activity state are:


DF_ACT_STATE_UNKNOWN = -1

Documentum updates the status of the workflow and activities as the workflow progresses. If an activity fails, for example, the r_act_state will show a value of “4″.

Package (dmi_package)

The workflow object doesn’t have any fields to relate it to the document(s) with which it is associated. This is the job of the “package” object. Each package references the id of the workflow and has a repeating attribute (called r_component_id) which contains the object ids of the content associated with that workflow.

Work Item (dmi_workitem)

Each activity for a workflow can have multiple performers and there are associated “work items” for each of them. The work item captures details (such as due date) about that activity for that performer. Each work item has a field (r_queue_item_id) which is used to associate it with a “queue item” in the user’s inbox.

Queue Item (dmi_queue_item)

“Queue items” are Documentum constructs which are used to preset items in the user’s inbox queue. Queue items are not workflow specific objects, but there will be one associated with each work item.

Workflow Design

We have brushed up on the key Documentum workflow objects, but we aren’t quite ready to jump into the details of our workflow recovery implementation. There isn’t a “one size fits all” implementation, because a successful strategy depends on the specifics of the workflow process design. In order to demonstrate the implementation, we will use a simplified version of our process template.

The end goal of our workflow is to ensure that a document gets the appropriate reviews so that it can be made active to end users. We have an automated workflow activity which inspects custom attributes on the document to determine the appropriate reviewer and creates review tasks. Note that this activity re-runs whenever a task is completed so that we can re-evaluate the status.

All of our documents go through workflow when they are uploaded. Documents initially have a version label of “pending”, and once the classifications are approved, we replace that label with “active” which indicates that the document can go live.

Workflow Recovery Implementation

Now that we have covered the details of our workflow implementation, we can discuss the strategy to recover from outages. As mentioned above, we have experienced times when the Java Method Server becomes unresponsive. When this is happening and new documents are imported (or outstanding review tasks are acted upon), the workflow activities fail and the documents become “stuck” in a pending state. Our solution was to introduce a process which runs regularly to find and fix workflows which aren’t in a valid state.

Given the business requirements of our system, we decided to keep the logic simple and kick off a new workflow when a document is in an invalid state. This works for us because our aforementioned automated review activity will inspect the status of previously processed documents and act appropriately. However, the same concepts could be used to restart workflows anywhere in the process flow.

Our process is a standalone java application that runs every 5 minutes. It utilizes a framework for running periodic tasks which is part or our solution, but could also have been implemented utilizing standard Documentum job scheduling. At a high level, the process finds any failed workflows by looking at state codes on the workflow object, deleting these failed workflows, and then starting a new one for any pending document (without a workflow). The detailed code, which utilizes Documentum Foundation Classes (DFC), follows:

1. Delete any halted workflows (note that when a running workflow fails, Documentum can put that workflow in a halted state).

   We identify these workflow objects using a DQL query:

   SELECT r_object_id FROM dm_workflow w WHERE w.r_runtime_state=3

   We then iterate over the returned objects and abort the workflows using DFC:

                   
   IDfWorkflow workflow = (IDfWorkflow)session.getObject(new DfId(workflowId));
   if (!workflow.isDeleted() 
                && workflow.getRuntimeState()!=IDfWorkflow.DF_WF_STATE_TERMINATED) {
       workflow.abort();
   }
   
               

2. Delete any workflow with a halted or failed activity. Note that this is different than item 1 because the overall workflow runtime state can show as “running” even though there is a failed activity.

   We identify these workflow objects using a DQL query:

   SELECT r_object_id FROM dm_workflow w WHERE any w.r_act_state in (3,4)

   We then iterate over the returned objects using the same logic described in step 1.

3. Now that we have deleted any halted workflows, we find all the “pending” documents which don’t have an associated workflow and start one. This works for us because all of our documents go through a workflow and it is a straightforward way to cover the various failure scenarios. If this wasn’t the case, we would have implemented logic to keep track of the documents for which workflows were deleted and act on those.

   We utilize DQL queries executed via DFC to locate the documents in question:

   
SELECT DISTINCT r_object_id FROM dm_document d
WHERE ANY r_version_label = 'PENDING' and r_object_id not in
(SELECT DISTINCT d.r_object_id FROM dmi_package p, dm_workflow w, dm_document d
WHERE ANY r_component_id = d.r_object_id AND p.r_workflow_id=w.r_object_id AND w.r_runtime_state <=1)


   Note:This query locates documents with our ‘PENDING’ version label and then performs a sub-select to only return those which don’t have a workflow. We determine which documents have active workflows by referencing the document’s r_object_id with the r_component_id repeating attribute on the dmi_package table, which we then join on r_workflow_id to the workflow table to limit results to active runtime states.

   We then loop through each of these documents and execute DFC methods to start a new workflow. Below are the key API calls executed for each document. Note that there are also calls to utility methods which are show after the main code block. We assume that your logic would provide variables, methods, and values which appear in blue.

                   
   // Instantiate a workflow. Note that you can also use an "IDfWorkflowBuilder" 
   // object to simplify this, but our approach gives you more flexibility (e.g.
   // to set the workflow name and supervisor name).
   IDfWorkflow wf = (IDfWorkflow) session.newObject("dm_workflow");
   
   // Set the workflow name
   wf.setObjectName("workflow for document");
   
   // Get the id for the associated process template
   // this example assumes you write some logic to get the process by name
   // (e.g. select r_object_id from dm_process where object_name='xyz')
   String processId = getProcessIdByName("some process template");
   
   // Set the process id on the workflow
   wf.setProcessId(new DfId(processId));
   
   // Set the supervisor
   wf.setSupervisorName(supervisorName);
   
   // Save the workflow to the repository
   wf.save();
   
   // Now start the workflow as this doesn't happen by simply saving it.
   // We don't set any "performers" because there is an automated activity in our
   // workflow which does this. If you need it, use wf.setPerformers()
   wf.execute();
   
   // Even though we have executed this workflow, the start activity will not 
   // become active until we attach a package because our start activity
   // has an "input port" which means it requires a package as input. So we
   // create a package with a document...
   
   IDfList objectIds = new DfList();
   objectIds.append(new DfId(documentId));
   
   // Get the starting activity
   IDfActitivty startActivity = getStartActivity(processId);
   
   // Now lookup the name for activity rather than using the name on the 
   // IDfActivity object since it might be different due to
   // a known idiosyncracy 
   String startActivityName = getActivityName(process,startActivity);
   
   // Add a package to the workflow to associate the document with it
   IDfId pkgId = wf.addPackage(startActivityName, 
                               getInputPort(startActivity),
                               getInputPackageName(startActivity), 
                               "dm_document",
                               null,      // not using a "note"
                               false,     // for our example
                               objectIds);
   
   // End of processing... we now have an active workflow!
   
               

   The following utility method gets the starting activity for a workflow by looking for an activity type of “1″ which indicates that it is a start activity.

                   
   public static IDfActivity getStartActivity(String processId) 
                                              throws DfException {
       IDfProcess process = (IDfProcess)session.getObject(new DfId(processId));
       int activityCount = process.getActivityCount();
       // Iterate over the activities looking for the start activity
       for (int i = 0; i < activityCount; i++) {
           if (process.getActivityType(i) == 1) {  // 1=start activity
               IDfId activityId = process.getActivityDefId(i);
               return (IDfActivity)process.getSession().getObject(activityId);
           }
       }
       return null;
   }
   
               

   This utility method gets the name of the passed activity. Note that the Documentum workflow applications will not update the name of existing activity objects if it is changed. Therefore, we always refer back to the process object to get the name.

                   
   public static String getActivityName(IDfProcess process, IDfActivity activity)
                                        throws DfException {
       if (activity == null) {
           throw new NullPointerException("activity must be non-null");
       }
           
       // Get the id of the activity
       IDfId activityId = activity.getObjectId();
           
       // Iterate over the activities looking for the specified id.
       // we do this rather than use the value on the activity because that
       // value may not be correct if the name has changed!
       int activityCount = process.getActivityCount();
       for (int i=0; i < activityCount; i++) {
           IDfId id = process.getActivityDefId(i);
           // If the id is found, return the matching name
           if (activityId.equals(id)) {
               return process.getActivityName(i);
           }
       }
          
       // The activity is not used within the specified process. This is an error.
       throw new IllegalArgumentException("dm_activity '" + activityId
                   + "' not member of dm_process '" + process.getObjectId() + "'");
   }
   
               

   This utility method gets the name of “input port” for the start activity. In Documentum workflows, ports are logical points of information exchange between activities and must be provided to add a package to a workflow.

                   
   public static String getInputPort(IDfActivity activity) throws DfException {
       for (int i = 0; i < activity.getPortCount(); i++) {
           if (activity.getPortType(i).equals("INPUT")) {
               return activity.getPortName(i);
           }
       }
       return null;
   }
   
               

   Finally, this utility method gets the name of package for the start activity. This must also be provided to add a package to a workflow.

                   
   public static String getInputPackageName(IDfActivity activity) throws DfException {
       // Iterate over the ports looking for the input port
       for (int i = 0; i < activity.getPortCount(); i++) {
           if (activity.getPortType(i).equals("INPUT")) {
               // Get the name of the package associated with the input port
               return activity.getPackageName(i);
           }
       }
       // If no package is located, return null
       return null;
   }
   
               

Conclusion

We got into the gory details of Documentum workflows and looked at some DQL and DFC code, but the fundamental idea is simple: find any workflows which are in a bad state and restart them. These details are all necessitated by the need to make our system resilient to outages. If you rely on the Java Method Server as we do, hopefully these details will assist you in making your system more resilient too.

While this wouldn’t be necessary in an ideal world, the reality is that systems with complex integration points can and will have problems. Depending on the criticality of the system, you may want to consider investing in defensive strategies to minimize the impact of these problems.

Blue Fish was recently selected as one of the fastest growing companies in one of the fastest growing cities in the country. Each year, the Austin Business Journal identifies the Austin Fast 50 – the 50 fastest growing companies in town. Blue Fish ranked 10th on their list for 2008.

Being selected to the list is fun, because it shows that even after 9 years, we’re still making our clients happy and they are giving us more and more projects to work on.

On a related note, I recently returned from a conference of entrepreneurs where I saw some old friends and made several new ones. It was great to hear them talking about all the positive things going on in their companies and how they are still doing well despite the economic downturn. It’s true that lots of companies are hunkering down for a cold economic winter, but others see the need to control costs as an opportunity.

Here at Blue Fish, many of our clients hire us to help them increase the productivity of their knowledge workers by improving the way they manage and work with content in the enterprise. Our solutions have helped our clients cut costs, improve sales, and reach more customers. This is exactly what companies need to do during tough times, so I don’t see any reason that we won’t continue to thrive regardless of the macro economic conditions out there.

Learn more about it by reading Synchronizing Web Content Across Web Publisher Environments.

Read more about how to Further Streamline BOF Module Deployment During Development.

Introduction

Several of the articles on our website discuss the technical, business-related, and planning challenges inherent in migrating enterprise content from one repository to another. This article focuses on the unique challenges of migrating web content from one Web Publisher environment to another. We’ll identify the most common web content migration scenarios and discuss options for addressing each of them. We’ll also explore the elements unique to Web Publisher which must be taken into consideration in a web content migration effort.

Based on client requests and feedback from our peers, we’ve chosen to address the most common web content migration scenarios:

1. To eliminate the need to re-create the latest content in a test or development environment, migrate web content from one environment to another.
2. To reduce the risk of testing in a production environment, enable the migration of templates from a development or testing environment into production.

What our clients really want is the ability to keep their development, testing, and production environments fresh with the latest and greatest content and templates. Ideally, our migration approach would allow for low-effort, incremental migrations.

Migration Scenarios

Most web content migrations fall into one of the following common scenarios:

Refresh Content

The most common web content migration request we hear from our clients is to refresh (or synchronize) the content in one environment with that of another. For instance, since the content in a production environment is frequently updated and maintained, users often request that we migrate that content from production to a development or test environment to facilitate testing with real-world content scenarios. The reverse is also true. Content authors might create new content in a development environment while testing a new web site feature. To avoid having to recreate that content they often request that the content created during development be migrated to production.

Refresh Templates

Similar to testing new web site features, developers often make tweaks and changes to templates to support web site changes. Ideally, developers would make the change in a test environment before putting the change into production. For most developers, this means a manual effort of exporting templates, presentation files and rules files from a testing environment and then importing them into the production environment.

Migrate Entire Repository

Another reason to migrate your content is to upgrade some or all of the software components or hardware platform in your Documentum environment. You’ll need a migration plan that allows you to move all of the content and supporting objects and relationships from the source repository to the target. This scenario is present in all types of content migrations and is not unique to web content, so we will not address it in this article. Look for a future article to address this scenario in greater detail.

Web Publisher Migration Challenges

There are a number of concepts in Web Publisher that make it different from other Documentum applications. These concepts add unique challenges to migrating web content, beyond what you might experience in a traditional Documentum migration. We can group these concepts into three categories: Environmental, Relationships, and Content.

Environmental

The following elements are a part of any Documentum repository. The good news is that all of these environmental artifacts, with the possible exception of Categories, can be migrated using DocApps or DAR files (using Composer). Optionally, you could also use a third party migration tool to migrate some or all of your environmental artifacts.

Environmental artifacts are generally the same artifacts found in a typical Documentum repository. These artifacts change infrequently when synchronizing two repositories, so we don’t necessarily need to concern ourselves with migrating these objects after our initial development or implementation effort. Below we’ve identified some of the key environmental artifacts and their role in a Web Publisher environment:

XML applications: One useful feature of Web Publisher is two-way synchronization of properties and content using XML applications. The XML Application enables automatic population of a piece of content’s property values when the content file is checked in or imported into Web Publisher. With two-way population, the opposite is also true. An author can set property values on a piece of content at the time of creation and the corresponding content values will be updated.

Lifecycles: Web Publisher uses a document lifecycle to define the various states that content will occupy during its life. In an out-of-the-box Web Publisher environment, content in its draft state is in the Work In Progress (WIP) lifecycle state. When a piece of content is ready for review and approval it is promoted to the Staging lifecycle state. After the content is reviewed, it is promoted to the Approved state. In addition to these standard lifecycle states, many people choose to define custom lifecycles with custom lifecycle states to support their business process.

Workflows: The workflow defines the promotion path, typically from one user or group to another, that the content will take as it passes through the various lifecycle states. A process workflow will automatically promote a web page to the next state in the lifecycle when certain approval tasks have been completed.

Permission Set Templates & User Defined ACLs: EMC Documentum provides a number of features to manage permissions within Web Publisher. All Documentum objects are assigned a single Access Control List (ACL). The ACL assigned to an object sets the access permissions and restrictions for the object. Permission set templates, which are a special case of ACLs, are applied to a document based on its lifecycle state. Alias sets can be used as placeholders for user names, group names, and ACL names. A combination of alias sets and permission set templates set the ACL for web content. Additionally, some Web Publisher implementations will use custom, user-defined ACLs to set access permissions and restrictions for certain objects as dictated by business processes and security policies.

Custom Folders: In a typical Documentum repository, most folders are of type dm_folder, the standard, out-of-the-box type of folder. But Web Publisher uses a custom type of folder called wcm_channel_fld. When you are migrating documents into a Web Publisher cabinet, you have to make sure that any folders that get automatically created are created as wcm_channel_fld rather than the default dm_folder.

Categories: One of the ways Web Publisher allows users to organize content is via Categories. All Web Publisher solutions will have at least one category instance, the Functional Taxonomy, which organizes content by page type according to the template with which the content was created. Typically, each page type in the system is represented by a category. Categories created to organize content templates by page type are created as functional taxonomy nodes. To associate content to its corresponding category, Web Publisher creates a wcm_category relationship. In essence, the category is another type of folder (a subtype of dm_folder). This type of category is migrated with the corresponding content template in a DocApp or DAR file.

Categories can also be used to define a custom hierarchical taxonomy that can be published to a website as XML. In this case, the root level category is typically a sibling of the functional taxonomy described above. The challenge with this type of category is that it can’t be migrated using a DocApp or DAR file (using Composer). For this reason, it’s important that our migration solution support the migration of this special folder sub-type.

Relationships

Templates are an integral part of the Web Publisher solution. Templates enable authors to create and edit content in a familiar form-like interface. Because templates often drive the content on a web site, it is not uncommon for them to change rather frequently to keep a web site looking fresh and interesting or to incorporate new page features or layouts.

Templates (or document types) are the basis for creating new content in Web Publisher, and are composed of the following set of files:

• A defining template upon which new files are based
• A rules file which defines the fields in the content creation form (optional)
• A presentation file which transforms the XML content into a web page (optional)
• A preview/thumbnail file which uses a graphic to identify the templates to content authors (optional)

Web Publisher uses relationships to connect templates to their supporting rules, presentation and preview files. The image below illustrates the relationships between a sample template and its supporting files:

Figure 1:
Sample relationships between a template and its supporting objects

To learn more about creating document types, refer to the Web Publisher Developers Guide or the article on this website titled “Web Publisher Template Basics”

Just as the templates have relationships to their supporting files, Web Publisher creates relationships between the content and each of its supporting objects. The diagram below illustrates some of the common relationships using a sample product page to show the relationships between a piece of web content and its supporting objects.

Figure 2:
Relationships for Sample Product Page

Below is a summary of some of the most important relationships that Web Publisher content might have:

_{* eWebEditPro is an optional HTML editing tool packaged for use in Web Publisher}

_{** Rules Editor is an optional tool that simplifies creation of rules file XML}

Content

We mentioned that the template, together with the rules file, enables the content author to enter the content in a standardized form. The form simplifies adding elements like images and links, and ensures consistency across all pages of a certain type. Web content created using a template and rules file is stored as XML in Web Publisher. To maintain the reference between the content and any images or included text files on the page, Web Publisher embeds the document id of the image or text file into the content XML file.

Figure 3:
Image IDs embedded in sample content XML

These embedded object IDs are used within the template to display which object has been selected by the content author when using the out-of-the-box graphic image (or text) selector provided with Web Publisher.

Migration Tools and Web Publisher

The best way to address the challenges of migrating web content from one Web Publisher environment to another is to use a migration tool that is up to the task. Although we may be a little biased, we think that Migration Workbench is the best tool to use for Documentum migrations, and it has several features that were specifically designed to synchronize WCM repositories.

As we stated earlier, one of the main motivators for moving web content from one environment to another is to keep development, testing, and production environments synchronized and fresh with the latest and greatest content and templates regardless of where it was created. This implies an ongoing, or iterative, process rather than the “one-time push” that is commonly associated with content migration initiatives. Therefore, you want to select a tool that supports an iterative approach – meaning you configure the migration process once and run it repeatedly at specified intervals. With each run, any new or updated documents will be migrated to the target system, and any deleted documents will be removed from the target system. Migration Workbench is the only tool we know of that does this.

But supporting an iterative approach is just the first item on the list. For a successful web content migration, the tool you use will also need the following capabilities to address the unique challenges of web content migrations.

Environmental

While DocApps and DARs can be used to migrate most of the environmental elements we discussed above, you also need to consider the effect these elements have on content.

For instance, it’s possible that some of the content in your source repository will be in the middle of a workflow. Depending on your business requirements, you may want to migrate only the content that has been completed and approved. If that’s the case, you’ll need a migration approach that will allow you exclude certain content from the migration based on some criteria. Migration Workbench uses migration rules to evaluate property values, such as the lifecycle state, to determine which content to migrate.

You may have a business case for migrating content that is in an intermediate lifecycle state – it’s neither in its initial lifecycle state nor in its final state. For instance, you may want to migrate all content – whether it’s in WIP, Staging, or Approved. Intermediate states, such as Staging, are often “non-attachable”. If content is in a non-attachable state, it cannot be imported into the system in that state. Instead, the content must be imported into the system in an attachable state, such as WIP, and then promoted to the intermediate (or non-attachable) state. The migration tool should support setting an object to an initial, attachable state, such as WIP, and then auto-promote it to a desired state, such as Staging. This is one of the advanced features of Migration Workbench.

Earlier in this article we discussed how Web Publisher uses a combination of alias sets and permission set templates to set the ACL for the content in a repository. The permission sets and ACLs are generally moved from one repository to another using custom DocApps or DAR files. Your migration tool must allow you to define the rules by which a permission set template was used to attach an ACL to a piece of content in the source system, and then re-apply the correct ACL after the content is imported into the target system. Migration Workbench can determine and attach the correct ACL depending of the lifecycle state or other properties of the document.

Your migration tool must also deal with the fact that Web Publisher uses a custom folder type called wcm_channel_fld. Most migration tools will automatically create folders when neccessary, if the folder does not already exist when trying to link a document into it. But Migration Workbench allows you to specify what type of folder should be created. This ensures that all the folders in your repository will be the right type (wcm_channel_fld) rather than the default.

Finally, since some categories cannot be moved in a DocApp or DAR file, your migration tool must be able to migrate your complete category folder structure prior to migrating the web content into those folders. Migration Workbench can migrate custom folder types such as categories, keeping all the custom attribute values intact.

Relationships

Because Web Publisher relys so heavily on relationships to tie together the different components of a web page, the migration tool must be able to migrate these relationships easily. Recall the diagram of object relationships earlier in this article; each arrow represents one relationship and all of these relationships must be reassembled in the target system for each piece of web content. This means the tool must maintain a source-to-target mapping for the migrated object IDs, migrate the relationship object and then update the parent and child fields with the new IDs that were generated by Web Publisher when the parent and child objects were imported. Following this sequence is essential.

Other relationships link one migrated object to a non-migrated object, such as workflows which are usually moved from one environment to another using a DocApp or Composer. In this case, the tool must be able to update the migrated relationship object with the Web Publisher-generated ID for the migrated object and then perform a lookup into the target repository to get the ID of the non-migrated object.

Migration Workbench supports both of these scenarios.

Content

To update embedded object IDs, your migration tool should maintain a source-to-target mapping for the object IDs so it can update the IDs embedded in the content XML file with the new ID assigned when the related image or content file was imported. Sequence is important; this step must occur after the related object is imported so the new ID is known. You may also opt to ignore the embedded object IDs. As long as you migrate both the content and the embedded content into the appropriate folders, you’ll find that the resulting web page will render appropriately. If the embedded IDs are not updated in the target content object, then content authors can expect to see errors similar to the one below when editing content. Reselecting the embedded content and saving will resolve the error.

        The previously selected Documentum object 09000405800dc63 is unavailable.  The object is unavailable either because it was not 
found, or you do not have the correct permissions to open the object.  
Select an object and click Save to overwrite the previous selection.  
Otherwise, make no selection and click Save to continue using the same object.
    

Summary

Our goal in writing this article was to highlight some of the unique challenges inherent in web content migrations between Web Publisher environments. Hopefully you have gained some insight that will help you anticipate these challenges and plan for them in your next web content migration initiative.

Experience has taught us that selecting the right migration tool is key to addressing the challenges unique to web content migrations. Ideally, a migration tool will enable everything from simple, iterative migrations of content to entire repository migrations. Here at Blue Fish, we’ve been working hard at solutions to overcome these challenges. Visit our product information page to learn more about how Migration Workbench can be a part of your web content migration solution. And for more information on the Blue Fish Agile Migration Methodology, including the concept of an iterative approach which we call incremental migrations, please see the article titled “Using Migration Workbench to Address Common Content Migration Issues”.

Additional Resources

For additional information, try the following documentation:

• Content Server Administration Guide
• Web Publisher Administration Guide
• Web Publisher Development Guide
• Web Publisher User Guide

Introduction

In a previous article,
Automate BOF Module Deployment During Development.
I detailed how to install Documentum Business Object
Framework (BOF) Modules using an undocumented Ant task
provided by Documentum’s Application Builder. With the
BOFPackaging Ant task, it is possible to install and update
modules in an automated manner, freeing us from having to
maintain these modules manually through docapps during
development.

While the automated BOFPackaging task represents a
significant improvement over manual deployments, it too can
be inefficient at times. Namely, the BOFPackaging task will
completely re-install all of a module’s Java archive (JAR)
files, including all third party dependencies (e.g., XML
parsers, Log4j, etc.) which are unlikely to have changed.
Often these static libraries can total several megabytes in
size, and the update process will waste several minutes -
even upwards to an hour over a poor network – updating them
unnecessarily. Considering that my fellow developers and I
will re-deploy a module numerous times during development,
the time wasted updating library files that haven’t been
modified quickly adds up. As such, we’re discouraged from
deploying many, small updates which are otherwise more
efficient and less error-prone, and instead will batch
enhancements into larger updates that are more difficult to
isolate, test, and troubleshoot.

To address this problem, I created a generic, commandline
application, FileLoader, which can add new files or update existing files in a
docbase. I use the FileLoader
tool within my Ant build scripts to update only the JAR
files that are likely to have changed while developing my
project’s modules. In this article, I’ll describe how my
FileLoader works and how Documentum makes these efficient
updates possible.

For your reference, links to all source code can be found at
the end of this article.

Review of BOF Object Schema and Folder Structure

Before we dive into the implementation of the FileLoader,
let’s review the object schema and folder structure of a BOF
module.

As of DFC 5.3, Documentum BOF code is stored in the
repository and dynamically loaded by DFC as needed by client
application code. Service Based Objects (SBOs) are hosted in
a single “global repository”, providing the same
functionality to multiple repositories, while Type Based
Objects (TBOs) are hosted and configured in the repository
that uses them. Additionally, Documentum introduced the
concept of a “simple module”, which is a unit of executable
code stored in the repository. Like TBOs, simple modules are
hosted in the repository that uses them.

To support this module framework, a few new object types
were introduced: “dmc_module”, “dmc_jar”, and “dmc_java_library”.
The “dmc_module” type is a subtype of “dm_folder” and represents
a BOF module. A “dmc_module” is comprised of the “dmc_jar” and
“dmc_java_library” objects that are contained within it.
(Technically, “dmc_java_library” objects are associated with
“dmc_module” objects through dm_relation objects, enabling the
Java libraries to be reused across modules. But it’s common
practice for each module to have its own “dmc_java_library”
object linked directly under it within the repository.) The
“dmc_jar” type is a subtype of dm_document and represents
individual JAR files. The “dmc_java_library” type is another
subtype of “dm_folder” and represents the set of “dmc_jar”
objects that are contained within it. A module will
typically include a “dmc_jar” object for the module’s
interfaces, a “dmc_jar” object for the implementation of the
module’s interfaces, and a single “dmc_java_library”
containing all of the module’s runtime dependencies.

Every repository has a System cabinet, which contains a top
level folder named Modules. By convention, modules are
installed under this folder according to their type. For
example, SBOs are installed under “/System/Modules/SBO”, and
TBOs are installed under “/System/Modules/TBO”. For simple
modules, you can reuse existing or create additional
subfolders of “/System/Modules” that match your modules’
types. For example, if the repository uses Java-based
evaluation of validation expressions, the associated modules
would appear under “/System/Modules/Validation”. The hierarchy
of folders under “/System/Modules/” is referred to as the
repository’s module registry, or simply its registry.

The following screenshot is an example of one repository’s
module registry as seen through Webtop.

Figure 1:
Subfolders of Modules representing BOF types.

Please see the “BOF Infrastructure” section of the
Documentum Foundation Classes Development Guide for a full
description of BOF Modules.

Hot Deployments

When any of a module’s JAR files are updated in the
repository, Documentum will automatically update the BOF
object caches on all client machines to include the updated
JAR files, and the DFC BOF
ClassLoader
will redeploy the classes loaded within a client
application’s JVM without requiring an application restart.

FThis hot deploy behavior unlocks even more value from the
FileLoader utility. Using the FileLoader, we can checkin new
versions of a module’s JAR files, and all applications
(e.g., Documentum Administrator, Webtop, Documentum Java
Method Server, and custom web applications) will immediately
pick up the new behavior without any interaction required.
This is a tremendous time saver for developers.

How does the Business Object Framework know when a module’s
JAR files have been updated? In DFC 5.3, they used an early,
internal version of the Aspects feature that is now publicly
available starting with the release of Documentum 6.0. By
inspecting the r_aspect_name repeating attribute on all
“dmc_jar” objects, we see that the Business Object Framework
attaches a change monitor Aspect
(“com.documentum.fc.bof.bootstrap.DfModuleItemChangeMonitor”)
to all JAR files loaded into the repository. When a “dmc_jar”
object is updated, the change monitor Aspect is invoked and
notifies a modification manager
(“com.documentum.fc.bof.util.IDfModificationManager”) of the
change. The modification manager encapsulates a distributed
system that ensures all running DFC client applications are
notified of the change. The BOF ClassLoader within the DFC
client application will then reloads all classes that
originated from the modified JAR file, and the client
application acquires the new behavior.

I should warn that despite this sophisticated framework,
client applications may occasionally fail to pick up the
changes made to a module. When this happens, shut down the
client application, delete the repository’s BOF object cache
on the client machine
(“[install drive]:Documentumcache[DFC version]bof[repository name]“),
and restart the client application. DFC will then
refresh the client machine’s BOF cache upon startup and pick
up the changes made to the module.

The FileLoader Commandline Tool

The FileLoader is a relatively simple commandline tool. You
can use it to load or update a set of documents in a
repository. For each file processed, the FileLoader will
look for a corresponding document in the repository using
the docbase location provided and the name of the file on
the client machine to make a call to
“IDfSession.getObjectByPath(docbaseFilePath)”. If the files
don’t already exist within the target repository, then new
documents will be created and linked into the correct
folder. If the files do exist, then they’ll be saved, minor
versioned (default), or major versioned as desired. For new
documents, it is possible to specify the documents’ object
type and content type as well.

The following table describes the options supported by the FileLoader:

The files to upload should be passed to the FileLoader as arguments
after the last commandline option. A sample usage might look like

            

C:java .. com.bluefishgroup.util.FileLoader --docbase=db1 --username=steve --password=password --docbaseFolder=/System/Modules/TBO/bf_custom_doc/ C:/dev/build/bf-tbo-int.jar C:/dev/build/bf-tbo-impl.jar


        

Finally, for existing files that should be minor- or major-versioned,
the FileLoader follows Documentum’s recommended practice of leveraging
the IDfCheckoutOperation and IDfCheckinOperation.

The code sample below represents FileLoader’s main file processing logic:

            

/**
 * Uses loginName to establish a session, creates folders as necessary
 * to build up the path in the docbase, and then uploads each file
 * in the collection.
 */
protected void execute() throws Exception {
    IDfSession session = getSession();

    // First create the folders in the path.
    createFolders();
    
    // Now upload each file, versioning existing content as appropriate
    for (int i=0; i < files.length; i++) {
        // Local file to load.
        File file = createFile(files[i]);
        // Object name.
        String objName = file.getName();
        // Full docbase object path.
        String docbaseFilePath = docbaseFolder + (docbaseFolder.endsWith("/") ? "" : "/") + objName;
        
        System.out.println();
        System.out.println("        File:  " + file.getAbsolutePath());
        System.out.println(" Object name:  " + objName);
        System.out.println("Docbase path:  " + docbaseFilePath);
        System.out.println();

        IDfSysObject doc = (IDfSysObject) session.getObjectByPath(docbaseFilePath);
        if (doc == null) {
            System.out.println("Creating new file in docbase.");
            doc = (IDfSysObject) session.newObject(objectType);
            doc.setObjectName(objName);
            doc.link(docbaseFolder);
            doc.setContentType(contentType);
            doc.setFile(file.getAbsolutePath());
            doc.save();
        } else {
            System.out.println("Updating file in docbase.");
            checkout(session, doc);
            checkin(session, doc, file);
        }
        
        System.out.println("Loaded.");
    }
}


        

To use the FileLoader in your Ant scripts, I suggest creating an
Ant target to encapsulate the java task that will invoke the FileLoader:

            

<target name="_run-file-loader" depends="compile" description="Updates files in the docbase.">
	<property name="fileloader.optional.args" value="" />

	<java classname="com.bluefishgroup.util.FileLoader" fork="true" failonerror="true">
		<classpath>
			<path refid="project.class.path" />
		</classpath>

		<arg line="--docbase=${docbase.name} --username=${docbase.username} --password=${docbase.password}" />
		<arg value="--docbaseFolder=${fileloader.location}" />
		<arg value="--version=${fileloader.version}" />
		<arg line="${fileloader.optional.args}" />
		<arg line="${fileloader.files}" />

		<!-- We need the lava.library.path set to pickup the DMCL DLLs -->
		<sysproperty key="java.library.path" value="${documentum.lib.dir}" />
	</java>

</target>


        

Then to update TBOs, SBOs, and simple modules, simply call the
“_run-file-loader” target with the correct parameters.

Here’s the Ant target included that I’ve written to update this article’s sample TBO:

            

<target name="update-tbo" depends="init" description="Updates document TBO jar files in the docbase (no 3rd party dependencies).">
	<antcall target="_run-file-loader" inheritRefs="true">
		<param name="fileloader.location" value="/System/Modules/TBO/${tbo.module.name}" />
		<param name="fileloader.version" value="minor" />
		<param name="fileloader.optional.args" value="--contentType=jar" />
		<param name="fileloader.files" value="${tbo.int.jar.file} ${tbo.impl.jar.file}" />
	</antcall>
</target>


        

Here’s the Ant target included that I’ve written to update this article’s sample SBO:

            

<target name="update-sbo" description="Updates SBO jar files in the docbase (no 3rd party dependencies).">
	<antcall target="_run-file-loader" inheritRefs="true">
		<param name="fileloader.location" value="/System/Modules/SBO/${sbo.module.name}" />
		<param name="fileloader.version" value="minor" />
		<param name="fileloader.optional.args" value="--contentType=jar" />
		<param name="fileloader.files" value="${sbo.int.jar.file} ${sbo.impl.jar.file}" />
	</antcall>
</target>


        

Finally, here’s the Ant target included that I’ve written to update this article’s sample simple module:

            

<target name="update-module" description="Updates module jar file in the docbase (no 3rd party dependencies).">
	<antcall target="_run-file-loader" inheritRefs="true">
		<param name="fileloader.location" value="/System/Modules/Lifecycle/${lifecycle.module.name}" />
		<param name="fileloader.version" value="minor" />
		<param name="fileloader.optional.args" value="--contentType=jar" />
		<param name="fileloader.files" value="${lifecycle.module.jar.file}" />
	</antcall>
</target>


        

Conclusion

By taking advantage of the cache update and hot deployment features
built into Documentum’s Business Object Framework, the FileLoader
utility can save you valuable minutes each time you update a module
by updating only the JAR files that are likely to have changed.
These minutes multiplied over the entire development team and length
of the release can quickly add up to hours or even days of overall
effort. That’s a significant savings that eliminates most of the
tedium from BOF development and can sometimes make the difference
between project success and failure.

Further Reading

• Article source code – bof_ant_task_samples.zip
• Automate BOF Module Deployment During Development, by Steve McMichael
• A Practical Guide to the Business Object Framework, by Jason Duke
• Documentum Foundation Classes (DFC) 5.3 Development Guide: Chapter 3 – Using the Business Object Framework (BOF)
• Apache Ant

Most companies that implement Documentum have big plans for it. They want to standardize on Documentum in an effort to reduce IT costs by eliminating the proliferation of one-off content management solutions.

This story is very familiar by now. Large organizations have standardized on databases, operating systems, ERP systems, programming tools, application servers, middleware – the list goes on and on. Standardizing on a common technology stack can keep support and maintenance costs down, and it’s a cornerstone strategy for CIOs everywhere.

But there are some drawbacks to this strategy. I can’t think of very many “one size fits all” solutions to today’s trickiest IT problems. Most hard problems have lots of nooks and crannies where unique needs are lurking. These nooks and crannies are the niche markets where specialty providers flourish, and these niche markets tend to be too small for the big guys to worry about. The big guys are great at solving the 80% of the problem that is common to all of their customers, but they struggle to address the 20% that makes each of their customers unique. The real challenge that companies face when standardizing on an enterprise solution is that each division or business unit might have its own 20%, and when you add them all up, you have a lot of needs that don’t get addressed with the out-of-the-box solution.

Here’s a case in point. Documentum can be deployed across your entire enterprise. It’s scalable enough to handle billions of documents. It can be distributed globally in a federated environment with replication across multiple data centers. It’s got an enormous feature set that’s been developed over more than 15 years. If you are going to standardize on a single ECM platform, you can’t do much better than Documentum.

But even Documentum doesn’t solve every problem out-of-the-box. Take airlines, for example. Documentum out-of-the-box can handle their normal business documents just fine. But airlines also have huge technical manuals called Aircraft Maintenance Manuals. These manuals can have more than 50,000 pages each, and a unique manual is required for each aircraft in the fleet. That adds up to a lot of pages! Because these manuals are so large and complex, they require an army of authors, editors, and reviewers. Add to that the regulated nature of the airline industry, and you’ll find that creating and managing these huge manuals is one of the biggest content management challenges out there. And Documentum doesn’t have an out of the box solution that can do it. The airlines have to customize the Documentum platform to meet their needs.

Now don’t get me wrong – I don’t blame EMC. The airline market is too small for EMC to focus exclusively on it. And this is my point. Most companies will have some unique requirements that are really important to them, but not very important to the big IT vendors out there. These companies face a real challenge when trying to deploy an enterprise content management solution. If they think too narrowly, the solution won’t meet all their needs. But if they think to broadly, they could end up with a customized version of Documentum in each department. And that defeats the purpose of a standard solution.

So what’s a CIO to do?

Josh McJilton has just published an article that could help. As a consultant and project manager here at Blue Fish, Josh has helped some of our clients address this very issue. He’s discovered that there are actually different types of “common IT solution”, and he’s created a model to help you determine which one is right for your business.

Read all about it in Uncommon Thinking About Common Solutions.

Introduction

When people talk about common solutions, you probably start thinking about a few specific technologies right away. A few examples of the ones most often thought of are Java, .NET, Oracle and Documentum. While it’s true these technologies can be used to implement common solutions, it’s not quite as easy as it sounds.

Here at Blue Fish, our experience with clients and our own independent analysis has led to the conclusion that there are at least three different types of common solutions. These types, or approaches to building a common solution, highlight differences in how common solutions are built, deployed and maintained. This article will explain the three different approaches we’ve identified and present a model we’ve developed to help you select the right approach for your common solution.

Understanding the Different Approaches to Common Solutions

The three approaches we’ve identified are specifically targeted at enterprises with departments, divisions or business units large enough to make their own IT decisions. This generally means they’re also large enough to create custom solutions to address their specific needs. The goal is to find a common solution across these departments that both meets their unique needs and minimizes development, deployment and support costs.

In this environment, identifying the right approach to a common solution is critical. Implement a common solution that doesn’t provide the flexibility departments need to support their unique business processes, and they won’t adopt it. Deliver a common solution that requires significant department specific customization to be useful, and you’ll end up with a bunch of department specific solutions that don’t look very common to your support and maintenance teams. These outcomes highlight the need to understand the different approaches to common solutions and the importance of selecting the right one for each enterprise solution.

It’s important to note that it will very likely make sense to use different approaches for different enterprise solutions. Organizations can successfully implement one approach for their enterprise accounting solution but may need to use a totally different approach to achieve the same success for their enterprise content management system. Since most of the solutions Blue Fish builds for our clients are built on Documentum, I’ll focus on Documentum based solutions for much of the article. However, the reality that there are different approaches to a common solution and the model we present for selecting the right one applies to many other technologies and solution domains.

So, let’s dive right in and talk through the three different approaches.

The Configurable Solution Approach

The ‘Configurable Solution’ approach is what most people think of when picturing a common solution. It’s the archetypal common solution because technically it is the same solution for everyone. This approach gives everyone the exact same technology solution, but then presents a few, well defined configuration points that enable departments or users to tailor it to their needs. All requests for new or modified functionality are reviewed, consolidated and implemented by a central group that owns the solution.

One great example of a configurable solution is Microsoft Word. It’s the same technology solution installed the same way throughout your organization. However, each department or individual user can configure Word to best suit their needs. For example, departments can create their own templates and macros to support their business processes. Individual users can organize the toolbars and configure the interface to help make them more productive. In our example, the central group that owns this solution is Microsoft. If you need new functionality that Word doesn’t provide, you can’t just go change the code yourself. You have to find a way to get that need on Microsoft’s radar and hope that it makes it into the next version.

Similar to Microsoft Word, Documentum can be deployed as a configurable solution. Out of the box, Documentum comes with all the features necessary for a basic content management system. You could use the default object types and configure your users, groups, roles and security using the out of box capabilities, and be up and running with no customization whatsoever. However, we’ve never seen Documentum deployed that way. Here at Blue Fish, we’ve been helping clients deploy and integrate Documentum for over nine years, and have never deployed it completely out of the box. That said, with each release Documentum moves closer to a completely configurable solution. For example, Documentum 6 added several new configuration points for scoping what menus, templates and other object types are visible to a particular user. This was a common type of customization in the past. Some day, Documentum may achieve the goal of providing a completely configurable solution, but for now some level of customization is almost always required.

The Customizable Platform Approach

The ‘Customizable Platform’ approach is the default way most organizations build common solutions. This approach leverages a core of commonality to build a starter kit of sorts. With this approach, the common solution isn’t complete by itself, but rather provides departments with the “80% solution” they can use to build the solution they need. A central team owns and maintains the core platform, and each department is free to customize this platform as they see fit. The central group isn’t necessarily involved, or even aware of the department specific customizations.

My favorite example of a customizable platform is the Linux kernel. There are many different distributions of Linux. A few examples of the more common distributions are Red Hat, SUSE, Fedora, Debian and Ubuntu. At their core, all these distributions share the same code. This core, officially known as the kernel, serves as a common starting point for every distribution of Linux. Each distribution is empowered to modify, extend and even customize the kernel as they see fit to package the solution that fits their needs. In this example, one guy from Finland, Linus Torvalds, owns the core platform and each distribution owner (e.g., Red Hat, SUSE, Ubuntu, etc.) is responsible for building and supporting their customized solution.

Just like Linux, Documentum can be deployed as a customizable platform. EMC expects their customers to extend and customize Documentum, which is why they invest so much in developer tools like WDK and DFS. EMC understands that Documentum can rarely be deployed with configuration alone. The vast majority of Documentum customers use it as a customizable platform by customizing core functionality such as the user interface. Unfortunately, in large enterprises you often find individual departments customizing the platform to meet their specific needs. Over time, the department specific customizations grow further and further apart, and the enterprise ends up with multiple individual solutions that, although they share a common platform, don’t look so common to your support and maintenance teams. This is how many efforts to implement a common enterprise content management system fail. A company buys Documentum thinking it will be common in the way Microsoft Word is common, and they don’t plan for the necessary level of development and support.

The Extensible Framework Approach

The ‘Extensible Framework’ approach provides a set of components and tools that enable departments to assemble focused, complete solutions that meet their unique needs. This approach leverages techniques from both the ‘Customizable Solution’ and ‘Configurable Platform’ approaches to provide users with the ability to assemble exactly what they need, but prevents the expense of divergent solutions by enforcing standards. The standards define basic rules for assembling a solution, documentation for how components interact with the core framework and the specification that new components must align to in order to operate with the framework. Using this approach, departments assemble exactly what they need from an existing set of components. If the functionality they need is not provided by an existing component, they can use the provided specification to create a new component to plug into the framework.

A great example of an extensible framework is the Firefox Web Browser. Firefox provides a core solution for browsing the Web. However, if you need to extend the functionality of Firefox to provide custom functionality, you can create your own extension, or ‘plug-in’, to the core solution. There is no need to create a completely custom build of the browser, you simply implement the ‘plug-in’ according to the Firefox extension framework standards, and then plug it into the core solution. You can continue to get the value of new releases of the underlying platform without the pain of constantly rebuilding and retesting your ‘plug-in’ because it is insulated from platform changes through the extension framework. In our example, the extensible framework solution (i.e., Firefox) is owned by the Mozilla Foundation. All changes to the Firefox itself are managed by Mozilla. Any custom functionality implemented through ‘plug-ins’ is managed by the individual or business who originally implemented the ‘plug-in’.

Similar to Firefox, many components of the Documentum platform can be deployed as an extensible framework. Documentum components such as lifecycles, workflows and the Business Object Framework (BOF) provide the ability to extend the core platform without creating custom builds. However, it’s important to note that the entire Documentum platform cannot be directly deployed as an extensible framework. Many components, such as the user interface, do require the creation of customized builds in order to deploy changes. This doesn’t mean Documentum can’t be deployed as a 100% extensible framework; it means that you must invest in design and development to create your own extensible framework for the components that typically require customization. Once you decide to do this, you can build an extensible framework that will enforce enough consistency across solutions to drive down support costs. Because this solution approach will provide extension points that enable departments to build their own components, each department can construct a solution that meets their needs. For example, you could construct a configuration driven, rules-based Documentum framework that uses services to perform tasks such as document numbering, document filing, lifecycle event management and security definition. These core services are shared and any department can plug them in to their solution if they meet the department’s needs. However, if a department needs to number their documents according to a different standard they can implement their own document numbering extension, and plug it in to their solution. The custom extension can then be shared with other departments who may have similar requirements. Because the new extension aligns to the extension framework standards, other departments will be able to plug it into their solutions in a predictable, repeatable way.

Comparing the Approaches

Now that we understand the three models, there is one more important concept to discuss: governance. Applying the appropriate governance model is critical for driving value out of a common solution and governance boils down to one key thing. The way you manage and support differences, or business process exceptions, within your solution is the most important aspect of governing a common solution. We’ll discuss governance in more detail in the next section. For now, the following table sums up the three solution approaches we’ve discussed so far, and provides some initial insight into the characteristics of an effective governance model for each approach.

Selecting the Best Approach for Your Organization

Understanding the different approaches is half the battle. Unfortunately, it’s the easy half. Identifying and successfully implementing the most appropriate approach for your situation is the real challenge. Do this well and you can drive down solution development and support costs while protecting, and even enhancing, the competitive advantages built into your business processes.

So how is the savvy CIO supposed to select the appropriate approach that fits his/her unique business? Our experience tells us that it mostly boils down to solution governance. But governance doesn’t just mean locking down the system and forcing all changes through an authoritarian change control process. As the table above hinted at, there are different ways to govern your solution. The key thing to understand is that no single governance approach is always right or always wrong. Depending on your organization’s size and governance capabilities, one may be more appropriate than another and ultimately this should impact the solution approach you select.

A Governance Based Selection Model

Below, Figure 1 depicts a model we’ve developed for helping select the right approach to your common solution. The model measures the amount of business process commonality in the departments that will use the solution on the X-axis, and the number of times the solution will be implemented (i.e., instances of the solution) on the Y-axis. This is not a quantitative model. There is no discrete amount of commonality or number of instances that should move every business from one approach to another. A thoughtful analysis of your business and an understanding of your roadmap for the future can’t be replaced by any model. However, this model will help you think about the most important drivers for selecting the right approach to your common solution.

Figure 1:
Our selection model leverages a governance-based approach to analyze the number of solution instances within your organization and the amount of business process commonality across those solutions.

We view this model as governance-based because measuring the amount of business process commonality is another way to look at the number of exceptions to that commonality that are present in your business; the more commonality, the fewer exceptions and vice versa.

Managing Exceptions

I introduced the concept of exceptions in the table that summarized the three approaches to common solutions. Let’s define exactly what an exception is so that we can discuss their importance in selecting the right approach for a common solution.

Exceptions are essentially the inverse of business process commonality. Let’s use the business process of creating, reviewing and approving purchase orders to identify a specific example. Assume that your hard working team of analysts has defined a standard purchase order process to be used across your enterprise. However, one business unit isn’t able to implement that process exactly because of a specific requirement with a key supplier. They can use the majority of the process as-is, but must make one key change to support integration with their primary supplier. This change to the standard purchase order process is an exception.

A typical goal of business process standardization is the identification and elimination of exceptions. For some common solutions, such as your accounting system, word processing technology or desktop operating system, this is an achievable goal. However, other common solutions, such as Content Management systems, are more likely to contain exceptions that business units will want to be supported. Regardless of the specific solution, new exceptions will be discovered throughout the lifecycle of the solution. New business units with new requirements will come online and adopt the solution; your business processes will evolve to adapt to a changing market; there will always be new exceptions, and therefore it is critical that you have a plan for managing them.

The process used to identify, evaluate and ultimately decided how, if at all, to support exceptions is one of the most important aspects of successful solution governance. As we discussed when describing the different approaches to common solutions, each approach requires a unique governance model.

Let’s look at each approach individually. As we discussed earlier, the ‘Configurable Solution’ approach typically requires high business process commonality across your organization. This is because the ‘Configurable Solution’ approach generally supports the least number of process exceptions. The governance required to support this type of solution must be centralized and empowered to challenge the nature and value of exceptions that are brought forward for implementation. Otherwise the central team will be swamped with changes and the ‘Configurable Solution’ will quickly suffer from feature bloat.

On the other hand, the ‘Extensible Framework’ approach requires this strong, centralized governance only for the core framework and the standards that define how components interact with that framework. This core framework and set of standards will change far less often than a complete, configurable solution. By distributing the capability to extend the core framework through well defined components, the central team significantly lightens the load on their governance responsibilities.

Similarly, the ‘Customizable Platform’ approach requires relatively little governance compared to the ‘Configurable Solution’. Exceptions are directly built into the individual department solutions through their own customizations. The central group that owns the core platform is responsible for making changes and extending that platform. Because each department is free to customize their own solutions as they see fit, the departments aren’t required to go through a formal governance process to implement the features they need.

A Detailed Example

Let’s use an example company, Acme Inc., to look at how this selection model is applied.

Acme, Inc. is a large, distributed enterprise with many divisions and departments that require content management. Acme is looking for a way to meet the diverse needs of its business units at the lowest total cost. Acme has selected Documentum as their common technology for content management and now needs to decide how to roll it out to the many divisions and departments across their enterprise.

Acme’s CIO has decided that initially, this common content management solution will only be used to store and manage business critical documents required to support regulatory compliance. Additionally, all implementations of the common solution will standardize on Documentum’s Web-based interface to minimize differences in the way users access and interact with the solution. This is beginning to sound like a relatively standardized solution, right? Might be a great candidate for the ‘Configurable Solution’ approach you say? Not so fast! Let’s take a closer look.

As we’ve said, Acme is a large, global enterprise. They’ve grown to their present size through a series of strategic acquisitions over the course of several decades. While there are only a few distinct lines of business across the organization, the business units within each of these business lines are scattered across every region of the globe. Further, each of these business units have their own unique history and, to a certain extent, culture. The business units themselves are also generally large enough to make their own IT decisions and implement their own custom solutions. There appears to be a great opportunity to drive standards and reduce costs through a common solution.

Using the selection model presented in this article, Acme creates an initiative to analyze the two key dimensions of the model:

1. The number of business units, departments and divisions that will implement the common content management system
2. The amount of commonality in the business processes used to support the generation and management of regulatory compliance documents

The number of implementations is relatively easy to determine. Acme has 50 business units around the globe with as many as 10 different departments within each business unit. Ultimately, there will be several hundred unique departments using the common solution.

The amount of business process commonality is more difficult to measure precisely. However, Acme realizes that there are significant differences in the regulations and compliance laws across the many regions where their business units operate. While the basic process of generating and managing compliance documents can be abstracted to look common, the details are often different enough that many, legally-required exceptions begin to materialize.

Using the governance based selection model, Acme determines that they are in the upper quadrants based on the number of departments they anticipate using the solution. Based on the number of exceptions initially identified they decide that they actually have relatively low commonality across their compliance document management business processes and plot themselves into the ‘Extensible Framework’ quadrant.

How do we know Acme arrived at the right answer for their common solution? What does Acme need to do to make sure their investment in a common, extensible framework has a positive impact on the overall business?

If Acme had not used the selection model and simply decided to use the ‘Customizable Platform’ approach they would have quickly experienced a maintenance and support nightmare. By building only the customizable platform of the content management system, each individual business unit or department would then be responsible for customizing that platform to address their specific requirements. With several hundred departments to come online, the core platform support organization would eventually have to worry about several hundred unique solutions each time an update to the core was ready to be released. This situation would almost certainly overshadow any benefits gained from sharing a common core platform. For this reason, the selection model only recommends the ‘Customizable Platform’ approach when the total number of instances, or implementations, is manageably low.

Had Acme decided that they could manage the regulatory difference across the regions, they could have selected the ‘Configurable Solution’ approach. However, this would require that a central team identify and drive out enough exceptions to achieve a high level commonality in their compliance document management business processes. For Acme, achieving that level of commonality may not have even been possible as many of the process differences are driven by regulatory statues that are non-negotiable. Additionally, because the ‘Configurable Solution’ approach requires a central team to manage the initial implementation and all subsequent changes, all the process variations would have to be identified and implemented as configuration points by the central team. The team will certainly miss a few necessary configuration points with the first release. Additionally, some of the configurations will almost certainly need to be refined or corrected. Every department will have to wait in line for the central team to address their specific need before they can use the solution. Depending on the capacity of the central team, the departments could end up waiting so long that they give up and implement a custom solution themselves. The entire strategy of a common solution is lost once departments abandon the common solution in favor of their own custom solution.

By selecting the ‘Extensible Framework’ approach to their common solution, Acme has positioned their common solution for success by defending against two threats:

1. Mass solution proliferation of customized versions of a common platform
2. Low adoption of a configurable solution delivered by a capacity constrained central team

To be successful with their extensible framework, Acme must understand and plan for the up-front development investment that is required and must implement an effective governance process. The up-front development will focus on building the core framework and the interface between it and extensions developed by business units and departments. The governance model must ensure that the core framework is not customized when deployed and that extensions align with the standards and are shared across implementations.

Summary

If you take nothing else away from this article, remember that there are different types of common solutions. Whether it’s the ‘Configurable Solution’, ‘Customizable Platform’ or the ‘Extensible Framework’ approach, selecting the right type of common solution is a critical first step on the road to a successful implementation. From there it’s on to the fun part; designing and building the common solution that fits your organization like a glove.

Jason Duke, a Technical Lead in our Custom ECM Solutions practice, has written an overview of Documentum Composer, EMC’s latest toolset for creating and deploying Documentum configurations and and other application components.

Learn what Jason thinks by reading Introduction to Documentum Composer.

Marc Perlman just published a new case study / how to article about a project we recently completed for Rio Tinto Minerals. Our client wanted their intranet (manged by Documentum Web Publisher) to contain a Photo Library where users could find approved corporate photos to use in presentations, reports, and other documents. Marc and team created a solution that makes it easy for a user on the intranet to find and preview images before they download them. Marc’s solution uses Web Publisher’s categories (think taxonomies) so that photos can be easily tagged against multiple hierarhical keywords.

To learn more, read Creating A Photo Library Using Web Publisher Categories.

Dan Burton, one of our Solution Architects, just wrote an article that discusses some of the challenges we have faced when trying to translate, internationalize, and localize applications built on the Endeca platform. Not only must you worry about translating the user interface components, but you have to plan for how to store the data and filter out results that are in languages your users don’t understand. There are some great lessons learned in this article.

Dan has all the details in Challenges Internationalizing Endeca Applications.

Introduction

As more and more applications reach out to a global audience, internationalization is becoming an increasingly important aspect of these applications. Since Endeca applications are focused on Information Access, considerations about internationalization are often critical success factors for these globally targeted applications. This article discusses a few of the challenges surrounding building effective internationalized applications on the Endeca platform, and a few approaches to tackling those challenges.

In general there are two primary areas in an application where internationalization becomes an important consideration: the underlying data and the user interface. Endeca’s own documentation does a good job of discussing best practices for handling data in multiple languages in Endeca’s forge process (see the Chapter entitled “Using Internationalized Data” in the Endeca IAP Developer’s Guide). As such, I’m going to focus more in this article on some of the challenges in working with the Endeca pipeline configuration and application user interface.

Interface Languages, Record Languages, & Interest Languages

Imagine an Endeca-powered web application that allows users to search for documents in a company knowledge base. Since company’s customers are all over the world, the application is translated into several languages. The most popular documents within the knowledge base are translated into multiple languages as well. The application gives the user the option to select which languages they can read, so that any documents not written in those languages are filtered out when the user searches the knowledge base.

There are several different ways that languages play an important part in this example. One of the important distinctions that needs to be made in any internationalized application is the difference between interface languages, record languages, and interest languages.

The interface language is the language that a particular user chooses to use when they are using the application. All the text controls and labels in the application should be translated into this language when displayed to the user. For instance, if a user selects “français” as their interface language, they might see a button labeled “Paramètres” instead of “Settings”.

The record language is the language describing a particular record or document. Most often this is the language a particular document or record is written in. Usually there is just one record language per record, but sometimes there can be several. For instance, a single User’s Guide document might contain sections written in different languages that contain the same basic material.

The interest languages are the languages that a user selects when using the application that dictate which content is shown to that user. Some applications tie this to the interface language, but that may not be appropriate for all applications. Users may be interested in content in multiple languages even though they must necessarily choose a single interface language. In the example above, not all documents are translated into all languages, so some content may only be available in the default language. Additionally, it may be valuable for application administrators to have access to records across all languages available in the system.

For the various representations of languages in an application, it is highly recommended to use ISO standard language codes. This will make the language determination clear and consistent. For the majority of cases, the familiar 2-letter ISO 639-2 codes
¹
are sufficient.

Properties & Dimensions

When dealing with an internationalized application, there are a couple different ways that you can structure your properties and dimensions. In all cases, the goal is to provide the user with dimension values in their selected interface language. For these examples, consider a dimension that describes geographical information called “Geography”. We’ll just define the first level of hierarchy to keep things simple.

Dimensions with Resource Key Values?

User interface frameworks that provide support for internationalization generally have the concept of a resource key. The resource key is a string that is used to look up (generally from resource bundles) the string to display to a user. For example, the resource key might be “n_america”, which would provide the string “North America”, or “Amérique du Nord”, or “Nordamerika” back to the application to display depending on whether the user’s interface language is set to English, French, or German, respectively.

However, when localizing an Endeca-powered application, it’s not so straightforward. Let’s examine how we could naively approach rendering our Endeca dimensions using resource bundles. In this approach we make each value of the dimension a resource key, and then use that resource key to look up the appropriate string to display to the user for that dimension value at render time. This is shown in the Developer Studio screenshot below.

Figure 1:
Using Resource Keys as Dimension Values

This has the advantage of keeping your dimension configuration simple and straightforward. It also means that the only place you need to maintain translations for the dimension values is in resource bundles handled by the user interface.

However, this approach has several shortcomings as well. First it means that the data in your records has to be tagged using the resource key value only, rather than the various translations of a value. This may or may not be problematic, depending on how your data is set up. (You could set up synonyms for each dimension value for each language to get around this, but it rather defeats the point of keeping your dimension configuration simple and isolating the translations in resource bundles.)

It also means that you can’t effectively use dimension search with this approach. Since the dimension values are actually the resource keys (which are never displayed directly to the user) the user can’t search using the translated values, which only exist in resource bundles handled by the user interface framework and are not available to the Endeca engine.

Another minor drawback is that the display order for Alpha-sort dimensions may appear jumbled, since the alphabetic sort order for the resource keys won’t always match the sort order of the translations into a given language. This can be dealt with in the user interface code by having it re-sort the values appropriately after they are translated, but it is additional work that you may want to avoid.

Dimensions per Language

A more complex, but more complete approach is to instead create, for each intended dimension, a separate dimension for each available interface language. Suppose our application will support English, French, Spanish, and German as interface languages. In that case, the configuration would look something like the following screenshots.

Figure 2:
Defining Dimensions per Language

Prefixing each dimension with its 2-letter language code makes them easy to identify. Notice also that I have selected “French” as the dimension language. Selecting “French” as the dimension language tells Endeca to treat values in this dimension as French words, which helps dimension searches against this dimension be more effective, as rules for matching words in French can be applied.

If the application requires dimension search to work in multiple languages, this is a preferable approach, since the dimension values themselves will be translated into the available interface languages. We can also maintain dimension ordering in the Endeca configuration, and dispense with any special user interface processing that might otherwise be required. How the data is mapped into these dimensions, and what classification synonyms may be required still depends on how the inbound data is structured, but since we are maintaining the translated dimension values here, there is no harm in maintaining translations as appropriate.

3.3 Record Language as a Property and/or Dimension

Recall that in addition to the interface language, and the record language, we also need to consider the user’s interest languages. The specifics of the application will dictate how the user’s interest languages are determined. For example, they might be stored in a user profile, assigned by an administrator, or selected automatically based on the user’s interface language, locale, or other settings.

In order to provide the user with the appropriate options, the record language will need to be represented as a property and/or dimension value on each record. This will allow the application to use the user’s interest language to filter their results accordingly. This is achievable simply by making the record language a filter-enabled property. However in many cases, especially when the user is allowed multiple interest languages, implementing the language as a dimension is more appropriate, as it brings all the power of Endeca dimensions – Guided Navigation based on language, refinement statistics, multi-valued selections, etc.

Just like the previous example of Geography, you will want to translate the dimension values for the Language dimension into multiple languages themselves! Following our previous example, you could create a dimension called “en:Language”, one called “fr:Langue”, one called “es:Lengua”, etc. These would contain values that correspond to the translations of the possible record languages into each of the possible interface languages.

Figure 3:
Multiple Language Dimensions

There are a few things to notice about this example. First, notice that we have made the two-letter ISO language code a classifiable synonym for each dimension value. This makes it easy to standardize the incoming data on using those codes for the record language. Notice also that Italian is not one of the interface languages that we have created dimensions for, but it is listed as one of the dimension values. This is because (in our hypothetical application) we are not offering an Italian translation of the user interface, but we do have some records in Italian that will be in our index. It may often be the case that an application has a small number of interface languages, but a much larger number of record languages.

Manually creating all these dimension values for each interface language dimension in Dev Studio can be tedious, especially when there will be many different record languages in the index. One option we have employed is to make these language dimensions autogenerated. Then have a Java manipulator in your pipeline look at the two-character language code for each record and set the values for each of the Language dimensions to their appropriately localized value using the Java Locale object. For example if the current record has “en” as its language, the manipulator sets a property called “fr_lang” to “Anglais”, “en_lang” to “English”, etc. You need to store the set of available interface languages so your Java manipulator would know which languages to translate the two-letter code into. Then you can map “en_lang” to “en:Language”, “fr_lang” to “fr:Langue”, etc. If you unexpectedly get a document with a new record language, it would automatically show up in the language dimensions. If you add a new interface language, however, you’d still need to create the new language dimension for it and update your list of interface languages for the Java manipulator so it would get populated appropriately.

Rendering the User Interface

In the scenario where we create a language-specific dimension for each logical dimension in our index, we will then need to filter the dimensions displayed to the user so that only those corresponding to the user’s interface language are displayed.

Delegating to the User Interface?

The simplistic way to do this is to issue a query that returns all dimensions, and delegate to the user interface the job of only displaying the ones appropriate for a user’s selected interface language. If we’re sticking to a naming convention like the one shown above for our dimensions, this is straightforward to implement. In this case, the user interface simply displays only the dimensions that are prefixed with the code that matches the user’s interface language.

However, this may have an extremely detrimental affect on application performance. If our application has 20 available interface languages, we are (nearly) multiplying the amount of work that the MDEX engine has to do and the amount of information being transferred over the network by 20 for every query that is executed.

Optimizing the Query

A slightly more involved, but much more performant solution is to take advantage of the setSelection() method of the ENEQuery object to specify which properties and dimensions to compute and return. The dimensions returned for a given user can be restricted to those matching the user’s chosen interface language. It may be necessary for the application to dynamically determine and/or cache which dimensions are appropriate for which interface languages so that it can take full advantage of the reduction in MDEX engine work and network bandwidth.

You should also set the language for each query that you send to Endeca using the setLanguageId() method of the ENEQuery object. For example if you know you are querying only French documents, set the query language to “fr” so French stemming will be used in the search.

Conclusion

As you can see from some of the complexities we’ve discussed, properly internationalizing an Endeca-powered application can be full of new and interesting challenges. I hope this article has provided a helpful head start for those embarking on such an endeavor. If you have any questions or comments, I encourage you to comment on this article.

References

1. ISO 639-2 codes

Overview

As part of the new Documentum 6 platform, EMC has introduced Documentum Composer, a new application for
packaging and deploying Documentum artifacts. It is meant to replace Documentum Application Builder and
Documentum Application Installer with a single application for developing, building, and deploying
Documentum projects into a repository.

It is important to note that Composer is a completely new application, built from scratch using the Eclipse
platform. It improves upon DAB and DAI in a number of ways, including an open, standardized XML file format;
a more robust and customizable compilation and installation paradigm; and improved visibility for developers
into the application’s operation and data.

Overall, Composer is a huge leap forward for the Documentum installation paradigm, but it is still maturing
as an application. In many cases, Composer will prove a more useful tool than DAB or DAI, but there are
probably still cases where DAB and DAI might be needed, at least temporarily, where Composer’s functionality
or reliability fall short. Careful planning and attention to detail can help to make Composer a valuable
tool for predictable, repeatable installation into Documentum repositories, despite some of the gaps and
pitfalls still present.

This article aims to provide an initial introduction to the Composer application, including some core
concepts, a discussion of major changes from the world of DAB and DAI, and some tips and tricks to make
using Composer easier and less problematic.

Note:
This article was written based on Documentum Composer 6.0 SP1. Subsequent Composer versions are known
to improve on some of the shortcomings identified here.

Core Concepts

Artifact

An
artifact
is a single unit of Composer code which usually describes a single object to be installed into a Documentum
repository. An artifact might be a custom object type, a document, an ACL, a user, a group, or any number of
other Documentum objects.

An artifact can be imported into a Composer project from an existing repository, or it can be created from
scratch using various Composer editors and local file system content.

Project

A
project
is a single, installable collection of artifacts. It corresponds roughly to the DocApp concept from the
DAB/DAI world. The only way to install artifacts into a repository using Composer is to install a Composer
project containing those artifacts. Unlike DAB/DAI, it is not possible to manually checkout or checkin
individual artifacts using Composer; one must install the entire project into the repository.

There are a number of ways to create and populate a project using Composer:

• A project can be imported into Composer from an existing repository project or DocApp.
• A project can be imported from an existing Composer project on the file system.
• A project can be created from scratch using artifacts from a repository as well as artifacts created
  from scratch.

Workspace

One big improvement over DAB/DAI is that Composer provides support for keeping multiple projects open at the
same time, by way of the Eclipse workspace paradigm. A
workspace
is a collection of projects which can be viewed and developed actively at the same time within a single
instance of Composer.

Although composer does not prevent you from dragging and dropping artifacts from one project to another,
it is NOT supported. In some cases it will work, but in most cases necessary prerequisites, dependencies,
or installation parameters will be lost. Always examine the XML for any artifacts that have been dragged
and dropped into a Composer project, to be certain that locations and other references are intact and
correct. It may be necessary to clean and build your project to discover some problems, while others
will not be evident until installation or later.

Note:
It is strongly recommended to keep your Composer workspace separate from any other Eclipse workspaces that
may already exist.

Installation Parameters

In order to make it possible to drive certain installation behavior based on parameters entered at the time
of installation, Composer utilizes
installation parameters
. An installation parameter can be created to represent a folder location, for example, and then that
location can be entered at runtime when installing the project into a repository.

Some features of Composer require installation parameters in order to function, but in these cases the
parameter is created and given a default value automatically. In other cases, you may wish to create your
own installation parameters.

If a project utilizes multiple installation parameters, it can be useful to provide values for all of these
parameters using a single file. This file is called an
Installation Parameter File
, and it can be specified as part of the installation wizard.

Major Changes from DAB/DAI

Overall, Composer is a vast improvement over DAB/DAI, because it provides so much more visibility into and
control over what is happening every step of the way. The architecture is far more robust and extensible,
and this can only mean good things for the future of Documentum configuration management.

The application is still maturing, however, and it shows. There are still many ways to accidentally “break”
your artifacts or your project. Some much-needed custom editors are still missing, occasionally requiring
manual editing of XML which is error-prone and time-consuming. Other editors are buggy or not fully
functional, and at times the installation parameters functionality can be unreliable. Hopefully these issues
will be addressed in future releases.

Some of the key improvements and changes are discussed below.

Open File Formats

All artifacts that make up a Composer project are represented using a standardized XML file format.
Documentum has abandoned the proprietary DAB/DAI file formats, which completely obfuscated their contents.
The new XML-based files provide visibility into what is in the project, making it easier to troubleshoot and
resolve problems. Furthermore, the XML format makes it simple to track and compare revisions to your
Composer project using source control.

Eclipse Platform

You might be familiar with the Eclipse Integrated Development Environment (IDE), which is popular among Java
developers for a variety of reasons. But Eclipse is more than just an IDE; it is also a powerful open-source
software development platform upon which a wide variety of rich client applications can be built.

Composer is built on the Eclipse platform, and the installation includes a bundled Eclipse environment.
Composer is deployed as a set of Eclipse plugins within this embedded Eclipse environment, such that running
the eclipse.exe executable automatically launches Composer.

Figure 1:
Composer Main Window

Composer leverages the
Eclipse Modeling Framework
(EMF), which is “a powerful modeling framework and code generation facility for building tools and other
applications based on a structured data model”. Like the rest of the Eclipse platform, EMF is open source.
It provides a wide variety of modeling and code generation capabilities which enable Composer to be
leveraged across multiple platforms and environments.

Although it is possible to run Eclipse-based applications as plugins within your Eclipse IDE, it is not
recommended to install the Composer plugins directly into an existing Eclipse environment in this way.
Composer is meant to run within its own separate installation of Eclipse.

Compilation Paradigm

One of the benefits of the Eclipse platform is that it offers a number of paradigms that are familiar to
developers. Composer utilizes the standard Eclipse build paradigm, with its own custom builder, in order to
provide real-time validation and compilation of artifacts before they are installed. Composer Projects must
be “built” before being “installed”, enabling offline discovery of some problems before attempting to
install into the repository.

The project directory structure is broken out in support of this compilation paradigm, with separate
directories for artifact source code, artifact content, and compiled binaries. Refer to the
Anatomy of a Composer Project
section for a full discussion of the project directory structure.

By default, Eclipse is configured to automatically build your Composer project, which means that as artifact
source files are saved, Eclipse will attempt to compile the project and will report validation or
compilation problems. If you Clean your project, all compiled resources will be deleted and rebuilt
automatically. If you turn off automatic compilation, you will need to remember to manually build the
project. If you attempt to install your project, it will be built automatically.

Figure 2:
Composer Project Menu

The
Problems
tab reports errors, warnings, and informational messages reported while compiling the artifacts in your
project. These kinds of problems typically mean something is wrong with your artifacts or their metadata.

Figure 3:
Composer Problems Tab

The
Error Log
tab reports errors encountered within the Composer or Eclipse code. Errors point to a Composer bug or
configuration problem, and are typically unrelated to your project. Many errors are simply informational
and do not adversely affect Composer’s execution. Composer 6.5 minimizes the prevalence of these errors
as compared with Composer 6.0.

Figure 4:
Composer Error Log Tab

Installation Paradigm

In the DAB/DAI world, DocApps and DocApp artifacts could be checked out and checked in. In Composer, the
paradigm is one of “installation”, with no explicit control over checkout/checkin of individual artifacts.
This means it is not possible to manually checkout and checkin project artifacts to and from the repository
without installing the entire project.

To install a Composer project, right click on the project in the Navigator pane, and select
Install Documentum Project…

Figure 5:
Composer Project Context Menu

Selecting
Install Documentum Project…
will launch the project installation dialog, which gathers installation options prior to installing the
project to a repository. You must first login to a repository before the wizard will allow you to continue.

Figure 6:
Composer Project Installation Wizard

Better Logging

Another major improvement over DAB/DAI is Composer’s logging. Composer provides much better and more
detailed logging of every activity, as well as full Java stack traces when something goes wrong. Together
these provide greater visibility into what’s happening as it is happening.

Whereas DAB/DAI provided decent logging of the installation process, there was no logging of compilation or
other activities taking place while working on a project, such as import or modification of existing
artifacts. Composer provides logging for all of these activities, making it much easier to track down issues
before they make it into a repository.

Composer’s log file can be found in the %TEMP%/documentum directory. Composer creates a new
log for every Composer session, so it is typically most expedient to sort the directory by date and look
at the most recent log file found.

Headless Eclipse Mode

Composer can be run in “headless Eclipse mode”, which enables command-line scripting and automation of
project installation. This feature, new to Composer, provides for self-contained, fully automated deployment
of projects into a repository, without the need for a previous DFC installation or DMCL client.

Anatomy of a Composer Project

Files and File Types

There are a variety of files and file types that make up a Composer project.

• Eclipse project files (
  *.project,
  *.classpath)
• Composer project files (
  *.dmproject,
  *.template)
• Artifact XML source files (
  *.aliasset,
  *.module,
  *.aspecttype,
  *.folder,
  *.format,
  *.group,
  *.jardef,
  *.javalibrary,
  *.job,
  *.lifecycle,
  *.method,
  *.module,
  *.acl,
  *.procedure,
  *.relation,
  *.relationtype,
  *.role,
  *.sysobject,
  *.type, and so forth)
• Installation Parameter definition files (*.parameter)
• Content files (*.* in the /content subdirectory)
• DAR Definition file (*.dardef in the /dar subdirectory)
• Compiled artifact files (*.artifact in the /bin subdirectory)
• Compiled DAR file (*.dar in the /bin-dar subdirectory)

Folder Structure

Each Composer project is broken down into a standard folder structure, with different folders serving
different purposes. Some folders contain XML source code, some contain content files, and others contain
compiled descriptors and binaries. The project folder structure can be seen in the Navigator pane of
Composer.

Figure 7:
Composer Navigator Pane

Hidden Resources

By default, the Composer interface hides certain resources that are not part of the core user-facing
functionality. It can sometimes be useful to reveal these hidden resources in order to troubleshoot and
resolve problems, but ordinarily it is best to keep these resources hidden to avoid tampering with them
inadvertantly.

In order to function properly, Composer relies on a hidden project that it creates in your workspace
automatically the first time you try to create a new project. This hidden project is called
DocumentumCoreProject, and it must be present in your workspace in order
for Composer to function properly.

Note:
If you wish to create a new workspace, simply moving your custom projects into it will not be sufficient.
You must also move the DocumentumCoreProject, or else let Composer create a new one by creating a new
project.

In order to reveal the hidden DocumentumCoreProject and other hidden resources, launch the Filters dialog
from the pulldown menu in the upper right corner of the Navigator pane.

Figure 8:
Composer Navigator Pane Pulldown Menu

The Filters dialog can be used to enable or disable various filters, which will have the effect of hiding or
revealing various resources in the Navigator pane.

Figure 9:
Composer Filters Dialog

Figure 10:
Composer Filters Dialog with All Filters Disabled

The Navigator pane shown below has had all filters disabled, revealing all hidden projects, folders, and
files.

Figure 11:
Composer Navigator Panel with All Filters Disabled

Folders

The
/Artifacts
folder contains the XML artifact “source files” from which your Composer project is built.

The
/content
folder contains the binary and other content files used to back artifacts that require content, such as
documents. The folder structure within the
/content
folder is created automatically by Composer as content is imported into the project from repositories and
the local file system. The
/content
folder structure is based on a hash structure, but it is possible to move a content file to a more
user-friendly location within your project by updating the content location reference in the artifact XML
file for the artifact. This can be useful in order to provide well-defined content locations within your
project, so that an Ant build, for example, could deposit updated BOF JARs or customized XML files into the
content location, then easily install the project containing those artifacts into a repository.

The folders below contain “compiled”
.artifact
files, built based on the XML and other project metadata described above. These folders would typically NOT
be checked in to source control or distributed in a source code package.

• /dfs-bin
• /dfs-gen-src
• /bin-dar
• /bin

The
/dar
folder contains XML source required by the project as a whole, including an object type definition and
default installation settings for the project.

• dmc_dar.type
  XML describing the dmc_dar object type, which is needed in order to install a project into a repository.
  The contents of this file typically do not need to be edited directly.
• default.dardef
  XML describing various default settings to use for installation into a repository. This file contains
  references to many other artifacts, and can sometimes require editing in order resolve broken references
  or to include missing artifacts.

The
/src
folder is created automatically by Composer. By default it is empty. It may be possible to leverage this
folder to develop Java source code to be used by your project, but this is beyond the scope of this article.

The
/Installation Parameter Files
folder is created automatically by Composer. By default it is empty, but it can be used to develop parameter
files containing parameter values to be used upon installation.

More About Artifacts

Artifact Editors

Composer includes custom editors for a wide variety of Documentum artifacts. These editors enable you to
specify the artifact metadata and configuration using intuitive controls. Composer comes preconfigured to
utilize custom editors for all of the following artifacts:

• Alias Sets
• Aspect Modules
• Aspect Types
• Formats
• Installation Parameters
• Jar Definitions
• Java Libraries
• Jobs
• Lifecycles
• Methods
• Modules
• Permission Sets
• Procedures
• Relation Types
• SysObjects
• Types

There are additional artifacts for which there are no custom editors. Some of these artifacts are hidden,
while others are not. It is much easier to create these artifacts by importing them from an existing
repository, but if necessary they can be created from scratch by editing XML files. These include Folders
and Relations, among others.

Artifact XML

In some cases the editors are insufficient or unstable, in which case it becomes necessary to edit the XML
source directly, using the System Editor or Text Editor. To open any artifact using an
editor other than the default, simply right-click on the artifact and select Open With.

Figure 12:
Composer Artifact Context Menu

The list below provides an overview of some of the key XML elements and attributes you may need to work
with.

• sysObjects
  This element can be found within the <dataModel> element within an artifact’s XML, typically a
  folder artifact. It specifies which objects should be linked into that folder. The references here must
  correspond to artifacts within your project.
• location
  This attribute can be found within the URL specified by the href attribute of the <sysobjects>
  element. It defines the folder path in the repository where this artifact should be linked, if
  applicable. Sometimes this value must be manually updated to change the location an artifact is linked,
  but don’t forget to update the <sysObject> element references to the artifact in the corresponding
  folder’s artifact XML, or the location change may not work!
• artifactURI
  This attribute can be found within the URI specified by the uri attribute of the
  <Artifact:Artifact> element. It is a unique identifier which has no purpose other than to uniquely
  identify the artifact within the project. If your artifact was imported from an existing repository,
  this value will be the folder path from that repository. If your artifact was imported from the file
  system, this value will be the path to the content on the local file system. It is okay if this value
  refers to a location that is no longer available — it is an identifier only and is not used to specify
  any installation locations.
• filePath
  This attribute of the <contentStore.contentEntries.value> element specifies the path within the
  project to the binary content file to use for this artifact. This value can be modified directly in the
  XML to point to a well-defined location, rather than the default location defined by Composer; just be
  sure the location you enter corresponds to real content.

Some Technical Details

Version Compatibility

It is important to understand which versions of Composer, DAB, and DAI are capable of interacting with
certain repository versions.

Composer provides support for all of the following situations, although in some cases an interim “migration
repository” must be used.

• Create a new Composer project by importing a 5.3 DocApp archive on a file system
• Create a new Composer project by importing a 5.3 DocApp from an existing 5.3 repository
• Create a new Composer project by importing a 5.3 DocApp from an existing 6.0 repository
• Create a new Composer project by importing an existing Composer project
• Install a Composer project into a 5.3 repository.
• Install a Composer project into a 6.0 repository.

For more details concerning DocApp migration, consult the
Migrating a repository DocApp
and
Migrating a DocApp Archive
sections of the EMC Documentum Composer Version 6 SP1.5 User Guide.

Alhough Composer is a great new paradigm shift, it is one which is still incomplete. Because of this, EMC is
still supporting DAB and DAI, and has released a version of DAB/DAI which is supported against both 5.x and
6.x repositories. To interact confidently with both D5 and D6 repositories, you must use version
5.3 SP5.5 Update 1
of DAB and DAI.

Source Control

One of the benefits of Composer’s new XML file format for artifact source is that it is now possible to
track an artifact’s version history by checking the artifact’s XML source file directly into a source
control system as changes are made.

In DAB/DAI, there were no source files representing the artifacts, and indeed the only view into these that
could be generated outside a repository was a DocApp archive, which was made up of binary files that could
not be easily compared or reviewed. This made it necessary to track an artifact’s version history directly
in the repository where it was installed and versioned, which was often insufficient and very limiting.

With Composer, it is simple to manage the entire Composer project source in a source control system.

Automated Build and Deployment

Another benefit of the Composer model is the ability to script the compilation and deployment of a project
and its artifacts. This makes it possible to develop truly integrated scripts for building and deploying
Documentum projects in an automated fashion. As this model matures, we should start to see projects with
fully integrated Ant or Maven scripts for building Java source code and then utilizing the output of that
build to deploy updated JARs and other artifacts into a repository.

This automation capability will make it easy for development teams to automatically deploy new software
releases to the repository, in addition to any automated deployment to application servers, method servers,
and elsewhere.

Tips and Tricks

• In certain situations, it may be necessary to inspect and modify the artifact XML files in order to
  resolve various problems or work around certain Composer limitations. Ordinary usage, however, should
  not require delving into the XML.
• When importing folders and objects, you must also import the folders containing those folders and
  objects, all the way up to the root cabinet, or you risk the folders being linked into the application
  folder instead of the location you intended.
• Composer does not seem to be capable of installing folders with a specific ACL set on them — only
  SysObject seem to allow ACL specification.
• ACLs set on Folders imported from another repository will NOT be preserved when imported into the
  Composer project. You must rely on installation parameters to set ACLs on objects.
• Installing a BOF module will unlink/unconfigure any previously installed BOF module with the same name
  automatically.
• Composer logs almost everything it does to a temporary log file in
  %TEMP%/documentum. Each Composer session starts a new log file in this folder, so often the best way to see the most
  recent log is simply to sort the files in the folder by date and look at the most recent one.
• Often Composer will report an error in the UI, but without much detail. In these cases it can be useful
  to look in the log. The number of misleading errors should be reduced in Composer 6.5.
• Composer sometimes reports success when in fact there has been an error along the way. It is therefore
  recommended that you always review the log file after any major activity, especially the installation of
  a project into repository.
• When moving from one version of Composer to another, the recommended approach is to create a new
  workspace, import your Composer projects into that new workspace, and do a clean build.

Conclusion

Composer is a powerful new development platform for building Documentum projects. Although it is still
maturing and may not be right for every project, for those willing to invest some time to understand its
capabilities and limitations, Composer offers a level of flexibility and visibility that until now has not
been possible when building and deploying artifacts into a Documentum repository.

Additional Resources

For additional reading, try the following documentation:

• EMC® Documentum® Composer Version 6.0 SP1.5 User Guide (P/N 300005253A02)
• EMC® Documentum® Composer Version 6.0 SP1.5 Release Notes (P/N 300006232A02)
• EMC® Documentum® Composer Version 6.0 SP1 Plugin Developer Guide (P/N 300006231A01)
• EMC® Documentum® Composer Version 6.0 SP1 Language Pack Installation and Release Notes (P/N
  300006233A01)
• 
  Creating DFS Services Using Documentum Composer
  
  (from Paul Warren’s blog)
• 
  Documentum Composer: Integrating BOF Development into your workflow
  
  (from Paul Warren’s blog)
• 
  Installing Documentum Composer on the Web Tools Project
  
  (from Paul Warren’s blog)
• 
  Items tagged with ‘Composer’
  
  within the Documentum community on EDN

Introduction

Blue Fish’s Web Content Management consultants are frequently asked to build something that brings an extra little spice and twist to a client’s website or Intranet. In doing so, the goal is always to build an elegant technical solution that can be easily administered by our client using Documentum’s Web Publisher. The more straight forward and simple the solution, the more likely our clients will be to adopt it and be able to support it. Often, it is the simple solution that results in an elated client.

One recent variation on a typical WCM solution was when we were asked to design and build a solution for a client’s corporate intranet photo library. While the intranet would allow employees to browse for forms, policies, news, and events just like any other website, the photo library would allow employees to click through image categories to find relevant pictures. This client, like so many others, had loads of images – of sites, community projects, machinery, logos, workers in the field, etc – spread across file shares, one off image hosting solutions by department, and even on CDs in shoe boxes and filing cabinets.

Instead of using Flickr or some other off the shelf solution that would have drawn all these digital assets together into one nice location for the employees, the client asked for two things:

1. The photo library had to be neatly integrated into the intranet; Why should the employees leave the intranet to visit Flickr or a file share to find an image?
2. The administration of the photos had to take place in Web Publisher; Why make content administrators learn another system other than Web Publisher specifically for the photo library?

Additionally, it was obvious that sometimes an image could be categorized multiple ways, so the photo library had to support a hierarchy of categories across which an image could be tagged multiple times.

In the discussion that follows, I’ll show you how we designed a solution that uses Documentum Web Publisher to meet those requirements. Armed with Web Publisher and knowledge of web applications, perhaps your next WCM solution can incorporate a little something special — like a photo library.

Solution Overview

Our client really wanted employees to browse and download digital images on the intranet, not on another website or file share. The experience of “leaving” the intranet to download a corporate logo was going to cause user adoption problems for both the intranet and the photo library. Additionally, the new design for the intranet looked great – so why not wrap the photo library with the same look and feel as the rest of the site?

All the pages on the intranet, like all well designed WCM solutions, featured a variety of common elements such as a header, footer, breadcrumbs and side bar navigation.

Figure 1:
Common elements on an intranet page

We were able to create a photo library that reused these common components while employees browsed the image categories.

Figure 2:
Common elements on the photo library category page

Maintaining the common look and feel, these elements were all still present when a user chose a category and was selecting images from the thumbnail gallery view.

Figure 3:
Common elements on the photo library thumbnail gallery page

Additionally, the client wanted the administration of the photo library to be handled within Web Publisher. Web Publisher’s out of the box category functionality was the perfect solution for implementing the administrative tools for the photo library due to the following category capabilities:

1. Web Publisher’s categories are easily created to reflect the hierarchal taxonomy required for the photo library. Each category has as many children, grand children, and deeper descendents as needed by the client’s changing business needs. Alternatively, a category can be a single node with no children at all.
2. Web Publisher allows any web content maintained in Web Publisher to be linked to multiple categories with just a few simple clicks.
3. Web Publisher’s interface requires no customization to allow content administrators to create, edit, and delete categories. Additionally, no modifications are needed to allow content administrators to associate images with any number of photo gallery categories.

It was just a matter bringing the following components of the solution together to create an easily administered gallery that looked and felt as great as the rest of the intranet!

Category Creation and Administration in Web Publisher

Take a look at any out of the box deployment of Documentum Web Publisher and you’ll notice the Categories node in the left hand side of the application. The Functional Taxonomy node is a child of the Categories node and is how the templates are organized. By selecting a template category, an administrator is shown the documents created using the template or templates in that category.

To create the hierarchal taxonomy of the photo library, a sibling to the Functional Taxonomy, named Photo Library (Figure 4), needs to be created under the Categories node. The Functional Taxonomy node allows users browse web content associated with specific templates and, thus, functional categories. Similarly, through the new taxonomy node, users will be able to browse photo library images associated with specific photo library categories. Once the taxonomy node is created, the library category hierarchy can quickly be created using standard Web Publisher functionality.

Figure 4:
Photo Library Taxonomy Node with Children Categories

When creating new gallery categories, an administrator can not only name the category but set the display title, locale specific translations, and permissions.

Figure 5:
Creating a new category

Once the photo library taxonomy structure is defined under the Categories node, there are a few remaining steps.

First, the administrator must associate this new taxonomy with a Web Cabinet. This step only needs to be done once when the initial taxonomy is created and setup. Apply the taxonomy to a Web Cabinet from the Web Cabinets tab of the taxonomy’s properties view.

Figure 6:
Associating the Photo Library taxonomy with a Web Cabinet

Subsequent changes to the taxonomy structure (for example, when a user creates a new gallery category) won’t change this association.

Once the taxonomy is associated with a Web Cabinet, Web Publisher will publish the structure as an XML file to a Taxonomy folder under the Web Cabinet. This XML file can be promoted and published to the Web Server to be consumed and displayed on the final website.

Figure 7:
Web Cabinet with the published XML file

        

<category name="Argentina_sites" className="Generic" 
    type="dm_category" internalId="0b000405800073a8">
   <details title="Argentina">
      <description></description>
      <owners>
         <owner name="dmadmin"/>
      </owners>
      <operations/>
      <languageInfo/>
   </details>
   <category name="Argentina Operations" className="Generic" 
       type="dm_category" internalId="0b000405800073b7">
      <details title="">
         <description></description>
         <owners>
            <owner name="dmadmin"/>
         </owners>
         <operations/>
         <languageInfo/>
      </details>
   </category>
</category>
    

For our client’s photo library solution, we created a .NET web control that consumed an alphabetically sorted XML file on a Web Server and displayed the category structure as a photo library “site map” on the intranet.

Figure 8:
The photo library XML consumed and rendered on the intranet

Associating images with categories

Now that the structure of the photo library categories has been created and published and is consumable XML on the website, the solution needs to allow a content administrator to tag an image as belonging to one or more categories.

Web Publisher allows a user to associate web content with an existing category a few different ways:

1. Select the category and create the new content from the category view. This ensures that the content will be related to both the Functional Taxonomy (via the template used for the content creation) as well as the Photo Library category.
2. Create or import the web content using any other manner. Then, from the Properties view, select the “Categories” tab and associate the image with the category or categories.

Figure 9:
The Categories Tab on an existing piece of Web Content

That’s it! Provided the SCS publishing configuration is set to export the relationships, the dm_relation of type dm_category_assign will be published when the content is published.

Displaying the gallery

Now the solution has an easily modified, published, and consumed XML representation of the photo library category structure. Additionally, the solution provides a simple mechanism to associate images with one or more categories. All that is missing is some website code – in the form of a JSP, taglib, .NET control, or other mechanism – to display the images that have been associated with a selected category.

For our client’s solution, Blue Fish created a simple ASPX page that read the category ID from the Request URL, queried the published web content metadata, and rendered the associated images in a thumbnail gallery view. Because the published XML representation of the photo library categories contains the names and titles as well as the unique ID, it was straightforward to embed the ID in the clickable links of “site map” view discussed earlier.

Figure 10:
Thumbnail Gallery View

Closing Thoughts and Considerations

In addition to creating a photo library, a similar solution could be created to catalog any type of file – or mixed media types including photos, video, and audio. Another enhanced solution would allow further detail to be collected about each image. Instead of associating the image directly to a category, an image “detail” template could be used to capture additional metadata including a description, long title, separate scaled thumbnail image, and more. This web content would be associated with the category. Using this metadata, the thumbnail gallery view could display even more details.

The possibilities for enhancing a WCM solution using Web Publisher’s out of the box category functionality are numerous. Given a little planning and preparation, the discussion in this article can help turn an ordinary web experience into something a bit unique.

• Automate BOF Module Deployment During Development – Steve has a great article on how to streamline development cycle times by automating the deployment of Documentum BOF modules.
• Introduction to the WDK Automated Test Framework – Thomas Hughes has reviewed the new WDK Automated Test Framework.
• Improving Web Publisher Usability – Lisa Hill shares some of our tricks for making Web Publisher easier to use.
• Bullet Proof Markup: Designing for Web Content Management Systems – Alex Bilstein writes about how to design your HTML markup to work best with a WCM system.
• Using Documentum Lifecycle State ExtensionsGlen Tarrant writes about how to add custom functionality to lifecycles via lifecycle state extensions
• Introduction to Documentum IRM Services – Terence McDevitt gives us an overview of Documentum’s Information Rights Management solution which can keep people from reading your documents even after they are exported from the docbase.
• Enabling High Availability for a Complex Documentum Based Application – Jamie Balling explains how he’s been able to increase application uptime by integrating some custom monitoring components into his solution.

Overview

For the past several months, I’ve been working on a global enterprise content management solution Blue Fish built for a Fortune 100 client. The system serves as the document repository for tens of thousands of sales and marketing users, spanning dozens of countries and supporting eight languages. This is a critical business application that is always being used by someone somewhere. There is no room for application outages and documents must always be available.

The solution relies heavily on Documentum, and a number of services and features are critical to proper functioning. These include:

• The Documentum Content Server
• Site Caching Services (SCS)
• SCS Post-Syncronization Scripts
• Documentum Workflows
• Media Transformation Services (MTS)
• Web Publisher and Documentum Administrator
• The Documentum LDAP Synchronization Job

Due to the necessary complexity of the overall system, we knew that a dedicated support group and an industry-leading monitoring solution would be required to ensure smooth operations. We therefore worked with our client to ensure that the IT infrastructure was robust and launched the application with a suite of monitors which confirm that machines are up, that services are running, and so forth. However, we soon found that this type of high-level monitoring was not sufficient given the complexity and criticality of the application.

This article gives some insight into the challenges we encountered and the solutions we implemented to monitor this critical application. Please note that this article assumes familiarity with some technical aspects of Documentum.

Support Infrastructure and Challenges

As previously mentioned, the application has a dedicated support team which utilizes a monitoring tool called OneSight from Empirix. This tool has the ability to monitor typical server statistics, such as availability, memory usage, disk usage, etc. It also has the ability to monitor Windows services and check URLs for expected responses.

Even though we launched with monitors for nearly every service, machine, and website involved in the architecture, this monitoring proved insufficient. Several production issues arose which highlighted weaknesses in the monitoring infrastructure. For example, we found cases where service monitors were showing success even though the service was hung, and we found instances where HTTP monitors showed success when the web component they were monitoring had critical errors.

Although Documentum provides out of the box notifications for notable repository events, such as job failures, these didn’t meet our needs, either. In the client’s environment, these notifications are directed to an email address which is intended to be checked regularly by the support team. In practice, this was difficult to manage. The appropriate response from the support team can vary drastically depending upon the severity of the problem being reported, and often times high priority notifications would get lost among less important ones. Also, reviewing an email inbox introduced a manual delay and increased possibility for human error.

Finally, given all the integration points across the system, the team sometimes had trouble answering the question of whether there were any issues across the system and pinpointing a component which was not functioning properly.

Monitoring Solution Overview

In response to these challenges, we took a big picture view of the architecture and designed a monitoring solution. This solution leveraged the existing monitoring infrastructure, but added custom monitoring components tightly integrated with the system to provide insight into whether things are not only running, but are successfully performing their functions.

The monitoring solution still utilizes Empirix for monitoring non-application-specific things. This includes machine availability, CPU usage, memory usage, free hard disk space, and so forth.

For monitoring items that are more tightly coupled to application logic and architecture, we created a custom application health dashboard. This dashboard was simple to create and consists of a single JSP page served by the web application itself. The data for the dashboard comes from a collection of monitoring modules (implemented as java classes) which perform business-level checks on the components for which they are targeted. Each of these monitoring modules reports a status of OK, warning, or failure and is displayed as a row on the dashboard. The dashboard also displays an overall application status which represents a rollup of certain critical modules.

We then setup URL monitors in Empirix for our application health which hit this application health dashboard and inspect the page for certain key strings, such as “APPLICATION STATUS: FAILED”. When these monitors detect a failure, they create support queue items for the support team. Individual modules can be queried to allow fine grain monitoring and severity reporting. In the case where the overall application status shows as failed, we report at the highest severity.

Figure 1:
Screen shot of section from application health page

Although this custom monitoring solution does indeed overlap with some capabilities of the traditional monitoring tools, it provides distinct benefits which greatly improve the supportability of the system. The solution provides a focused view of the application health and also gives the business team visibility as they wouldn’t normally have access to back-end monitoring systems. This helps them better communicate and manage issues. The solution also provides the application team greater control over monitoring. They have the best understanding of the severity and impact of specific application component issues, and the dashboard gives them immediate insight which helps expedite resolution of issues.

Documentum Integration Points

The monitoring solution is broadly scoped, covering almost all of the external processes and application components that interact with the core Documentum services. However, to keep the discussion focused, the rest of this article focuses on the strategies employed for monitoring the Documentum components.

Content Server

The support team had already setup Empirix monitors for the Content Server host and Windows service. We augmented this with a module that establishes a connection to the repository and executes a simple query via Documentum Foundation Classes (DFC). This connectivity is verified whenever the application health page is served by the web application, and it therefore provides confidence that each app server instance (there are several that run in a clustered environment) is successfully communicating with the content server.

Site Caching Services (SCS) Publishing Jobs

There are several SCS publishing jobs which push out several types of data from the content server for consumption by the web application and other application components. These jobs publish taxonomies (from Content Intelligence Services), web content, and -most critically-documents which have been approved for distribution. Implementing monitoring for these publishing jobs was especially important, as multiple issues had been experienced where publishing jobs failed or hung, and the support team wasn’t aware until users started complaining!

To prevent this problem, we built a generic module that can check on the status of any web publishing job. This module has logic which queries the repository tables to determine the status of the last run for the job. Furthermore, it compares the time of the last successful run against a time threshold configured in the application. If the job hasn’t run within the expected timeframe, a failure is reported.

Some additional effort was required to retrieve accurate publishing job status from the repository. The publishing configuration object (dm_webc_config) is updated with the time of the last full publish and the time of the last successful incremental run. However, this object is not updated when the job runs but there is no data to publish. This presents a problem for our monitoring solution as we don’t want to “cry wolf” just because there was nothing to do! Therefore, this monitor also queries the job table (dm_job) to retrieve the last completion time and job status. The job status will be “COMPLETED” if it ran successfully.

SCS Post-Synchronization Script Execution

Some of the SCS jobs kick off post-sync scripts that ensure that application caches are kept up to date when new content and metadata are published. These critical scripts are kicked off by specifying a parameter in the “agent.ini” file on the SCS target machine. In our case, the script executes a java process.

We built logic into this java process to reports on its status by recording success and error events to a special “application health event” table in the database. We have a monitoring module which reads this table, determines when the job last ran, and reports a failure condition if it was unsuccessful. Since this process should be kicked off after the publishing job runs, we also check the time of the last publish to make sure it executes successfully within the expected time interval after that publish. If it hasn’t run, we report a failure.

Java Method Server (for workflows)

Documents posted to the system go through an approval workflow before they are made available to end users. To accomplish this, custom “methods” were developed and integrated with Documentum workflow steps. The Documentum architecture utilizes the Java Method Server (an embedded Tomcat web server) to trigger these methods.

Basic Empirix service monitoring was in place for this Tomcat instance, but it proved insufficient. On several occasions, the service appeared to be running successfully even though our custom methods were not being triggered successfully.

In response, we created an additional custom “health check” method. This method accesses the same application context object that the workflow methods utilize and confirms that it has been successfully initialized. It also performs a simple query to the docbase using DFC to verify that it is properly functioning. We then have a module which invokes this method to make sure it is functioning properly.

Here are the technical details of this monitor for those of you who like that kind of stuff :

The implementation of the health check method is straightforward. In the “execute” function, we perform our checks and throw an exception in case of any errors. This exception will cause the method server to report that the launch failed. The implementation of the monitoring module is a little more involved…It utilizes DFC to create an IDfSession and calls the “apply()” function. We pass the “DO_METHOD” API command and the name of our health check custom method as arguments to this function. This function returns an IDfCollection which includes the attributes “launch_failed” and “timed_out”. If either of these attributes has a value of “T” we report a failure.

Media Transformation Services

Media Transformation Services (MTS) is a Documentum product that falls under the “Content Transformation Services” umbrella. The system utilizes MTS for a couple of specialized functions such as thumbnailing and assembling PowerPoint presentations.

MTS has its own dedicated machine and services, so we setup Empirix monitors to make sure these things are up and running. We also added a health check module which checks the size of the MTS “queue”. We determine what is in this queue by querying the dm_queue table for rows with a name of “dm_mediaserver”, which is the standard name for media server entries. We report an error if we can’t access the queue or we determine that the queue size is building up without any reductions over a configured time interval.

LDAP Synchronization Job

The system is used in a large corporate environment and utilizes LDAP to retrieve user and group information from Active Directory. We rely on the LDAP Synchronization job, which is an out-of-the-box job used to keep user accounts up to date in Documentum.

We have not yet implemented custom monitoring targeted at this component. When the job fails, the standard Documentum notification email is sent to the email address defined for the Content Server. So far, this has been acceptable, but there is room for improvement. The job has failed and we have had issues with particular user accounts as a result. We plan to add a module which leverages the logic used to monitor our publishing jobs. It would query the dm_job table for the time of the run and status for that job.

Web Publisher and Documentum Administrator

Web Publisher and Documentum Administrator are instrumental components of the system, and their availability is therefore critical to the overall health of the system. We have found that standard Empirix monitors setup to monitor the machines and web servers have done a fine job keeping tabs on the health of these two components.

Conclusion

If there is one major theme from our experiences supporting this application, it is:

“Running” does not necessarily mean “working”!

Many monitoring systems simply determine whether a machine is up or a process is running, but this simplistic approach to monitoring only tells half the story (if that!). Make sure your monitoring confirms that critical application components are actually doing what they are supposed to do.

Application monitoring should grow and improve as experience with the application mounts: when problems occur which expose blind spots in the monitoring solution, learn from them and improve the monitoring to ensure a more robust and stable system.

Overview

As content becomes more readily accessible to enterprise users, it also becomes more difficult to control distribution, particularly in situations where content is shared outside of the corporate network. This is especially concerning when the nature of the content could give competitive advantage to outside parties.

In order to minimize the potential impact of sensitive data falling into the wrong hands, organizations are looking to implement Information Rights Management (IRM) solutions. IRM allows companies to gain tighter control of how content is distributed both within the organization and external via partners, vendors, and customers.

EMC Documentum provides the leading platform for storing and managing content in the market. With a recent acquisition, Documentum now includes IRM capabilities in its product offering. These services ensure that Documentum is able to provide a truly secure content management and distribution system.

What is IRM?

Traditionally, organizations have approached data security by ensuring that malicious persons or entities are kept out of the network and resources attached to it. This type of security paradigm is known as perimeter security. While this approach has served the enterprise well, it does not cover situations in which persons within the organization are sharing sensitive data to unauthorized users outside of the network (e.g. emailing, FTPing, etc.)

Information Rights Management (IRM) attempts to solve this problem by securing the content, not just the resources that house the content like file servers or web sites. With IRM, enterprises can mark selected content as protected and authorize select users or groups of users to perform specific actions on the content, including copying, printing, and emailing. Any request to perform an action on protected content is first sent to the IRM policy server, along with the credentials of the person attempting the action. If the policy server determines that the user’s action is authorized, the content is unlocked and the user can finish their work. Otherwise, the user is denied from acting on the content.

A typical example of where IRM’s power becomes apparent involves a content author working in an office application such as Microsoft Word. Prior to IRM, authors wishing to “lock down” their content either used the built-in password protection in Word or third-party encryption tools. In either case, the user needed to provide information to consumers as to how to open the protected content. With IRM, authors can take advantage of plug-ins built into their applications allowing them to protect content and assign rights policies. Consumers receiving this protected content need not know how to unprotect it. This responsibility is left up to the IRM client embedded in their applications (this client communicates with the policy server to determine if the user has the rights to unprotect the content).

Information Rights Management generally encompasses the following concepts:

• Principals – IRM defines Principals as those entities that are authorized to perform given actions on content. Typically, a Principal is a user on the network. In some IRM implementations, a Principal could be a group of users, another network (e.g. a partner) or a resource on the network such as a mobile device.
• Policies – Policies define what actions are acceptable for a given piece of content and a particular Principal or set of Principals. Some IRM products can also allow content owners to create policies which revoke content privileges that were once established. For example, a Sales and Marketing group may release a product’s data sheet and make it available for printing by the sales team. If a new release of the product is introduced, the Sales and Marketing content owners can revoke rights on the old version of the product data sheet, forcing the sales team to request the latest copy.
• Encryption – Content that is considered “protected” under IRM generally has strong encryption applied to it. This encryption prevents unauthorized users from making any sense of the data. Content encrypted with IRM products can only be unlocked using valid encryption keys known to the IRM policy server and the authorized client.

Documentum IRM Services

History

In 2006, EMC Documentum entered into the IRM space with its acquisition of Authentica. At the time of the purchase, Authentica was amongst the leading players in the relatively immature IRM market (SealedMedia and Liquid Machines being other notables). In addition to its stand-alone IRM offering, Authentica also provided support for products like Microsoft Office. This integration allowed for users to easily create IRM protected content from tools they were already familiar with. EMC’s purchase of Authentica was in line with their bigger picture plan of tying together the storage, software and security portions of the business. EMC clearly saw the benefits of rights management in the enterprise, and reeling in Authentica would complete the security picture between Content Server and enterprise users.

Architecture

Documentum IRM services can be deployed in an enterprise with incredible ease (although planning for this is a much larger task – see “Considerations” below). The core component, IRM Server, only requires connectivity to a database server for storing policy and key information. Enterprise users can easily encrypt content via a number of clients that securely connect to the Server. Finally, enterprise services can leverage the Documentum IRM software development kit (SDK) to programmatically encrypt/decrypt sensitive content. Refer to Figure 1 for a contextual diagram of a typical IRM Services architecture.

Figure 1:
Documentum IRM Services Architecture

Three key components make up the IRM Services architecture:

Features

Documentum got several IRM features “out of the box” with the purchase of Authentica. Included in these features:

• Dynamic Rights – A key feature in the Documentum IRM offering is the ability to change document policies on the fly. Imagine a scenario where a document owner decides to give a vendor access to a document. When the vendor opens the document, the Documentum IRM Policy Server is contacted to determine if the user opening the document has the appropriate rights to do so. Now suppose that the vendor’s partnership agreement expires. The content creator can revoke the user’s permissions without having to modify or delete the user’s copy of the content. This notion of dynamic policy assignment is critical for enterprises wishing to truly manage access to content that is distributed outside of the corporate network.
• Offline Rights Management – Documentum IRM Services provides functionality to allow offline usage of content, while continuing to respect any policies that have been defined for the document. For example, documents can be made available to users with set expiration dates. Once a user has authenticated against the document’s policy, IRM Services embeds this information in the document and will self destruct the content at the specified expiration date.
• Stand-Alone Product – While Documentum IRM Services integrates with the existing Documentum product line, the IRM Services component is a stand-alone product. Thus, customers do not need to have Documentum in place in order to take advantage of the features provided in IRM Services.
• Plug-Ins for Common Office Applications – Documentum provides a number of plug-ins that allow end users to define IRM policies for documents they create in applications such as Word, Outlook, Excel, PowerPoint and Adobe Acrobat. There is even a plug-in for RIM Blackberry devices for users needing to access IRM protected content on the go.
• Support for Dynamic Watermarks – Documentum IRM Services allows protected documents to carry watermarks, which are useful for identifying the status of a printed document (e.g. “Confidential”, “Draft”, etc.). In addition to standard watermarks, Documentum IRM Services can place dynamic watermarks on documents when certain actions are taken. For example, if a user prints a document, IRM Services can embed a dynamic watermark on the content with the login name of the user who is printing the document. Thus, if an unauthorized hard copy of the data is found, organizations can immediately track down those responsible.

Since the Authentica acquisition, EMC has extended IRM Services to better integrate with the Documentum stack. Some notable enhancements include:

• Direct Integration with Content Server – Documentum IRM provides rights management features from within Documentum Content Server. This integration provides an easier means of applying IRM policies to documents with limited changes to existing Documentum-based architectures. In addition, current business processes (e.g. workflows, lifecycles, etc.) can now take advantage of IRM with minimal modifications.
• Software Development Kit (SDK) – Documentum provides an IRM Services SDK in order to allow third parties to extend IRM functionality to their own applications. Access to this SDK should help facilitate any IRM customizations needed in an enterprise Documentum application. As an example, an organization may have a custom desktop integration with Documentum. In order for users to open IRM protected content, the desktop customization could utilize the SDK to pass the user’s credentials in to the Policy Server to determine whether or not the user should have rights to work with the content.

Architectural Considerations with IRM

Roll out of an enterprise IRM solution is not something that happens overnight. A tremendous amount of planning must go into the initiative, from infrastructure build-out with corporate IT to business change management for the users who will be interacting with IRM. Here are some key points to consider when implementing an IRM solution:

• Sending Content over the Wall – While most organizations will want to exclude outside parties from accessing sensitive information, there are often situations where access must be granted. For example, if a company is working closely with a manufacturing partner, it is very likely that content will be shared with that partner. Documentum IRM Services can be deployed in a manner that keeps policy information secure, while still exposing services to the outside world (this could be done either by setting up IRM Server in a DMZ or having Repository Server directly serving protected content). This decision should be made as early in the planning stages as is possible.
• Enterprise Services Need to “Crack the Safe” – Large organizations typically have services that perform actions on content (full-text indexing for search, backup/recovery, etc.). An enterprise wishing to roll out an IRM solution must understand at which point they want content protected. If an IRM policy is applied when content is created, enterprise services such as indexing will need to have a way to unprotect that information when it is read. Depending on the number of services, customizations required to decrypt data could derail the IRM initiative. Organizations could consider protecting content at the point of distribution to alleviate this problem.
• Who Controls the Policies? – Companies implementing an IRM solution are likely going to have a well-defined set of rules for when policies are applied to content and which rights principals will have to that content. In this situation, the usage of policy templates helps ensure that users aren?t making the decision about what rights to give to whom. Of course, this power can be left in the hands of the user if the company prefers.

Summary

As the enterprise is increasingly becoming connected to the outside world via technologies such as email and web services, securing sensitive information is quickly showing up on CIOs’ priority lists. A key component to fulfilling an organization’s security mandate is to ensure that critical content remains under tight control of the enterprise, whether that content is distributed internally or externally.

Implementation of an Information Rights Management solution helps companies gain confidence that content is secure, no matter where it resides. Documentum’s IRM Services provide a powerful solution to this problem, giving incredible management options for content. In addition, IRM Services tightly integrates with existing tools and platforms within the enterprise such as Documentum Content Server and Microsoft Office.

With the continued demand for organizations to distribute content within the enterprise as well as outside the firewall, the need to control this content is ever increasing. Information Rights Management attempts to fill this need by providing dynamic policy control that follows content, no matter where it goes. While implementation of an enterprise-wide IRM initiative is a significant undertaking, with proper planning and execution, an organization can ensure that only the right people are able to view or manage sensitive information.

Introduction

For years, Web Publisher was the sole interface for Web Content Management solutions built on the Documentum platform. Recently, through the release of Page Builder and the Contributor UI, EMC has provided alternative interfaces to this core product aimed at simplifying the content authoring process. While these interfaces are great tools for certain types of content creation tasks, unfortunately, they do not replicate the full capabilities of Web Publisher. For users that have to perform a wide variety of web content management tasks, Web Publisher is still the only tool available.

One of the biggest complaints of Web Publisher (and hence the push to develop alternative interfaces) has been that its usability is not up to par with other enterprise web content management tools. With D6, EMC has begun to address this usability issue in Web Publisher itself with incremental usability enhancements. For example, users will notice that with D6 the check boxes of the 5.3 stack have been removed and in place of them are right-click menus with CTRL- and SHIFT- click enabled multi-select.

Figure 1:
D6 Web Publisher Usability Enhancements

While these enhancements are welcome improvements, there is still much that can be done to make Web Publisher a more usable tool, both in D6 and on the legacy 5.3x stack. This article will outline some of these simple enhancements and best practices that you can put in place today to better leverage your investment in Web Publisher (and hopefully give your users a nice sigh of relief)

Scoping Down Interface Elements

One of the easiest ways to make Web Publisher more useable is to simply reduce the amount of options that users see. Much of the perceived problem with Web Publisher is not that the individual actions themselves are so difficult, but simply that there are too many choices – too many options from which users can select. While proper use of the standard Web Publisher roles (content author, content manager, web developer, and administrator) can go a long way towards reducing the complexity for users that don’t need it, even greater impact can be had through smart application of security.

Folder Security

One way that users often interact with content is via the Web Cabinets. This view allows users to browse a folder structure that houses content for the system. Even if users don’t have access to the Web Cabinets node within Web Publisher explicitly, content authors frequently must browse the folder structure when creating links to other documents within the content editor. A great way to reduce the complexity of the interface is to remove from view those folders in which users have no business poking. For instance, nearly every site has some sort of folder structure for “includes files” – those artifacts that web developers maintain for the look and feel of the site (CSS, Javascript, headers, footers, etc.). By simply restricting access to the root folder that houses these items, the interface has one less element for users to think about. With a rich and robust security model (as well as a well thought out folder structure), this concept can often be greatly expanded to further reduce the folders that a given user encounters when browsing the web cabinets. The fewer folders a user sees, the easier it is for them to find what they are looking for.

Categories Security

Another popular view for interacting with content within Web Publisher is the Categories view, or more specifically, the functional taxonomy. This view has content organized by what “type” of page it is (according to the templates used to create the content). A well structured functional taxonomy can be very powerful, allowing users to browse all “press releases” or “product pages” in one place regardless of how they might be organized on the physical website.

In any given site, there are often different types of users that interact only with certain types of pages. For instance, certain users may be tasked to ONLY create press releases and are not working with any other type of content. Further, certain templates (the home page for instance) might be restricted such that only trusted users have the ability to create content of this type. If security is applied to the nodes in a well thought out category structure, users will only be presented with the categories that are relevant to them, reducing from the interface irrelevant options. Not only does this improve the user’s experience, if you never show the option to a user to begin with, then your administrators don’t have to deal with the inevitable question of “why don’t I have access to that?”

Reduce the Number of Clicks

Everyone is busy these days – users want to get into the content management system, perform their tasks, and get out. Unfortunately, Web Publisher is a “click” intensive application. Anything that can be done to reduce the number of clicks a user has to perform in order to get their job done will be greeted with a huge welcome. Over the years, we at Blue Fish have identified several clicks that can be removed regardless of the other details of the full solution.

Auto-start in Classic View (5.3 only)

If you have already moved to D6, then you can skip right over this suggestion. If, on the other hand, you still are leveraging your investment in Web Publisher 5.3, then you are probably already familiar with the “Streamline” view. The Streamline view can be considered Documentum’s first foray into addressing the usability concerns of the Web Publisher interface by providing an alternative way to navigate the repository. Out of the box, Web Publisher 5.3 is configured such that the Streamline view is the default view into which a new user of Web Publisher is launched. Unfortunately, most users find the “Classic” view much more useful. While a user always has the ability to set their user preferences up to start in the alternative Classic view, many users (especially novice ones) don’t know about this option and so are launched into the Streamline view every time they log in. At best, this is an annoyance – at worst, something that can become so confusing that user adoption suffers. A very simple customization can take away all of this concern, and just launch the users directly into the Classic view.

Auto-start workflow if only one available

Workflow is one of the most powerful concepts within Web Publisher. While Web Publisher comes with a few sample workflows, these are often customized to create workflows that make sense for a given organization. Having read through the section above on reducing interface elements, you can probably guess that I am also suggesting applying proper security to workflows – so that users only see the workflows that should be made available to them. Once this is done, in many cases, the majority of users will only have one workflow available to them. Yet when launching a workflow, Web Publisher always launches a dialog that forces the users to choose “which workflow” even if there is only one available. My thought is, why give me a choice if there is only one option to choose? A simple customization can make it such that when a user selects to “Start Workflow”, if there is only one workflow available, the system will just go ahead and launch that one. No need to make a user pick if they really don’t have a choice anyway.

Pre-check “Edit Now”

When an author is creating new content, after selecting a template, they are taken to the new content attributes screen. On this screen, users enter metadata for the content – things like the object name, title and keywords, in addition to any effective dates or expiration dates that should apply to the content. At the bottom of this page is a check box with an inconspicuous “Edit Now” label. If users check this box, the template editor automatically launches when the user says OK to the content attributes. If the user forgets to check this box, the new content attributes page is closed and the user has to locate and select their newly created content, and then launch the template editor manually. Again, with a simple customization the “Edit Now” option can be checked by default so users are saved these unnecessary steps. And if they ever didn’t want to launch the content editor, they could still uncheck the box – but for the 99% of cases where you actually want to launch the template editor, one less click required.

User Focused Template Design

Often when Blue Fish comes on board to work with a new client, it is not to create a new system from scratch, or even to port a solution from a legacy content management system into Documentum. In fact, many times our clients have existing Web Publisher solutions in place, but they are just so hard to use that the business is ready to throw them out. This isn’t the fault of the Web Publisher tool itself, but instead is the result of a few poor design choices in the implementation of the system. One area in particular that tends to suffer is the templates themselves.

Documentum is improving the experience of working with Web Publisher templates by creating a richer in-context editing experience in newer versions of Page Builder, but even for those sites that are still leveraging the tried and true Web Publisher Editor templates, keeping the user experience in mind during development can go a long way towards creating a system that is so straightforward that an in-context editing experience (and the limitations that these solutions still impose) is simply not necessary for a highly usable content authoring system.

Top-Bottom Layout

We all know that when a user encounters a web page they will start reading at the top and work their way down. We also know that users may get lazy on the way and loose interest, so information architects will tell you to place the most important items that you want them to see at the top, and then leave the ancillary information for the end of the page “below the fold”. While this is implicitly obvious, it is an important part of web page design – and it should be a real consideration in template design as well. The template should follow the same top-bottom design as the web page, with elements appearing in the form in the same order that they appear on the website. Primary page information should come first, followed by auxiliary or sidebar content. It may sound obvious, but when a content author is filling out that form, it should feel natural and map seamlessly to the fields on the page to create the best possible user experience.

Preview Format

There are two primary mechanisms for an author to view their content changes from the template editor: Webview and Preview. Webview is by far my recommended approach as this actually launches the current page on the WIP website, and you can see all changes within the full site context (header, footer, navigation, etc.). Preview on the other hand, launches the rendered page within a Web Publisher preview application. The drawback of this is that since it is not launching the actual website, those elements of the page that are determined by the site context could never be accurately portrayed.

The preview format can be very useful, however, in the case of templates that produce content fragments instead of full content pages. Take for instance a template that creates a re-usable callout box that is included on multiple pages throughout a site. The actual published rendition of this template would not be a full page, but instead just an html fragment. This fragment, if viewed directly on the website, would probably look pretty strange without the page that holds it. In these cases, a separate and distinct preview rendition can be created. This rendition – while not actually published to the web site – can show the page snippet in a “mocked up” site context. Seeing the fragment in the correct positioning on a page with the correct styles applied goes a long way to instilling confidence in the user that they have in fact created their content correctly – once again, improving the usability of the CMS.

Figure 2:
Sample use of the Preview format

XML Applications for Metadata Synchronization

XML Applications are certainly not a new feature of Web Publisher, so I have never understood why they are so infrequently utilized. XML Applications at their heart are simple XML documents (though there is a little XML App configuration tool – which in itself could use a lot of work in the usability department – that developers can use to create and modify them) that when applied to a content template can be used to synchronize metadata to content template fields. Why might one use these? Well, let’s take the simplest example of a page element (and one that affects nearly every web page in your repository) – the page title. Every time a user creates a new page, after selecting their template they are immediately launched into the content attributes screen where (after entering the name of their file) they enter the Title for the page. They check edit now (or have it pre-checked if you have taken my suggestion above), click OK, and the template editor is launched. What is the first field at the top of the template editor for them to fill out – you guessed it, Page Title! Having JUST entered this information on the previous screen, without an XML Application in place, they now have to enter this information again! Also consider three months down the road when that page title needs to change – now the user has to make sure to remember to update it in two places instead of just one. While this is certainly doable, the experience does leave something to be desired.

A simple XML Application configured for two-way synchronization of the content’s Title attribute with the template’s Page Title field eliminates this need for double work. Now the experience is that after the user fills out the Title on the attributes page and launches the template editor, the Page Title comes pre-filled! If they make a change to this field in the template editor, the change is also reflected in the metadata – pretty slick. Keep in mind that while this is a simple usability enhancement for the page title, it can (and should) become an integral part of template design when you start to think about content that is reused dynamically on other pages. Take for example a product page – users may specify when creating the content a title, short description, and perhaps even a thumbnail image for the product that is then displayed dynamically on a product listing page. While annoying, without an XML Application, the title could certainly be entered independently in both the metadata and the page content. And perhaps the same would hold true for the short description – but what about the thumbnail image? While there are great widgets available in the template editor for selecting an image, there really isn’t a corresponding concept for the attributes page. Without this synchronization, users would have to manually (and correctly) type out the path to the image they wanted to include. Add a simple XML App and voila, users can simply pick this image when filling out their content template, and it will automatically show up in the metadata – no tricky error prone user experience required.

Reduce Content Author Decisions

Part of the benefit of a powerful content management system like the one Documentum provides is that it allows developers to customize and extend the core framework to meet a particular organization’s needs. These customizations can build proprietary business logic right into the system, really tailoring the tool to fit the business. Unfortunately, with Web Publisher, these sorts of customizations often require a deep level of technical Documentum expertise to implement that many organizations simply don’t have in house.

Here at Blue Fish, we have been growing our Documentum expertise for years, and so we feel very comfortable making these sorts of enhancements. The good news for your organization is that even if you don’t possess these years of experience, Web Publisher provides some great methods for customization that are closer to configuration that true custom software development. These strategies can allow an organization to get some business logic coded into the system with minimal investment – in many cases eliminating the need for an expert content author to make certain decisions by teaching the content management system how something needs to be done and allowing the system to perform the task for them.

Use Folder Mapping to Place Content in Web Cabinets

One very powerful concept that is built into Web Publisher is that of folder mapping. The thought underlying folder mapping is that all pages within the content management system should have a “correct” place where they live in the folder structure. Content shouldn’t be organized haphazardly (like it often is in homegrown systems that have taken on a life of their own after years left to their own devices) – but should instead consist of a logical structure based on the content and the rules of the business. Web Publisher’s folder mapping capability allows a developer to define these business rules of where content should be placed by simply enhancing the rudimentary folder mapping script that is provided out of the box. By having the system place folders in the correct spot, you know that content is always filed away correctly – authors don’t have to remember where content needs to be put, and administrators don’t have to come along behind them to pick things up that have been put in the wrong place.

Assign Workflow Based Upon the Content

I’ve mentioned that Web Publisher comes with a few sample workflows – and we’ve already discussed a simple customization to have Web Publisher automatically launch a workflow if there is only one available to a given user. The usability enhancements with workflow don’t have to end there, however! Nearly every organization I have worked with has some business rule surrounding who should be the approver for a given piece of content. It may be that all product pages go to the product marketing team, or all press releases go to corporate communications – or perhaps it is even simpler and all pages go to the same group of approvers. Whatever the case, there is nearly always some business rule that defines the “correct approver” for a given piece of content – and I always like to say, if you can articulate the business rule, you can code it. In a well designed content management system, the CMS should know quite a bit about that piece of content that you are launching on a workflow – enough that is should be able to determine the proper approver of the content for you. If this business logic can be coded into the workflow, then instead of users having to select the approver (and risk them selecting the wrong one, causing the workflow to be re-routed, and introducing inevitable delays in getting the content out to the web site) the system can automatically route it to the correct person (or even better, group of persons). One less decision for the content authors to make, one more way the system usability is improved.

Conclusion

While Documentum is making a huge investment in improving the content authoring experience, your organization doesn’t need to wait for the next release of Web Publisher to improve usability. With a little forethought and some graceful design tweaks, even today Web Publisher provides the basis for a powerful (and highly usable) web content management solution.

Introduction

The WDK Automated Test Framework was originally created by Documentum for internal testing of WDK components and WDK applications. Although the framework is not an officially supported EMC product, it is made available to partners and developers via the
EMC Developer Network (EDN).
The WDK Automated Test Framework is available in two versions:

• Version 1.0 is compatible with WDK 5.3 SP5 (or later service packs)
• Version 2.0 is compatible with WDK 6.0.

The test framework includes a web-based user interface which allows you to create new tests by recording actions in your custom Webtop or WDK application. The framework is tightly integrated into WDK and runs on any supported WDK browser. It records WDK mouse selection, keyboard shortcuts, right-click menu actions, and supports UCF transfers for importing, viewing, or versioning documents. The user interface allows you to playback recorded tests and verify the behavior of your application without requiring any command-line interactions or editing any configuration files.

Setting up the Framework

The testing framework works with any Webtop or WDK application, but requires that you deploy your application as an unpacked webapp directory, rather than a single WAR file. The unpacked directory is necessary because new tests will be saved as XML and Java files inside the webapp directory. Login to your application and verify that everything is working correctly before you move on.

Download the correct version of the WDK Automated Test Framework from EDN; version 2.0 of the framework is required for Webtop 6.0. Unpack the test framework files into the directory which contains your webapp. The test framework only adds new files, so you should not need to overwrite any existing files. Unpacking the archive is the only step necessary to setup the test framework for testing a standard version Webtop that does not utilize any custom WDK controls.

Once this archive is unpacked, you may access the WDK Automated Test Framework UI at:

    http://{host}:{port} /{webapp}/component/testtool

where {webapp} is the name of the deployed Webtop or WDK application. For example, if you are testing a Webtop instance running on port 8080 of your local machine, then you would point your web browser at:

    http://localhost:8080/webtop/component/testtool

Now that you have verified that the test framework has been correctly deployed, it is time to specify the parameters for recording test cases.

Configuring Recording Parameters

The WDK Automated Test Framework UI generates two artifacts for each recording: an XML configuration file and a Java class. The auto-generated Java class extends com.documentum.web.test.ComponentTestCase and defines a sequence of actions to perform. The XML configuration file is created inside the webapp directory structure under /config/testcase/ and defines values that are used to automatically navigate the WDK application and to verify expected values.

To get started, click the [Test Recorder Launcher] link at the top of the main test framework page to access the Recorder Launcher, and then update the settings which will be used to record your new test case.

Figure 1:
The Test Recorder Launcher

Figure 2:
Test Recorder Configruation Settings

1. The most important elements to be configured are the names and locations of the new XML configuration file and the associated Java class. The “classname” is the name of the new Java class file and “testcaseiId” is the name of the new XML configuration file.
2. The “packagename” value indicates what Java package will be declared in the new auto-generated Java class. The “path” value is a directory inside the webapp directory structure and should match the “packagename”.
3. The “configpath” is the full directory path of the XML configuration file which will be created by the Test Recorder.
4. The “URL” setting controls the starting URL for the test case. By default this will be the root WebTop component, but it is possible to choose a specific component, for example “/component/search”.
5. Set “auto login” to “Yes” and enter valid “user”, “password”, and “docbase” login information. Your session is automatically logged in using this information, these values will not be recorded in the test case, and you can specify a different login when you playback the test.
6. The “clientDataFolderPath” specifies a directory on your local machine which contains files used during testing, for example a sample document to be imported.

Now that you’ve successfully set up the testing framework, it is time to record a test.

Recording a Test

A simple Webtop test has three stages:

1. Start Recording
2. Perform Validation
3. Finish Recording

1. Start Recording

Click the [Record] button located in the bottom right corner of the Test Recorder Launcher to start recording your test. A Recorder Monitor window will be opened and you will immediately be prompted to click [OK] in order to start recording your test actions.

Figure 3:
Click [OK] to start recording

For this introductory example, we will record the steps to perform an import and then modify an attribute. Follow these steps to record your test:

• Navigate into a temporary folder and import a sample document. Note that the standard Webtop UCF client is utilized for the import.
• Use Webtop to view the properties of the imported document, assign a new value to the subject attribute, and save the changes.

Figure 4:
Import a test document

Figure 5:
Modify the Subject Attribute

This sample document is now ready to be validated in the next stage of our test.

2. Perform Validation

An important part of testing is validating that the correct data is displayed to the user. To validate that the test document has the correct attribute value assigned, open the standard Webtop properties viewer and click the “Turn On Inspection Mode” link (located in the bottom right corner of the Recorder Monitor).

Figure 6:
Click – Turn On Inspection Mode

Inspection mode allows you to inspect the values of any WDK component shown by the webapp and record a comparison-check of these values as part of your test. The test will fail if a different value is encountered during playback. When inspection mode is active, a red box is shown as you move the mouse pointer over elements on the properties page. Continuing our example, click the “Subject” attribute field to open a validation box and perform a comparison against the current value.

Figure 7:
Select the “Subject” attribute for inspection

Figure 8:
Dialog when inspecting the “Subject” attribute

You may stop inspection mode by clicking the “Turn Off Inspection Mode” link in the bottom right corner. Close the property inspector and return to the folder browser in Webtop.

Figure 9:
Click – Turn Off Inspection Mode

Let’s validate that the correct version label is displayed in the document list. To do this, enable inspection mode when the document list is displayed:

Figure 10:
Select the version label for validation

Figure 11:
Validate the current version label

As before, click [OK] to record your validation values and click “Turn Off Inspection Mode” to end inspection mode.

3. Finish Recording

Now that we’ve performed a few validation tests, let’s clean up the content repository. Using Webtop, delete the test document you previously created and then click the “Stop Recording” link in the bottom right corner of the Recorder Monitor. This ends your test recording. Now click the [Compile] link to compile and hot-deploy the Java class associated with your new test. Close both the “Compilation” and “Recorder Monitor” windows to get back to the main test framework UI.

Figure 12:
After recording is is finished

Figure 13:
Compile the Java class for your test

Executing the Test

New let’s execute the test we just recorded. Note that new tests are not immediately visible for playback. To run a recently recorded test you must click the [Refresh ConfigService] button in the bottom left corner of the Test Launcher and then switch to view by “testcase”.

Figure 14:
Click [Refresh ConfigService]

Figure 15:
Select View By: testcase

Select the new test in left side tree and then click the right-arrow button to add the test to the current Test Plan. Set “auto login” to “Yes” and enter appropriate authentication information.

Figure 16:
Add a test to execute

Figure 17:
Enter authentication information

Click the [Launch] button to start playback. After playback finishes, the test results viewer shows success or failure for each test, along with timing information. You can click on individual tests to view more information, which is particularly helpful for finding out why a test failed.

Figure 18:
Monitor playback, in progress

Figure 19:
Results, after playback

Figure 20:
Details of an executed test case

Tips & Tricks

Use the [Save State] button to record custom settings.

The settings in the Test Recorder Launcher will be reset to default values when the webapp server is restarted or your browser session ends. The [Save State] button saves the current configuration values out to a file, and you may reload these values anytime using the [Load State] button. Once you establish the recording configuration appropriate for your environment, be sure to save the state. This enables you to quickly reload it later when you want to record a new test case.

The test recorder can be finicky.

Plan out the sequence of actions before you start recording and keep your tests straightforward. You can break one large test into a series of smaller tests and play them back in sequence using a custom testsuite. It will be easier to re-record individual sections of a large test when necessary rather than the entire test.

Pay attention to which version of Webtop is supported.

For example, version 2.0 of the WDK Automated Test Framework (Build Number: 6.0.0.089) only indicates support for Webtop 6.0. We have experienced intermittent errors when using Webtop 6.0 SP1, and these errors were resolved by switching to Webtop 6.0 with no service pack.

Avoid reusing testcase names.

Although the test framework supports hot-deploy of Java classes, updating an existing test can cause Tomcat to throw an exception or otherwise fail to reload the Webtop application. You may need to restart Tomcat, and Tomcat could end up in an invalid state which requires you to manually empty the Tomcat work directory. If you simply append a new version number to any test that is re-recorded, you can avoid issues caused by hot-deployment of existing Java class files.

The recorder may include unintended comparisons.

This issue may require new tests to be manually tweaked before they will playback correctly. For example, we have seen the recorder unexpectedly check for a specific value of the r_access_date attribute. This works fine when the test is first recorded, but fails each time the test is played back. To remove these unintended validations, you must edit the XML configuration file and remove the fragments related to the fields that should not be validated (r_access_date in the above example). Because of the way the auto-generated Java classes are structured, it may even be necessary to modify the Java class in order to remove the validation step. Straightforward tests are less likely to include unintended comparisons.

Do not leave the test framework on a production system.

The test framework allows you edit and hot-deploy Java classes without providing authentication information, which poses a security risk. Make sure you update your deployment process so that you can easily deploy without the framework files or delete all files and directories associated with the test framework after deployment. The WDK Automated Test Framework User Guide provides details on which files must be deleted, as well as an ANT snippet for packaging a WAR file which excludes all of the test framework files.

Advanced Usage

Once you understand how to create and playback a basic test, you may want to move on to more advanced test cases. Here is an overview of the test framework’s more advanced features:

Setup and TearDown

When recording a new testcase, you can designate tests to be run as a setup or teardown step. As with standard JUnit-based Java test cases, the setup is performed before executing your test and the teardown is performed after the test execution completes. This allows you to encapsulate common setup or teardown behaviors into a special test which can be reused in new tests.

Figure 21:
Specify Setup and TearDown tests

Variables

You can define input and output variables which will be passed between tests at runtime. For example, you might specify a variable that holds the r_object_id of a test document so that it can be deleted by a teardown test. When sequencing tests together, the output variables from one test are automatically passed into the next test as input variables. You can specify default values which may be overridden when tests are launched in the UI. Managing variables does add complexity, but it also provides flexibility for creating advanced tests.

Modify a Test Case

Once you have created a useful test, you may want to integrate the java and configuration XML file into your build process. The compiled class may be deployed inside of a jar file, rather than hot-deployed from the source code. The com.documentum.web.test package is provided by the standard webtop.jar, so no additional libraries are required to compile the web test classes. The variable names can also be cleaned up so that it’s easier to re-use tests.

Custom WDK controls

If your application uses custom controls, then you may need to customize the testing framework in order to record tests. The WDK Automated Test Framework Development Guide provides details on how to instrument the test framework to handle custom events and to inspect custom controls. If you do not provide support for a custom control, the test recorder includes an “unknown control” comment message in the generated java test class.

Tracing

If you are already familiar with WDK trace flags, then you may be interested to know that the test framework introduces three new flags for tracing:

• com.documentum.web.test.Trace.TESTRECORDER
  

  This flag traces the recording and production of a test case.

• com.documentum.web.test.Trace.TESTCASE
  

  This flag traces the traversal of test suites and test cases and the execution of test cases.

• com.documentum.web.test.Trace.TESTSTEP
  

  This flag traces the traversal of test steps and the execution of test steps.

I encourage you to visit the EMC Developer Network (EDN) for the latest version of the WDK Automated Test Framework and to access the developer forums.

Conclusion

The WDK test framework provides a robust, reproducible mechanism for performing automated end-to-end WDK testing. Because value inspections can be performed on any WDK control, you can test any functionality
invoked – directly or indirectly – from the UI,
including complex Documentum functionality such as lifecycle state transitions and custom TBO behaviors. Although custom WDK components require additional customization of the test framework, the many benefits of automated WDK testing should easily make up for the additional effort.

Other Resources

• EMC Developer Network (EDN) – Documentum developer community
• WDK Automated Test Framework – Download the framework

Introduction

One of the key requirements of any agile development methodology is
continuous integrations.
In the 9 years I’ve been writing software professionally, I’ve noticed a dramatic difference in the overall
quality of software produced by projects that employed fully automated continuous integrations over projects
that didn’t. When combined with a full suite of automated integration tests, the benefits of continuous integrations are:

• Immediate detection of broken or incompatible code changes.
• Immediate unit and integration testing of all changes.
• Constant availability of a “current” build for testing, demo, and release purposes.
• Increased efficiency, predictability, and quality of project development cycles.

For releases of Documentum prior to Documentum 6 service pack 1 (D6 sp1), EMC’s only officially
supported mechanism for creating Documentum Business Object Framework modules (BOFs) is through
the Documentum Application Builder (DAB) application, and the only supported mechanism for
installing the DocApp archives generated by DAB is through Documentum Application Installer (DAI).
As of D6 sp1, EMC is shipping a new application, Documentum Composer, that can be used to install
and update BOFs. Of these applications, only Documentum Composer can be executed through a
command-line interface, and none support installing or updating BOFs directly from their Java
archive (JAR) files – you must somehow get the BOF into a repository before you can then create
an archive of the BOF to install into another repository. The net result of this is that there
are no officially supported mechanisms for creating, updating, and/or deploying BOF modules in a
fully automated manner from source code to a target repository.

Thankfully, DAB ships with an Ant task, “BOFPackaging”, that we can use to deploy new modules to a
Documentum repository. With just a basic understanding of both the Apache Ant build tool and the
representation of a module within a repository, using the “BOFPackaging” task is pretty intuitive.
For your reference, I’ve included a
sample project
which demonstrates the BOF installation and
update operations in Ant for Service Based Objects (SBOs), Type Based Objects (TBOs), and simple modules.

Overview of BOF Object Schema and Folder Structure

First I want to provide a brief overview of the structure of BOF modules deployed in a repository.

As of DFC 5.3, Documentum BOF code is stored in the docbase repository and dynamically loaded by
DFC as needed by application code. Service Based Objects (SBOs) are hosted in a single “global”
repository, providing the same functionality to multiple repositories, while Type Based Objects
(TBOs) are hosted and configured in the repository that uses them. Additionally, Documentum introduced
the concept of a “simple module”, which is a unit of executable code stored in the repository. Like
TBOs, simple modules are hosted in the repository that uses them.

To support this new module framework, a few new object types have been introduced: “dmc_module”,
“dmc_java_library”, and “dmc_jar”. “dmc_module” is a subtype of “dm_folder” that represents a BOF module.
Its custom attributes identify the module type, its implementation class, the interfaces it implements,
and the modules it depends on. Other attributes provide version information, a description of the
module’s functionality, the minimum Java VM version the module can run under, and the developer’s
contact information. Please see the Documentum “Content Server Object Reference Guide” for a full description
of the attributes available on the “dmc_module” object type.

A “dmc_module” can contain “dmc_java_library” objects and “dmc_jar” objects. The “dmc_java_library” object
type is also a subtype of “dm_folder” and contains “dmc_jar” objects. The “dmc_jar” object type is a subtype
of “dm_document” and represents Java archive (JAR) files in the repository.

Every repository has a System cabinet, which contains a top level folder named Modules.

By convention, modules are installed under this folder according to their type. For example, SBOs are
installed under “/System/Modules/SBO”, and TBOs are installed under “/System/Modules/TBO”. For simple
modules, you can reuse existing or create additional subfolders of “/System/Modules” that match your
module’s types. For example, if the repository uses Java-based evaluation of validation expressions,
the associated modules would appear under “/System/Modules/Validation”. The hierarchy of folders under
“/System/Modules” is referred to as the repository’s “module registry”, or simply its “registry”.

Figure 1:
Subfolders of Modules representing BOF types.

For more information on the structure of deployed modules, see the “BOF Infrastructure” section of the
Documentum Foundation Classes Development Guide.

Using the “BOFPackaging” Ant Task

As mentioned previously, Documentum’s Documentum Application Builder ships with the “BOFPackaging”
Ant task that we can use to deploy or update modules to a repository. After installing Documentum
Application Builder, you will find the Ant task in the “DDS.jar” file under the
“[Documentum install dir]Shared” directory. Simply add the “DDS.jar” file to your Ant project’s
classpath (“dctm.classpath”), and we’re ready to begin.

First we’ll need to register the custom Ant task using Ant’s taskdef task in an init
target so that we can use it within our Ant script.

            

<target name="init" >            
	<taskdef name="BOFPackaging"
		classname="com.documentum.bof.ant.BOFPackaging"
        classpathref="dctm.classpath" />            
</target>


        

Next we’ll need to create a new Ant target and leverage the “BOFPackaging” task to install
our custom module, which in this example will be a TBO. I’ve included sample SBOs and simple
IDfModules in this article’s sample project as well.

            

<!-- install TBO -->
<target name="install-tbo" depends="init" description="Installs the document TBO to repository">            
	<BOFPackaging contactInfo="steve@bluefishgroup.com"
			description="A Blue Fish document"
			generateWebService="false"
			minVMVersion=""
			moduleName="${tbo.module.name}"
			moduleType="TBO">
		<primaryClass name="com.bluefishgroup.sample.tbo.impl.CustomDocument" />
		<addToPackage versionPolicy="minor">
			<jarFileSet downloadable="true" file="${tbo.int.jar.file}" jarType="interface" />
			<jarFileSet downloadable="true" file="${tbo.impl.jar.file}" jarType="implementation" />
			<javaLib name="${tbo.module.name} TBO dependencies" sandbox="false">
				<jarFileSet minVMVersion="" downloadable="true" jarType="implementation" dir="${lib.dir}">
					<include name="jdom-1.0b9.jar" />
				</jarFileSet>
			</javaLib>
		</addToPackage>
	</BOFPackaging>
</target>


        

The following table summarizes the important “BOFPackaging” task attributes and sub-elements that
should be modified to install your own module. For the others values, simply follow the templates
I’ve provided in the sample project.

The following are screenshots of the resulting module as viewed in Webtop.
The module itself resides at “/System/Modules/[module type]/[module name]“,
which in this case is “/System/Modules/TBO/fnd_custom_attr_doc”. The TBO’s
interface and implementation jars are in the module’s root folder:

Figure 2:
TBO interface and implementation JAR files.

The TBO module’s dependencies are stored under the Java library folder
(“dmc_java_library”), “bf_custom_doc TBO dependencies”.

Figure 3:
TBO dependency JAR files.

Conclusion

Using the “BOFPackaging” Ant task, you can easily automate the installation
and update of BOF modules into any repository. This practice will free you
from manual BOF updates, removing what is often the single greatest barrier
to continuous integrations on Documentum projects and resulting in greater
overall efficiency, predictability, and quality of development cycles.

Further Reading

• Article source code – bof_ant_task_samples.zip
• A Practical Guide to the Business Object Framework, by Jason Duke
• Documentum Foundation Classes (DFC) 5.3 Development Guide: Chapter 3 – Using the Business Object Framework (BOF)
• Deploying Modules Outside of a Docapp (EMC Developer Network) – open source Ant task requiring Java 1.5.
• Continuous Integration (Wikipedia)
• Continuous Integration, by Martin Fowler

Introduction

Documentum Lifecycles provide a way to change a document’s properties as it progresses
through its natural course of “life”. For example, when a document is approved, you might
want its approval_date attribute to reflect the date of the signoff. Lifecycles can automate
this type of action easily. Additionally, lifecycles can call custom code that can perform
additional actions, such as sending a notification that the document was approved.

With Documentum version 5.3, EMC introduced a feature called Lifecycle State Extensions
that makes certain types of custom lifecycle actions easier to code and support.
Prior to this version, any custom lifecycle code you wrote would be filled with
hard-coded references and configuration data. But with the addition of
Lifecycle State Extensions, Documentum now provides a place in the repository
to store that type of configuration data.

What are Lifecycle State Extensions?

A lifecycle state extension is a special type of Documentum object relationship (dm_relation)
that allows developers to associate custom attributes with a specific lifecycle state.
State extensions are useful for storing lifecycle state-specific information that is
needed by custom lifecycle state transition code. For example, consider a document
approval lifecycle for architectural plans which requires the notification of various
government agencies as the document enters different lifecycle states. The lists
of agencies to notify can be stored in state extensions that are associated with
the corresponding lifecycle states. Custom code can then access the agency lists
and generate notifications automatically, tag the documents, or perform any other
task that can be written in code.

Documentum provides an out-of-the-box state extension object type: dm_state_extension. This
object type is intended to be subtyped to add as many custom attributes as are needed to a
lifecycle state. The dm_state_extension object type is a subtype of dm_relation and can
therefore be associated with a lifecycle (dm_policy) object via its parent_id attribute.
The state extension type’s state_no attribute associates the extension with a particular
state in the lifecycle. The state extension’s child_id simply points back to itself,
creating a one-way (hanging) relation. The diagram below shows the lifecycle state
extension-related object types.

Figure 1:
Lifecycle State Extension-Related Object Types and Relationships

For more information on the basics of dm_relation objects and their binding rules,
please see the excellent dm_developer article, Object Relationships.

If you’re considering using lifecycle state extensions to store Documentum users or groups then
you may want to check into Documentum alias sets which will likely provide a better solution,
especially for defining ACL accessors and workflow participants. Here’s another excellent
dm_developer article that explains how to use alias sets for these purposes: Using Documentum Alias Sets.

Creating a Custom Lifecycle State Extension Subtype

The dm_state_extension object type is provided out-of-the-box so that it can be subtyped
to add your custom state-specific attributes. You can create a custom subtype of dm_state_extension
by using the Documentum Application Builder (DAB), by executing DQL statements, or if you’re using D6 by using the
new Documentum Composer administrative interface.
The first two methods are described below.

The first step is to determine the size and type of the custom attributes you want to add.
Then you can create the new object subtype using your method of choice, DAB or DQL.
This example shows the creation of a custom subtype that meets the requirements of
the fictional architectural document lifecycle described in the introduction.
The new subtype in this example, becuse it is intended to extend the QCReview lifecycle state (Quality Control Review),
is named qc_lc_extension. This new object type has two custom attributes:
requires_ext_notification (Boolean), and notify_entities (String(128), repeating).

Creating a Custom State Extension Subtype Using Application Builder

1. Using Documentum Application Builder, open the existing docapp that will contain the new lifecycle state extension object type (or create a new DocApp).
2. Select Insert->Object Type from the main menu.
3. Type a name for your custom state extension object type in the “Type name” box.
4. Select the supertype, dm_state_extension.

Figure 2:
Creating a custom subtype of dm_state_extension in DAB

Perform the following steps for each custom attribute that you want to add to the new state extension subtype:

1. Select your new custom state extension object type in the left-hand tree view and then select Insert->Attribute from the main menu.
2. Enter a name for the new custom attribute.
3. Select values for data type, length, and whether the attribute is repeating.

Figure 3:
Adding an attribute to the custom state extension type

4. Finally, close the object type configuration dialogs and check in the new object type.

Note: At this point you should restart the content server to refresh the data dictionary cache.

Creating a Custom State Extension Subtype Using DQL

An alternate method for creating a new subtype of dm_state_extension is by using the
Documentum query language (DQL). Use a DQL tool like Samson or DA to create your custom
state extension object type. The DQL statement below creates a custom state extension
object type named “qc_lc_extension”.

        
CREATE TYPE qc_lc_extension (
requires_ext_notifications BOOLEAN,
notify_entites CHAR(128) REPEATING)
WITH SUPERTYPE dm_state_extension;

    

Note that the above DQL statement is a simplified example that does not configure many
of the available options for the object type’s attributes. For complete information about
creating objects types via DQL, please see the Documentum Content Server DQL Reference,
available from EMC.

Setting the Lifecycle’s extension_type Attribute

A lifecycle’s extension_type attribute must first be set to the name of a state extension
object type before a state extension of that type can be used. This lifecycle attribute is
not exposed via the DAB client and must be set via DQL, the API, or DFC. However, DAB is
still a useful tool for uninstalling the lifecycle, a task which must be executed prior
to setting its extension_type attribute. A lifecycle can utilize only one state extension
object type. Follow the steps below to set a lifecycle (dm_policy) object’s
extension_type attribute.

1. Uninstall the lifecycle using DAB (click the ‘Uninstall’ button).
2. Use a DQL statement similar to the one shown below to set the lifecycle’s extension_type attribute to the name of your custom state extension object type:

            
UPDATE dm_policy OBJECT SET extension_type = 'qc_lc_extension' WHERE object_name = 'Architectural_Doc_LC';

        

Note that “qc_lc_extension” is the name of the custom state extension object type created in section 2 of this article.

3. Validate and re-install the lifecycle using DAB (click the ‘Validate’ and ‘Install’ button).

Creating Instances of a Custom State Extension

Once you’ve created a custom state extension object type, you can create instances of the object
type to extend states in a lifecycle. The section below explains how to configure a state extension’s
attribute values to extend a given lifecycle state.

These are the state extension attributes that must be populated to extend a lifecycle state:

• relation_name – Set this value to ‘BP_STATE_EXTENSION’ for all state extensions.
• state_no – Set this value to the index of the lifecycle state that you want to extend. The index is 0-based (first lifecycle state is index 0).
• parent_id – Set this value to the r_object_id of the lifecycle being extended.
• child_id – Set this value to the r_object_id of the state extension instance. The relationship points back to itself.
• permanent_link – Set this value to TRUE if you want all future versions of the lifecycle to have the same state extensions.

The example DQL statements below show how to create an instance of a custom lifecycle state extension:

1. First create the lifecycle state extension object, populating all of its attributes except for child_id.

            
CREATE qc_lc_extension OBJECT
	SET state_no = <index of state to extend>,
	SET parent_id = '<policy_id>',
	SET relation_name = 'BP_STATE_EXTENSION',
	SET permanent_link = TRUE,
 	SET requires_ext_notifications = TRUE,
	APPEND notify_entites = 'Travis County Commissioner',
	APPEND notify_entites = 'Travis County Tax Assessor',
	APPEND notify_entites = 'City of Austin Zoning Board';

        

2. Next update the state extension’s child id to be its own object id. This is the ID that was returned from the previous CREATE OBJECT query.

            
UPDATE qc_lc_extension OBJECT
	SET child_id = '<id of this state extension>'
	WHERE r_object_id = '<id of this state extension>';

        

Accessing State Extension Attribute Values from Custom Lifecycle Code

State extensions are useful because they can store custom attribute values. These values
can be easily accessed in custom Java DFC code that is called from a lifecycle state transition.
This section shows how to use DFC to retrieve the correct state extension for an object entering
a particular lifecycle state.

Note: For more information on creating custom Java lifecycle modules which can be called from
lifecycle state transitions, see this excellent DM Developer article: Custom Java Lifecycle Modules.

Documentum provides a handy computed lifecycle attribute, _state_extension_obj[[index]], which is
a built-in mechanism for getting a handle to the appropriate state extension for a given lifecycle state.
Computed attributes are read-only and are evaluated by the Content Server at runtime. When you query
a lifecycle for its _state_extension_obj value for a particular state index, it will return the
object_id of the corresponding state extension object (if one exists). For example, consider
the following DFC code:

        
// Get the lifecycle's computed attribute value
// for the state's extension id
String stateExtensionIdString = lifecycle.getString(
			"_state_extension_obj[" + stateIndex + "]");

    

The code above uses the lifecycle’s getString() method to return the computed attribute value
for _state_extension_obj[<state index>]. The return value is a string containing the r_object_id
of the corresponding state extension object instance.

The code example below is a custom implementation of the userPostProcessing() lifecycle state-transition method.
It gets the appropriate lifecycle state extension and then accesses the extension’s
attribute values and logs them out.

        
public void userPostProcessing(IDfSysObject obj, String userName, String targetState) throws DfException {
        
   DfLogger.info(this, "Start userPostProcessing(..)", null, null); 
   DfLogger.info(this, "Lifecycle Name = " + obj.getPolicyName(), null, null);        
   DfLogger.info(this, "Target State = " + targetState, null, null);
        
   // Get the lifecycles's object ID from the transitioning object
   IDfId policyId = obj.getPolicyId();
   DfLogger.info(this, "Lifecycle State Name = " + obj.getCurrentStateName(), null, null); 
        
   int stateIndex = obj.getCurrentState();
   DfLogger.info(this, "Lifecycle State Index = " + stateIndex, null, null); 
        
   IDfSession userSession = obj.getSession();
   IDfPersistentObject lifecycle = userSession.getObject(policyId);
        
   // Get the computed attribute value for the state's extension id
   String stateExtensionIdString = lifecycle.getString("_state_extension_obj[" + stateIndex + "]");       
        
   DfLogger.info(this, "Lifecycle State Extension ID = " + stateExtensionIdString, null, null);        
IDfPersistentObject lifecycleExtension = userSession.getObjectByQualification("qc_lc_extension where    r_object_id = '" + stateExtensionIdString + "'");
        
   boolean requiresExternalNotifications = lifecycleExtension.getBoolean("requires_ext_notifications");
   DfLogger.info(this, "Requires External Notifications = " + requiresExternalNotifications, null, null);
        
   // If this object requires external notifications then
   // log out which entities to notify
   if (requiresExternalNotifications == true) {
      int length = lifecycleExtension.getValueCount("notify_entities");
      String thisNotifyEntity;
      for (int i=0; i<length; ++i) {
         thisNotifyEntity = lifecycleExtension.getRepeatingString("notify_entities", i);
         DfLogger.info(this, "Notify Entity [" +  i + "] : " + thisNotifyEntity, null, null);
      }       	
   }          
}

    

The output log for the code’s execution is shown below.

        
 Start userPostProcessing(..)
 Lifecycle Name = Test_Lifecycle
 Target State = QCReview
 Lifecycle State Name = QCReview
 Lifecycle State Index = 1
 Lifecycle State Extension ID = 3704a32080000100
 Requires External Notifications = true
 Notify Entity [0] : Travis County Commissioner
 Notify Entity [1] : Travis County Tax Assessor
 Notify Entity [2] : City of Austin Zoning Board

    

Notice that the method’s targetState parameter is the friendly string name for the lifecycle state,
not the state’s index which we need in order to get the lifecycle’s computed state_extension_obj[<i>]
attribute. So we instead call the transitioning object’s obj.getCurrentState() method which returns
the state’s integer index. This works fine because we are in the userPostProcessing(..) method,
however the table below shows that the object’s current state may not be what you’d expect within
other lifecycle state transitions methods.

Figure 4:
Comparison of targetState vs. getCurrentState() values for an object transitioning into the QCReview lifecycle state

Wrap-up

Lifecycle state extensions are a little-known feature in Documentum that can be very useful
when you need to maintain custom state-specific attribute values for a lifecycle.
When you use state extensions in your solutions, remember to thoroughly document
their role and maintenance as their presence won’t be immediately obvious in
Documentum’s out-of-the-box administrative tools and applications.

Overview

This article will introduce you to some general strategies for writing flexible markup as well as provide some specific examples that you can take away for your next project. First let’s look at just a few of the modern trends in web design:

• minimize class usage
• minimize tag usage
• semantic markup
• reusable markup

Look familiar? The first three trends are certainly accepted and practiced on a regular basis. Reusable markup, however, is often paid lip service but rarely practiced. What I’m going to show you is how to take the first three trends and set them aside so that we can achieve the fourth.

Reusable markup is not appropriate for every project. Semantic markup, minimal tags, and minimal classes are generally a good thing. However, there are times when flexibility and markup reuse are more important.

Let’s look at a few examples where this is the case. When designing for a content management system (CMS), it is especially important to ensure that your markup is both flexible and robust. Many content management systems use a template based architecture for building pages, and therefore your markup must be capable of supporting diverse designs without significant modification.

Or you may be working in an environment where the HTML cannot be modified, but you have full control of the CSS. In such a situation, it is extremely important that your HTML can accommodate changes. While it’s true that no two projects will use the exact same markup, a little forethought can go a long way when reusing code.

General Strategy

Element ID’s and Clases

Element ID’s differ from classes in a number of ways. First of all, ID’s must be unique in an HTML document. Duplicate ID’s might appear to work at first glance, but your markup won’t validate and you will likely run into problems when working with Javascript and CSS. When you consider some unique components might later be repeated on a page, it’s important to choose your ID’s carefully.

Another consideration is that ID’s carry more weight (10 to 1 ratio) when measuring CSS specificity. Take the following markup and CSS for example:

        
<div class="box">
    <p id="para" class="review">
    ...
    </p>
</div<

    

        
#para {
    width: 400px;
}
    div.box p.review {
    width: 200px;
}

    

The first rule (#para) will be honored even though the second rule (div.box p.review) is declared subsequently and contains more selectors. This is because the ID selector counts as 10 points (an ID), while the second rule counts only as a total of 4 points (2 elements and 2 classes). Of course, you could override the ID selector by using the , but such a technique should be used sparingly.

ID’s have their place in markup – they’re easier to target with Javascript and sometimes their greater specificity makes sense in CSS. Nevertheless, try to limit ID’s to the main page components such as the header, body, aside, and footer. Otherwise, consider using a class just to be safe.

Styling Element Selectors

When designers first began taking advantage of CSS, they used classes almost indiscriminately which resulted in markup only marginally preferable to inline style. As techniques progressed, most designers have switched to using as few classes and ID’s as possible while targeting child elements using descendent selectors. This is generally the preferred method, but something to be careful of when designing for user generated content. Let’s look at the following example which includes a div with a child blockquote:

        
<div id="my-body">
  <blockquote>
    Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Proin in tellus quis eros imperdiet tempor. Morbi et sem lacinia nisi molestie auctor. Mauris rhoncus augue eu eros. Cras porta dui nec odio.
  <blockquote>
</div>

    

It might be tempting to target this using the following:

        
#body blockquote {
  background: #EFEFEF;
  border: 1px solid #CCC;
}

    

While this works fine and keeps your blockquote element free of a class, this method runs the risk that any user generated content will also receive the same styling which may not be your intent. Even though it’s somewhat antithetical to recent markup trends, you might want to be more explicit:

        
<div id="body">
  <blockquote class="review">
    Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Proin in tellus quis eros imperdiet tempor. Morbi et sem lacinia nisi molestie auctor. Mauris rhoncus augue eu eros. Cras porta dui nec odio.
  <blockquote>
</div>

    

        
#body blockquote.review {
  background: #EFEFEF;
  border: 1px solid #CCC;
}

    

This ensures that any user generated blockquote will receive generic style, but your review blockquote will receive your intended style.

Anticipating User Generated Content

Many sites today deal with user generated content in one way or another. CMS’s typically offer rich text editors that offer users the opportunity to add just about any type of content to a page. Whether it’s a content management system, a social network, or an e-commerce site, it’s a good idea to anticipate the nature of the user generated content that your design must accommodate.

First, a good practice to employ is a CSS “reset” file. Since different browsers have different default values for HTML elements, it’s a good idea to normalize all elements before you apply any specific style. A reset will style all common HTML elements with consistent values. For example, one browser might apply a 1em margin for headings, while another browser might apply 2ems. Using a reset file in this case might set the margin to 0ems for all browsers, giving you a common starting point for your custom style. Without a reset file, you might run into some unexpected inconsistencies between browsers.

Next it’s a good idea to set up a base CSS. Your base should create a consistent, simple style that you can use as a starting point for your design. Additionally, this base style can be used for any user generated content which will likely be some type of general rich text. This is why it’s important to be careful when styling your element selectors so that you don’t unintentionally affect any user generated content.

One final technique you can employ when dealing with such content is to take care of max-widths, word-wraps, and overflows.

Setting a max width in the content area of your site helps to prevent the horizontal dimensions from being pushed beyond their intended boundary. This can happen with large images, unbroken character strings, and user generated inline style. For most modern browsers, you can simply set the max-width in CSS. For example, if your content area is 600 pixels wide div with an id of “article”, you could do the following:

        
<div id="article">
  ...
</div>

    

And in your CSS declare:

        
#article {
  max-width: 600px;
}

    

Keep in mind however that this does not work in IE6, so if you are supporting that browser version you’ll need to set max-width using external Javascript or with the following Javascript in CSS:

        
#article {
  width: 600px;
  _width: expression(document.body.clientWidth > 601? "600px" : "auto");
}

    

One nice CSS property found in IE6, IE7, and Safari 3.1 is word-wrap, which was introduced by Microsoft and is now part of the CSS3 working draft. The word-wrap property accepts the value of break-word, which tells the browser to break any text string that reaches the horizontal boundary of its container, and wrap the string to the next line. Taking our “article” div as an example again, we can declare:

        
#article {
  word-wrap: break-word;
}

    

Unfortunately this doesn’t currently work in Firefox or Opera, but is expected to be supported in Firefox 3.1 and an undetermined future version of Opera. There are some workarounds for Firefox, but they are not trivial to implement and the result is less than ideal. As a defensive measure for the browsers, you can set the overflow-x property to either hidden or scroll depending on the your desired effect. This will at least ensure that your dimensions are respected and any content that is too wide for the div will trigger the overflow according to your design needs.

Examples

The following examples will work in all modern browsers1. For convenience I have embedded screen captures in this article, however you can download the working examples in a separate file.

Buttons

One of the most common web site elements is the button. Let’s take a look at a plain button element:

        
<button>Click Me</button>

    

Figure 1:
A plain button with no styling.

Often a plain button will do just fine, but you might want to match the visual design of the site. Or maybe you just want the button to look consistent across different browsers and operating systems. Let’s take a look at the same standard button with some additional styling:

        
<button>Click Me</button>

    

        
button {
  background: #EBF4F8;
  border: 1px solid #B6CCD2;
  color: #0671AB;
  padding: .7em;
}

    

Figure 2:
A button with CSS style.

That’s better! It has a bit more style and appears consistently across browsers and OS’s. But what happens if you don’t know what the final design is going to look like, and another developer is waiting on the markup? You could give that developer what you have now, and go through the whole process again if there are any design changes. Or, you could add add a little more markup and allow yourself some design flexibility.

Let’s say the client decides they prefer something a bit softer with rounded corners. That’s simple enough you think, you can just create a background image and add that to your CSS rule. But that doesn’t allow for variable width content – for that we’ll need something extra. The extra markup, or “hook”, provides an additional element to which we can attach another background image. We’ll employ the “sliding doors” technique and attach one image to our button element and one image to our span element. This will ensure that our button will scale horizontally.

        
<button><span>Welcome to our Intranet</span></button>

    

        
button {
  background: url('button-right.png') right center no-repeat;
  border: none;
  color: #0671AB;
  height: 30px;
  margin-right: 7px;
  padding: 0 7px 0 0;
}
button span {
  background: #EBF4F8 url('button-left.png') left center no-repeat;
  display: block;
  line-height: 15px;
  margin: -2px 0 0 0;
  padding: 7px 0 7px 10px;
}

    

Figure 3:
A buton with markup hooks and background images that scales horizontally.

And now we have a button with rounded corners that scales to any horizontal dimension. By planning ahead and adding your hooks, you can be more certain that future design changes won’t require any markup modifications.

Lists

Another common website element is the list. First let’s take a look at a standard, semantically marked up unordered list of items:

        
<ul>
  <li>Item One</li>
  <li>Item Two</li>
  <li>Item Three</li>
  <li>Item Four</li>
</ul>

    

Figure 4:
A plain unordered list with no styling.

Nice and clean. Now let’s add a bit of style. We’ll take the same markup and give the whole list a blue background and separate each item with a horizontal line using the following CSS:

        
ul {
  background: #EBF4F8;
  border: 1px solid #B6CCD2;
  color: #0671AB;
  padding: .5em 1em;
  width: 300px;
}
ul li {
  border-bottom: 1px solid #B6CCD2;
  background-image: url('bullet.png');
  background-position: center left;
  background-repeat: no-repeat;
  list-style: none;
  padding: .5em 0 .5em .5em;
}

    

Figure 5:
An unordered list with with CSS style.

That’s looking better. But the separator on the last item looks a bit out of place with nothing below it. There are a number of ways we could fix this. We could use Javascript to navigate the DOM and remove the border from the last list item, but that seems a bit heavy-handed. We could add a class called “no-border” to the last item, but that’s not a very semantic name and we may want to do something different in the future that doesn’t make sense when called “no-border”.

Let’s examine the common design problem here. Very often a designer will want to style the first or last item in a list differently – for example, adding or removing a border, changing the background image, etc. Sometimes these changes come late in a projects life cycle, and it may not be so easy to change the markup. To anticipate this, we can add a class called “first” to the first item in the list, and a class called “last” to the last item in the list. Let’s look at the new markup and CSS to achieve this:

        
<ul>
  <li class="first">Item One</li>
  <li>Item Two</li>
  <li>Item Three</li>
  <li class="last">Item Four</li>
</ul>

    

        
ul li {
  border-bottom: 1px solid #B6CCD2;
  background-image: url('bullet.png');
  background-position: center left;
  background-repeat: no-repeat;
  list-style: none;
  padding: .5em 0 .5em .5em;
}
ul li.last {
  border-bottom: none;
}

    

Figure 6:
An unordered list with classes on the first and last items.

Now, we’ve removed the last border. That looks better! While we’ve only used the last class in this case, it’s a good idea to include both just in case. This first/last method can be used for any type of repeating element on a page and be very convenient when you need to fine tune the design.

We can also apply the technique we learned about buttons to provide additional hooks that we might want for each item. In this case we want two images on each list item – one for the bullet and one for an arrow on the right. Here is our final markup and visual design:

        
<ul>
  <li class="first"><span>Item One</span></li>
  <li><span>Item Two</span></li>
  <li><span>Item Three</span></li>
  <li class="last"><span>Item Four</span></li>
</ul>

    

        
ul li {
  border-bottom: 1px solid #B6CCD2;
  background-image: url('bullet.png');
  background-position: center left;
  background-repeat: no-repeat;
  list-style: none;
  padding: .5em 0 .5em .5em;
}
ul li.last {
  border-bottom: none;
}
ul li span {
  display: block;
  background: url('list-arrow.png') center right no-repeat;
}

    

Figure 7:
An unordered list with markup hooks for background images.

Boxes

Boxes are everywhere in web design. They’re used to organize content on the page and highlight information. All modern browsers can handle some level of style without resorting to markup hooks. Let’s look at an example box with some text and simple styling:

        
<div>
  <p>
    Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Proin in tellus quis eros imperdiet tempor. Morbi et sem lacinia nisi molestie auctor. Mauris rhoncus augue eu eros. Cras porta dui nec odio. Donec interdum, tellus id volutpat faucibus, mi lacus tempor tortor, eget pellentesque mauris nunc ut ante. Vivamus nec sapien ac lorem volutpat consequat. Quisque dolor. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Nullam lacinia erat. Maecenas vestibulum.
  </p>
</div>

    

        
div {
  background: #EBF4F8;
  border: 3px solid #B6CCD2;
  color: #333;
  padding: 1em;
  width: 400px;
}

    

Figure 8:
A box with CSS style.

That looks nice and clean. But once again let’s look at the challenge presented by rounded corners, as it’s such a common design technique. Just like our button above, we could use a single background image and fix the height and width of the div to match. Unfortunately that doesn’t work with dynamic content and therefore is not very flexible. We could create two images – one for the top half and one for the bottom half, but that doesn’t scale horizontally. Let’s be sure our box can handle anything and allow it to scale both horizontally and vertically.

Because a single HTML element can only have one background image attached, we’ll need five elements to achieve our box – one element for the container and one element for each corner. We’ll nest them like this:

        
<div>
  <div class="upper-left">
    <div class="upper-right">
      <div class="lower-left">
        <div class="lower-right">
          <p>
            Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Proin in tellus quis eros imperdiet tempor. Morbi et sem lacinia nisi molestie auctor. Mauris rhoncus augue eu eros. Cras porta dui nec odio. Donec interdum, tellus id volutpat faucibus, mi lacus tempor tortor, eget pellentesque mauris nunc ut ante. Vivamus nec sapien ac lorem volutpat consequat. Quisque dolor. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Nullam lacinia erat. Maecenas vestibulum.
          </p>
        </div>
      </div>
    </div>
  </div>
</div>

    

Now we’ll add some CSS to make it all work. Each corner element will have its own background image while the container has a border to make it all appear seamless.

        
div {
  background-color: #EBF4F8;
  border: 3px solid #B6CCD2;
  width: 426px;
}
div .upper-left {
  background: url('box-upper-left.png') left top no-repeat;
  margin: -3px 0 0 -3px;
}
div .upper-right {
  background: url('box-upper-right.png') right top no-repeat;
  margin: 0 -3px 0 0;
}
div .lower-left {
  background: url('box-lower-left.png') left bottom no-repeat;
  margin: 0 0 -3px 0;
}
div .lower-right {
  background: url('box-lower-right.png') right bottom no-repeat;
  margin: 0 0 -3px 0;
  padding: 1em;
}

    

Figure 9:
A box with markup hooks for background images to create rounded corners and scalable dimensions.

And there is our fully scalable box with rounded corners. It works, but that’s a lot of effort just to get rounded corners. And it gets trickier if your box will appear on different backgrounds or if you need a gradient inside the box. There’s got to be an easier way, right?

Looking Forward

Many of the strategies discussed in this article should be useful for years to come. Anticipating user generated content, selecting ID’s and classes appropriately, and generally thinking through your entire design are approaches that are not going to disappear. However, recent advances in CSS suggest that many of the markup hooks described above will not be necessary in the coming years. In fact, the current CSS3 working draft obviates the need for all them. Safari 3.1, Firefox 3, and Opera 9.5 all have some level of support for CSS3 and promise more to come. Even Microsoft has begun introducing some CSS3 support in their IE8 beta.

Let’s take another look at the examples above and see what we can do with a browser that supports some of the CSS3 recommendations.

Buttons

The new border-radius property allows rounded corners without the need for images or markup hooks. You can use this today with experimental support in Safari 3.1 and Firefox 3.

        
<button>Click Me</button>

    

        
button {
  border: 1px solid #B6CCD2;
  background-color: #EBF4F8;
  color: #0671AB;
  padding: .7em;
  -moz-border-radius: 10px;
  -webkit-border-radius: 10px;
}

    

Figure 10:
A button with CSS3 border-radius applied.

Lists

If you’ll recall in our example above, we had to use markup hooks and extra classes to achieve or design goal. Now let’s see what we can do with a plain semantic list.

        
<ul>
  <li>Item One</li>
  <li>Item Two</li>
  <li>Item Three</li>
  <li>Item Four</li>
</ul>

    

        
ul {
  background: #EBF4F8;
  border: 1px solid #B6CCD2;
  color: #0671AB;
  padding: .5em 1em;
  width: 300px;
}
ul li {
  border-bottom: 1px solid #B6CCD2;
  background-image: url('bullet.png'), url('list-arrow.png');
  background-position: left center, right center;
  background-repeat: no-repeat;
  list-style: none;
  padding: .5em 0 .5em .5em;
}
ul li:last-child {
  border: none;
}

    

Figure 11:
An unordered list with CSS last-child style.

Looks great, and no extra spans or classes were needed! Using the last-child pseudo element supported by most modern browsers (CSS2), and the multiple background image support of Safari 3.1 (CSS3) we can achieve the same effect.

Boxes

Now we’ll take a look at our last example, the scalable box with rounded corners. Remember all the hooks and images we needed before? Here’s the only markup we need if we’re using CSS3:

        
<div>
  <p>Phasellus adipiscing iaculis nisl. Phasellus ultricies odio a lacus. Proin sollicitudin adipiscing velit. Donec tempor tortor at elit scelerisque tincidunt. Suspendisse potenti. Donec egestas est ut elit.
  </p>
</div>

    

        
    div {
      background: #EBF4F8;
      border: 3px solid #B6CCD2;
      color: #333;
      padding: 1em;
      width: 400px;
      -moz-border-radius: 10px;
      -webkit-border-radius: 10px;
    }

    

Figure 12:
A box with CSS3 border-radius style.

Once again we use the experimental border-radius property and our rounded, scalable box is now simple to create.

As you can see, it’s already possible to eliminate all of the aforementioned hooks with a shipping browser (Safari 3.1). There are other CSS features coming soon that I didn’t even touch on in this article such as animations, transformations, variables, and more. It will take some time, but all the major browsers will eventually get there and the users will be sure to follow. Until then, a little extra markup will help you on your way.

Conclusion

While the benefits of semantic and minimal markup are great, writing reusable code can be more important in some situations. If you’re creating page templates for a CMS or working in an environment where the markup is not always available for modification, you may want to consider adding some extra hooks. The defensive CSS strategies are always good to keep in mind when dealing with user generated content. Whether or not these techniques are appropriate for your next project, is something that you’ll have to decide.

Introduction

Endeca has thrived by continually updating their flagship search offering, the Information Access Platform, to keep ahead of the game in the arena of world-class search and navigation engines. And the upcoming version 6.0 is no exception.

This article will describe some of the most notable enhancements in version 6.0 that you may be interested in if you are currently using a previous version of Endeca’s Information Access Platform (IAP). It assumes a basic familiarity with the Endeca concepts and components.

XQuery

One of the most fundamental and interesting new pieces of functionality in Endeca IAP 6.0 is the inclusion of an XQuery evaluator in the MDEX Engine.

XQuery (also sometimes called XML Query) is a W3C standard language that is designed to easily query across XML data. It utilizes the XPath standard that is also used in XSLT, but combines that with a full-featured language syntax to provide extensibility in data processing. This gives XQuery the ability to perform complex operations on XML data that would have previously required code in other languages leveraging purpose-built libraries for processing XML. The technical details about XQuery are beyond the scope of this article, but there is a wealth of information about it online. The XQuery 1.0 Specification (http://www.w3.org/TR/xquery/) and this XQuery Tutorial (http://www.w3schools.com/xquery/) are good places to start.

Endeca chose to implement an XQuery evaluator as part of its new MDEX engine to provide the ability to write extensible code that is deployed in the engine instead of in the application, and can be bound to web-service interfaces exposed by the engine. This has a number of benefits.

Firstly, much of the code that was previously written in the user interface layer or application layer that dealt with building queries to the engine, parsing results, re-querying, etc. can now be pushed into the engine layer of the application. This means that the other layers of the application can focus on business logic and presenting results to the end user, instead of needing to be concerned with the specifics of working with the engine to manage queries and results.

Providing the ability to bind XQuery functionality to web-services also opens the platform up to the option of using additional programming languages on the client side. Previously, writing code to interface with the engine in anything besides .NET and Java was potentially costly and error-prone, but now, any language that is able to invoke a web service can be used to build the client portion of an Endeca-driven application. These web services can be built either as SOAP-style services, or as simpler REST-style services, and virtually all modern languages have mature frameworks for handling both.

This ‘engine-layer’ code can also now be shared across multiple applications or clients, even those written in multiple languages. A Java-based web application and a native C++ Windows application could both query the same engine, and both take advantage of the shared code that now resides in the engine itself.

Pushing this functionality into the engine layer can have significant performance benefits as well. Consider an application where there are multiple, prioritized classes of data, and the application is to search the first class of data, and only if there are no results in the first, then search the second, and so on. If this code were written in the application layer, there could be several sets of communications with the engine before any results are ultimately presented to the user. However, if this same ‘cascading search’ functionality were implemented in the engine using XQuery, there would only be one request to the engine, and one response, regardless of how many of the underlying classes of data needed to be searched.

Another benefit this brings is the potential for reuse, not just within a single organization, but across the community of Endeca users. Code that is written specifically to reside in the MDEX engine and is not tied to a particular application or user interface is much more likely to be reusable in a broader context. Endeca is already encouraging users of the Early Access release of IAP 6.0 to contribute reusable XQuery code to EDeN, the Endeca Developer Network.

Performance and Scale Enhancements

With the 6.0 version of the IAP, Endeca is continuing its tradition of continual performance improvements from version to version. With this release, special focus was given to some of the more complex edge cases of search and navigation.

One area of focus is navigable dimensions with extremely deep taxonomic hierarchies. This is likely to be an especially important improvement for users who have content management systems with deep and complex folder structures or other hierarchical organization schemes.

Another area of focus is on dimensions with multi-valued attributes. Again, rich content management systems that may have, for instance, many keyword tags, categories, or authors, on a product or document are likely to see more gain from these changes than simpler, flatter data sets.

Actual performance improvements for a particular application will vary, but can be very significant. In internal testing, Endeca has observed gains in throughput (queries per second) of 30 – 60% over the current release 5.1 product on many benchmark tests.

Large scale implementations will benefit from major enhancements to management of index data on disk and in memory. This can be especially impactful in cases where extremely rich records are used for findability, but the many of the details of that record are not required to display to the user. For instance, in a large chemical or parts catalog, an individual component may be described by hundreds of parameters which are potentially important in narrowing a search to that particular item. However, once found, the most important details such as price and SKU may be simple and straightforward.

These scalability enhancements contribute to the throughput gains mentioned earlier. Other key metrics also show notable improvements. One example is large-scale engine process startup times, where Endeca is observing dramatic reductions of 70 – 95% relative to the 5.1 release.

Platform Support

With the 6.0 release, Endeca is completing its transition from 32-bit to the modern 64-bit server architecture. Optimizations have been made to the full 64-bit versions of the Indexer and MDEX Engine for improved performance on 64-bit platforms. Forge remains a 32-bit application, though the gains in moving to 64-bit would be slight, as it is more often bottlenecked by disk contention or source data systems than processor and memory resources.

The 6.0 release will support the 64-bit versions of Windows, Sun Solaris, and Red Hat Enterprise Linux.

Merchandising Workbench

One new product that is now available alongside IAP 6.0 is Merchandising Workbench. Merchandising Workbench provides additional functionality on top of Endeca’s Web Studio that is designed specifically for merchandisers. This functionality removes the complexity of managing business rules from the task of the merchandiser, and provides a simplified interface for performing the most common merchandising tasks, including a workflow specifically designed for streamlining the process of creating effective product landing pages.

CAS v1.2

The latest version of CAS (Content Acquisition System) provides several enhancements to crawling content management systems, including a graphical interface for managing CMS crawls and the ability to run multiple CMS crawls simultaneously, which can be a huge boon to systems that need to pull data from multiple CMSs.

RAD Framework .NET v1.1

The new version of the Rapid Application Development Framework for ASP .NET has a number of bug fixes and enhancements. This framework is intended to jumpstart the process of building a .NET-based Endeca application, and provides code for many common components that can be easily reused. A Java RAD framework is under development, and is currently in the beta stage.

Availability

The Early Access release of IAP 6.0 was made available in May 2008. The EA release is intended for production use by select customers who see immediate, compelling value. The EA release is also available as a technology preview for current customers. Endeca is currently projecting general availability of the 6.0 release later in 2008. For more detailed information, I encourage you to check out EDeN at http://eden.endeca.com/

I hope this article has provided some helpful insights into the latest version of Endeca’s flagship product, and how some of its new features may be relevant and important for you. If you have any questions or comments, I encourage you to comment on this article.

Introduction

A core component of the Migration Workbench product (aka, Workbench) is its embedded rules engine, which provides a language for expressing transformations on migration objects. To achieve the transformations often required in migration efforts, a user of Workbench will author migration rules that are intepreted and executed by the embedded rules engine. These rules are authored in an expressive rule language which users can use to read and write properties of source and target objects involved in the migration. The Workbench also permits users to introduce custom utility methods written in Java that can be called from the context of rules. Custom utility methods are helpful for several reasons, which include:

1. Common transformation code can be encapsulated in utility methods and used across rules files without having to duplicate transformation logic. For example, a common transformation rule may be to convert a comma-delimited first name (e.g, Smith, John) to its typical colloquial version (e.g., John Smith) — this would be perfect behavior to encapsulate in a Workbench custom utility method.
2. In some cases the MWB rules language is not expressive enough to achieve all of a migration’s transformation requirements. Custom utility methods can be used in these special cases.

Prerequisites

This document assumes that the reader has the MWB product, including the IDE (Integrated Development Environment) plugins, installed. It is also assumed that the reader can create a valid MWB migration project and perform a batch execution using that project.

A Simple Example

To demonstrate how to introduce and employ a utility method in a migration project, we will write an extremely simple utility method. The method will print a Hello message to standard out, which will appear in the Workbench console during execution. The Hello message will be followed by a string representation of a target document that it receives as a parameter:

1. In the Workbench IDE, make sure that you are in the Java perspective (See Referenced Images 1.)
2. Create a new source folder. Give it any name you like. For example, “src/custom” (See Referenced Images 2.)
3. Create a new Java package to contain your class. You can give it any name you like but for this example we will use “utility.sample.hello” (See Referenced Images 3.)
4. Create a new Java class in this package called “HelloUtil” (See Referenced Images 4.)
5. Copy and paste the following code as the implementation for this class:

                       
   package utility.sample.hello;
   
   import com.bluefishgroup.mwb.object.IMObject;
   
   public final class HelloUtil {
   
   	public static void printHello(IMObject object) {
   		System.out.println("Hello " + object);
   	}
   }
   
                   

   We now have an implementation for our utility method. Note that the printHello() method is a public static method. This allows us to access the utility method in a static manner from the rules file. We’ll demonstrate the invocation of this method later. Note that the utility method accepts an IMObject instance as a parameter. The source and target attributes on our document classes implement this interface; we’ll see how to invoke the method with a valid argument later in this example.

   But first let’s make sure that this class is compiled into the proper location in the MAR/project so that the rules engine can access its implementation at runtime.

6. Ensure that the target output folder for compiled classes is the META-INF/classes directory of the MAR/project. Right click the project folder and select Properties from the pop-up menu. Select “Java Build Path” from the navigation tree and then select the “Source” tab. You should see the following dialog; ensure that the Default output folder is the project name followed by “/META-INF/classes”:
   

   
   Figure 1:
   The default output folder for a Workbench project.

   At this point our utility method should be available when writing our rules. Open any rules file in the project and add the following import statement to the project:

                       
   import utility.sample.hello.HelloUtil
   
                   

   Note that for any custom utility class that we wish to use in a rules file, a corresponding import statement needs to appear at the top of that file. Now we can write a rule in the rules file that invokes the utility method:

                       
   rule "Print hello message for every document"
   when
   	document : Document( )
   then
   	HelloUtil.printHello( document.target );
   end
   
                   

   This rule matches every Document available in the rules engine during execution, then invokes our custom printHello() method to print a message for the target side of each Document. Recall that the printHello() method takes an IMObject instance as a parameter. For Document instances in the rules engine the source and target attributes are valid values to pass in as IMObjects.

   When you are finished you should have a rules file that looks roughly like:

                       
   package com.bluefishgroup.mwb.external.vocabulary.acme
   
   import com.bluefishgroup.mwb.external.vocabulary.trial.mfacade.Document
   import com.bluefishgroup.mwb.external.vocabulary.trial.mfacade.Document.Source
   import com.bluefishgroup.mwb.external.vocabulary.trial.mfacade.Document.Target
   import com.bluefishgroup.mwb.external.vocabulary.trial.mfacade.Document.Custom
   
   import utility.sample.hello.HelloUtil
   
   rule "Print hello message for every document"
   when
   	document : Document()
   then
   	HelloUtil.printHello(document.target);
   end
   
                   

   We are now finished and can execute a batch that references this rules file. The console should display our debug message for each migration Document. The output we will see should look something like:

                       
   
   Hello MObject(091c963880020a0c,[FIFTH_DOCUMENT.txt])
   Hello MObject(091c963880020a0f,[FIRST_DOCUMENT.txt])
   Hello MObject(091c963880020a12,[FOURTH_DOCUMENT.txt])
   Hello MObject(091c963880020a06,[FIFTH_DOCUMENT.txt])
   ...
   
                   

Using Third-Party Libraries In Rules

The previous example demonstrated a simple but perhaps not very useful example of adding custom behavior to rules transformations via custom utility methods. In more realistic use cases we may need to employ more complex Java functionality than just message printing. It is likely that a reasonably complex transformation would need to invoke a third-party or otherwise external Java library, for example. In this section we will show how to invoke Java behavior that is contained inside a JAR file from the context of rules:

1. Add your third-party JAR to the “lib” folder in your MAR/project. For this example, we will use the Apache commons-lang library. After adding the JAR to the lib folder, make sure to add the JAR to your project’s build path. This step will allow us to use auto-completion and rule validation features when using methods in commons-lang:
   

   
   Figure 2:
   Adding a third-party library to a Workbench project’s build path.

2. Now we can use commons-lang utility methods in a rules file. The IDE will even provide us with auto-completion:
   

   
   Figure 3:
   Example of auto-completion for a utility method in a rules file.

At runtime the MWB loads classes from JARs in the “lib” directory and makes them available to the rules engine so we should see our rules that use 3rd-party libraries executed successfully.

Invoking Third-Party Libraries From Custom Utility Methods

In the previous example we invoked a 3rd-party library’s utility method directly from a rules file. Of course we could have also written a custom utility method that invoked behavior in one or more 3rd-party libraries — in which case we would still need to provide the 3rd-party library JARs in the project’s “lib” directory to ensure its classes were available at runtime.

Performance Concerns

One concern in developing and invoking custom utility methods from rules is the potential negative impact on rules engine performance. A method invoked in the right-hand side of a rule will be potentially invoked several times within a rules engine session. If care is not taken to make custom utility methods fast, the execution time of the MWB rules engine could grow with the introduction of a utility method.

Conclusion

Custom utility methods offer a powerful way to develop and encapsulate behavior that can be used from within Migration Workbench transformation rules. These utility methods can employ third-party libraries to achieve transformation or other logic (logging, for example) not easily achieved in the rules language. The usage of custom utility methods can have a negative impact on rules engine performance, but when care is taken they offer a powerful means of introducing extra behavior in MWB rules.

Referenced Images

1. Use the Window menu to switch to the Java perspective.
   

   
   Figure 4

2. While in the Java perspective, right-click the MWB project and choose New -> Source Folder. At the prompt, type the name of the source folder. For example, “src/custom”.
   

   
   Figure 5

   

   
   Figure 6

3. Use Right+Click, New -> Package to create a new package. Then fill in the dialog that will pop-up:
   

   
   Figure 7

   

   
   Figure 8

4. Right+click the Java package and select New -> Class. Fill in the class details and select OK. The new class should appear in the selected Java package.

Read more in Planning for the Data Migration that’s Lurking in Your Content Migration.

Introduction

Migrating enterprise content from one system to another requires undertaking two separate yet intertwined migration efforts: moving the actual content files and moving the related data. When planning a migration initiative, it’s easy to overlook the data migration that is implied in the process. Yet the majority of problems encountered during content migrations are actually problems with the data, which includes the obvious metadata as well as the not so obvious data components such as ownership, security and permissions, folders, workflow and lifecycle state, version stack, and audit history. This article describes the types of problems you can expect to see when moving the content’s data and explores options for planning and performing the migration of the data that is related to your content.

How Big is the Iceberg?

Upon first glance, migrating content from one repository into another – or even from a filesystem into a repository – seems simple. There are tools that allow you to map the attributes from the source system to their corresponding attributes in the target system, then push a button and watch the content move. However, in practice it’s rarely this simple. To understand the potential pitfalls in even the simplest content migrations, let’s turn to the familiar iceberg metaphor.

Figure 1:
The Tip of the Iceberg

The visible portion of the iceberg represents the attributes that can be mapped in a straightforward fashion. For example, object_name in the source system might map to object_name in the target system, while doc_no in the source system might map to document_number in the target. But lurking below the waterline is a continent of challenges and issues – issues related to attribute transformations, attributes requiring enrichment, and exceptions. We can cast these challenges as different strata in the iceberg; the deeper you go, the more complex or challenging the issues become.

Figure 2:
The Full Iceberg

Dealing with the Iceberg

Each layer of the iceberg presents its own unique challenges and must be planned for differently. The one common theme is that your approach and plan must anticipate that you will find problems once you begin looking at migrated content and data in the target system.

Straightforward Attribute Mappings

Many of the attributes in your source repository will map seamlessly to your target repository. Since these mappings are, well – straightforward, this part of your migration specification usually gets the most attention. Your intuition for how to plan this part of the migration will probably serve you well – it boils down to doing a little bit of analysis as you plan.

At Blue Fish we’ve developed a set of best practices that has served us well across a broad range of content migration efforts. Start with your target repository’s object model and understand where each attribute will come from. If your source system is a Documentum docbase or another formal content repository, it’s likely that many of the attribute values will map directly from attributes in that system. If your source is a filesystem, there will be fewer explicit attributes to pull from. In either case, some values may require transforming, which we will discuss in the next section.

Next, analyze your source documents. Does your legacy system have all the metadata that the target repository requires? If not, we deal with those missing values in the next two sections.

Assess the quality of the metadata. Do the values exist? Are they complete? Are they valid? Are they in the expected format? For example, are all dates “mm/dd/yyyy”, or are some “dd/mm/yyyy” or “dd-mm-yyyy” or “dd-MMM-yy” or “Month dd, yyyy” or “mm/dd/yyyy hh:mm:ss” and so on. Were users consistent in how they tagged and classified documents? Did the legacy system enforce consistency? An invalid or incomplete value moved into a non-critical target attribute may be a nuisance, but a missing or invalid value moved into a mandatory target attribute will cause your migration to fail.

Things to watch out for:

• A migration is a great opportunity to do some housekeeping to ensure that you migrate only relevant content. How will you identify and exclude documents that you don’t want to migrate? For example, will you migrate only major versions (v1.0, 2.0, 3.0…) and leave behind the minor revisions (v1.1, 1.2, 2.1, 2.2…)? Should you screen out documents for products no longer in service? Will you archive documents more than 7 years old instead of migrating them to the new system?
• Having analyzed the quality of your metadata, don’t place too much reliance on that analysis (more on this later). You must plan to encounter exceptions resulting from inconsistent or incomplete metadata values and be prepared to handle them when they arise.

Attribute Transformations

The first submerged layer of the iceberg consists of the complex mappings we call attribute transformations. For example, if in the source system the author_name is “John Smith” but it must be “Smith, John” in the target, the value must be transformed during the migration into the new format.

To use a transformation, you must be able to programmatically derive the value of the target attribute from information in the source repository, such as the document’s folder path, owner, or other attributes. Many transformations are expressed in terms of conditional statements. For example, you may need to set an archived attribute to true only if the document is more than 3 years old.

Some common types of transformations we’ve seen include:

• String manipulation – repeating the example above, in the source system the author_name is “John Smith” but it must be “Smith, John” in the target
• Date arithmetic – take the creation_date attribute in the source, add 180 days to it and set it into a target attribute called review_date
• Combining attributes – the first_name and last_name in the source system are combined to create the target value called full_name
• Splitting attributes – splitting street address, city, state, and zip code out of a single address attribute in the source system and putting them into individual target attributes
• Parsing – starting with a date field with the format “01/31/98″, pull out the year and reconstruct the 4 digit year of 1998
• Deriving attribute values from other attributes – if a document is in a source system folder called “Invoices”, then the doc_type is “invoice”; if the file name contains the string “-APP”, then the doc_status is “APPROVED”

Planning for this part of the iceberg is similar to planning for the straightforward mappings. In fact, you can often do the analysis for transformations at the same time you are analyzing for the straightforward mappings. The main reason that we think of transformations as more challenging is that they can frequently get out of control, adding a lot of complexity to what you may have thought was a simple migration. It’s not uncommon to build up a transformation that evaluates the values of three or more source attributes. In a recent migration, I recently had a transformation that required evaluating three document level attributes and one folder level attribute to determine whether a document’s status was “draft” or “released” in the target system.

Things to watch out for:

• Inconsistently applied naming conventions – if you are deriving an attribute value from the name of the content file, don’t expect 100% compliance with the stated naming convention. If human beings created the file names, there will be exceptions.
• Attributes that were not used for their intended purpose – some of the more interesting exceptions arise from well-intended users hijacking attributes and using them for something different on “just these 5 documents” or for “just this one customer”.
• If you use the folder path as part of your transformation, how will you deal with documents that are linked into multiple folders?
• Mappings that require complex transformations (if you have these cases, you need to know before you select your migration tool, since most tools can’t do this sort of thing). For example, version stacks that need to be joined or split, separate documents in the source that need to be treated as renditions of the same document in the target, and attribute values from the current version needing to be applied to all versions in the version stack.

Enriched Attributes

After you deal with attribute transformations, you’ll face an even greater challenge: target attributes whose values cannot be mapped or programmatically derived from information in the source repository. Typically, this is the case when either the source system has no corresponding attribute or the source attribute was not mandatory (and therefore not populated). Enrichment is basically a last resort when there is no programmatic means for providing the target attribute values.

These attributes require users to “enrich” the data in a manual or semi-automated fashion, supplying the missing data based on their knowledge of the content, the system, and the related business processes. For example, if a document is missing its title, the author of that document must go back and add a title before the document can be migrated.

When planning enrichment activities, give some thought to who is best suited to do the work. Since enrichment activities require users with an understanding of the business context, plan to have the right business user(s) enrich the right set of attributes. You can divide the content into enrichment groups and route each group to the user(s) with the relevant knowledge. This is especially useful if the schedule is tight since it allows you to parallelize the enrichment process, Or if knowledge about specific attribute values is segmented across departments, you can slice the enrichment process and have different users enrich different attributes. And make sure you work with the appropriate managers to allocate time for the selected business users.

Even though knowledgeable users are providing the missing values, don’t expect them to get it right on the first try. Build in safeguards. For example, if you are using an Excel spreadsheet to capture missing attribute values, use drop down lists to ensure consistency and completeness and avoid data entry mistakes. If a drop down list must match the data dictionary in the target system, make sure you export the latest version of the list before starting enrichment, and periodically refresh the list if needed.

In addition to drop down lists, provide a default value so that the attribute value cannot end up blank. As a last resort you could default to the object owner’s name for future resolution. Have the enrichment process provide immediate feedback for potential errors. For example, use the built-in data validation capabilities of Excel to prompt the user to correct errors immediately. And lastly, be prepared to validate or otherwise check the results of enrichment activities.

To determine the duration of the expected enrichment activities, run a sample set through your enrichment process. Since attribute enrichment is basically a linear process, you can safely project a completion date based on the time it took to complete the sample set, after adding in a reasonable amount of schedule contingency.

Don’t try to use a statistical sample (again, more on this later). Instead, query your source repository to determine exactly which documents are missing necessary attribute values. You’ll need a separate query for each attribute. In a Documentum repository, an example would be “select * from dm_document (all) where title is nullstring”.

Finally, select a migration tool that directly consumes the enriched attribute values. This speeds up the process and avoids translation or transcription errors.

Things to watch out for:

• Don’t forget about older versions – just because the current version of a document has all its attributes doesn’t mean that the older versions do. These older versions will cause the migration to fail if you don’t have a way to deal with them.
• If your migration tool uses a staging area between the source and the target, you might be tempted to enrich the data at this point. Beware, because if you have to export your data again, you’ll overwrite your previously enriched attributes.
• Business users are busy people and will not be thrilled with having to enrich their data, especially their old data they don’t use anymore. Plan your diplomatic request for their time and allocate more time than you think it will take.

Exceptions

Now we are into the deepest and darkest region of the iceberg. This is where most migrations go wrong, because the approach failed to expect the unexpected. Most exceptions won’t be discovered until you start migrating data into the target repository. Identifying and addressing them is where you will spend most of your time, so you’d best plan for it.

The following list of examples is by no means complete but will give you a flavor of the variety and severity of exceptions you can expect to encounter.

• Documents with missing or incomplete attribute values
• Unexpected formatting, such as inconsistent capitalization and white space at the beginning of a string
• Attributes with values that don’t exist in the target repository, such as owners who have left the company or IDs for products that have been discontinued
• Documents whose file names don’t conform to naming conventions or otherwise break your attribute transformations
• Unexpected data, such as a company name in the author field
• Immutable objects – by default, objects in a Documentum repository become immutable when they are not the current version.
• Interdependencies and potential circular relationships with users, groups, ACLs, relations, and virtual documents – each object can have dependencies on other objects in the source repository as well as the target repository. For example, if you are migrating the document owner as part of the document object’s metadata, that owner needs to exist as a valid user in the target system before you try to import the document.
• Migrating the workflow state of a document that is somewhere between the initial and final workflow states – usually these intermediate states are “non-attachable”, meaning you cannot set the document to this intermediate state after importing it.
• Old content files whose file format or version is not supported by the target system

There are four ways of dealing with exceptions:

1. Correct the source data – Sometimes you can resolve an exception by updating the source data. If a document’s name doesn’t conform to the naming convention, update it so that it does. This approach may not always work, since the source system likely contains immutable objects or regulated content that can’t be changed without an approval process.
2. Update the migration specification – if your migration tool supports conditional transformations, you may be able to resolve some exceptions by updating the migration specification to treat the document in question differently. For example, if a document does not conform to the naming convention, you can encode the migration tool to detect this case and apply a different naming convention. The danger of this approach is that your migration specification can quickly get out of hand and will soon have so many exceptions encoded into it that debugging and maintenance become a nightmare.
3. Further enrich the metadata – if your migration tool supports external enrichment, you may be able to resolve your exceptions by manually specifying the exact values for each exception. This is likely the most time consuming approach.
4. Do nothing – at some point, the effort it takes to handle an exception may not be worth it. An example might be a document is no longer in active use and is just being migrated for archive purposes. If it has several exceptions or requires enrichment, you may decide not to migrate the document after all, or to migrate it into a “parking lot” folder to be dealt with at a later time.

One of the things that makes exception handling particularly hard is that some exceptions will be “hiding behind” other exceptions. What I mean by this is that a document might have two exceptions, but you won’t know about the second exception until you resolve the first exception. For example, you may find a set of documents whose document_type attribute is not populated. To resolve this exception, you decide to change your migration specification, populating the document_type based on the naming convention of the folder the document is linked into. You then re-run your migration, only to discover that some folders don’t conform to the naming convention.

We’ve found the most effective way of dealing with exceptions is to find them early and fix them iteratively. Start performing migrations right away, perhaps into a test environment. Then run your migration again – you’ll find new exceptions that need to be resolved. Plan on having many, many iterations. Most migration tools can’t process documents incrementally, so either select one that can or plan each iteration to have enough time to roll back your migration and rerun it from scratch.

At Blue Fish, we use a methodology we call Agile Migrations, where we perform frequent, incremental migrations during which we only re-migrate the small number of documents that have changed. Having worked on migrations for years, I can tell you that this beats the old “big-bang” approach hands down.

Additional planning steps that will keep you out of trouble include:

• Keep an exception log so that you can track all the issues that need to be resolved
• Plan on keeping detailed records of the status of each document – your business users will want to know which documents have migrated successfully and which still have issues that need to be resolved
• Try to segment your high priority documents from your low priority documents, migrating the critical documents first. The low priority documents can be migrated on a different schedule.

The Perils of Sample Data Sets

One of the most common migration planning mistakes is to employ an approach that runs one or more sample sets through the migration process and then projects a migration timeline based on the analysis of the samples. If you’ve learned one thing from this article, hopefully it is that it’s very hard to predict how many errors and exceptions you’ll encounter during the migration process. Even a statistically valid sampling of the data will give you a false sense of security since you are certain to run into exceptions that were not found in the sample set.

We learned this lesson the hard way. Several years ago we embarked on an effort to migrate 600,000 documents from three Documentum source repositories and one fileshare into a new consolidated Documentum repository. Determined to exercise due diligence in the planning stage, we started off by compiling a list of 150 assumptions about the state of the content and metadata. For example, we assumed that every document had an object name. We scripted DQL statements and programmatically validated all of these assumptions.

After validating the assumptions, we divided the documents into 350 batches. The plan was to run 20 of the batches, representing 6% of the total content to be migrated, through the migration process. We expected this test drive to uncover 80% of the exceptions. However, all 20 batches had their unique problems that required specific handling. At this point we realized we could not predict the number of exceptions and shifted our approach to expect exceptions and facilitate addressing them as they arose.

In summary, the smart and sane approach is to expect exceptions and be prepared to handle them when, not if, they arise.

Conclusion

Before tackling a content migration initiative, remember the iceberg. The straightforward attribute mapping is just the tip. Each layer has its own challenges and requires different planning. Exceptions take most of the time, and planning to respond quickly to the exceptions boils down to three things:

1. Expect exceptions.
2. Have the right resources available and prepared to respond to the exceptions when they arise.
3. Migrate early and often, finding and resolving exceptions iteratively.

Following this approach and setting the right expectations up front will prepare everyone for the challenges. When data issues and exceptions are spotted in the target system and quickly addressed, the migration team and business users alike will realize the process is “working as designed”. Everyone will see the progress and feel confident that the migration effort will be successful and will be completed on time. And you will feel a sense of accomplishment and relief knowing you successfully charted the iceberg and navigated around it instead of colliding with it.

To learn more about the tools and processes we use here at Blue Fish, visit the Content Migrations section of our Web site.

Read more at An Introduction to Web Publishing Jobs in Documentum.

Overview

Documentum provides a number of out-of-the box administrative jobs that help administrators ensure the health and integrity of the Documentum environment. For example, there is a job to check the consistency and integrity of the objects in the repository and a job to remove the old audit trail entries.

In addition to the standard jobs that ship with the content server, there are several jobs delivered with Web Publisher and Site Caching Services (SCS) that automate the clean-up and publishing of content in web publishing environments. For example, the dm_WebPublish job publishes content from the repository to a website.

This article will provide an overview of the jobs used in the web publishing environment: what they are, recommended configuration, and troubleshooting tips for some of the more common errors.

What is a job?

Jobs in Documentum are used to automate the execution of a program or script. Each job in Documentum represents an automation schedule and execution frequency for something called a method, which in turn holds the reference to the program or script to execute.

Jobs are configured in Documentum Administrator (DA) under Administration->Job Management->Jobs. To view the properties, schedule, method, and object information for a job, click on the information icon in DA.

Selecting the “Pass Standard Arguments” on the method tab page will result in the following default argument values:

The method uses these arguments to log into the docbase to accomplish its intended purpose.

Figure 1:
Pass Standard Arguments on the Method tab of the Job Properties

Another argument commonly used in job configuration is the window_interval. This argument defines a range of time in which the job can start outside of its scheduled execution time. For instance, if a job is scheduled to run every day at 2pm and the window_interval is set to 60 (minutes), then the job can start at any time between 1pm and 3pm. This interval can be useful if the Content Server is down at the scheduled execution time, then the job will run if the current time is within the current window interval. Using my example of 2pm, if the content server went down at 1:30 and was restored at 2:30, the job execution process would attempt to run the job in the remaining half hour window (2:30 – 3:00). Setting the interval to 1440 (24 hours) will guarantee the job will run, but may also negatively impact performance.

The execution of jobs is handled by the agent exec process. On the Content Server, the agent exec process runs continuously, polling for new jobs to execute. By default, the agent exec process polls the repository for jobs to execute every 60 seconds. When the process finds more than one job to execute in a cycle it will execute each job, sleeping 30 seconds between each. The number of jobs executed in a single polling cycle is configurable; by default, the agent exec process will execute up to three jobs in a polling cycle. To change this value, modify the max_concurrent_key value in the dmcl.ini. For more information on the agent exec process and modifying the polling interval, see the Content Server Administrator’s Guide.

Jobs for Web Publishing

Now that we’ve introduced the basics about Documentum jobs, let’s discuss the jobs used in the web publishing environment. Web Publisher and SCS provide us with a few jobs to aid in creating, promoting, and publishing content. To help the content authors with content creation, we have the Create_Dynamic_Content job and the WcmObjectBagJob. For automatic promotion of content and publishing we have the Monitor_Lifecycles job and the dm_WebPublish_[r_object_id] job, respectively. Lastly, the dm_SCSLogPurge job is used for housekeeping of the WebCache logs. Let’s dive into each one of the jobs…

Create_Dynamic_Content

The Create_Dynamic_Content job is used to update certain types of dynamic web pages with the latest content.

In web content management, there is often a requirement to create dynamic web pages – pages that update automatically based on other content in the repository. One of the most common types of dynamic page is a listing page; for example, a web page listing all of the latest press releases or announcements. Each time a user creates or removes a press release, you would expect the press release listing page to be updated automatically.

At Blue Fish, we prefer to create dynamic pages by having the web page query the SCS database. This provides real-time updates and follows a familiar paradigm that most web developers are familiar with. If you create dynamic web pages the way that we do, you won’t need the Create_Dynamic_Content job.

Web Publisher does, however, provide an alternative approach to creating dynamic web pages in which XDQL queries (Documentum queries that produce XML fragments) are embedded into Web Publisher presentation files (XSL stylesheets). Whenever the file is published, the XDQL query is run and the listing page is updated with the latest press releases.

The Create Dynamic Content job is used to refresh these web pages on a regular schedule so that the pages are always up to date.

To use this job, Web Publisher must know which pages are dynamically generated using the XDQL method. To indicate that a file has dynamically generated content, its template must be flagged to automatically refresh when related content is modified. To set the flag, select the Content Refresh checkbox on the publishing tab of the content’s Properties page. This type of content is created by the transformation process performed by the Create_Dynamic_Content job, which transforms the content using the presentation file containing the XDQL. The job will only perform this transformation on content in the Active lifecycle state.

An administrator can configure the Create Dynamic Content job to run as frequently as is appropriate for the content types. The job run frequency should be set based on the freshness requirements of the dynamic content. For instance, a current press release listing page might be updated daily, but an employee listing might only be refreshed monthly. Since the Create Dynamic Content job can only be configured once for all content types, frequency should be set based on the content with the highest freshness requirement.

Note that the Create Dynamic Content job does not invoke an SCS publish operation. For the content to be published to the Active website, configure the dm_WebPublish job to run at periodic intervals.

dm_WebPublish_[r_object_id]