ETL 01: Documentum -> MuleSoft -> Filesystem

This post is the first in a series focused on Extract-Transform-Load tools and techniques that I will discuss on this blog.

MuleSoft is an excellent tool to integrate real time updates (such as approved documents) from one system into another system.  MuleSoft has a very rich developer community with lots of examples, a good YouTube channel, and training …. I recommend this free 8 week course to learn the fundamentals.

We can use the Documentum REST interface to run a Documentum query, then either store the results or extract related PDF files from the system.

You can image the use cases for this set of tools:

  • Migrate content into or out of Documentum
  • Synchronize master data between systems
  • Publish content from Documentum to Box, Dropbox, Microsoft SkyDrive, Google Drive, etc.


Anyone new to the Documentum REST will want to use Postman to investigate some of the URLs that we use in this post:

Get repositories:


Run a Query:

http://localhost:8080/dctm-rest/repositories/repo1?dql=select r_object_id,object_name from dm_document where folder('/HR',descend)

Get Content:



MuleSoft Configuration to Query Documentum and save to a CSV file

The Documentum REST query output has the attribute values located at:

so we can use a transform-message to put the properties into a nicely formatted CSV file as follows:


 <flow name="dctm_query_to_csv">
 <http:listener config-ref="HTTP_Listener_Configuration"
 path="/dctm2" doc:name="HTTP" />
 <http:request config-ref="HTTP_Request_Configuration"
 path="/dctm-rest/repositories/repo1" method="GET"
 <http:query-param paramName="dql"
 value="select r_object_id,object_name from dm_document where folder('/HR',descend)" />

 <set-variable variableName="OriginalPayload" value="#[payload]"
 doc:name="Variable" />
 <set-payload value="#[flowVars.OriginalPayload]" mimeType="application/json"
 doc:name="Set Payload" />
 <dw:transform-message metadata:id="d8d90bbc-3265-4882-93e8-34b942bed718"
 doc:name="Transform Message">
 <dw:input-payload />
 <dw:set-payload><![CDATA[%dw 1.0
%output application/json

 rows: payload.entries map ((entry , indexOfEntry) -> {
 <dw:transform-message doc:name="Transform Message">
 <dw:set-payload><![CDATA[%dw 1.0
%output application/csv
payload.rows map ((row , indexOfRow) -> {
 "object_name": row.object_name,
 "r_object_id": row.r_object_id
 <logger level="INFO" metadata:id="04754dd7-e7cb-4e3a-a151-18c5ed1bbd42"
 doc:name="Logger" />
 <file:outbound-endpoint path="\data\mule"
 outputPattern="dctm.csv" responseTimeout="10000" doc:name="File" />


MuleSoft Configuration to Query Documentum and save PDF files

Similarly, we can run a query and extract all the PDF renditions of those files to the file system.

In this example, create a folder: c:\data\mule (or change the location int he configuration below.

Here is the MuleSoft Configuration:


<flow name="dctm_query_export_pdf">
 <http:listener config-ref="HTTP_Listener_Configuration"
 path="/dctm3" doc:name="HTTP" />

 <http:request config-ref="HTTP_Request_Configuration"
 path="/dctm-rest/repositories/dv4_ls_d2_cs_reg" method="GET"
 <http:query-param paramName="dql"
 value="select r_object_id,object_name from dm_document where folder('/HR/Form/Employment form',descend)" />

 returnClass="java.lang.Object" doc:name="JSON to Object" />
 <foreach collection="#[payload.entries]" doc:name="For Each">
 <set-variable variableName="r_object_id" value="#[]" doc:name="get r_object_id"/>
 <set-variable variableName="ContentURL" value="#[&quot;/dctm-rest/repositories/dv4_ls_d2_cs_reg/objects/&quot; + flowVars.r_object_id + &quot;/contents/content&quot;]" doc:name="ContentURL"/>
 <logger message="#[flowVars.counter] - #[flowVars.r_object_id]" level="INFO" doc:name="Logger"/>
 <http:request config-ref="HTTP_Request_Configuration" path="#[flowVars.ContentURL]" method="GET" doc:name="content obj">
 <http:query-param paramName="format" value="pdf"/>
 <json:json-to-object-transformer returnClass="java.lang.Object" doc:name="JSON2"/>
 <set-variable variableName="href" value="#[payload.links[1].href]" doc:name="href" doc:description="Extract the URL"/>
 <set-variable variableName="href2" value="#[flowVars.href.substring(43)]" doc:name="href2" doc:description="Extract the URL portion after the host name"/>
 <logger message="href = #[flowVars.href]" level="INFO" doc:name="Logger"/>
 <http:request config-ref="HTTP_Request_ACS" path="#[flowVars.href2]" method="GET" doc:name="Request PDF"/>
 <file:outbound-endpoint path="/data/mule" outputPattern="#[flowVars.r_object_id].pdf" responseTimeout="10000" mimeType="application/pdf" doc:name="Save PDF"/>
 <logger level="INFO" metadata:id="04754dd7-e7cb-4e3a-a151-18c5ed1bbd42"
 doc:name="Logger" />


This post should help you get started with:

  • Using Documentum REST to run queries and extract content
  • Using MuleSoft to process the data returned from REST to store as a CSV file or PDF files

Future ETL blog posts will include:


Custom Export Vdoc Widget – Part 1

The “custom export” use case has been implemented many times before, here is one example:

I don’t want to re-invent the wheel, but I think producing a custom D2 exporter is unique enough to warrant a fresh look:

  • Use a D2 widget to set up user options
  • Export to a file share
  • Export a virtual document using nodes as folder names

This implementation can form a base for anyone wishing to do their own


The user would activate the VdocExport widget, then select the virtual document they wish to export.  Selecting the Virtual document will update the widget to verify the top document that should be exported and default the exported folder name.

D2 Screenshot

Clicking export will export PDF renditions of all the documents in the Virtual Document structure:


The servlet will return back to the user with the number of documents that have been exported.



VdocExport.jsp – provide the user the list of options to export and track the virtual document – servlet that exports the related PDF files

The full project is here (look in the folder VdocExport)

Future features and follow up blog posts:

  • Part 2: Allow the user to select the base shared file path (maybe from a list)
  • Use the D2 configurations for export (which would include the signature page and overlays)
  • Add the ability to export virtual document snapshots
  • Better logging – right now the user doesn’t know what went wrong
  • Build and download a giant zip file containing all the documents in the export path
  • Export the metadata into a separate excel document
  • Let the user decide which renditions to export
  • Integrate with the P2 publishing engine
  • Code cleanup




Document Viewer – Part 1: Multiple Monitors

Multiple Monitors and Giant 4K Screens

Many of us use multiple monitors for a variety of reasons:

  1. Published studies have shown that a dual monitor setup can increase productivity by 20 to 50 percent (depending on what you’re doing). We know that D2 takes up a lot of space, its difficult to see the D2 screen AND the document.
  2. Real multi-tasking requires enough screen space to keep two or more apps in full view simultaneously. Our authors are always in email, D2, Word, Excel, and 5 other applications.
  3. Copy/Paste between documents is much easier when you don’t have to alt-tab between them and scroll up or down so much. Tables are especially tough.

Having a second monitor to display your document is really nice:


Deficiencies with the D2 PDF Viewer

D2 has a nice PDF viewing widget, this is something that we haven’t seen in Webtop or other Documentum .

But the D2 PDF viewing widget has several limitations:

  • It is difficult to use with multiple monitors
  • We would like to size the document viewer without affecting my D2 window or the widgets
  • It doesn’t automatically update when i click on a new task
  • Readability (also addressed by  Brava, Arender, and a widget by Trana1)
  • We may want to extend the viewer with additional functionality (see future release section below)

How to Do It

The widget itself does not need to display anything within, but it has to…

  1. Subscribe for the document selection event
  2. When the widget gets the event, it opens a window with the URL to the document

Subscribing is described in the D2 Developers Guide:

          selectObjectCallback, true);


Opening a window is also pretty common, we want to make it always open in the same external browser window.,'viewPDF_target');

The D2 download servlet will provide the PDF to a browser.  It has a bunch of parameters that we can use, we want to make sure that content_disposition=inline.

Here is an example URL:





The code is in github in the “widget-core” subfolder “docview”:


Future Releases

Future releases of this widget may include:

  • Keyboard shortcuts (at least activate the PDF window)
  • Better hyperlink handling for links within a PDF
  • Remember the location of the window as a default for the next time it runs
  • Zoom to quarter panel with a function key
  • Preferences Screen
  • Annotations
  • Macros
  • Caching – for faster performance
  • PDF Form filling
  • Other document formats
  • Key from image
  • Better integrated search
  • Full screen mode (minimize the title bar)
  • Really optimize for processing tasks

There may be other widgets that could be displayed in an external viewer as well.




Adding Charts to D2 Part 3

In the first two parts, we discussed:

Part 1: Starting with Google charts

Part 2:  Clicking part of the chart calls a query form that tells D2 to list the documents


Here, in Part 3, I release the code and discuss a couple implementation options.

The code is on github at:


Please branch it, improve it, add comments, and use it in your own projects.

Included in the github are images that you may choose to use for the widgets you create.

There are areas in which the code can be cleaned up, but having it in a single JSP page makes it easy to use.

The configuration might be better as objects in the repository, or eventually D2-Config components, and that may come in the future, but using a properties file is efficient and helps demonstrate the configurability.

In order to run this in eclipse, you can run this in a separate tomcat instance, then point to it from the D2 widget configuration like this:


In future sections about google charts I may address:

  • adding a refresh button
  • adding a drop down value to show
  • better formatting options
  • improving performance
  • Charts with more than 1 value
  • Separating the code from the JSP
  • Cleaning up the configuration

Let me know in the comments if you have any additional thoughts or features.

Adding Charts to D2 Part 2

Part 2 describes how to send a message to D2 to load the doclist widget

After creating the graph in Part 1, we want to make the graph clickable and send a message to the D2 interface to update the Doclist widget with the documents that the user clicked on.

This helps me itemize the specifics of each pie piece or bar chart:

  • Why do we have so many draft documents?
  • Which documents are taking so long in the workflow?
  • Which items are in an error state?

Now, because the results in the doc list, I can do standard Documentum D2 actions:

  • View, Edit, Check in, or double check the results
  • Delete old draft documents that never got started
  • Start a workflow or terminate an existing workflow and restart
  • Email the owner and ask a question

So how do we do this in D2?

1. Capture the click

In the Google chart documentation, we can add a selectHandler:, 'select', selectHandler);

function selectHandler() {
    var selectedItem = chart.getSelection()[0];
    var value = data.getValue(selectedItem.row, 0);
    var param_name = "<%=sParamName%>";

2.Send a message to D2

There are a couple good resources for learning D2 Widgets:

And it is good to look for some examples too.

For our project, we need to configure a standard D2 Query Form ahead of timethat will define the search. Then the updateDoclist() function will post a D2_ACTION_EXECUTE message.

Our Message will be created like this:

function updateDoclist(param_name,param_value,queryFormConfigName) {

//To update the doclist a new OpenAjax message will be build to be posted in the Hub using // the D2-OAH API
 var messageToSend = new OpenAjaxMessage();

//In the message, we need to define what properties will be sent. Here a_status and // r_object_type
 messageToSend.put("list", param_name);

//We set the a_status value
 messageToSend.put(param_name, param_value);

//set the query form config name which will be used to update the doclist
 messageToSend.put("config", queryFormConfigName);

//Then we define what service and what method in the service will be called. We call the //Search service and the runQueryFormSearch method.
//Calling this service will update the user's last search object
 messageToSend.put("eService", "Search");
 messageToSend.put("eMethod", "runQueryFormSearch");

//When the service call will be finished, we can define what action will be executed. Here, //an event will be posted.
 messageToSend.put("rType", "EVENT");
 //As the last search has been updated by the web service call, we will post the D2_ACTION_SEARCH_DOCUMENT event to display the search results

//The message is now ready, it can be posted in the Hub
 d2OpenAjaxHub.sendMessage("D2_ACTION_EXECUTE", messageToSend);

The message will then tell the doclist widget to run the query form with the right parameter (such as lifecycle state).

In Part 3, I provide a link to the code on github and provide some more discussion on some of the glue between the widget and D2.




Adding Charts to D2 Part 1

I’ve been experimenting with Google Charts to add some color to our D2 screens.  This is a good choice when building adding a dashboard widget to the 8 useful items to include on your D2 Welcome Page.  The eDRG reporting does nicely add charts and reports, but in this example, I will use the Google Charts   This also allows the developer to add additional functionality as needed.

This post will be in 3 parts:

  1. Creating a simple Google charts widget
  2. Adding call-backs to view documents when clicked
  3. Code walk-through

The widget requirements are pretty simple:

  1. This should act like a normal D2 custom widget, creating a session.
  2. The widget should automatically build a chart without the user requesting the chart be run.
  3. The query for the chart should be in a configuration file.  The configuration
  4. The chart should be clickable, which will update the document list.

…in the comments, please add more requirements, perhaps I will add a future release of the charts widget.

To implement Google charts in D2, we will need both Java and HTML, for simplicity, I am using a JSP file now.

The steps will be to do the following:

  1. Build the rest of the Google chart HTML.
  2. Get the parameters from the URL, retrieve the configuration, and get a session.
  3. Execute the query using the session building a data set for the Google chart javascript to use.

1.  Build the basic HTML

we start with the simple HTML from Google.

Keep track of this link in case Google charts change, or you use a different charting tool

Using JSP, we will build this section on the fly:

var data = google.visualization.arrayToDataTable([
[‘Task’, ‘Hours per Day’],
[‘Work’,     11],
[‘Eat’,      2],
[‘Commute’,  2],
[‘Watch TV’, 2],
[‘Sleep’,    7]

var options = {
: ‘My Daily Activities’

Of course this could be done with a separate REST call, which would make sense if you give the user other UI elements to pick different charts or drill down capabilities.

2. Get Parameters, get a session,

The D2 widget configuration will pass parameters like this:


After getting the parameters from the URL, the widget will get configuration items.  For this simple example, they are stored in a .properties file, which the system grabs from WEB-INF.  Alternatively, it could get the information on the URL or from an object in the repository.

The config properties file will have the title, query, the chart type, and any other information we might need in the future:

title=Docuemnt Status Counts
query=select a_status as param1,count(*) as cnt from dm_document where a_status != '' group by a_status

3. Execute the Query

As we go though the results of the query, we output

DfQuery query = new DfQuery();
IDfCollection colObjects = query.execute(idfSession,DfQuery.DF_READ_QUERY);

boolean isFirst = true;

while ( {
    String value = colObjects.getString("param1");
    String count = colObjects.getString("cnt");
    if (!isFirst) {
    out.println("['" + value + "'," + count + "]");
    isFirst = false;


 var options = {
          title: '<%=title%>'

This chart is only good for pie charts, but we can add other chart types.

Go to Part 2 — update the doclist widget.



8 useful items to include on your D2 Welcome Page

Many users benefit from a well organized Welcome Page when they log in, but we can design a workspace that helps them quickly:

  • Feel comfortable in the new space
  • Get an overview of the system and its capabilities
  • Find a document
  • Get to the right space

The following D2 Widgets are options that your users may wish to have on their landing page:

1. Welcome and Quick Help Page

A short help page can aid users that are not always on the system.  This page should show the number of tasks you have, and link to common items like Browse, Process Tasks, Create New, and Import.

This page may also contain announcements.

The page can be written in HTML or JSP and should have links to the help documentation, training tools, and to people who can help for people who are just lost.

2.  Quick Search

Adding the quick search at the very top of the page makes it easy for people to get to work finding a document right away.

This is ideal when it brings the user to the document list widget with facets shown.

3. My Searches

We generally create several query forms and saved searches for our customers.  By providing the My Searches widget, users can quickly get to those searches to find their documents.

The power of the saved searches lies in the ability to create lots of search capabilities after a project has started.

4. My Favorites

When users work on the same set of documents, the My Favorites widget is an excellent place to start.

5.  Task Browser

By quickly seeing a summary of all the tasks that are waiting, the user can determine whether they should perform those tasks right away.

6. Status Report

For some users, the first thing they want to do every day is to run a status report that summarizes:

  • New documents
  • Workflow status
  • Documents in my queue
  • Documents due to be reviewed in the next 2 months
  • Stale documents

By providing a status report that is run automatically on login, users can quickly see the health of the content system.

7. Business Specific Query Form

Every D2 implementation is unique, so query forms provide a way to ask users for specific attributes to be used in a specific query.

Seeing an established and recognizable form front and center upon login) helps users immediately feel at ease with the system.

8. Highly Targeted D2 Custom Widget

For some implementations, the out-of-the-box widgets are not enough.

Similar to #1, the Welcome and Quick Help Page, we can do the following:

  • Provide Tasks information
  • A link to create a new document
  • A link to import a documents
  • Links to other widgets, reports, queries, or tools

Additionally, we might:

  • Provide an integration with another system
  • Present graphical illustration of the documents, lifecycles, workflows, or relationships
  • Use other tools to present information about the repository, such as other javascript libraries, graphs, or user interfaces

Additional Helpful Hints

  1. When we define the Welcome Page, it should be set up as the leftmost View so they see it right away.
  2. Users that no longer want to start on the Welcome Page, can create an intelligent link that brings them to the Browse or Tasks view.
  3. Users can still personalize the workspace to add and remove components that are applicable to them


Question:  What have you used for your welcome page?