Archive: April ’08

Clearwater Seafoods Speeds up Data Warehouse Development 17 Apr ’08

Thanks to Clearwater Seafoods for providing a case study on how WhereScape RED sped up their development.  We really appreciate when our customers are prepared to talk about how WhereScape RED has assisted them. 

WhereScape RED Enables Clearwater Seafoods To Adopt a Self-Service Reporting Model

RED Slashes Time in Building Data Warehouse Prototype by 30 Percent.

From its humble beginnings in 1976 when founders John Risley and Colin MacDonald, with one pickup truck, operated their company as a small lobster distributor in Halifax, Canada, Clearwater Seafoods has grown to become a premier supplier of premium shellfish and sea food globally.  Today, with a large fleet of freezer processor vessels and several processing plants throughout eastern Canada, Clearwater Seafoods has delivered 70 million pounds of premium seafood –including clams, scallops, shrimp, lobster, crab and ground fish –to 30 countries.

As part of its corporate structure, Clearwater operates separate companies, located in Canada, the U.S., U.K., Belgium, and Argentina, to sell and distribute their seafood around the world.  These distinctive operations, however, were adding to the complexity and accuracy of corporate-wide reporting.  Each business unit was reporting differently through a variety of methods that included spreadsheets and manual input of data. 

Since Clearwater sells their products worldwide, business analysis reports such as profit margins, inventory, and sales are critical to their business. As a global company, fuel and other costs as well as foreign exchange rates affect how the company evaluates its profit margins, according to Tony Scallon, business systems analyst at Clearwater Seafoods,   As a company selling internationally, Clearwater needs to know the true cost and profitability of what they are selling by region.  For instance, if Clearwater sold a particular seafood for $5 a pound  in the West Coast of the U.S., are they making money for that region once fuel costs, exchange rates, harvesting and production costs are factored in. With each company using different reporting tools and formats, this type of business analysis reporting would be time consuming and challenging. With its IT staff of 21 employees, Clearwater Seafoods had to dedicate two IT professionals full time to running business analysis reports for managers throughout the company. Depending on the complexity of the reports, it would take several hours to numerous days to complete reports.   

To improve its reporting capabilities, the decision was made to centralize all of its data into one data warehouse.  Clearwater uses Great Plains as its core ERP system that includes financials, sales and inventory management with Siebel CRM as the front end system used for order entry, and MS SQL Server as the RDBMS.

‘We selected WhereScape RED to manage our entire data warehouse,” said Scallon.   With RED, Clearwater Seafoods can now manage all their schema DDL (data definition language), into one meta data repository which is fully documented. 

“RED automated the time consuming process of writing SQL scripts and documentation and slashed the time to do this by more than 20 percent.  If you are manually writing SQL scripts, you also are susceptible to numerous errors. WhereScape RED not only automated this process but it provided the documentation where you could compare the  data to assure accuracy.” RED also has a fully integrated scheduler that  performs ongoing loads and logs all activities during processing.

From an IT perspective, WhereScape RED allows Clearwater to develop data transformations when needed and to automatically generate the documentation that shows all the data lineage based on the data that is loading into the warehouse.  These transformations allow the application of  business rules to be applied to  make a company’s data usable. For instance,  Clearwater may want to run a report on the total weight, in tons, of how much fish they distributed. However, the data they have been capturing was only based on the number of boxes packed. 

With WhereScape RED, a prototype of the sales data mart, with analysis cube and reporting services, was built within 30 days.  Scallon projected the prototype would take at least 90 days and was “very impressed” how quickly it took to prototype.  “WhereScape Red proved to be very scalable and an all in one tool that we needed in building our data warehouse,” he said.

With RED, data rates were proving faster than ever since Clearwater was able to build multi-dimensional cubes by leveraging Microsoft Analysis Cubes.  Once its data warehouse is in full production, scheduled for April 2008, Clearwater expects maintenance on the data warehouse to be easy and fast with RED since many of the mundane maintenance processes are all automated.

Another key benefit that may not be as tangible to measure is the peace of mind that RED provides, says Scallon.  “I no longer have to validate the data and worry about the accuracy of the scripts.  RED automates all that and gives me the time to focus on more strategic IT initiatives that will impact the company for the future.” 

The Business Problem:

• Business required more sophisticated business analysis reporting

• Separate business units within Clearwater Seafoods were all reporting differently

• Reports took several hours to several days to prepare

• Remove the need for IT department to produce reports and make it into a self-service model

The WhereScape Solution:

• A prototype was built with RED within 30 days compared to the projected 90 days estimated by Clearwater’s consultants 

• Reporting is now a self-service model with users now specifying and running their own reports

Return on Investments:

• Faster and Accurate reporting  across its separate business units operating throughout North America and Europe

• Able to run more sophisticated analytics and “what if” business scenarios

• More productive use of IT staff with two IT professionals, previously running reports, now reassigned to more strategic IT projects

Compudigm goes under 15 Apr ’08

It is never a good thing to see a company you know go under.  Compudigm, who developed visulaization software primarily for the gaming industry, went into receivership on March 28.  Their last accounts filed with the New Zealand companies office (they were based in Wellington New Zealand and Las Vegas Nevada) show a loss of 2.5m NZ (approximately 2m US) in 2005, and a similar loss in 2004.  They had recently sold the rights to the gaming software to Bally Technologies.  It looks like Bally is going to roll the Compudigm software into their own business intelligence solution so we will have to wait and see what happens to current Compudigm customers, which include some WhereScape RED customers. 

Design and the Elastic Mind 14 Apr ’08

There is a really cool exhibition on at MoMA at the moment - Design and the Elastic Mind.  As a data geek there were two works that really stood out for me - Bradford Paley’s Alice in Wonderland (check out his TextArc site) and Jonathan Harris’s I Want You To Want Me.  Both totally innovative and cool ways of showing immense amounts of data.  It shows how far we have to go as an industry in making data available, usable and interesting.

WhereScape sponsors Gartner Business Intelligence Summit 2008 11 Apr ’08

WhereScape was a sponsor at the recent Gartner BI Summits in Chicago and in Sydney (in conjunction with our Australian distributor, MIP).  Both were highly successful events for us, and the concept of decreasing the time to value was again well received.    There were more than 50 vendors at the Chicago Summit, but we were joking with our Gartner rep that with all the vendor consolidation going on it will be hard to get that number next year.

Migrating Between RDBMS Platforms 11 Apr ’08

A powerful feature within WhereScape RED is the ability to migrate a metadata repository from one RDBMS platform to another.  This will allow you to take an existing data warehouse environment on one database, such as SQL Server, and move it to another like Oracle.  While rarely used, this functionality is of immense value should you need to move RDBMS platforms at any stage after the data warehouse has been created.  

The following steps can be performed to move an existing WhereScape RED metadata repository from one RDBMS platform to another.  Note: this only moves the metadata structures and content, it does not move the data contained in the data warehouse. 

Prior to performing the metadata migration you may wish to perform the following tasks to ease the process;

  1. Upgrade to the latest version of WhereScape RED, to ensure you have the latest functionality. 
  2. Archive the Scheduler Audit and Detail log tables if you do not need to bring across this information.  The archive process moves the records from the audit trail log to an audit trail archive table.  This will reduce the amount of data that needs to be imported into the target metadata repository.  To perform the archive, go to the Scheduler tab within WhereScape RED and choose from the menu:  Logs à Archive the audit and detail logs.

In the example below we are moving from SQL Server to Oracle;

  1. Create an Oracle metadata repository via WhereScape Setup Administrator tool, using the same version as the SQL Server repository you are going to migrate.
  2. From the SQL Server repository, unload the metadata.  This can be done within WhereScape RED using menu option:  Backup à Unload the metadata to disk.
  3. Connect to the Oracle metadata repository using WhereScape RED.  Load the SQL Server metadata unload, ignoring the warning about the database being different.  This can be done using menu option:  Backup à Load the metadata from disk.  This process can take a long time for a large metadata environment.
  4. Verify column data types have been converted.  WhereScape RED will convert most data types for you during the load process.  However, there may be some data types that still need to be manually adjusted.  This can be easily verified by creating all of the tables (Load, Dimension, Stage, Fact, and Aggregate).  Those tables with bad data types will fail during table creation.  If any data types need to be manually converted, please inform WhereScape Support and they will ensure that future releases of WhereScape RED accommodate those data types.
  5. Change all transformations to Oracle supported functions.  Common transformations will be OK.  However, RDBMS specific transformations will need to be converted to the appropriate syntax on the target RDBMS.
  6. Regenerate or Rebuild all unmodified procedures. 
    • If you created simple join statement in SQL Server using ANSI join syntax, this will not translate in an Oracle environment.  You will need to rebuild the join syntax in Oracle via the wizard.
    • Outer join syntax will need to be rebuilt as the SQL syntax is different.
  7. Custom procedures will need to be recreated using the appropriate procedural language. eg: PL/SQL
  8. Change Load Table where clauses if required.
  9. Change View where clauses if required.
  10. Change Connection details if required.
  11. Change Load Table connections (for file and script loads) to UNIX if a UNIX scheduler is being used.
  12. Change Script connections (for script loads) to UNIX if a UNIX scheduler is being used.
  13. Regenerate any Load scripts.

While not a common situation, it shows the flexibility of WhereScape RED and the advantage of using a tool to generate the bulk of the underlying code.  What can be achieved in a very short time using WhereScape would be a substantial undertaking, if not impossible, in any hand-coded or proprietary environment.