WhereScape RED Builds data warehouses, fast.

WhereScape, in conjunction with Forrester and Composite Software, is running an Agile BI webinar.  As well as our very own Mark Budzinski, the other presenters will be Boris Evelson, VP and Principal Analyst at Forrester Research and Rorbert Eve, EVP of Marketing with Composite Software.

The idea for the webinar really came about from the research Boris conducted on Agile BI, and more specifically meta data driven development of data warehouses,  which featured both WhereScape and Composite.

WhereScape and Composite have totally different approaches to solving the same set of problems.  It should make the event interesting as both vendors will have to articulate our own value propositions in a manner that clearly differentiates our offerings.  This can only be useful for people looking at different approaches to being agile.

The webinar is on Wednesday August 4th, 11am PT/ 2PM ET.  More information and registration is provided here.

We recently invited Dr Ken Collier (theagilest.com) down to New Zealand to work with our development team.  As you can guess from the name of his website Ken is an agile kind of guy.  And not one of the crop of recent converts  – he has been working in agile almost as long as he has in information management.

There were some fascinating discussions.   WhereScape has always been a “rapid” development environment, and very much in alignment with the Agile Manifesto:

Manifesto for Agile Software Development

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more

And the associated principles:

Principles behind the Agile Manifesto

We follow these principles:

Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.

Welcome changing requirements, even late in
development. Agile processes harness change for
the customer’s competitive advantage.

Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.

Business people and developers must work
together daily throughout the project.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

Continuous attention to technical excellence
and good design enhances agility.

Simplicity–the art of maximizing the amount
of work not done–is essential.

The best architectures, requirements, and designs
emerge from self-organizing teams.

At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.

But there are certainly things we can do to better support common agile practices.  As a data warehouse development environment we cannot and should not attempt to provide project management functionality, but we can look to see how we can better enable an agile project.

This is the key reason we are working with Ken, and the first deliverables from these discussions are being worked on now.

Getting back to the title…one of the many things Ken talked about that resonated with us was the concept of being constantly deployable.  To illustrate the point he challenged us with “What would happen if we cut the funding on your project at the end of the week?”  It is a question that really got to the heart of being agile - nothing about scrum, sprints, continuous integration testing etc etc – but a question about BEING agile.

It is certainly a question we will ponder on our own consulting engagement and best practice guides, and one we will use with our customers who want to be more agile. 

Watch this space for more information on how WhereScape will enable agile practices.  We will see what we can do to help you if someone asks…. are you ready to deploy this afternoon?

On Saturday I had my 40^th birthday party in the Dominion Bar in Mt Eden, Auckland, New Zealand.  The theme was “70’s meets Charity Shop Glam” and I must say there were some fantastic outfit efforts.

A Big thanks to all the WhereScape chaps who attended with their lovely, lucky ladies and also to Michael Whitehead who turned up as a “70’s boss” in full stay press grey business suit and tie(he managed to escape before the photo was taken).

There are actually a lot fewer wigs in the picture than you would think….can you spot them ?

So I will admit I am little behind on my reading…but I just got around to looking at a blog posting on the BeyeNETWORK from Bill Inmon entitled “Incremental Data Warehouse Development – The Only Way to Fly”

I totally agree with what he says.  The SDLC approach for data warehousing doesn’t work, never has worked, and never will work.  How can you use an approach predicated on solid requirements when you know you cannot get them? 

The more I read articles like this, the more I am convinced that WhereScape is on the right track building in support for agile data warehousing practices.  This doesn’t need to mean Agile with a capital “A”, but agile in the context of small, fast pieces of work.  Watch this space - as well as working on a huge amount of new functionality in WhereScape RED to make it even faster to build data warehouses, we have a new product under development as well.  More on that later, but I can say it will be totally focused on helping developers build data warehouses faster.

And thanks to Raphael Klebanov for sending the link through.

WhereScape RED natively supports type 1, 2 and 3 dimensions. But what on earth is a type 6 dimension? It is a combination of a type1, type 2 and type 3! Basically there are two columns used to track a type 6 attribute – one holds the newest value, and one holds a point in time value that references a start and end datetime. This is a great way of being able to:

• choose whether to report or filter by an employee’s current department or their department when an event / fact was recorded.
• choose whether to report or filter by a customer’s current suburb or their suburb when an transaction / fact was recorded.
• do a “where are they now?” type of report eg where are employees working now, that worked in this location 2 years ago.

WhereScape RED can build these type 6 dimensions without resorting to customizing the code. The standard WhereScape RED procedures will generate change detection code for time variant data (type 2 and 3 attributes) and RED will add change tracking columns to manage date ranges in those dimensions.
Just to summarise:

• A type 1 slowly changing dimension (a Normal dimension in RED’s dimension wizard) will overwrite historical changes with the newest data – one record for each unique business key.
• A type 2 slowly changing dimension (a Slowly Changing dimension in RED’s dimension wizard) will create new records each time an attribute that is marked as slowly changing does not match the “current” value for that business key. Unique start and end date ranges will manage which record is relevant at any point in time.
• A type 3 dimension (a Previous Values dimension in RED’s dimension wizard) will hold the current and the previous attribute values for a tracked attribute in two attributes in the dimension.
• The mysterious date ranged dimension in WhereScape RED is essentially a type 2 dimension that relies on the source system to maintain and pass changed data with date ranges.

So how would you build a type 6 dimension in RED using standard RED code? By copying a source attribute into two dimension attributes, one to hold the current value and one for a point in time value (with suffixes of _cur and _pit). This dimension should be built as a type 2 dimension in RED, with the _cur attributes marked as type 1, whilst the _pit attributes selected as type 2 attributes.

Thanks to Mr Kimball for writing about them here.
And thanks to Andrew Cann and the chaps from Mater for explaining why type 6’s were useful to them!

We are moving away from the concept of the free Practitioner License of WhereScape RED.

The original idea was that we would happily provide free copies of WhereScape RED to individuals, but would charge organizations if they wanted to go into production with the output.  I guess we should have anticipated it, but whereas the majority of people respected this, the minority have caused an issue.  We have now had too many instances where people have used the free software to build entire data warehouses.  We were pretty relaxed when this was an individual practitioner doing this, but fair to say we got up tight when BI consulting companies were charging for projects, using our software to build solutions faster, and leaving the end customer with an unlicensed solution.

So…from V6.1 (which is in beta now) the practitioner keys will no longer work.  Evaluation Keys are not impacted.

If you have a practitioner key you can continue to use it (within the original restrictions) with pre 6.1 versions.

If this is going to cause you a real pain, and you were using a practitioner key for valid reasons, then give us a call.  We will see if we can come up with a solution that works for you.

WhereScape has made the finals of the the Duncan Cotterill Innovative Software Product of the Year category at the New Zealand Hi Tech Awards.  We do our development out of New Zealand, and these awards are about as big as they come in that part of the world (in tech anyway, some would argue the Young Farmer of the Year is bigger - after all they get televised, and are held in Gore).

Compeition is hot, with four software companies still standing.  Judging happens in April, with the winners announced at a gala dinner on 7th May in Auckland, New Zealand.

Once again, Steve, Jason and Doug and new recruit Jake took on the2010 Kona Colville Classic, a 72km (45 mile) mountain bike race around the tip of the Coromandel Peninsula (http://en.wikipedia.org/wiki/Coromandel_Peninsula).  The section of the race around the top is not accessible by road so careful riding and packing of tools was required.  On the day, we all got round the course safely in one piece and with various levels of physical discomfort!

We all beat our previous 2009 times largely due to our increased fitness, strength and in part due to the fact there was no rain this year so no mud bath.  Photos show before (pre race nervous smiles at Colville) and after (rehydrating and carbo loading in Coromandel Town)

before:

after:

According to post race banter, the 2011 race will continue to feature some WhereScapers on bikes……     

WhereScape’s Chuck Kelley has written a blog for BeyeNETWORK on the performance trade offs between joins and storage. 

The question he asks is:

What is the best performing way to load a fact table that has 20 different dimensions?

As he sees it, there are 3 options:

1. Create the table from a single FROM … WHERE clause
2. Create a view that does the FROM … WHERE clause and join that to the input for the fact table
3. Create the table with multiple staging steps joining

The full blog is available at http://www.b-eye-network.com/view/12675

We are thrilled to be taking part in a Teradata webinar next week.  It is called Production Deadline Insomnia: Beat the Clock with Maximum Results.  As the title implies the webinar will be be about the unfortunately all too common scenario of having project deadlines that look too tight.

The presenters will be:

Dan Mosher, Director of Enterprise Data Warehousing, United Rentals,

Chris Twogood, Director, Appliance Product Marketing, Teradata Coroporation, and our very own

Mark Budzinski, VP and GM, WhereScape USA

The format is an interactive panel, where the presenters will discuss the challenges and solutions within the business world where time waits for no one.

Details are:

Wednesday, February 3, 2010

1pm-2pm (EST), 10am-11am (PST)

Registration is at http://www.teradata.com/t/webseminar.aspx?id=12969

The BBBT is one of those events you wait a long time for.  The brain child of Claudia Imhoff, the Boulder BI Brain Trust is an opportunity for vendors to spend quality time with some of the BI industry’s leading analysts.  It is a popular and sort after event - I understand the slots for 2010 are basically gone already.  Our slot was booked in the beginning of the year, and we got to present on the 20th of November.

We have met with and briefed a number of the analysts before, but this is a little different.  First off it is face-to-face in Claudia’s office.  Secondly it is for half a day - much longer than a standard briefing.

Talking with a room of highly intelligence and opinionated analysts has its challenges.  At one point a tweet came through that it was like being in a room full of ADD sufferers - and that was from some one sitting there.  As well as the physical audience, analysts also dial in from around the US and the world.  I hate to think what it is like to try and keep up with all the questions and conversations if you are not actually present.

The BBBT is held in Boulder, Colorado.  I had not been to Boulder before, and I have to say I thoroughly enjoyed visiting.  The Boulderado was a great hotel, and the restaurants and ambiance of the city was superb.

I may admit to sneaking in some geocaching the day before -always good to add a new state.  Dinner the night before with Claudia and the crew from B-eye-network was a highlight, and there may be some truth to the rumor it finished at a Tequila bar.

The actual presentation is pretty free format.  For the first half we choose to talk about WhereScape and WhereScape RED - why we started the company, what we wnated to achieve, who was buying our software, and what they were achieving with it.  All the way through there are tweets coming from around the world and in the room - sort of instant and constant polling of how interesting you are. 

For the second half we choose to talk about our future directions.  For us this was the interesting part.  We talked through our ideas of the market and the new things you will be seeing from WhereScape over the next year or so.  We are excited about our roadmap, and it was invaluable to get feedback and insights at an early stage from a bunch of savvy experienced people.

Overall the experience was great.  We got heaps of feedback and ideas, and I hope the analyst community gained a greater understanding of what customers can achieve with WhereScape.  Thanks to Claudia and Dave Imhoff for hosting the event, Shawn Rogers and Richard Hackathorn for their instant-blogs (both were up before the end of the session) and all the participants for their interest and comments.

Installing RED on a Windows 64-bit server, with Oracle as a repository or source system

When installing RED on 64-bit Windows, the installer selects the following installation folder by default:

C:\Program Files (x86)\WhereScape\

This is where 32-bit applications normally reside on a 64-bit Windows platform.

However, if Oracle is involved, either as the data warehouse repository, or as a source system, installing RED in “Program Files (x86)” causes a problem.

The problem is that Oracle doesn’t like directories that have a parenthesis in the name - applications using Oracle cannot be located in a directory with parenthesis in the name, and this includes RED using an ODBC connection to Oracle.  The result…RED will not be able to connect to Oracle.

This issue can be prevented by changing the installation directory of RED to:

C:\WhereScape

For more information about this Oracle issue, see

http://tiredblogger.wordpress.com/2007/11/21/oracle-data-access-components-odac-with-64-bit-development/

64-bit SQL Server data warehouse repository with Oracle as a source system

If you are running a 64-bit SQL Server data warehouse, and need to connect to one or more Oracle source systems, you will require both the 32-bit and 64-bit Oracle Clients installed on the 64-bit Windows Server:

1) 32-bit Oracle Client

The RED client is a 32-bit application, and requires a 32-bit ODBC source to connect to an Oracle database. 

The 32-bit Oracle Client is required so you can set up a 32-bit ODBC source using the “Microsoft ODBC for Oracle” ODBC source.

2) 64-bit Oracle Client

For better data load performance, create a SQL Server “Linked Server” to connect to an Oracle source system.

The RED scheduler will use the Linked Server for the data loads.

64-bit SQL Server requires the 64-bit Oracle Client and ODAC (Oracle Data Access Components) to create the Linked Server - these are both included in the Oracle Database 10g Release 2 Client (10.2.0.4.0) for Vista and Windows 2008 (64 bit).

For installation instructions for the 64-bit Oracle Client, and setup of the Linked Server, see

http://www.mssqltips.com/tip.asp?tip=1433

We have tested this setup using the following software versions:

64-bit SQL Server 2005 as the DW repository on the 64-bit Windows Server

32-bit Oracle 10g Client installed on the 64-bit Windows Server

64-bit Oracle 10g Client installed on the 64-bit Windows Server

Oracle 9iR2 as a source system

Oracle 10g as a source system

The SUBWAY® Franchisee-Owned and Operated Company Is Using WhereScape RED to Build Its Microsoft SQL Server Enterprise Data Warehouse

SEATTLE–WhereScape, a provider of a comprehensive Integrated Development Environment (IDE) for data warehousing, announced today that IPC, an independent SUBWAY® franchisee-owned and operated purchasing cooperative, has purchased WhereScape RED to enable the rapid development and deployment of its Microsoft SQL Server-based enterprise data warehouse. The announcement was made at PASS Summit 2009, the premier event for SQL Server professionals, where WhereScape is exhibiting in Booth #514.

IPC, headquartered in Miami, Florida, is implementing its new SQL Server data warehouse to gain additional business insights from its terabytes of Subway® Card processing data, finance and supply chain data in support of SUBWAY’s 25,000 restaurants across the U.S. and Canada. IPC is responsible for procuring all of the SUBWAY food, packaging, equipment and services for the US and Canada through negotiating price, supply, and distribution terms while improving quality, enhancing competitiveness and ensuring the best value to SUBWAY restaurants and their customers.

“IPC looks at all supply chain opportunities that will result in better operations, savings or quality for SUBWAY franchisees,” said Anthony Ronconi, IPC Director of Information Technology. “This is a continuous process with IPC often receiving requests from its members to evaluate taking on new functions. As data volumes continue to grow, our enterprise data warehouse will be key in helping us provide timely data analysis and reports that enable management to make insightful and timely business decisions, benefiting all SUBWAY store owners.”

According to Ronconi, IPC’s initial data warehouse development efforts, while successful, proved too time consuming making it difficult to meet dynamic business requirements. This resulted in the company performing due diligence looking for a cost-effective solution to aid his team. “WhereScape RED’s ‘prototype and iterate’ data warehouse-building approach is ideal for IPC as our business requirements and needs are constantly evolving and changing. In addition, WhereScape RED has greatly increased the productivity of our developers and our results to date have impressed both our IT group as well as our stakeholders. We have been very pleased with the purchase and progress we are now making,” Ronconi said.

IPC’s enterprise data warehouse consists of several massive modules including card model, store model, finance and supply chain. For example, just one of the activity fact tables contains more than one billion records, with the enterprise data warehouse containing almost two terabytes of data and growing daily. “The data contained in each of these modules is quite complex, and WhereScape is enabling us to effectively manage that complexity,” Ronconi said. Two major modules?card model and store model?have completed testing and have been placed into production. The supply chain module is expected to be in production by year-end, and the finance data will be incorporated into the data warehouse in a multi-phased approach throughout next year.

Ronconi added that IPC has also been highly impressed with the metadata-based documentation-creating capabilities of WhereScape RED. “We have complex reporting relationships in the IPC data warehouse between business units and models, without which proper documentation is very hard to understand for a user, yet alone communicate to other IPC technical resources,” he said. “As we make changes to our data warehouse, WhereScape allows us to automatically regenerate the documentation to reflect the changes.”

“We are pleased to welcome IPC to the WhereScape customer family and honored to be playing a key role in this high profile data warehouse development effort,” said WhereScape Founder and CEO Michael Whitehead. “WhereScape RED ensures that IPC and other organizations can not only quickly and cost-effectively develop data warehousing solutions, but also easily adapt and change the environment as business needs dictate. IPC is in good company, joining more than 100 other SQL Server customers enjoying the productivity, ease of use and cost benefits of the SQL Server and WhereScape combination.”

Fully Integrated with Microsoft SQL Server

WhereScape RED provides native support for SQL Server as well as IBM DB2, Oracle and Teradata. The same WhereScape integrated development environment is used to build SQL Server data warehouses as well as Analysis Services cubes. The benefits are simple – organizations only need to learn one development environment, end to end metadata from the source system through to the cube layer (and where supported the client tool as well), integrated workflow and scheduling for the data warehouse and the cubes, automatically maintained lineage information and documentation from source system extract through to the data warehouse and the cubes.

About IPC

IPC is an independent SUBWAY® franchisee-owned and operated purchasing cooperative. IPC negotiates the lowest cost for goods and services, while improving quality, enhancing competitiveness and ensuring the best value to SUBWAY® restaurants and their customers. Moreover, IPC is committed to delivering returns to its members year after year. By organizing as a cooperative, the business initiatives are set by the owner members and are benchmarked by the active participation of an elected Board of Directors. For more information, please visit, www.ipcoop.com.

About WhereScape

WhereScape provides a product, WhereScape RED, which enables organizations to ensure that the structure of their data always meets the changing needs of the business. WhereScape RED is the only comprehensive Integrated Development Environment (IDE) for data warehousing that supports the entire data warehouse management life cycle, integrating source system exploration, schema design, metadata management, warehouse scheduling and enhancement into a single, simple integrated design.

More than 300 customers worldwide are using WhereScape RED on a variety of platforms. Projects performed using WhereScape RED typically come in under budget, ahead of schedule, with improved performance, greater transparency and built on more solid foundations over the systems they replace. WhereScape has head offices in Auckland, New Zealand, Portland Oregon, and Wokingham, UK. For more information, please visit www.wherescape.com.

All products or company names herein may be trademarks of their respective owners.

We are thrilled to have one of our newer customers, Dan Mosher from United Rentals (NYSE: URI), with us at the recent Teradata Partners User Group conference in Washington DC.

While he was there Dan very kindly sat down with Mark Budzinski, WhereScape’s Sales and Operations VP for North America and Shaun Rogers from the b-eye-network to record a podcast.

Some fo the great quotes include:

“WhereScape promised a lot and the product has delivered.  We are very happy with the amount of time it is saving us in development, as well as the documentation it is producing and the built-in scheduler.  I am very happy with the purchase.”
 
“We estimate the development lifecycle is 20-25% of what it was previously when we were hand-coding.”

 ”WhereScape also gives us the ability to utilize people who are not necessarily programmers, but people who are familiar with our data.  The majority of my staff has been with the company for at least 10 years so they are very familiar with where the data is coming from, and with their knowledge of SQL, knowing what they want to achieve.  It has been a good fit for us.”

 ”We always do documentation, but as most people find, we would make a change but often fail to go back to fix the documentation everywhere.  The ability to click a button and have it generate a web page that is reflective of our data warehouse has been great for us.”

You can hear the entire podcast here.

Conference season is in full swing, and we have just completed two conferences - Teradata Partners User Group in Washington DC and the DW 2.0 Asia Pacific Summit and Workshop. 

Teradata Partners was a great event.  It was a real pleasure to have Dan Mosher from United Networks with us there.  He took the time to meet with analysts including Claudia Imhoff and Colin White, and took part in a pod cast with WhereScape’s Mark Budzinski and Shaun Rogers from the b-eye-network.

DW2.0 took place in Sydney, Australia.  WhereScape in conjunction with MIP were Platinum sponsors of the event.

Coming up WhereScape is speaking and exhibiting at the Europe 2009 Data Management, Information Quality and DW/BI Conferences that take place on the 2-4 November at the Park Plaza Hotel in London.

At the same time we will be exhibiting at PASS (Professional Association for SQL Server) Summit Unite 2009 at the Washington State Convention and Trade Center in Seattle.

If you are passing through either of these conferences please look us for a chat.