WhereScape Builds data warehouses right, now

The other day I set out to build a small data mart combining support data and development data from two database sources and several other sources.

I needed to quickly find out which tables in the two database sources I needed to use.

My development system has 104 tables and my support system 244 tables.

So I pointed WhereScape 3D at both sources.

Very quickly I had:
- complete ERDs of both sources
- profiling results for all tables and columns
- identified the 12 tables in the develpment system I needed
- identified the 13 tables in the support system I needed

Less than 30 minutes after starting, I was ready to build my data mart using WhereScape RED.

I’m sure it would have taken me at least a day to figure out both source systems without 3D.

Here’s the ERD I ended up with for the subset of the develpment system I needed:

And here’s the one for my subset of the support source system:

A busy week for the WhereScape Europe team kicked off on Thursday 29th September. WhereScape Europe hosted a breakfast workshop in central London, and the venue for this auspicious event was the Penthouse of New Zealand Embassy.  What a venue it was, with fantastic views over London, as the photos below prove.

For More Information:

Scott R. Humphrey
Humphrey Strategic Communications
(503) 644-9709
humphrey@strategic-pr.com

Pacific Northwest BI Summit to Feature Leading Business Intelligence Experts

The 10^th annual Summit kicks off July 21 and will again be headlined by
Claudia Imhoff, Colin White, Jill Dyche, William McKnight and Shawn Rogers

Portland, OR – July 19, 2011 – The lineup is set for the sold-out tenth annual Pacific Northwest BI Summit, set for July 21-25 in Grants Pass, Oregon. The Pacific Northwest BI Summit, presented by Humphrey Strategic Communications, is an executive networking retreat bringing together five of the world’s leading BI and data warehousing experts with leading software vendors to discuss the latest industry trends, as well as predict future ones.

The Pacific Northwest BI Summit will once again be headlined by Dr. Claudia Imhoff, president of Intelligent Solutions; Colin White, president of BI Research; William McKnight, president of McKnight Consulting Group; Jill Dyche, partner with Baseline Consulting/DataFlux; and Shawn Rogers, vice president of EMA Research. Topics to be debated in spirited roundtable discussions include “Collaboration, Social Data Analytics and Self- Service—This is Not Your Daddy’s BI”; “Big Data and Big Analytics—It’s Not About Terabytes Anymore”; and “BI and Analytics in the Cloud—Is Cloud Really the New Promised Land?”

“The BI landscape has changed dramatically over the past 10 years due to technical innovations and acquisitions. The Pacific Northwest BI Summit provides an excellent forum for the exchange of ideas to help make sense and keep pace—if not stay ahead—of this fast-changing marketplace,” said Claudia Imhoff. She and Colin White are charter members of the Pacific Northwest BI Summit. “The conversations around these formal presentations as well as during our informal gatherings are some of the best I have all year long. I truly look forward to this event more than any other during the year.”

Added Donald Farmer, QlikView product advocate who will be attending his sixth Pacific Northwest BI Summit, “The ability to network with leading influencers in such a small and relaxed setting is unmatched in the industry today. The Pacific Northwest BI Summit affords an opportunity to share ideas with fellow vendors—both established companies and emerging ones as well—in a very informal, yet highly creative environment. I anticipate seeing this event show up on my calendar each and every year.”

A small, diverse group of vendors fill out the attendee list, taking advantage of the rare opportunity to mingle with these true industry experts and influencers in an intimate, relaxed setting. Attending the 2011 Pacific Northwest BI Summit will be representatives from Composite Software, DataFlux, IBM, JackBe, Lyzasoft, ParAccel, Predixion Software, QlikView, SAP, Talend, Teradata and WhereScape.

TechTarget, a leading technology media company, will once again serve as media sponsor of the Pacific Northwest BI Summit.

Gartner’s Merv Adrian to Discuss Future of BI

As part of the Pacific Northwest BI Summit’s 10 year celebration, featured speaker Merv Adrian, Gartner Research VP, Information Management, will offer a keynote address entitled “Extreme Information: Challenges and Opportunities for Large-Scale Data Warehousing, BI and Analytics”. In his presentation, Adrian will examine the definitions and issues involved with managing extreme information and explain why big data is only a small part of the overall concept of extreme information. Adrian will also look at the issues around managing this information with today’s systems and will look at the new technologies and methods for managing extreme information.

Ever wondered how to run an Oracle Procedure when a RED Job completes with a Linux or UNIX Scheduler?

Here’s a simple way that doesn’t require a separate script.

The procedure I want to run is very simple:

CREATE OR REPLACE PROCEDURE dss_test
( p_sequence IN NUMBER
, p_job_name IN VARCHAR2
, p_job_id IN NUMBER
) AS
v_result INTEGER;
BEGIN
v_result := WsParameterWrite('TEST_IT', '1', 'Test');
RETURN;
END dss_test;

This procedure writes a new RED parameter, just as a test. But you could get this procedure to do anything, including calling other RED APIs to restart a job if it fails, etc. The three input parameters give you all the information you need to call any RED Scheduler API from the procedure.

The all you have to do to run this procedure when a job succeeds or fails is paste this command into the job success / failure command boxes:

echo "exec SchemaName.Dss_Test($JOB_SEQ$,'$JOB_NAME$',$JOB_KEY$);" | sqlplus /

Note: If your Linux or UNIX scheduler is not running under an OS Authenticated user, you’ll need to change this to:

echo "exec SchemaName.Dss_Test($JOB_SEQ$,'$JOB_NAME$',$JOB_KEY$);" | sqlplus username/password

When my job completed, I could see a new parameter called TEST_IT in RED.

That’s it! Magic!

In spring of 2010, I began working on a new venture with WhereScape’s client, a procurement organization for a major restaurant chain. The project was developing an Enterprise Data Warehouse (EDW) in a Data Vault “flavor”. The organization had a new management team which understood the benefits of using an agile methodology, so it perfect for me as I had always felt that this organization was well positioned to benefit from adopting an agile approach to data warehousing.

The organization was already a Wherescape RED customer, and already knew all of its great features. While adopting agile techniques, using scrums or even using Wherescape RED is not “the” answer, it can certainly be part of the answer – and they were open to its use. 

The key principles of Agile data warehousing are tightly aligned with WhereScape Pragmatic Data Warehousing™ (PDW) methodology. The following shows the PDW methodology principles in conjunction with the Scrum method that the organization adopted at the beginning of the new EDW.

1. Divide and conquer
A big bang tactic to data warehousing usually ends in disaster. One of the main principles of Scrum is to divide the project (Product) into smaller 2-4 weeks chunks (Sprints) and even smaller 24-hours pieces. “Simplicity–…–is essential”. Projects in Wherescape RED are easily divisible by business area, “star”, “uow”, even individual Data Warehouse objects.

2. Adjust to business users’ needs
WhereScape RED “welcomes changing requirements, even late in development”. Instead of adopting a strict change management process, WhereScape adapts the Agile approach to change management where the company’s stakeholders can easily change their minds according to their changing business needs. Scrum’s relatively short Sprints allow the team to adjust to business’ needs more easily.

3. Support the team
“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done”. Building a team that was not just a collection of individuals working under the same management was one of the immediate benefits of Scrum. We, as teammates, select our own tasks (or choose to work together on tasks). The tasks we choose are small, taking up to a day to complete.

4. Customer satisfaction
Everyone on the team adopted the main goal of agile methodology: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”.  At the daily 15-minute stand-up meeting, each of us talked about what we worked on the previous day, what is planned for today, impediments , and what kind of help we need from a business owner to complete the task.

5. Deliver working components regularly
In other words, “a potentially shippable product at the sprint end”. This way the business community sees and uses the results of the DW/BI development very early and supports, including financially, further iterations of the EDW. Wherescape RED’s elaborate mechanism of versioning and deployment works well here.

6. Fail fast
If a Sprint fails, the failures are fast and cheap. Scrum/ Wherescape RED allows dismissing the development iteration midstream. The team recovers and generates another Sprint in virtually no time.

7. Short timeframes
Strive for iterations of one to two weeks. Iterations of this length provide more opportunity to govern the project effectively due to the greater feedback provided by regular delivery and business users’ input “Deliver working software frequently … with a preference to the shorter timescale.”

8. Involve users early
This is the only way to keep them focused and involved. The business community is a key stakeholder on any DW/BI project. The sooner you involve them, the sooner you find out what their requirements are. The product owner is one of the pillars of the Scrum: “Business people and developers must work together daily throughout the project.”

9. Test throughout the lifecycle
Quality Assurance (QA) is an integral part of every Sprint to avoid common data quality challenges. Testing is embedded into the WhereScape methodology. “Delivered outcome must be correct”.

10. Embrace standards
Embrace common development standards via WhereScape RED functionality and company-adopted governing software. This includes both end-user and technical documentation sets auto-generated by WhereScape RED. Now our client goes all the way with burn-down charts, elaborate storyboards, pivotal tracker, and other highly visible attributes of Scrum.

The bottom line is that now, months into the project, Agile/Scrum/WhereScape RED have blended very well, truly helping the client in reaching its main goal of building a high quality, reliable and well-documented EDW. For more information on the WhereScape RED product, please visit wherescape.com

Note: All the italicized quotations are from the agile manifesto.

I was talking about agile development with one of our software developers, who also used to be a data warehouse consultant.     He made the point that designing and building a data warehouse is harder than designing and building software.  The culprit – data.   The conclusion we came to was that rather than adopting the same practices we should be looking at the best way to adapt the Agile Manifesto to the more complex problem of implementing a data warehouse, otherwise we run the risk of falling into the same traps that the agile movement set out to avoid.

We are not talking about Agile BI.  That is the easy part; it just makes total sense.  Collaboration and responsiveness to change are prerequisites to success.  We can’t imagine a better way of building a report, graph, dashboard etc than sitting down with users, with their data, and showing what is possible.

It is when the data isn’t there that the process becomes harder.  That’s where we see the challenges for agile in the data world.

Agile in the data world needs to extend past the BI layer and into the data warehouse itself.  We have all seen situations where it would have been considerably easier to make changes to the data layer rather than twist the BI tool to perform unnatural acts.

One answer to agile data warehousing is to gather all the data you need, organize it around the business (rather than source systems or requirements).  Once you have this data layer, or enough of it, in place you can start to rapidly build out an agile data  layer – we see these called data marts, dependent data marts,  reporting tables, presentation layer objects, sandboxes and a myriad of other names.  There is nothing wrong with this – in fact we endorse this strategy and actively promote it to organizations with existing data warehouses).  If you have a robust, functional, well designed data warehouse in place, then encouraging usage is not a bad aim.  Stephen Brobst, Michael McIntire and Edward Rado talk about the concept of sandboxing in Teradata environments in their 2008 article Agile Data Warehousing with Integrated Sandboxing.  They go further than I would on governance advocating allowable residency of 90 days for sandbox tables, but in principle we are in agreement.

But what about when you don’t have an existing data warehouse?  What if you cannot wait until you have that data layer completed?

We believe Agile is a great fit for the data world – as long as we embrace the differences that exist between the data and software development worlds rather than ignore them.  The agile manifesto was and is aimed at developing software.  We develop data warehouses.

One of the key differences – we start with data from day one, software developers do not. 

WhereScape’s answer?  Get the data into the equation quicker.  In the information world, we can’t deliver working software without data.  Our designs are theoretical until they are populated.

Yes, WhereScape RED combines data modelling and data integration with the data to remove silos and reduce handovers.  Yes, it integrates meta data into the process for automated documentation and lineage to reduce development time and simplify change.  But the practices are more important than the software.

For Agile to have the same impact it in the data world as it has in the software development world, (and I could argue it should have a greater impact!) we need to make sure we adapt rather than adopt the common practices – while ensuring we adhere to the underlying principles.

We have just put up the archive of a webinar we did last month on a meta data driven approach to data warehouse development.

The webinar description is:

After nearly two decades, databases, development tools, and design approaches have matured greatly to support Business Intelligence. Yet, Data Warehouse projects still struggle to meet the needs of business users and often require resources beyond the reach of many companies. Agile development techniques address the underlying root causes of the roadblocks to success. Unfortunately, most BI development tools were not designed to support this approach. Presenters will illustrate the agile approach, using the industry’s preeminent agile data warehouse development platform, and motivate attendees to consider just how successful agile DW development can be for most businesses, regardless of size.

Why you should attend:

• Learn how to make your current data warehouse nimble and responsive to the dynamic needs of the business.
• Don’t have a proper data warehouse? It’s not as expensive and resource intensive as you think to build one.
• Doing too much via hand coding, without any documentation? A metadata generated approach accelerates the development process.
• Build faster, and manage more effectively by getting agile.

What you’ll take away:

• Free one-hour private consultation focused on your Data Warehousing challenges.
• Data Warehouse Prototyping Whitepaper

To access the webinar yopu will need to register and login here - it is hosted on a third party site.

Enjoy!

I got picked up for tweet I made recently during a webinar.   The webinar was Agile BI Made Easy: Two Proven Paths to Success that WhereScape put on with Composite Software.  This was a new concept for us – while Composite and we believe there are different and smarter approaches to solve common problems, our approaches are quite different, and people will generally end up picking one or other of us rather than both.  Makes for an a more interesting webinar, but that is another story.

Industry analyst Boris Evelson (@bevelson) was the headliner for the webinar, and he made a great pitch for how agile BI can be used in the face of deployment efforts that are often underestimated, have ever-changing user requirements, with a growing user base and rapidly growing volumes and complexity in the data.

I tweeted that it is not the “what” it is the “how” that is changing, which was my take on one of Boris’ slides.  I got picked up on it later by James Kobielus (@jameskobielus), who disagreed and gave a couple of examples that showed the “what” is changing as well.  Now of course he is right – there is a ton of new, innovative ”what” changing out there.  He gave the examples of predictive and current (I am sure he could have given more, but hey, he only had 140 characters).

So yes, he is right, but I still think it misses the point.  Thankfully there is constant innovation being carried out in the business intelligence space, and there is certainly room for a lot more.  But it was great to be actually talking about a problem that has been around for a long time and is generally ignored  – data warehouses take too long to build, and once they are built they are too hard to change.  This is a real problem in the data world.

There are a lot of reasons for this –data volumes, difficulties of scope control in enterprise data initiatives, hard to pin down requirements, refactoring large data sets etc etc.  But that doesn’t mean we should ignore the problem.  The current interest in agile bi and agile data warehousing techniques is overdue and healthy.   

It is great to see an age old issue being exposed and talked about.  And if we can do a better job of building, rebuilding and managing data warehouses it only makes the “what” story even better.

As an aside I can recommend Boris Evelson’s excellent research Agile BI Out Of The Box:  Reduce Development Time And Effort With Metadata-Generated BI Applications , which was the impetus behind the original webinar.  If you would like a copy contact us or leave a comment and we will send one through. 

We recently invited Dr Ken Collier (theagilest.com) down to New Zealand to work with our development team.  As you can guess from the name of his website Ken is an agile kind of guy.  And not one of the crop of recent converts  – he has been working in agile almost as long as he has in information management.

There were some fascinating discussions.   WhereScape has always been a “rapid” development environment, and very much in alignment with the Agile Manifesto:

Manifesto for Agile Software Development

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more

And the associated principles:

Principles behind the Agile Manifesto

We follow these principles:

Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.

Welcome changing requirements, even late in
development. Agile processes harness change for
the customer’s competitive advantage.

Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.

Business people and developers must work
together daily throughout the project.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

Continuous attention to technical excellence
and good design enhances agility.

Simplicity–the art of maximizing the amount
of work not done–is essential.

The best architectures, requirements, and designs
emerge from self-organizing teams.

At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.

But there are certainly things we can do to better support common agile practices.  As a data warehouse development environment we cannot and should not attempt to provide project management functionality, but we can look to see how we can better enable an agile project.

This is the key reason we are working with Ken, and the first deliverables from these discussions are being worked on now.

Getting back to the title…one of the many things Ken talked about that resonated with us was the concept of being constantly deployable.  To illustrate the point he challenged us with “What would happen if we cut the funding on your project at the end of the week?”  It is a question that really got to the heart of being agile – nothing about scrum, sprints, continuous integration testing etc etc – but a question about BEING agile.

It is certainly a question we will ponder on our own consulting engagement and best practice guides, and one we will use with our customers who want to be more agile. 

Watch this space for more information on how WhereScape will enable agile practices.  We will see what we can do to help you if someone asks…. are you ready to deploy this afternoon?