VLDB Solutions Blog

Teradata 6690 Announced

2012-03-09T10:24:00.000Z

Our friends over at Teradata have just announced the new Teradata 6690:

http://www.teradata.com/News-Releases/2012/New-Teradata-Platform-Reshapes-Business-Intelligence-Industry/

I'm not convinced it will re-shape the BI industry (but what do I know), but as the first of Teradata's hybrid SSD+HDD offerings it certainly looks like a step change from the previous HDD-only offerings.

A lot will hinge on the effectiveness of the Teradata Virtual Storage (TVS) software which "automatically moves data as temperature changes ensuring alignment to the most appropriate storage location". We'll see how that copes with the abuse inflicted on it in the real-world.

Curt Monash also has a nice write-up of a chat with Teradata's Carson Schmidt here, which covers the new Teradata 6690:

http://www.dbms2.com/2012/03/09/hardware-and-components-lessons-from-teradata/#comment-289620

With any luck we'll get to kick the tyres a little on the Teradata 6690 at the upcoming Teradata Universe EMEA event next month in Dublin. See you there :-)

Gartner Magic Quadrant for Data Warehouse Database Management Systems part 2

2012-02-20T20:38:00.001Z

It looks like those nice folks at Teradata have ponied up some cash so the world can read what Gartner have to say in their latest "Magic Quadrant for Data Warehouse Database Management Systems".

Thanks Teradata :-)

Gartner Magic Quadrant for Data Warehouse Database Management Systems

2012-02-15T13:24:00.004Z

The annual "Gartner Magic Quadrant for Data Warehouse Database Management Systems" was published last week.

Like many folk out there, I am as interested in what the industry analysts make of the report as the report itself. Curt Monash's comments are here: http://www.dbms2.com/2012/02/08/gartner-magic-quadrant-data-warehouse-2011-2012/

The comments from Curt I picked up on are as follows...

"Teradata really needs to do is evolve to a more pick-your-own-node-combination mix-match kind of offering"

Whilst I agree with this in some respects, and it's something I have always hoped to see, Teradata has always taken the reasonable view that their 'stuff' consists of a tightly integrated stack that only they can assemble, test and certify.

It would also cause support nightmares for Teradata's GSC. At present there is just the eponymous Teradata DBMS running on top of SUSE Linux (with legacy NCR MP-RAS), Dell servers and LSI or EMC storage to support. The BYNET interconnect for MPP systems is also a potential showstopper.

I think we'll but stuck with 2 AMP Teradata VMs and software-only single node Teradata Data Mart SMP servers as the choices for running on 'non-standard' hardware for the foreseeable.

Regarding IBM, Curt states "But Gartner does mention concurrency as a strength. I agree, especially if we presume that that was a reference to DB2 rather than Netezza."

Wise words. Concurrency was an issue that Netezza struggled with in the early days, but doesn't everyone? It's a lot better than it was, no doubt. Workload management on large data warehouse systems is never easy, but it tends to improve over time as products mature.

This really made me chuckle: "in Netezza’s defense, it has had to endure IBM’s post-acquisition on-boarding process."

As a long-term Netezza partner we baulked at the "on-boarding" required to maintain this status once Netezza was gorged by IBM.

The comments relating to EMC Greenplum seem entirely fair and reasonable. The only thing I would add is that until Teradata goes down the 'mix and match' route (see above), Greenplum is likely to maintain it's status as the only 'roll-your-own' MPP play anywhere near Gartner's Data Warehouse Database Management Systems leaders quadrant.

Although the Greenplum DBMS is promoted by EMC as part of the EMC Greenplum Data Computing Appliance, it is still very much available as a software-only offering that can be deployed on a wide range of OS, server and storage choices. This we like, a lot.

Perhaps the most notable of Curt's comments relate to Microsoft:

"there isn’t a single production reference for Microsoft’s Parallel Data Warehouse".

This comes as no surprise. Despite considering ourselves well connected in the DW/BI world, and despite posting on several forums, we had been unable to confirm a single PDW deployment out in the wild.

The 40 billion row Dataupia-powered SQL Server system built by yours truly at ITIS in the UK in 2009 looks even more impressive now ;-)

VLDB's Teradata Partner Page

2012-02-13T08:43:00.000Z

Our friends over at Teradata have been helping us recently with our Teradata partner page on the Teradata web site:

http://www.teradata.com/partners/VLDB-Solutions/

It's coming along quite nicely, so well done to all those involved.

Onwards and upwards...

EMC Greenplum, Teradata and Netezza ramblings

2012-02-11T11:29:00.001Z

Well, that was quite a week....despite following Prof. Marcus du Sautoy onto the stage at the EMC Data Science Series event in London, our presentation covering Search Science was well received. Moscow, Dubai and Hong Kong next? Who knows.

The prospects and web/digital media agencies we met with down in London town were very bullish. The number of inbound enquiries is also on the rise. Some of them are even converting into paying customers in double-quick time. Woo-hoo! It seems like they all say the same thing, along the lines of "we've tried all the other bid management tools, and they're all a) the same and b) underperform". Just what we want to hear!

El Reg wrote up a nice piece, as usual, on Teradata's Q4 performance. It seems our friends over at Teradata can do no wrong at the moment. Long may that continue.

The most interesting point in the article for me is that "the 2000 Series Data Warehouse Appliance machinery is what it tends to sell against Netezza and Exadata iron, and that this appliance business had nearly doubled in 2011 after 90 per cent growth in 2010."

This supports my theory that Netezza is probably the best thing that happened to Teradata (or maybe being spun off from NCR?). How so? Well, quite simply, competition is seen by most as a healthy thing.

Until my good friend Foster Hinshaw and his Netezza boxes arrived on the scene, Teradata largely had the parallel DBMS space to themselves, and it was therefore "reassuringly expensive". You either ponied up the money or struggled along with a traditional "general purpose DBMS + SMP server + SAN/NAS" approach, and we all know that doesn't work at scale. Don't we?

Despite assurances that they "won't survive 18 months", Netezza survived, flourished, managed an IPO and got bought by IBM all in less than 10 years. Quite impressive I'd say.

So how is this good for Teradata?

Well, first of all Netezza proved there was latent demand for a broader product offering in the datawarehouse appliance space than the narrow, premium offering previously only available from Teradata.

Secondly, due to the arrival of the upstart Netezza with their "performance, value, simplicity" message, Teradata was forced to re-appraise their product offerings, which quickly broadened into a much wider range than the premium priced "enterprise-only" platform. Just look at all the Teradata platform goodies on offer now - there's something for everyone.

Teradata now has more platforms being sold in more territories, more users, more revenue and more profit than ever before.

Part of this I chalk up to the arrival of Netezza on the scene. The 'big data' wave has also helped, no doubt. Also, let's never forget just how much those early Teradata developers got right over 30 years ago.

I can still remember how amazed I was when I ran my first BTEQ in the late 1980s at Royal Insurance in Liverpool. Wow - an answer in under 12 hours and no Cobol, this we liked a lot.

EMC Data Science Series

2012-01-27T08:56:00.000Z

I've always considered it dangerous when those sales types go out on the road unaccompained...and I've been proven right once again!

This time I've been volunteered for a speaking engagement at the upcoming EMC Data Science series event: http://www.datascienceseries.com/.

The free 'Big Data' event is taking place on Wednesday 8th February at the Hospital Club in Covent Garden, London.

For those that are interested, and why wouldn't you be, registration is available here: http://www.datascienceseries.com/EVBregister.php

Rumours that my colleague, Richard Jackson, is co-presenting are unconfirmed at this time.

See you in London :-)

Big Data - ebay @ Teradata Partners 2011

2012-01-06T21:09:00.000Z

So, a few days ago, we were talking through various consulting engagements we've had over the years...the kind of marketing collateral discussion you just can't avoid, no matter how hard you might try!

Some of the bigger systems we've encountered are at the likes of Vodafone, Verizon, BT and Nokia. All are big in their own right, but client confidentiality precludes saying any more than that, as you might expect.

Then talk turned to ebay. Not that we've had any dealing with that system. I just remembered that I'd scribbled some notes down from the various ebay sessions at the Teradata Partners 2011 conference in San Diego.

So, here's a few snippets relating to ebay from Teradata Partners:

ebay use a Teradata EDW, a Teradata high capacity appliance system and a Hadoop system (‘horses for courses’)
the Teradata EDW is 6PB and dual-active
the 'Singularity' high capacity system is 40PB and consists of 256 high capacity appliance nodes
the Hadoop system is 20PB
ETL is controlled by Ab Initio and metadata-driven
most feeds are daily with inputs landed on disk as fixed width files
the Teradata loading approach is Fastload/BTEQ
Teradata TD13 compression delivered a 50% IO reduction
maximum loading throughput is 12TB/hour
50TB/day of new data is received
100 trillion name/value pairs are stored in a single table
100 PB/day is analysed, mainly for web site optimisation

In addion to the system metrics above, some words of wisdom that I noted (and agree with):

“Keep atomic data, it supports deep insight’
“Data marts are expensive chaos, which cannot be cheap enough to justify, and lead to data drift”

Ebay seems to have overtaken Walmart as the 'grande fromage' of the Teradata world. They also like to share their story, which is nice.

We're big fans of Ab Initio and the FastLoad/BTEQ approach to Teradata ETL, so it's nice to know there are like-minded folk at ebay.

A 100 trilllion rows in a single table - I'll bet there's no FALLBACK on that baby :-)

El Reg Article Comments

2011-12-01T12:49:00.018Z

For those of you that aren't aware, there is a rather good IT news web site in the UK called 'The Register', or El Reg: http://www.theregister.co.uk

Every few weeks or so there is an article that yours truly feels the need to comment on, sometimes at length. No prizes for guessing that these articles are the ones that talk about databases, data warehousing, Teradata, MPP, 'big data' and the like. As if I could resist!

Anyway, here is a series of links to my article comments on El Reg over the last couple of years:

14/09/2010
http://forums.theregister.co.uk/forum/1/2010/09/13/michael_stonebraker_interview/

05/11/2010
http://forums.theregister.co.uk/forum/1/2010/10/25/teradata_appliance_refresh/

http://forums.theregister.co.uk/forum/1/2010/11/04/teradata_q3_2010_numbers/

24/11/2010
http://forums.theregister.co.uk/forum/1/2010/11/24/numascale_shared_memory_interconnect/

20/01/2011
http://forums.theregister.co.uk/forum/1/2011/01/19/hp_microsoft_frontline_appliances/

24/03/2011
http://forums.theregister.co.uk/forum/1/2011/03/24/intel_hp_itanium_defense/

06/05/2011
http://forums.theregister.co.uk/forum/1/2011/05/05/teradata_q1_2011_numbers/

06/07/2011
http://forums.theregister.co.uk/forum/1/2011/07/05/enterprise_db_hp_ux/

05/08/2011
http://forums.theregister.co.uk/forum/1/2011/08/04/teradata_q2_2011_numbers/

13/09/2011
http://forums.theregister.co.uk/forum/1/2011/09/12/postgresql_9_1_cloud_server/

04/11/2011
http://forums.theregister.co.uk/forum/1/2011/11/04/teradata_in_analytics_crosshairs/

15/11/2011
http://forums.theregister.co.uk/forum/1/2011/11/14/proprietary_interconnects/

29/11/2011
http://forums.theregister.co.uk/forum/1/2011/11/28/data_analytics/

Enjoy!

VLDB and Celebrus partnership

2011-11-03T09:02:00.004Z

Following on from all the good work achieved at Shop Direct, VLDB are delighted to announce a partnership with Celebrus (formerly Speed-Trap):

http://www.celebrus.com/partnerindex.aspx

Here's to many more successful implementations :-)

VLDB - a Partner at Teradata Partners 2011

2011-10-18T12:33:00.005+01:00

It was a delight to (finally!) attend this year's Teradata Partners event as a formal Teradata technology partner, rather than 'just' Teradata specialists.

Many thanks to our friends at Teradata for processing the partner paperwork before the event, during a very busy period for all at Teradata. The Teradata Partners event was as enjoyable and informative as ever.

Roll on Dublin in spring 2012 :-)

Teradata Partners 2011

2011-10-10T13:39:00.011+01:00

We finally made it back to Manchester airport from San Diego at 6pm last Friday.

Despite the relative conmfort of BA's 'World Traveller Plus' (aka upper economy), yours truly decided to watch films back-to-back instead of catching some shut-eye. The result? A very, very tired return to work today. Ho-hum.

Anyway, a few topics of note will be covered separately over the next few days/weeks:

TD14 features

Teradata 2690 appliance

Teradata OLAP connector

ebay's use of Teradata and Hadoop

Aster Data

Alison Torres 'Teradata Index Essentials' book

The main disappointment during this year's Teradata Partners event was the inclement weather that coincided with the night of the gala dinner. Sadly we had to move indoors to the Hilton hotel instead of the USS Midway.

Thursday's guest speaker was none other than Aron Ralston.

Teradata and Oracle Tuning Guide

2011-09-23T14:43:00.003+01:00

Are Teradata DBAs overpaid, or are Oracle DBAs underpaid?

Take your pick ;-)

Teradata Appliance Musings

2011-09-04T17:56:00.010+01:00

Some would suggest, quite reasonably, that Teradata has been an appliance ever since it's birth in the 1980's. I would tend to agree, sort of. Server, storage, OS, DBMS, cabinet...it's all there pre-configured and ready to go. Isn't it?

Well, yes, but...there has always been quite a bit of latitude when choosing the spec of Teradata's 'enterprise class' systems. This allows Teradata to ensure that the architecure of the system matches the requirements of the workload. Choices such as disk size, disk:node ratio (does sir require power or capacity?), RAM/node, number of nodes, etc. All very sensible.

It's not *that* long ago - maybe 8/9 years - that the concept of a 'data warehouse appliance' was born. Teradata's formal appliance offerings came along a few years later and represent various pre-packaged configurations. So as not to cannibalise the 'non-appliance' enterprise offerings, the Teradata appliances essentially use cheaper disk sub-systems and a 'lite' version of the TASM workload management capability. They also can't be expanded/upgraded as per the enterprise class offerings.

Most (all?) long-established Teradata customers seem to still be using enterprise class offerings for the EDW. No surprises there really. As a result we hadn't got our hands on a Teradata appliance out in the wild until recently.

However, we've now been using a Teradata 2650 appliance at a customer site for the last few months. Here's the official blurb: http://www.teradata.com/brochures/Teradata-Data-Warehouse-Appliance-2650-eb6183/?type=BR

A quick summary of the spec is as follows:

CPU

• 2 cores/node and 6 cores/CPU = 12 cores/node
• 2.9GHz CPU

RAM

• 96GB/node RAM – not so long ago this was only 2GB/node on NCR's 32bit MP-RAS
• 4GB RAM/core – 2x typical enterprise class value

VPROCs

• 4GB RAM/AMP - 2x typical enterprise class value

DISK

• 24 disks/node – ¼ typical enterprise class value
• 1 disk/AMP – ¼ typical enterprise value
• 10,000 RPM 300GB or 600GB disks - not 15,000 RPM enterprise class speed
• 18MB/s per AMP nominal scan performance - ¼ typical enterprise class value

COMPUTE

• 330 node specint – 3.5x typical enterprise value

On face value this looked like a compute-heavy, IO-light system by traditional Teradata standards. The proof of the pudding is in the eating however!

The Teradata 2650 was tasked with a variety of KPI reports and ad hoc queries submitted via a mainstream BI tool. User concurrency was in the 50-100m range.

I would summarise the results, quite simply, as 'most impressive'. Certainly better that we anticipated, although I can't go into detail due to customer confidentiality.

With the appliance offerings, it looks like Teradata are learning how to take advantage of an abundance of (cheap) compute power and RAM, rather than just a massively capable (expensive) IO sub-system. Long may it continue.

The Teradta 2650 appliance is an impressive platform and certainly delivers a lot of 'bang per buck'.

Shop Direct Wins Data Driven marketing Award

2011-06-26T19:46:00.010+01:00

Shop Direct recently won the 'Data Driven Marketing Award' at the Marketing Week Engage Awards 2011 (see page 31).

The press release from Celebrus (formerly SpeedTrap) is here: http://www.speed-trap.com/media/news/marketingweekawardwin.aspx

The award was for a campaign to 'Use browsing data to deliver a step change in the relevancy of customer contacts'. The objective was to 'gather almost real-time data' which enabled communication 'on a one-to-one basis based on current motivations rather than previous buying patterns'. The judges noted the outstanding ROI delivered in a short space of time.

So, what's the interest here for yours truly? Well, let's wind back a bit shall we....

Littlewoods in Liverpool and Great Universal Stores (GUS) in Manchester were both early Teradata adopters, having used the technology mainly for database marketing from the late 1980's. Part of this can be attributed to the early success of Teradata's north-west UK sales team, including our own Jim 'the phone' Clarke.

During 2001/02 there was a significant investment in Teradata at Littlewoods that included the deployment of Teradata's recently acquired CRM tool, known as 'Teradata CRM' or TCRM. Oooh, those branders were inspired ;-)

Yours truly was hired as the technical lead on the project, and the standards I wrote back then are largely in use to this day. Or so they pretend!

Littlewoods was subsequently sold to the Barclay brothers, merged with GUS, and became known as Shop Direct Group. Both Littlewoods and GUS had used Teradata for database marketing for a long time, so it was no surprise that the combined group continued to use Teradata.

The move towards increased online sales has meant that web data has become of keen interest to companies such as Shop Direct. How can you engage in meaningful customer interaction if you can't capture, understand and exploit the web data at the customer level? You can't!

The Speed-Trap deployment at Shop Direct, as anywhere, brought the online subject area to the data model - specifically Teradata's Integrated Web Intelligence (IWI) model. The main challenge this brings, unsurprisingly, is to provide combined access to business users to both the 'old' offline data and the 'new' online data. Additionally, there are the not-so-minor considerations of managing the volume of online data and understanding what it represents.

The marketing campaigns enabled through access to both offline and online customer data are already reaping rewards for Shop Direct, within a year of the Speed-Trap deployment, as noted in the press release.

Teradata are justifiably pleased with the project success and have invited Shop Direct's own James Perkins to speak at several events, including Teradata Universe 2011 in Sydney. And that's where I re-join the story...James and I are working on various opportunities to further exploit the web data to drive ROI higher.

I've a feeling this journey is only just starting to get interesting...

Database Express Editions

2011-05-16T22:11:00.018+01:00

We all like free stuff, right? Of course we do! What about free database software? So long as it works...most certainly!

Until a few years ago 'free' database software consisted of open source offerings such as PostgreSQL and MySQL.

Rather than watch the open source upstarts eat their lunch, especially at the bottom end of the market, the major database vendors such as Teradata, Oracle, IBM, Microsoft and EMC have jumped on the 'Express Edition' bandwagon.

So, what exactly is this 'Express Edition' thing all about? Well, DBMS express editions are typically 'crippleware' that supports most/all of the same functionality of the paid-for editions, but with limits on the number of CPU cores, the amount of RAM or the amount of data that can be stored.

Lets take a look at what's on offer in the database express edition world, starting with Teradata.

Teradata Express 13 for Windows

OS - Windows
CPU limit - 2 virtual processors (VPROCs)
RAM limit - 4GB
Database size limit - 10GB
Production use? - no

As the latest incarnation of the 'Teradata demo' software from the 1990's, Teradata Express is a fully-featured learning and small-scale development environment.

Teradata Express 13 for VMware Player

OS - Suse Linux
CPU limit - 2 virtual processors (VPROCs)
RAM limit -
Database size limit - 40GB or 1TB
Production use? - no

For those that want a bigger Teradata Express system than the Teradata Express for Windows offering, the newer Teradata VMware editions are a welcome addition. The Teradata Express for VMware editions also provide an opportunity to understand Linux-based Teradata systems.

EMC Greenplum Community Edition

OS - Unix, Linux, OS X
CPU limit - 2 sockets or 8 virtual CPU cores
RAM limit - unlimited
Database size limit - unlimited
Production use? - yes (subject to CPU limits above otherwise license required)

EMC's Greenplum Community Edition (CE) is the follow-on the the Greenplum Single Node Edition (SNE) from pre-EMC days. The Greenplum CE software is exactly the same as the software for a multi-node Greenplum MPP system, it just happens to run on a single SMP node.

Oracle Database XE

OS - Unix, Linux, Windows
CPU limit - 1 socket
RAM limit - 1GB
Database size limit - 4GB
Production use? - yes

Oracle XE is a breeze for Oracle newbies due to the web-based interface.

IBM DB2 Express-C

OS - Unix, Linux, Windows
CPU limit - 2 cores
RAM limit - 2GB
Database size limit - unlimited
Production use? - yes

Microsoft SQL Server Express

OS - Windows
CPU limit - 1 socket
RAM limit - 1GB
Database size limit - 10GB
Production use? - yes

Teradata Pricing Myth

2011-04-07T21:39:00.006+01:00

I was reading an interesting article in the latest Teradata magazine a few days ago about Teradata pricing:

http://www.teradatamagazine.com/v11n01/Viewpoints/Busting-the-Pricing-Myth/

A very well presented case, I thought. Teradata has always been regarded as "reassuringly expensive" which is not a label it may ever shake off, no matter how many articles, papers, beiefings and presentations suggest otherwise.

A recent conversation with a client re-inforces this belief..."we can't afford Teradata" I was advised. Funny that, given that they don't know what it costs, thought I.

The advent of data warehouse appliances and competition from the likes of IBM's Netezza (not DB2!), EMC's Greenplum, HP's Vertica (not NeoView!) and err, Oracle and Microsoft, seem to have had an impact on Teradata's pricing power.

I'm all for healthy competition, and I'm all for more Teradata sites out there.

For anyone wanting a FREE version of Teradata, take a look here:

http://www.teradata.com/t/teradata-express/

Enjoy!

Loading Greenplum From An External Table

2011-02-24T16:56:00.003Z

Ok, we've got Greenplum up and running and now we want to load some data. There are several methods available, of which we'll look at external tables.

This demo will load a simple file containing 2 fields and 10 records into an empty table.

Here is the input file:

gpadmin$ more test1.txt

1,aaa
2,bbb
3,ccc
4,ddd
5,eee
6,fff
7,ggg
8,hhh
9,iii
10,jjj

So, let's create an external table called 'test1':

CREATE EXTERNAL TABLE TEST1 (COL1 INT, COL2 TEXT) LOCATION ('gpfdist://gpdemo:8080/*') FORMAT 'TEXT' (DELIMITER ',');

We now have a table that maps to an external file. This table can be queried just like a normal table:

test=# select * from test1;

col1 | col2
------+------
1 | aaa
2 | bbb
3 | ccc
4 | ddd
5 | eee
6 | fff
7 | ggg
8 | hhh
9 | iii
10 | jjj
(10 rows)

To load the 10 records from the external table 'test1' to the table 'test_load' we simply run an INSERT/SELECT statement via psql:

INSERT INTO TEST_LOAD SELECT * FROM TEST1;

The target table 'test_load' now contains the same data as the external table 'test1' from which it has been loaded:

test=# select * from test_load;

col1 | col2
------+------
2 | bbb
4 | ddd
6 | fff
8 | hhh
10 | jjj
1 | aaa
3 | ccc
5 | eee
7 | ggg
9 | iii
(10 rows)

So there you have it - a simple load into Greenplum from a file that is treated as an external table.

HP: bye-bye Neoview, hello Vertica

2011-02-15T08:49:00.002Z

HP has announced that it is buying Vertica, the latest DBMS offering from Michael Stonebraker:

http://www.theregister.co.uk/2011/02/14/hp_buys_vertica/

This would seem to make sense for HP following the demise of Neoview.

Who's next..Paraccel, Dataupia?

Greenplum Performance Monitor

2011-02-14T19:59:00.005Z

Following the recent installation of Greenplum Community Edition v4 on my Mac, I decided to give the Greenplum Performance Monitor a spin.

The install guide highlights are as follows:

gpperfmon_install --enable --password gpmon --port 5432
gpperfmon –setup
gpperfmon --start gpdemo

Once installed and started, the Greenplum Performance Monitor web UI is available on 'gpdemo' via https://gpdemo:28080/:

Logging is as 'gpmon' reveals a slick, fast-loading interface:

Summary - Greenplum Performance Monitor on OSX is easy to install, start and use. Woo-hoo!

Gartner Magic Quadrant for Data Warehouse Database Management System

2011-02-11T12:15:00.003Z

The latest Gartner Magic Quadrant for data warehouse platforms can be viewed here:

http://www.gartner.com/technology/media-products/reprints/teradata/vol3/article1/article1.html

Interesting to note the proximity of Greenplum to Netezza in the leaders quadrant, along with IBM, Teradata and Oracle.

Amazing what being bought by EMC can do for a company!

Perhaps most noteworthy is the dash to MPP. It was only a matter of time.

More comments will follow in due course...

Greenplum Community Edition (SNE) V4 for OSX

2011-02-09T14:01:00.020Z

VLDB's own demo Greenplum MPP system is currently running v3.x of the Greenplum DBMS software, which we had planned to upgrade at some point soon. Pesky client projects always seem to get in the way ;-)

Hang on a minute, we thought...we've got shiny new Apple MacBook Airs and Greenplum Community Edition (CE) v4 has just been released, so why not try Greenplum v4 out on Mac OSX. Why not indeed?

Greenplum Community Edition (formerly Greenplum Single Node Edition, or SNE) info is here:

http://www.greenplum.com/products/community-edition/

Greenplum Community Edition v4 can be downloaded here:

http://community.greenplum.com/

The Greenplum CE v4 installation on OSX was carried out using the install guide, plus the following excellent topic on the Greenplum community forum:

http://community.greenplum.com/forums/showthread.php?29-Installing-Greenplum-SNE-on-Mac-OS-X-10.5

You will need both!

Once installed, the Greenplum DBMS is started as usual in a terminal window on the Mac (snipped):

gpdemo:~ gpadmin$ gpstart

[INFO]:-Master Started...
[INFO]:-Successfully started 2 of 2 segment instances
[INFO]:-Starting Master instance gpdemo directory /gp/master/gp-1
[INFO]:-Command pg_ctl reports Master gpdemo instance active
[INFO]:-Database successfully started with no errors reported

The Postgres 'psql' SQL client connects to the 'gpperfmon' database interactively:

gpdemo:~ gpadmin$ psql gpperfmon

psql (8.2.14)

Type "help" for help.

To list the tables installed in the 'gpperfmon' database (snipped):

(42 rows)

To count the rows in a table:

gpperfmon=# select count(*) from system_now;

count
-------
1

(1 row)

To count the rows in a table non-interactively, exit interactive psql using '\q' and run psql from the command prompt:

gpdemo:~ gpadmin$ psql gpperfmon -c 'select count(*) from system_now;'

count
-------
1

(1 row)

To run the same SQL from a shell script and send the output to a log file:

gpdemo:sql gpadmin$ more system_now.sh

psql gpperfmon -f system_now.sql > system_now.log 2>&1

gpdemo:sql gpadmin$ more system_now.sql

select count(*) from system_now;

gpdemo:sql gpadmin$ ./system_now.sh

gpdemo:sql gpadmin$ more system_now.log

count
-------
1

(1 row)

The Postgres admin tool 'PGAdmin' was also downloaded, installed, configured and pointed at the newly-minted Greenplum Community Edition for OSX install.

Download PGAmin from here: http://www.pgadmin.org/download/

This took under 5 minutes from start to finish.

Conclusion: Greenplum Community Edition V4 - so far so good :-)

HP NeoView RIP

2011-01-25T00:35:00.003Z

HP has announced the end of the NeoView line of data warehouse appliances, as predicted last week by yours truly.

The El Reg article is here:

http://www.theregister.co.uk/2011/01/24/hp_kills_neoview_appliance/

Wonder if any will turn up on ebay?

Jim The Phone

2011-01-20T21:19:00.003Z

I'm writing this in a bar on a business trip to the USA...how apt, I'm sure Jim would have approved.

Anyway, let's start at the very beginning, a very good place to start.

I first met Jim Clarke back at Royal Insurance in Liverpool in the late 1980's. I had recently moved from working on IMS and Cobol in the application support team (yuk!) to the reporting function in the Information Systems (IS) division. As a graduate trainee we got moved around every 6 months, so this was not unusual. Little did I know that this move would shape the next 20 years, and maybe longer, for yours truly.

My new role consisted of support and development of the Householders Information System (HIS). This contained historical data relating to policy holders and claims, and was refreshed every 3 months from the IMS policy, claims and accounts OLTP systems. Forget real-time, daily or even weekly feeds. How about quarterly!?!?!?!

HIS ran on a new-fangled machine called 'Teradata', with the Teradata FastLoad, MultiLoad and BTEQ client software running on the IBM mainframe. No escape from JCL, but at least IMS was consigned to history. Phew.

Royal Insurance were very early Teradata adopters, thanks to Jim and the team in the northern Teradata sales office. The Teradata V1 architecture consisted of the Teradata DBMS running on top of a proprietary OS (Teradata OS or TOS), which crashed a lot, powered by Intel x86 chips which each owned it's own disk. I think we may have had about 40 CPUs at the time.

Little did I realise I had moved onto the world's first commercially available Massively Parallel Processing (MPP) technology. The name made sense – 'Tera' (byte) and 'data' = 'Teradata', and a terabyte seemed like a very, very big database at the time. SQL beat the hell out of Cobol, was all that I cared about.

Due to the 'reassuringly expensive' nature of the Teradata technology (ring any bells?), the biggest issue we seemed to have was a lack of space. No sooner was it made available than the users found more data they just had to have loaded. Some things never change in that respect.

As a consequence of the users insatiable desire for more space, Jim was always on hand to make sure an upgrade was available. He always made sure the Teradata SE team kept finding more data to load, he confided many years later.

After one particular splurge on more space Jim was stood nearby talking to his then wife on his mobile phone. Yes, as mobile phone in about 1989! 'How very flash' thought I, or words to that effect. I seemed to remember Jim being proud of the commission he'd just earned on the upgrade. It would have been like a lottery win to yours truly, so overhearing this smug Teradata salesmen telling his wife via mobile phone within hearing distance was a tad annoying, to say the least.

Frustrated with lack of career opportunities, I left Royal Insurance in '92 and went freelance, initially consulting to Bank of America in San Francisco, on Teradata and Nomad (remember that?). Jim and his team were very successful in the north of England. In addition to Royal Insurance, Teradata sites popped up at Littlewoods, Great Universal Stores (GUS), TSB, Makro, Co-operative Retail, Grattan, JD Williams and Asda, to name a few.

Jim stayed at Teradata, which was acquired by NCR, which in turn was acquired by AT+T. NCR was eventually spun off from NCR, and Teradata was spun off from NCR before going public a few years ago. Before his departure from NCR/Teradata in the mid-1990's, Jim was head of government operations responsible for selling to Her Majesty's finest (cough).

During 2001-02 I was engaged on a Teradata upgrade project at Littlewoods (now ShopDirect) in Liverpool. Jim was working for SAS at the time, and was also engaged on the project. Our paths never crossed, which seems most unlikely in retrospect.

In the summer of 2003 I was riding my bike slowly uphill past the car park of our local pub – The Shrewsbury Arms, aka “The Shrew”. A convertible Jaguar XJS rumbled slowly past, turned into the car park and stopped.

As it was a V12 engined XJS, identified by the badges on the exterior, I pedalled over and asked the driver where he took his car for care and attention. Jim took out a SAS business card and scribbled down the contact details for “RG Bate Engineering” on the back. As the owner of a V12 engined Jaguar E-Type, this was a very welcome bit of information. I said thanks and off I went.

My own E-Type has been entrusted to Bob Bate ever since Jim gave me his contact details. One day I was chatting to Bob and remembered I had been given his contact details by Jim, written on the back of a SAS business card. We were looking to expand our sales capability so I asked Bob if he would pass my details onto Jim and ask him to call me.

Jim phoned the same day and arranged to meet the next afternoon in The Shrew (where else?). As he began to tell me his story I smiled more and more. Eventually it became apparent...this is the man himself sat in front of me...”you're Jim The Phone”!?!?!?!

Some 14 years after out paths first crossed at Royal Insurance, and after both being engaged on the same project at Littlewoods a few years earlier (without knowing it), here we were sat in our local pub, the much-loved Shrew where Jim even had his own dimpled pint pot behind the bar. Jim happened to live in the next road to me, about a 5 minute stroll away. There is more Teradata talent in Oxton Village, but that's another story.

Jim and I became business associates and very good friends from this point onwards. After many sales missions and business trips together I never tired of his company. He seemed to know everyone, especially those that worked in every decent pub up and down the country between home and the south-east.

His self-appointed nickname was “septic knuckles” - a reference to the gusto with which he enjoyed opening a sales opportunity. During the volcanic ash cloud incident last year Jim managed to get himself stranded and unable to get home...from Majorca. Typical!!!

Due to his public school accent, attire, and general demeanour, Jim reminded me of the old-school actor Terry Thomas. As a result I would answer the phone “ding-dong” whenever he called. He got his own back with that iron handshake. God help those who weren't prepared for it. You could lose fingers if you weren't careful.

Although Jim had experienced health problems over the last few years, his career was very much on an upswing again selling software and services to the NHS. His sudden death before Xmas came very much as a shock to very many of us.

Jim, we sent you off in the style you would have appreciated – several beers in West Kirby sailing club and The Shrew. Even Bob Bate turned up. You were a true one-off and it was a pleasure to have known you and enjoyed so many pleasurable times together.

“Ding-dong”.

Microsoft and HP Data Warehouse Appliance

2011-01-20T19:00:00.004Z

Remember Microsoft bought Datallegro a few years ago? Remember HP provided the hardware for the Oracle Exadata v1 offering?

Well, since Oracle bought Sun last year things have moved around a little. Oracle now runs Exadata v2 on Sun kit, unsurprisingly, leaving HP out in the cold with regard to Exadata. Or maybe not? There's always HP's very own NeoView offering. Moving on...

Microsoft's project Madison aimed to roll the IP acquired in the Datallegro purchase into a scaled-out parallel version of SQL Server. It finally appeared as the catchily named "SQL Server 2008 R2 Parallel Data Warehouse" (in 2010) which promises "a highly scalable appliance that delivers performance at low cost through a massively parallel processing (MPP)" (sic).

The partnership with HP aims to roll the parallel version of SQL Server into a full-fledged data warehouse appliance so that punters/re-sellers/system integrators don't make a mess out of putting it all together. That's one justification anyway.

The El Reg article is here:

http://www.theregister.co.uk/2011/01/19/hp_microsoft_frontline_appliances/

Curt Monash's take over at dbms2.com is here:

http://www.dbms2.com/2011/01/19/sound-bites-on-hpmicrosoft-and-neoview/

The Microsoft press release is here:

http://www.microsoft.com/Presspass/press/2011/jan11/01-18HPMSAppliancesPR.mspx

The HP press release is here:

http://www.hp.com/hpinfo/newsroom/press/2011/110119xa.html

Only time will tell if this new HP/Microsoft offering takes a chunk of the business currently being fought over by Teradata, Netezza, Oracle and co.

HP Neoview RIP? Seems more likely than not. Will anyone notice? Doubt it.

VLDB SIGNS PARTNER AGREEMENT WITH NETEZZA

2010-11-10T21:47:00.001Z

VLDB Solutions Ltd, a leading provider of business intelligence, data warehousing and systems integration services, today announced a Professional Services (PS) partnership agreement with Netezza, a leading provider of data warehouse appliance technology.

VLDB's proven track record in delivering large-scale, high-performance data warehousing solutions for major blue chip clients in the financial, retail and telecommunications sectors will be further enhanced through closer working with Netezza's appliance technology for the benefit of both companies clients.

Welcoming the partnership agreement, Paul Johnson, Managing Director of VLDB commented:
"Having identified Netezza as a disruptive data warehousing appliance technology from the early days, we are delighted to formalise the relationship as Netezza's migration to the TwinFin architecture delivers further technological disruption that the team here at VLDB Solutions can exploit to deliver yet more successful projects.'

About VLDB Solutions Limited

Based in Liverpool, UK, VLDB Solutions Ltd is a highly experienced Systems Integrator (SI) specialising in data warehousing, business intelligence and data mining. Working with blue chip companies across many industry sectors, VLDB is a leading vendor-independent UK consultancy in the 'Very Large Database' space.

Working with client project teams, VLDB brings in-depth knowledge and experience of high-end massively parallel processing (MPP) technologies to solve the 'big data' challenge. VLDB helps clients to plan, select, design and deploy technology-based solutions to solve real-world business challenges.

For more information about VLDB Solutions Ltd, please visit http://www.vldbsolutions.com.

About Netezza Corporation

Netezza Corporation is the global leader in data warehouse, analytic and monitoring appliances that dramatically simplify high-performance analytics across an extended enterprise. Netezza's technology enables organizations to process enormous amounts of captured data at exceptional speed, providing a significant competitive and operational advantage in today's data-intensive industries, including digital media, energy, financial services, government, health and life sciences, retail and telecommunications.

For more information about Netezza, please visit http://www.netezza.com.

2010 Teradata PARTNERS User Group Conference & Expo

2010-11-08T21:06:00.000Z

Images from the 2010 Teradata PARTNERS User Group Conference & Expo, October 25 - 28 in San Diego, California.

http://www.flickr.com/photos/partners_user_group/sets/72157625217138048/

Go Teradata

2010-11-08T21:05:00.000Z

Data warehousing and analytics is not a booming business, but it is doing a lot better than other sectors of the IT economy. Teradata is benefiting from the increasingly complex data types and sources that companies are chewing on to help drive new business and keep current customers happy.

In the third quarter ended in September, Teradata's hardware and software revenues grew by 27 per cent to $243m. Data warehousing appliances sales are still around five to 10 per cent of sales, the company said, and no customers have yet converted from appliances up to the full-tilt Enterprise Data Warehouse clusters.

http://www.theregister.co.uk/2010/11/04/teradata_q3_2010_numbers/

Teradata Appliance Refresh

2010-11-08T21:02:00.001Z

Teradata might be the pioneer of data warehousing on cheap x64 server clusters and the use of appliance packaging to tune machines and their software to attack specific workloads, but Oracle and IBM want to eat Teradata's lunch. And its breakfast and dinner, too. That means Teradata has to keep upgrading its hardware and database software and partnering to bring more functionality onto its data warehouse and analytics appliances, making them more useful to the customers who shell out big bucks for them.

At the Partners user group conference in San Diego today, Teradata launched a completely refreshed lineup of entry, midrange, and high-performance data warehousing and analytics appliances, and also tossed in a new flash-heavy extreme performance machine to take on Oracle's new Exadata X2-8 appliances and IBM's Smart Analytics System (SAS) appliancesas well as the Netezza appliances that will soon become part of the Big Blue product catalog unless someone swoops in and tries to steal away Netezza. (This means you, NEC or Dell.) EMC is also putting the heat on Teradata with its own Greenplum Data Computing Appliances, and has much deeper pockets than a free-standing Greenplum could ever hope to have.

http://forums.theregister.co.uk/forum/1/2010/10/25/teradata_appliance_refresh/

Teradata Q3 Numbers

2010-11-08T21:01:00.001Z

Among new accounts, the appliance products sold by Teradata are being picked up by under half of the new customers. But as Mike Koehler, president and chief executive officer at the company, said in a conference call with Wall Street analysts, the appliances allow Teradata to chase new opportunities among smaller customers and for very specific niche data crunching needs.

http://forums.theregister.co.uk/forum/1/2010/11/04/teradata_q3_2010_numbers/

IBM to Acquire Netezza

2010-11-06T10:23:00.000Z

IBM to Acquire NetezzaExpands Business Analytics Capabilities Through Workload Optimized Systems

ARMONK - 20 Sep 2010: IBM (NYSE: IBM) and Netezza Corporation (NYSE: NZ) today announced they have entered into a definitive agreement for IBM to acquire Netezza, a publicly held company based in Marlborough, Mass., in a cash transaction at a price of $27 per share or at a net price of approximately $1.7 billion, after adjusting for cash. Netezza will expand IBM's business analytics initiatives to help clients gain faster insights into their business information, with increased performance at a lower cost.

http://www-03.ibm.com/press/us/en/pressrelease/32514.wss

IBM to buy Netezza

2010-10-13T21:57:00.000+01:00

"They won't last a year" warned a VP from one of Netezza's competitors in 2003. I've always chosen my own friends, so (predictably!) took little notice.

Well, they made a splash in the market, managed an IPO and are now getting folded into 'big blue':

IBM press release

Netezza press release

Now where did I put that options certificate...

EMC Greenplum Data Computing Appliance

2010-10-13T21:31:00.000+01:00

So, EMC have rolled their recent Greenplum buy through the marketing department and come up with the 'EMC Greenplum Data Coumputing Appliance'.

The biggest casualty looks like the Greenplum logo, which is now blue...well it is in the marketing email anyway ;-)

There's also plenty of nice content over at Chuck Hollis' blog.

Clearly a man after my own heart: "From a pure technology architecture perspective, the path ahead seems clear: MPP shared-nothing models that scale out in linear fashion that exploit the mainstream of industry standard technology."

Wise words indeed.

EMC Buys Greenplum

2010-08-11T19:25:00.000+01:00

In a move that surprised even some of our friends at Greenplum, EMC became their new employer last month as reported in El Reg:

http://www.theregister.co.uk/2010/07/07/emc_greenplum/

...and DBMS2:

http://www.dbms2.com/2010/07/07/more-on-greenplum-and-emc/

Curt Monash posted the following after meeting the folks at both EMC and Greenplum:

http://www.dbms2.com/2010/08/09/emc-greenplum/

We've been fans of Greenplum since meeting Luke and Scott whilst still in pre-Greenplum 'Metapa' days at a Teradata Partners show in Seattle. It is an MPP play after all, how could we not like it!

Let's hope EMC propels Greenplum's to a wider audience. That's all that's been lacking so far from where we're sat.

Good luck chaps!

P.S. For anyone that wants to try Greenplum we has a demo MPP system at VLDB Solutions.

Mark Hurd Leaves HP...shocker!

2010-08-11T18:49:00.000+01:00

Mark Hurd leaves HP, so what, where's the angle? Well, he was the CEO at NCR when NCR owned Teradata...that's the angle!

That man that used to run NCR, that used to own Teradata, left HP a few days ago as reported on El Reg:

http://www.theregister.co.uk/2010/08/06/mark_hurd_sexual_harassment/

The possible impact on HP was reported by the New York Times:

http://www.nytimes.com/2010/08/09/technology/09hp.html?_r=2&ref=technology

Mr. Hurd's tennis buddy Larry over at OraSun was less than impressed:

http://www.theregister.co.uk/2010/08/09/ellison_defends_hurd/

El Reg thane asked the obvious question - did HP over-react to the wrongdoings:

http://www.theregister.co.uk/2010/08/10/hp_board_overreact/

Having met the man himself and listened to him talk at several Teradata Partners events, it came as no surprise to me when Mr Hurd became the CEO at HP following Carly's departure. A steady hand to steer the ship through choppy waters and to cut costs was needed, in the opinion of the board, and that's what HP got.

I'm still surprised that HP's Neoview was a) allowed off the drawing board (should have bought Teradata instead) and b) has failed so badly. At least it meant HP never bought Teradata, which I always thought was highly likely when Mr. Hurd left NCR and became CEO at HP.

We live in interesting times indeed.

Data Warehouse Expansion/Upgrade via SMP Scale-Up

2010-06-09T13:14:00.000+01:00

Most data warehouses run on SMP servers with SAN/NAS as the storage.

How do we know this? Well there can't be many data warehouses running on a single CPU, surely? There aren't many running MPP - maybe a few thousand globally (Teradata - thousands, Netezza - hundreds)? That leaves the rest, probably tens or hundreds of thousands of systems running the traditional SMP stack.

So what happens when Johnny User inevitably complains about query performance, and/or lack of space to hold his wondrous creations? Fast is never fast enough and more space is never enough. This is hardly new news to most of us.

So, when the inevitable complaint reaches IT what happens? Well, most of the time an 'upgrade' is proposed. The sales guy promised it would be X times faster or Y% bigger, the users are footing the bill (ho, ho!) and the techies are more than happy noodling away adding better/faster/more 'stuff' to the system. An all round winner, surely?

This is the 'SMP scale-up' or 'fat node' approach in action - throwing more 'stuff' at the problem e.g CPUs, RAM, disks and making a fatter SMP node. But, crucially, it's a fatter single SMP node.

What are the benefits of this approach:

- uses 'commodity' x86 servers from HP, Dell, IBM etc
- uses existing SAN/NAS infrastructure
- DBMS skills abundant in-house
- OS skills abundant in-house

Even when there is compelling reason for a platform change, at least in the eyes of a 3rd party observer, this is often the last thing that will get discussed. Why? Well, 'turkeys don't vote for Xmas' and DBAs don't vote for change."Will this work?" asks users or management..."of course it will, after we've upgraded to version X and added more Y..." reply the techies.

So off they go and build a star schema with a plan to "add more stuff" as and when it's needed as the get-out-of-jail-card.

SMP data warehouses use general purpose/OLTP databases such as SQL Server or Oracle to support BI, which is not what they were designed for. Fundementally though, the underlying single SMP 'fat node' does not scale and every upgrade costs more than the last - diminishing returns in action.

'Fat node' drawbacks include the following:

- poor load performance
- poor backup performance
- poor query performance
- poor query concurrency
- poor unload performance
- poor query optimisation
- impact on shared SAN/NAS users

It's also hard to know why it doesn't work when the stack consists of a general purpose DBMS running on SMP tin and SAN/NAS storage, which is often shared ('not my fault' syndrome). Ironically, none of the components are normally broken, it's just resource contention inherent in the approach.

So, if the SMP 'fat node' stack is hard to tune and lacks scalability, why do most data warehouses end up getting built this way? Well, at the outset it looks/feels like the right thing to do ("we have all the parts and the skills") and there is often a mistaken belief that it will work. The rest of the industry can't be wrong, after all.

Data volumes in the past may have meant that most sites could actually get away with this approach, but this is increasingly not the case. The evidence in support of this is the recent advent of Oracle Exadata and Microsoft SQL Server 2008 R2 Parallel Data Warehouse ("Madison").

Oracle and Microsoft clearly recognise the limits inherent in the SMP 'fat node' approach to data warehousing.

The future is parallel ;-)

Footnote - I actually presented a paper on SMP and MPP scaling several years ago.

Data Warehouse Upgrade/Expansion Choices

2010-04-26T15:58:00.000+01:00

Data warehouses are designed to store data. Normally lots of data. Sometimes lots and lots of data. You get the point. The rate of growth of a data warehouse often exceeds initial expectations, especially those that meet user expectations.

Some data warehouses consequently become victims of their own success. Technology constraints then become an issue. "Too slow" or "not enough space" are the common user gripes. Are they ever happy!?!?!?!

Once up a time when a data warehouse platform had 'run out road', the predictable response was twofold - software and hardware upgrades:

software - upgrade the OS, upgrade the DBMS, maybe create more indexes.

hardware - more RAM, faster CPUs, more CPUs, more storage, faster storage.

A mixture of software and hardware 'upgrades' was/is often the knee-jerk response to end user gripes. In essence, whatever it took to buy more time. Not a real solution, more a deferral so that addressing the underlying problem can wait until a later date.

Upgrading and/or adding more 'stuff' to the traditional SMP+SAN/NAS stack is the time-honoured 'scaling up' of the single SMP 'fat node' that underpins the majority of data warehouse platforms.

So, what's the point???

Well, it has become apparent in the last few years that there are now more choices available to those who have run out of data warehouse road.

SMP scale-up is still a choice (see above), although diminishing returns will remain an issue.

Rip and replace with an MPP 'appliance' e.g. Teradata, Netezza...or even Microsoft's upcoming parallsl SQL Server edition (project Madison, the Datallegro purchase), Oracle's Exadata or HP's NeoView.

Enable the existing SMP stack with a complimentary MPP appliance e.g. Dataupia.

Rip and replace with 'roll your own' MPP .e.g GreenPlum.

Decisions, decisions...the good news is that there are many more choices available now when your data warehouse needs more oomph, more space, or both (most likely).

The bad news is, there is more choice!

With any luck I'll find time to talk about our experiences to date with each of these approaches, out there where it matters, in the real world.

Netezza April Fool Jolly Jape

2010-04-04T19:10:00.000+01:00

Those wags over at Netezza (or maybe just the UK's very own Tim Young?) have come up with a brilliant bit of April Fool humour:

Netezza Mind Activiated Queries

Netezza's nzThink beats the iPad hands down on my 'must have' list...fo' sho'!

Well done Tim and gang :-)

Netezza In-Database Analytics Announced

2010-02-24T10:35:00.000Z

Netezza has announced the ability to run 'number crunching' style analytics inside the DBMS, thus avoiding the cumbersome task of exporting data to run models etc.

It also neatly allows the power of the parallel engine to set to work on demanding tasks that would otherwise run on an SMP box.

As ever, read all about it at The Register:

http://www.theregister.co.uk/2010/02/24/netezza_data_analytics/

Enjoy!

Happy Xmas (and New Year soon)

2009-12-24T12:29:00.000Z

Well, the bookies have slashed the odds on it being a white Xmas in the UK, and who can blame them.

A few inches of snow have predictably brought much of the country to a standstill. Even the train broke down in the channel tunnel. Whatever next???

Anyway, hope everyone has a great time over the Xmas and New Year festive period, and fingers crossed for an enonomic recovery in 2010.

Greenplum Single Node Edition

2009-11-15T11:18:00.000Z

I've finally had to chance to follow-up on the Greenplum single node edition news.

See here for details from Greenplum:

http://www.greenplum.com/single-node/

Curt Monash's blog post on the subject is here:

http://www.dbms2.com/category/products-and-vendors/greenplum/

It makes a lot of sense from both the vendor's perspective and the user's perspective to start with an affordable SMP edition of an MPP product. And free is certainly cheap.

This approach allows the user to dip a toe in the water with no investment in lot's of tin for an MPP setup required from the outset.

Once happy with the technology it's comforting to know that scaling out from SMP to MPP can be achieved without changing the DBMS, ETL and applications. The cost and risk of such a change can be a major barrier to MPP adoption based on my experiences.

If things don't go well, the SMP edition it can be simply switched off as there is no software investment to re-coup. So, on the face of it the whole situation has been significantly de-risked from a user perspective.

The detail is as follows:

single SMP node

up to 2 CPU sockets

any number of cores (up to 8 on a VM)

any amout of storage

column and row storage supported

SQL And MapReduce supported

free community support or paid-for support available

OSX, Linux or Solaris versions available

It will interesting to see if this drives Greenplum adoption.

VLDB on Facebook

2009-11-06T18:14:00.000Z

It had to happen:

http://www.facebook.com/pages/VLDB-Solutions-Ltd/179585921832

No doubt there will be some light-hearted stuff posted on there from time to time.

Enjoy!?!?!

New Domain and Host

2009-10-25T22:57:00.000Z

The service at our old host has deteriorated to the point where a move to a new home became inevitable.

I also took the opportunity to change the domain from olap.bi to mpp.bi.

So, a new domain and a new host...the migration process seems to have screwed up the line breaks in the old posts.

Good old mysqldump!

Dearly Departed

2009-10-13T20:59:00.000+01:00

It is with great sadness that we learned of the recent passing of Mike Larkins. Although Mike had been ill for some time he never let it get him down.

As well as being a real old-fashioned gent, Mike was a Teradata and SQL guru and a wonderful trainer. He was happy to take a plane across the pond and come and help us out several times a year with Teradata training to our UK customers.

We were always very happy when Mike was coming as we knew the course would receive rave reviews from the students. Few things in life are as predictable.

Mike, we'll miss you.

It's 10 years since Peter 'Blakers' Blakeburn passed away. Peter's former colleagues from the LloydsTSB data warehousing team are organising a night out in Manchester in late November/early December.

I'll be raising a glass for the best manager I ever worked for, and I'm sure there are many others that feel the same way.

VLDB Solutions Ltd

2009-09-21T10:58:00.000+01:00

As some folks may have noticed, since leaving Dataupia I have been running a 'boutique' BI consultancy that goes by the name 'VLDB Solutions Ltd'.

The name sums up the company focus quite well.

We offer professional services primarily aimed at companies and organisations that have, or would like, a BI solution based on a very large database (VLDB).

The web site is here:

www.vldbsolutions.com

Netezza World Tour 2009

2009-09-21T10:35:00.000+01:00

The Enzee Universe World Tour 2009 hit London town last week. I won't dwell on the 5am start and 6am train to London too much.

Anyway, the event took place in the lower ground floor of the Mayfair hotel. This was a very clever move as no-one could get a mobile phone signal which meant the presenters at least got some attention. The lobby was rammed with folks making calls and trying to get online.

It was good to catch up with some of the Netezza 'old guard' that I hadn't seen for a few years. One of the highlights was Philip Howard forgetting Jim Baum's question during the keynote speech. Sorry Philip, that had to get a mention!

Anyway, I was very impressed with Jim's keynote, not having met him before he came across very well indeed. Needless to say Tim Young made the audience laugh more than once. How many references to Kylie Mingogue's bum would you normally get at a BI conference? Not many.

It looks like Netezza expects the new 'twin fin' hardware to be as disruptive as when Netezza originally launched in 2003. There are a lot more players in the market now, for sure. Will that be an issue for Netezza or will they be as disruptive as they'd hope? Only time will tell.

One thing's for sure...they need a bigger venue for next year's London event, with or wothout any increased attention cause by 'twin fin'. The Mayfair was simply too small to handle the number of attendees. That's the only small negative in an otherwise very enjoyable event.

Dataupia White Knight?

2009-09-04T18:48:00.000+01:00

Looks like Dataupia has found a buyer and several of the ex-staffers have been re-hired.

I expect public announcements will be made in the not too distant future.

Watch this space, as they say!

Dataupia Fire-Sale

2009-08-12T16:16:00.000+01:00

Well, it looks like there is a skeleton crew of 7-8 staff remaining with a fire-sale imminent, maybe to one of the major DBMS players???

I expect an announcement in the next 2 weeks.

Watch this space, as they say!

Last One Out Turn Off The Light?

2009-08-07T15:00:00.000+01:00

It looks like Dataupia have finally announced that they're looking for a buyer:

http://www.dbms2.com/2009/08/05/dataupia-is-officially-for-sale/

I have a feeling that it won't be sold for nearly enough to make the options worth anything...ho, hum.

Dataupia Gets Crunched

2009-06-10T10:00:00.000+01:00

The credit crunch well and truly bit Dataupia recently.

A pretty accurate version of events can be found here:

http://www.xconomy.com/boston/2009/06/09/drastic-cuts-at-dataupia-company-lays-off-majority-of-staff-while-hunting-for-new-investors/

Me thinks a disgruntled ex-staffer dished the dirt given the accuracy of the news story. Maybe even a VP?

Maybe one day I'll tell my version...now that would be interesting ;-)

Two of the biggest successes came from yours truly, Jonathan and Richard in the UK. The Subex system weighed in at 100 billion rows on Oracle and the ITIS system chipped in with 25 billion rows on SQL Server. Both pretty impressive numbers.

The first Subex POC system was built at my house and driven to the BT data centre in the back of my car. We took it slow and steady (about 85 mph) for the 4 hour drive down to Ipswich in case sudden braking decapitated either of us.

So, joining Foster at Dataupia never made me rich after all. good to hear he's making a full recovery after the heart surgery. Maybe we'll go flying in MA again when I'm next over in the US?

Ah well, back to high-end database consulting for me then, and an end to being a wage slave.

Whoa, there's still a recession out there!

Clearing a Data Pileup - ITIS article

2009-05-05T18:47:00.000+01:00

There's a nice piece called 'Clearing a Data Pileup' covering ITIS and their use of Dataupia and SQL Server on the information-management.com web site:

http://www.information-management.com/issues/2007_58/traffic_analysis_itis_olap_data_warehouse-10015250-1.html

Enjoy!

MPP Article at information-management.com

2009-05-01T16:12:00.000+01:00

Dataupia's very own John O'Brien has written a nice piece called "Degrees of Massively Parallel Processing" at information-management.com:

Click here to read John's article.

Oracle Buys Sun

2009-04-27T13:49:00.000+01:00

Looks like the folks at IBM passed up the chance to buy Sun, then along came Oracle instead.

What Oracle do with the acquired Sun hardware and software assets will no doubt be subject to much debate. Only time will tell.

The folks at 'El Reg' have covered it quite nicely with several recent articles:

http://www.theregister.co.uk/2009/04/20/oracle_buys_sun/

http://www.theregister.co.uk/2009/04/20/oracle_eats_sun/

For what it's worth, me thinks this is likely, but by no means guaranteed, to be a good thing as far as Oracle/Sun enterprise customers are concerned.

Where it leaves the open source and mySQL community is quite another matter.

MPP SQL Server System at ITIS

2009-03-09T15:48:00.000Z

The 25 billion row Dataupia-powered MPP SQL Server system at ITIS is starting to get some analyst coverage.

Here's a nice article on the TDWI web site:

http://www.tdwi.org/News/display.aspx?ID=9333

I'm pretty sure a few more articles will appear in due course.

25 Billion Row MPP SQL Server System Announced

2009-02-23T18:12:00.000Z

Here's another interesting project I've been working on recently at Dataupia:

http://www.dataupia.com/pr20090223_itis.php

Enjoy!

Intelligent Enterprise Oracle Exadata Article

2009-02-12T16:16:00.000Z

Curt Monash has written a relatively lengthy article on Oracle Exadata over at Intelligent Enterprise.

Click here for the article:

Oracle Exadata: Still an Unknown Quantity

It appears that there are still no customer reference sites and that on-site POCs are not happening.

I wonder why that is???

Oracle Exadata and Dataupia Article

2008-12-24T11:28:00.000Z

Philip Howard over at Bloor Research has written an article that discusses the merits of both Exadata and Dataupia when it comes to scaling an Oracle data warehouse.

Registration is required, but it's well worth a read:

http://www.bloor-research.com/research/incomparison/991/scaling-your-oracle-warehouse.html

My very brief take on Exadata goes like this:

high entry point in terms of database size - you can't buy 'a little'

high entry point in terms of price (see above)

expensive per TB

highly complex i.e. very un-appliance like

only a small sub-set of queries will benefit from 'end-to-end' parallelism - fact:fact joins within the storage layer are not supported

RAC is not an MPP enabler

How this joint Oracle/HP Exadata offering fits with the HP Neoview offering is anyone's guess.

Teradata Partners 2008

2008-10-19T11:32:00.000+01:00

Teradata recently announced several new features at the annual Teradata Partners 'love-in' event in Las Vegas.

The 'up to 30% performance improvements and 75 enhancements' constitute the lastest Teradata release - Teradata v13.

The key new features are as follows:

Teradata virtual storage

ELT enhancements (not ETL - transforms are done in parallel inside the MPP database, remember?)

geospatial capabilities

The full blurb from Teradata's web site can be found here:

http://www.teradata.com/t/page/184445/index.html

As ever, the devil is in the detail...more will follow when I've dug deeper.

20 Years in IT...

2008-09-26T18:35:00.000+01:00

...is that a good thing or a bad thing?

Either way, I realised this week that I started my IT career back in September 1988, which is 20 years ago. My how time whizzes by when you're having fun!

'Back in the day' I started out on an IBM mainframe green-screen writing Cobol against a hierarchical IMS DB/DC database in an OLTP environment. That was at Royal Insurance in Liverpool.

After a transfer to the 'IS team' I was soon up to my neck in all things Teradata, and that was back in 1989. The machine needed re-booting a lot, but it was still better writing SQL than Cobol. It probably still is.

After leaving Royal in '92 I headed to the Teradata team at Bank Of America in San Francisco for a year. That was great fun, and the Bay Area left a lasting impression. No surprises there then.

Back in the UK, I was a Teradata contractor at various place such as NCR/Teradata, Abbey National, British Airways, LloydsTSB and Littlewoods.

In 2002 I formed a boutique BI consultancy that carried out many successful Teradata, Netezza and SQL Server projects for companies such as Vodafone, Co-op, Iceland, Littlewoods, EMR, Park Group and others.

Here's to another 20 years...

'SQLBits' SQL Server Conference

2008-08-12T15:36:00.000+01:00

Given my involvement with SQL Server at Dataupia, I'll be attending the upcoming SQL Server conference called 'SQL Bits III'.

It's at Hatfield on Saturday 13th September. Best of all it's FREE!!!

Info here:

http://www.sqlbits.com/

See you there.

Microsoft To Buy Datallegro

2008-07-25T10:00:00.000+01:00

So, looks like Mircosoft plans to pony up something north of $100m to acquire Datallegro.

Interesting...several questions spring to mind immediately.

Why would a company that offers only Windows/SQL Server decide to buy a company whose competing solution runs on Linux/Ingres?

Why can't Microsoft build a parallel version of SQL Server themselves, given their huge resources?

How long will it take for Datallegro to re-appear as "Windows/SQL Server powered Datallegro"?

Does this purchase demonstrate that the likes of Microsoft concede that SMP systems simply can't scale, and that MPP is the only way to meet the data volumes in play?

I've believed that for 20 years.

How long before Oracle follow suit and buy an MPP database player?

Anyway, a big well done to Stuart Frost and his team, especially Anthony Howcroft that runs the UK - a fellow Everton supporter.

Has Datallegro Been Bought?

2008-07-24T19:11:00.000+01:00

A little bird tells me that the announcement will hit the wire tomorrow.

Watch this space!

Teradata Experts on LinkedIn

2008-07-14T10:20:00.000+01:00

There is an 'invitation only' Teradata experts group on LinkedIn:

http://www.teradata-experts.com/

My membership was recently approved, so it looks like it's open to gatecrashers as well...only joking, I've been using Teradata since the late 1980's and became a 'Jedi' (Teradata Certified Master) in 2002.

DM Review Dimensional Modelling Article

2008-05-16T10:40:00.000+01:00

This DM Review article from 2004 certainly struck a chord with yours truly when I first stumbled across it.

Part 1:

http://www.dmreview.com/issues/20040301/8183-1.html

Part 2:

http://www.dmreview.com/issues/20040401/1000939-1.html

"The World Is Not A Star".

Indeed.

Data Warehouse Appliances (Wikipedia)

2008-05-16T10:05:00.000+01:00

Here's a nice Wikipedia article that covers a bit about MPP, Teradata, Netezza and Dataupia, amongst others:

http://en.wikipedia.org/wiki/Data_warehouse_appliance

Interesting to note Dataupia as one of the 'Other players in the DW appliance space'.

So who else supports a live 150TB Oracle implementation ;-)

Dataupia/Subex Press Release

2008-02-03T14:13:00.000Z

Here's what's been keeping me out of trouble recently:

http://www.dataupia.com/pr20080122_subex.php

Keep checking back for more Dataupia news.

Teradata Resources

2008-01-12T09:00:00.000Z

Just a few Teradata bits 'n' bobs that some folks might find useful:

href="http://www.teradataforum.com/" target="_blank">Teradata Forum - email based Teradata forum that anyone can join.
href="http://www.teradataforum.com/" target="_blank">

href="http://www.info.teradata.com/DataWarehouse/eTeradata-BrowseBy.cfm" target="_blank">Teradata Documentation - Teradata manuals

in PDF form.

href="http://www.teradata.com/services-support/certified-professional-program" target="_blank">Teradata Certification - get yourself

some Teradata certificates and maybe become a Certified Teradata Master?
href="http://www.teradata.com/services-support/certified-professional-program" target="_blank">

href="http://www.ward-analytics.com/" target="_blank">Ward Analytics - UK based company specialising in Teradata value-add

applications such as Visual Edge, HeatSeeker, SLA Manager (SLAM), Active Workload Manager (AWM), Prism and performance History Database

(PHD).

target="_blank">Teradata Partners - well attended and highly regarded annual Teradata customer love-in.
href="http://www.teradata.com/teradata-partners/" target="_blank">

Data Temperature (Conspectus Magazine, February 2006)

2006-02-01T09:00:00.000Z

I was invited to write a paper for a UK magazine called "Conspectus".

The chosen topic was "data temperature":

Data Temperature Article (PDF)

Due to the restricted number of words available some of the content was a bit brief, but I think the main points came across OK.

It's interesting to note that 'data temperature' has become a bit of a 'hot' topic in the meantime (pardon the pun).

Netezza User Conference (Boston, May 2005)

2005-05-30T10:00:00.000+01:00

This was a guest speaking slot at the first Netezza user conference in Cambridge, Mass., USA.

The focus was on the Reach Telecom/Caudwell Communications Netezza implementation in the UK.

A case study was presented by yours truly which was videoed by the Netezza folks for posterity.

In essense, a rapidly growing virtual fixed-line telco had too much data for SQL Server, so yours trulyand his boutique SI rode into town, recommended Netezza and saved the day.

Job's a good 'un, as they say!

DAMA UK (London, March 2005)

2005-03-23T09:00:00.000Z

This was a presentation given at a DAMA (UK) event in London comparing symmetric multi-processor (SMP) and massively parallel processing (MPP) data warehouse architectures for business intelligence (BI).

No prizes for guessing which came out better!

Link to content on DAMA web site

Download presentation (PDF)

TDWI (Las Vegas, March 2005)

2005-03-02T09:00:00.000Z

This was an evening session that covered real-world IBM DB2, Netezza and Teradata experiences.

The full title was: "Selecting the Right Data Warehouse: Three Experts Discuss Hands-On Experiences with IBM, Netezza, and Teradata"

Myself, Tom Coffing (Coffing Data Warehousing) and Srinivasan Ramaswamy (Systech Solutions) were the speakers.

Here's the TDWI report:

http://download.101com.com/pub/tdwi/files/LasVegas_05_TripReport.pdf

Any excuse for a trip to Vegas ;-)

Revenue Assurance (New Orleans, January 2005)

2005-01-10T09:00:00.000Z

This was a presentation at a Revenue Assurance & Billing conference in New Orleans.

The focus was on the Netezza implementation at Reach Telecom which underpinned their revenue assurance (RA) efforts.

Reach Telecom was renamed Caudwell Communications, and was later sold to Pipex, which was then sold to and Indian company whose name escapes me.

I remember seeing that Teradata had a booth at this show.

Me: "I didn't know you had a revenue assurance offering".

Teradata: "We have thought leadership in the RA space".

Me: "Ah, no offering then..."