Are you using the “default” clustering INDEX or are you defining the correct INDEX with the CLUSTER Attribute?
This newsletter is dedicated to all DDL designers who do their best but then omit that last tiny bit. No-one really notices, or even cares, for years and years until…
Imagine a huge table up in the billions of rows. Imagine now that you have SQL that accesses this table and it must, as always, run fast. So what do you do? You create an index, RUNSTATS it and Hey Presto! Everything is sweet and dandy! Now imagine this happening again and again over time… What you finally end up with is a huge table with billions of rows now with ten indexes! Not too brilliant for insert and update but that is not the point of this newsletter.
An SQL, that had worked perfectly well, suddenly went pear shaped…
So now stop imagining, as this had already really happened at a customer site. We come to the crux: An SQL, that had worked perfectly well, suddenly went pear shaped (belly-up for the non-British English readers!) and started using a two column index with one matching column instead of a six column index with six matching columns! This change in access path caused death by random-IO to occur and it all went horribly wrong.
Now the question is why? What on earth happened for the DB2 Optimizer to make such a terrible decision?
1- RUNSTATS review
First idea was, of course, my favourite – Incomplete or not Full RUNSTATS data. In fact there were “bogus stats” from 2003 in the SYSCOLDIST, but even after all the bogus stats were deleted and a complete RUNSTATS with HISTOGRAM and FREQVAL performed, the access path remained stuck on the “bad” index.
2- DDL review
I then reviewed the DDL that created all of the objects and noticed that none of the indexes was defined with the CLUSTER attribute. The table itself was “as old as the hills,” but all of the indexes had been created and/or altered many times over the last ten years or so.
3- Redefine the “bad” Index as CLUSTER
The dummy CLUSTER
Now, as we all know, if no index is defined as CLUSTER DB2 picks one to be a dummy CLUSTER when it does a REORG. So you can end up with 100% clustering non-clustered indexes. In this case that was exactly what was happening. The “bad” index was, purely by fluke after many years of index maintenance, the “default” clustering index, however it was a *terrible* choice for a clustering index. Worse still: because of the fact it was non-unique with two columns and therefore small (well small in this case was still 40,000 pages!) it looked positively “good” to the DB2 Optimizer—hence the decision to abandon a six column matching index in favour of a single column one…
“Cleansing” with ALTER, REORG and the DB2 10 INCLUDE syntax
A quick ALTER of the original “first” index to get the CLUSTER attribute, a REORG scheduled for the weekend to get the data into *proper* CLUSTERing sequence and – Bob’s your uncle! Access path swapped back to the “good” index.
Now there’s still work to be done here as ten indexes is about seven too many, if you ask me.With DB2 10 it is possible to use the INCLUDE syntax to weed out some of the extra indexes and thus speed up all usage of this mega-table. But, for the right here and now, the job is done!
So now I am at the end of this sad story of how a little design “error” of just forgetting one little attribute on an index create statement caused major mayhem many years down the line… remember to check *all* of your tables and see if you have any beauties like this in your shop (surely not!)
Here’s a little SQL that will do the job for you:
-- -- QUERY TO LIST OUT ALL TABLES WITH TWO OR MORE INDEXES WHERE NO INDEX -- IS DEFINED AS CLUSTER -- SELECT A.CREATOR ,A.NAME FROM SYSIBM.SYSTABLES A WHERE NOT A.CREATOR = 'SYSIBM' AND NOT EXISTS (SELECT 1 FROM SYSIBM.SYSINDEXES B WHERE A.NAME = B.TBNAME AND A.CREATOR = B.TBCREATOR AND B.CLUSTERING = 'Y') AND 1 < (SELECT COALESCE(COUNT(*) , 0) FROM SYSIBM.SYSINDEXES B WHERE A.NAME = B.TBNAME AND A.CREATOR = B.TBCREATOR) ORDER BY 1 , 2 ;
Note that this query excludes the SYSIBM indexes as IBM also forgot to CLUSTER them!
As usual, any comments or questions please mail me!
TTFN
Roy Boxwell