Our story so far: 

Part 1 - 27-August through 04-September

Part 2 05-September through 09-September

Part 3 10-September through 11-September

Detective's log, Friday afternoon, September 12, the day before my getaway

I was headed out of town, and still no clear cut answers on this case.  We know what's wrong, but not quite how to fix it.  The answers we're getting from downtown are, how to put this nicely, not in accordance with the evidence.  We're prepared to escalate, bring out the big guns, and then when the smoke clears, we'll see who's left holding the bag. To coin a phrase.

I asked security to extend the test system being open for another 9 days.  That would take us beyond my 1 week trip (combining business and pleasure), which I thought should be more than enough.  But then, I didn't expect the case would be unsolved for 9 days to begin with.  The importance of this 18 day window will become more obvious later.  The initial security period had expired.

I sent a synopsis of the case to a trusted adviser, who returned this prognosis:

"If I clap once in my hand, it will be fast. If I clap 100,000 times in my hand, each clap will be fast, but overall, it will take some time to clap 100,000 times."

A memorable analogy; whether it would have helped to speed the detective work we will probably never know.

Detective's log, Monday, September 15

I concentrated on my current assignment, staking out the ASUG Operations Optimization conference in Nashville TN, leaving the case details of the runaway KSSK table to the duty officers.  No news really meant no news, that we didn't have a resolution.  Then my communications unit died, and I was forced to stop looking over their shoulders.

But here's what I saw later:

15.09.2008 - 16:46:33 CET - Call to Customer by SAP     
Customer Call
Performed on 15.09.2008 10:45:18
Contact person []
Status of discussion:
Customer reached
Subject of conversation:
I had spoken [] at [] we need to forward the case to development.

Detective's log, Tuesday, September 16

Nothing for breakfast. But.  Black coffee.  Cheese danish.

Email from our Basis team member.  She's found a note, wrong subject, but it smells like our problem (note 1065373 - "Program termination when you change material classification").  She wrote in our case records:

solution is a code fix that skips the bad kssk call if an earlier table is empty.  Thus it would avoid ever reading kssk if no match will be found, which is exactly our case.

Here is my favorite part of the SAP note:

"Due to a program error, an excessive amount of data is selected." 

Detective's log, Wednesday, September 17

We put the fix in to our development system.  (Not me, though, I went on vacation that afternoon).

Later, I found SAP moved the issue to another team:

Component

SV-BO-SCM

CA-CL-CL

Detective's log, Thursday, September 18

I noted that SAP updated their case log with the following entry (I'm paraphrasing here):

We can't get in. What's the password again?

We hit the squeal.  That's police slang for the siren, the emergency, the buzzer, the clanger, the, you get the picture.  Off cycle, we moved the suspect fix from quality to production.

Detective's log, Monday, September 22

We closed the books.

Here are a few keywords that might help the next poor sap that stumbles down the same dark alley:

• SAPLCCHA
• LCCHAU27
• FILTER_APO_DATA
• KSSK