Sunday four batch jobs that normally run in an hour had been stuck for 3 hours and had not completed the first unit of work out of many. Earlier in the day the on call DBA had applied a SQL Profile and cancelled the jobs and rerun them but it did not help. We picked a different SQL Profile, killed the jobs, and the jobs ran normally. How did we figure out the right plan to use for a SQL Profile?
The main clue came from the output of my sqlstat.sql script:
The good plan seemed to be 2367956558. EXECUTIONS_DELTA = 1 means that the SQL finished in that hour. Elapsed Average ms of 45177.105 was 45 seconds. 19790.066 was 19 seconds.
PLAN_HASH_VALUE END_INTERVAL_TIME EXECUTIONS_DELTA Elapsed Average ms
--------------- ------------------------- ---------------- ------------------
2367956558 20-APR-25 01.00.23.638 AM 1 45177.105
2367956558 20-APR-25 03.00.29.921 AM 1 19790.066
The other plans 2166514251 and 3151484146 seemed to have executions that spanned multiple hours. For example, the first 2166514251 line had EXECUTIONS_DELTA = 0 which means it didn’t finish in that hour. Plus, the elapsed time of 8995202.39 ms = 8995 seconds = 2.5 hours suggests that it was running in parallel, probably for the entire hour.
PLAN_HASH_VALUE END_INTERVAL_TIME EXECUTIONS_DELTA Elapsed Average ms
--------------- ------------------------- ---------------- ------------------
2166514251 20-APR-25 01.00.23.638 AM 0 8995202.39
So, it seems clear that 2367956558 is the best plan.
I could say a lot more about this incident, but I wanted to focus on the sqlstat.sql output. The values of EXECUTIONS_DELTA and Elapsed Average ms are keys to identifying plans with the best behavior.
Bobby
I want to know the name of the oncall dba. HaHa just kidding. Any new features using AI with Oracle or other databases?
Not me! I hope you are doing well. Maybe we will see you again some time in Phoenix? I’m working on a monitoring script using Python which is fun. They are trying to move Big Data into Kubernetes which is interesting. Anyway, I hope that you and your family are well.