Bobby Durrett's DBA Blog

IOUG accepted both my Toastmasters and Python talks for Collaborate 18. RMOUG also accepted them both so I will be doing these two talks in both February and April. I am a little surprised because I have had talks rejected by IOUG in the past. There are a lot of great speakers competing for speaking slots. This is my first time for RMOUG so I did not know how hard it would be to get a talk accepted. I put both talks in for both conferences not knowing if either would be accepted and both were at both conferences!  So, 2018 will be a busier year than normal for me in terms of speaking at conferences. My last conference was two years ago at Oracle OpenWorld where I spoke about Delphix, a tool that I use with the Oracle database. Next year I’m talking about two things that I feel passionate about. The talks are not about the Oracle database but they are about things I have learned that have helped me in my Oracle database work. They are about how Toastmasters has helped me improve my speaking and leadership skills and about why the Python programming language has become my favorite general purpose scripting tool. I am looking forward to giving the talks. If you are able to attend one of the conferences maybe you could check out one of my talks. Fun.

Bobby

The Arizona Oracle User Group (AZORA) is meeting this Thursday, November 16th, 2017 from 12:30 to 3:30 pm Arizona time.

Here is the link to the Meetup:

https://www.meetup.com/Arizona-Oracle-User-Group/events/244531599/

Here is a link to the user group site with more information:

http://www.az-ora.org/azora-meeting-nov-16-2017/

I’m looking forward to it. I hope to see you there!

Bobby

I was on call Sunday and got paged about a job that normally runs for 10 to 15 minutes but had run for 5 hours already. I used the coe_xfr_sql_profile.sql script from SQLT to speed up an insert statement that was part of the job. We restarted the job and it completed in a few minutes.

I have written a number of posts about the use of coe_xfr_sql_profile.sql. Sunday’s issue was most like the post on using hints with coe_xfr_sql_profile.sql.

The first thing I did was look at the execution history of the problem insert statement which had sql_id = ‘ba9w9cjy87hd8’. I used my sqlstat.sql script.

SQL_ID        PLAN_HASH_VALUE END_INTERVAL_TIME         EXECUTIONS_DELTA Elapsed Average ms CPU Average ms IO Average ms Cluster Average ms Application Average ms Concurrency Average ms Average buffer gets Average disk reads Average rows processed
------------- --------------- ------------------------- ---------------- ------------------ -------------- ------------- ------------------ ---------------------- ---------------------- ------------------- ------------------ ----------------------
ba9w9cjy87hd8      3341942963 05-NOV-17 07.00.03.091 AM                2         1302824.53        1162165      572.8985                  0                  2.271                61.1125             6077761               1021                    8.5
ba9w9cjy87hd8      3341942963 05-NOV-17 10.00.23.889 AM                3          846842.04     844443.333       108.995                  0                      0                      0          4475478.33         196.333333             5.66666667

Even though this job runs every weekend and we keep 6 weeks of history in the AWR ba9w9cjy87hd8 only showed up yesterday. I checked the SQL and it did not have any constants. It uses bind variables so the same sql_id should apply every week. My guess is that it normally runs so fast that it did not get recorded in the AWR on previous weeks but this week it chose a terrible plan. I see this a lot with SQL statements that have bind variables. I think that the bad execution plan gets chosen based on some abnormal bind variable value and then it sticks in memory.

The other thing I noticed while looking at the session for this job is that it was not using any parallel processes. I looked at the plan and there were no parallel steps. This system uses a lot of parallel query so I suspected that a parallel plan was the normal plan and the bad plan somehow came out with serial execution.

I checked our test databases to see if they had any execution history for this SQL and they did not. So, I was stuck trying to get a better plan and then using a SQL profile to force that plan on the insert statement. I ended up running the explain plan statement without giving the bind variable a value and it came out with a parallel plan. I used the outline hint from that plan to force the insert statement to run the parallel plan against a test database. I ran the insert with a rollback statement so it wouldn’t commit the new rows. Also I picked a bind variable value from the production database’s AWR to run with my test query.

Next I decided to run my test insert statement against production with the rollback statement. The first time I tried the statement it hung on a TM enqueue. I think this was because the insert was using an append hint and the running job was holding a TM lock. After we killed the production job I could run my test insert with the rollback statement.

I had to get the parallel plan into memory or into the AWR so that coe_xfr_sql_profile.sql would pick it up based on the plan hash value. It felt weird running an insert statement for a production table but I tested it on a test database first and made sure it would be rolled back.

I ran this select after the insert and rollback to get the plan for the insert in production:

select * from table(dbms_xplan.display_cursor(null,null,'ALL'));

Part of the output of the select included the plan hash value of the plan that my test insert used:

Plan hash value: 3809184161

Finally, I forced the good plan which had hash value 3809184161 onto the sql_id ba9w9cjy87hd8  using coe_xfr_sql_profile.sql. Then the job ran in a few minutes.

Bobby

I got two talks accepted for RMOUG Training Days in February. I mentioned these two titles in a earlier post:

• Python for the Oracle DBA
• Toastmasters for the Oracle DBA

These two talks are about topics that interest me so I am glad that RMOUG thinks that they are valuable to the conference attendees.

I plan to do the two talks for my DBA coworkers and shorter versions at Toastmasters so I should get some constructive feedback and practice before the conference.

Should be fun. Hope to see you in Denver next February.

My Python posts: url

My Toastmasters posts: url

Bobby

I submitted my two talks to Collaborate 18 through IOUG. These are the same two that I submitted to RMOUG Training Days. Hopefully one of the talks will get accepted at one of the two conferences but I don’t know. The nice thing is that they are both professional development talks and a user group meeting might be more open to that sort of talk than a vendor (i.e. Oracle OpenWorld) conference. But, there is a lot of competition out there so we will see. Hopefully I will get to put some of my new Toastmaster skills to work at a conference early next year. 🙂

Bobby

I was looking at the Python sortedcontainers package and it got me thinking. It is a long convoluted story and I am not sure that I can explain it clearly in a short blog post. I tried to explain all this to my wife in the last few minutes as we were driving up to a friend’s house last night and I’m sure it was confusing the way I explained it. But, I’m hoping that I can capture some of the ideas that I have thought about in a way that will be useful to others.

I have studied some computer science topics that do not directly relate to my work with Oracle databases and my review of the sortedcontainers implementation documentation tied together several things that I have studied and related them back to my work with Oracle performance tuning. I have not tested sortedcontainers to make sure that it does everything the web site says it does. But, I think it is the best Python package for doing balanced tree type of structures in memory. An AVL tree or B-tree keeps data ordered so you can quickly search for a range of key values and get them out in sorted order. Normal Oracle indexes are a type of B-tree but on disk with blocks cached in memory when queries access them. AVL trees are binary trees so each node points to at most 2 children. B-tree nodes can have many children. Sortedcontainers seem to work like a balanced tree with 1000 or so max children per node. I think it makes efficient use of Python’s built-in list objects. It seems to work well with caching within the CPU. I have not carefully reviewed the theory and tested all this out to prove that it is right but it seems likely that it is. I think it seems convincing because it ties back to other computer science topics that I have studied and to my experience with Oracle performance tuning.

I have been slowly working through an algorithms class on MIT’s OCW website. I am on a section about AVL trees. So, I was looking around at AVL trees in Python. I noticed that Rosetta Code had an AVL tree topic but no Python example until I added one. I also looked around on PyPI for an AVL tree Python package. Based on my search, I thought that bintrees was the most mature, but its web page has a note saying “Use sortedcontainers instead”. So, that made me think that sortedcontainers was the best balanced tree option in Python. The algorithms class talks about how to prove that you can work with AVL trees in O(log n) time. The sortedcontainers performance documentation has a complex explanation of its big O complexity. Also, I think that my class will discuss some of the concepts used in the sortedcontainers analysis in future lessons. So, that motivates me to go forward.

The assembly language book that I worked through helped me understand how to write programs that run faster because they make better use of the x86-64 processor’s cache and registers. Its creator seems to have designed sortedcontainers with CPU caches in mind. Right or wrong, in my mind this ties back to memory caches that affect Oracle database performance. How much of Oracle tuning relates back to how systems cache database blocks in RAM and where? You have the database block cache of course. You also have operating system filesystem cache which you might bypass with direct I/O. You may have high-speed memory cache within your SAN’s storage server. I don’t know about today but in the past disk controller cards and even disk drives themselves had memory caches. You might say, joking, that “cache is king” in database performance. At least, you have to say it is important to understand when and where database systems cache disk blocks in memory to understand why you are getting the performance you are seeing.

So, I guess my mind connected sortedcontainers with my algorithms class and assembly language book. I also connected sortedcontainers back to Oracle performance tuning. It makes me feel that digging into some computer science training is not a waste of time. It helps me to step back from Oracle database specific study and get a little theory. Also, my database work is focusing more and more on the x86-64 architecture and the Linux platform so looking at computer science on the same platform that I use for work has clear benefits.

So, I’m concerned that I have not made this post helpful to people who read it. Is it just about my experience or does it have a point for other people? Maybe the point is that it can’t hurt for an Oracle DBA to learn some computer science. Maybe you are like me and studied C.S. in school many years ago. Maybe you have learned Oracle on the job and don’t have a C.S. background. Maybe the message for both of us from my story about sortedcontainers and my “Aha!” moment is that there are benefits to studying a little computer science even if it does not directly relate to your job. There is only so much time in an Oracle DBA’s day and you can’t learn everything, but maybe it is worth putting some time into learning some C.S. theory if you can.

Bobby

I finally broke down and submitted my two talks to the RMOUG Training Days 2018. I’m not sure how likely I am to get my talks accepted but I learned a lot just through constructing my two abstracts. I have two talks that I have processed in my mind for a while:

• Python for the Oracle DBA
• Toastmasters for the Oracle DBA

These are just two things I’ve worked with and that I think have helped me in my job. Plus, they are two topics that interest me. Why talk about something boring just because you think other people want you to?

Anyway, it was fun to write the abstracts for two topics that I am passionate about. I have never been to RMOUG Training Days but I have heard a lot of good things. Hopefully I will get accepted and be able to get these talks out to the public.

I’m off next week so I don’t expect to blog until I get back to work. Maybe after I get back I can start to outline my Toastmasters talk in a few posts. I have given a couple of Toastmasters talks with similar material but this would be geared specifically for people who are more like me in the sense that they are Oracle DBAs.

Anyway, it was fun to get the abstracts turned in.

Bobby

I started to write this blog post and then realized that in the process of making a copy of my blog on my laptop I whacked my connection to WordPress. But, it was easy to resolve it by disconnecting Jetpack from WordPress, clearing my browser cache, and then reconnecting. Whew! I worried that messing with this stuff would cause some problem but the blog still seems to work.

My idea was to bring up a copy of this blog site on a VirtualBox VM on my laptop. I have thought about doing this in the past but did not have any good reason to do so. I have mainly acted like an end user for this blog software and to iPage, the company that hosts it. I have not tried to delve into the underlying PHP code or mess with the MySQL database. But, my database team has recently started supporting MySQL and I am trying to train myself up a bit so it made sense to play with duplicating this small MySQL app.

I duplicated the blog in three main steps:

1. Setup a VirtualBox VM
2. Setup a MySQL database
3. Setup the web site

STEP 1 – SETUP A VIRTUALBOX VM

I am still using the VirtualBox NAT networking that I described in an earlier post. I created a VM called Blog with this ip and port forwarding details:

Blog ip

10.0.2.17

Blog port forwarding

61014 to 22
61015 to 80

I duplicated an existing Oracle Enterprise Linux 7 VM and modified the ip and setup the port forwarding. Then I did a yum update to catch up all the existing packages. I also disabled the firewall so that it would not get in the way.

yum -y update
systemctl disable firewalld
systemctl stop firewalld

STEP 2 – SETUP MYSQL DATABASE

I downloaded mysql57-community-release-el7-11.noarch.rpm from http://dev.mysql.com/downloads/repo/yum/ and ran the following commands to install MySQL:

yum localinstall mysql57-community-release-el7-11.noarch.rpm
yum install mysql-community-server
service mysqld start
service mysqld status

Then I ran a series of SQL commands logged into MySQL to setup the user and database for the blog:

grep 'temporary password' /var/log/mysqld.log

mysql -uroot -p
use temporary root password

ALTER USER 'root'@'localhost' IDENTIFIED BY 'Bl0gC0py!';

CREATE DATABASE blogdb;

GRANT ALL PRIVILEGES ON blogdb.* TO "blogdbuser"@"localhost"
IDENTIFIED BY "Bl0gC0py!";

FLUSH PRIVILEGES;

EXIT

Next I took a backup of my blog database that I got from iPage and made the following string replacements in the SQL commands:

Replace all instances of

https://www.bobbydurrettdba.com

with 

http://localhost:61015

Also

bobby@bobbydurrettdba.com

replace with

bobby@blogvm.com

and

@bobbydurrettdba

with

@blogvm

Finally

bobbydurrettdba

with 

blogvm

I was trying to make sure that I didn’t whack anything in my real blog and that it worked with my localhost:61015 web site host name and port. I had to add two lines to the beginning of the sql script to make it work:

use blogdb
SET sql_mode = '';

I ran the script like this:

mysql -v --force -ublogdbuser -pBl0gC0py! < database.sql > database.out 2> database.err

I checked database.err and it only had a warning about using the password on the command line.

STEP3 – SETUP WEB SITE

Now that the database was setup and loaded with data I worked on the web site.

First, I installed the Linux packages for php which pulled in the web server as a dependency:

yum install php php-common php-mysql php-gd php-xml php-mbstring php-mcrypt 

Edited httpd.conf to setup web server:

vi /etc/httpd/conf/httpd.conf

Replace all instances of AllowOverride None or none
with AllowOverride all

Added VirtualHost lines at the end:

Added host to /etc/hosts

vi /etc/hosts

add this line 

127.0.0.1  blog

I forgot to mention in step 1 that when I created the VM I renamed it to blog using hostnamectl.

Next I created the web site directory and populated it with files from my ftp backup of my website.

mkdir /var/www/html/wordpress

copy my blog files to /var/www/html/wordpress

Next I edited the WordPress configuration file to work with the database:

cd /var/www/html/wordpress

vi wp-config.php

/** The name of the database for WordPress */
define('DB_NAME', 'blogdb');

/** MySQL database username */
define('DB_USER', 'blogdbuser');

/** MySQL database password */
define('DB_PASSWORD', 'Bl0gC0py!');

/** MySQL hostname */
define('DB_HOST', 'localhost');

Finally I restarted the web server and set the web server to automatically start on reboot:

systemctl restart httpd.service 

systemctl enable httpd.service

Finally, I tested the web site at http://localhost:61015 and it looked a lot like this blog site.

Bobby

I looked around for some other ways to do this but decided to just code this up in Python. It connects to a remote Linux server using ftp and recursively copies all the files and directories back to a Windows machine.

Here is the source:

This is an example of the ease of use of Python for quick scripting. It uses a low-level ftp library called ftplib.

Bobby

I pushed out a quick change to PythonDBAGraphs to automatically save the data for any graph that you make so that you can redraw the graph later. This is better than saving an image file because the redrawn graph lets you see details about the points on the graph when you hover the mouse over the points.

Now when you generate a graph you get a line like this:

Saving data in C:\temp\ASH active session count for MYDB database.txt

When you want to see the graph again you run show_saved.py like this:

python show_saved.py
Enter name of data file to be restored: C:\temp\ASH active session count for MYDB database.txt

Bobby