Importing and Working with Exported AWR/ASH data in an Oracle database in a VirtualBox VM

A lot of my performance tuning work involves analysis of ASH and AWR data.  Frequently, I do not have direct access to the databases in question.  Sometimes, I ask clients to run eDB360 on their databases and send me the results, but sometimes I also want to work directly with ASH or AWR metrics.  So, I ask for an export of their AWR repository.
Oracle distributes a pair of scripts in $ORACLE_HOME/rdbms/admin.

• awrextr.sql exports the AWR repository (including the ASH data), optionally for a range of AWR snapshot IDs. 
• awrload.sql the export into a staging table and then moves it into the AWR repository in the current database from where you can reference it via the usual DBA_HIST% views and work with it in the usual way.

I import this into an Oracle database in a virtual machine in VirtualBox and work on it there.  This note explains how I create and set up that machine and then work with the data.

Getting Started

1. Build Oracle VirtualBox Linux VM

I use Frits Hoogland's vagrant builder to create a standard Oracle database on a Linux VM in VirtualBox.  I always build the latest version of Oracle because I can import an AWR export into a higher version of the database, but not a lower one.

• Frits' blog describes the basic process: A total unattended install of Linux and the Oracle database (https://fritshoogland.wordpress.com/2016/08/02/a-total-unattended-install-of-linux-and-the-oracle-database/).
• Install Vagrant on the host machine (download from https://www.vagrantup.com/).
• Download Vagrant Builder from https://gitlab.com/FritsHoogland/vagrant-builder
• Modern Oracle distributions should be downloaded Oracle Technical Resources (formerly OTN) https://www.oracle.com/technical-resources/ and put in the ansible\files directory, 
• It can take less than an hour to build a new virtual box with a freshly built Oracle database at the version of your choice.

2. Make Life Easier For Yourself

I sometimes install other utilities into the virtual machine, such as git, using yum.

yum install git

The VirtualBox is only going to run a single Oracle database, and I am very lazy.  So, I set up the environment in the shell profile by adding the following commands the ~oracle/.bash_profile.

export ORACLE_SID=oracle
export ORAENV_ASK=NO
. oraenv
export ORAENV_ASK=YES

3. Share a Folder on the Host Machine

I usually leave the AWR export file on my host machine rather than copy it into the virtual machine.  I create a VirtualBox shared folder so that the VirtualBox VM can reference a folder on the host machine.  In this case, I have shared c:\temp on my host Windows laptop and called the filesystem temp.  Note that it is auto-mounted and permanent.

In the VirtualBox, the shared folder has been mounted as /media/sf_temp.

[oracle@oracle-database sql]$ df /media/sf_temp
Filesystem     1K-blocks      Used Available Use% Mounted on
temp           974661916 340278844 634383072  35% /media/sf_temp

The sf_temp directory is owned by root and accessible only to the vboxsf group.  Therefore, you must add any user who needs to reference this directory to that group.  In this case, Oracle must be made a member of the vboxsf group.  For safety, this should be done with the usermod command rather than by editing /etc/group directly.

usermod -G vboxsf -a oracle

You can see the result of this command in the /etc/group file

grep oracle /etc/group
vboxsf:x:994:oracle

When logged in as Oracle, the id command will also report the groups of which it is a member.

[oracle@oracle-database ~]$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),100(users),994(vboxsf),54322(dba)

Create a directory in the database to reference this directory on the filesystem.

CREATE OR REPLACE DIRECTORY TEMP as '/media/sf_temp';

4. Oracle FILESYSTEMIO_OPTIONS Parameter

Initially, I got the following error when trying to run a data pump import from the shared folder:

ORA-39000: bad dump file specification
ORA-31640: unable to open dump file "/media/sf_temp/awrdat_xxxx_xxxx.dmp" for read
ORA-27041: unable to open file
Linux-x86_64 Error: 22: Invalid argument
Additional information: 2

On VirtualBox, the Oracle parameter FILESYSTEMIO_OPTIONS will be set to setall by default.  To work around the import error set it to none and restart the database.

ALTER SYSTEM SET FILESYSTEMIO_OPTIONS=none SCOPE=SPFILE;
SHUTDOWN IMMEDIATE
STARTUP

• See also https://oracle-base.com/articles/misc/direct-and-asynchronous-io

5. Data Pump Import

Now you can import an AWR export (previously exported with awrextr.sql) with the script awrload.sql.

cd $ORACLE_HOME/rdbms/admin
sqlplus / as sysdba
@awrload

Working with Imported Data

The AWR data exported from another database will be visible through all the usual database views.  It can be distinguished by having the database ID of the original database.  If, like me, you are used to mining ASH data with SQL scripts, then they will work if you specify the DBID.  However, I have a couple of tips for working with imported AWR data.

SQLDB360 (EDB360 and SQLD360)

Sometimes I run an eDB360 report on the imported data.  The reports and graphs of ASH data and the system time model are particularly useful.
I can generate AWR reports for any pair of snapshots in the imported repository, not just the ones generated by the client and provided to me.  eDB360 generates AWR reports for peak periods by various criteria.
Having installed git, I can use it to download the latest version of SQLDB360.

cd ~/
git clone https://github.com/sqldb360/sqldb360.git
cd ~/sqldb360

Although, I could have used curl instead.

cd ~/
curl -LO https://github.com/sqldb360/sqldb360/archive/unstable.zip
unzip unstable.zip

Obtain the DBID of the database that you have just imported by looking at one of the AWR views

col max(startup_time) format a26
col min(begin_interval_time) format a26
col max(end_interval_time) format a26
SELECT dbid, db_name, count(*), max(startup_time)
FROM   dba_hist_database_instance
GROUP BY dbid, db_name
/
SELECT dbid, min(begin_interval_time), max(end_interval_time)
FROM   dba_hist_snapshot
GROUP BY dbid
/

In this case, my database is ORACLE, and XXXXXXX is the database that I have just imported.

      DBID DB_NAME     COUNT(*) MAX(STARTUP_TIME)
---------- --------- ---------- --------------------------
1884370030 ORACLE             9 20-JUN-19 10.51.50.000 AM
3743596248 XXXXXXX            4 05-MAY-19 01.03.57.000 AM

      DBID MIN(BEGIN_INTERVAL_TIME)   MAX(END_INTERVAL_TIME)
---------- -------------------------- --------------------------
1884370030 18-JUN-19 07.03.44.000 PM  20-JUN-19 11.02.49.684 AM
3743596248 18-JUN-19 05.00.13.342 PM  19-JUN-19 11.00.20.020 AM

Create a custom configuration file in the sqldb360/sql directory

• Set the database ID with the DBID from the export.
• Specify edb360_conf_date_from/to parameters to match the range of dates in the import.

- this is a sample custom configuration requesting to produce just one section
-- for all possible DEF values please review edb360_00_config.sql

-- section to report. null means all (default)
-- report column, or section, or range of columns or range of sections i.e. 3, 3-4, 3a, 3a-4c, 3-4c, 3c-4
DEF edb360_sections = '';

-- range of dates below superceed history days when values are other than YYYY-MM-DD
-- default values YYYY-MM-DD mean: use edb360_conf_days
-- actual values sample: 2016-04-26
DEF edb360_conf_date_from = '2019-06-18';
DEF edb360_conf_date_to = '2019-06-19';

-- use if you need tool to act on a dbid stored on AWR, but that is not the current v$database.dbid
DEF edb360_config_dbid = '3743596248';

Run EDB360 referencing the custom config file

@edb360 T custom_config_01.sql
@sqld360 5dcbnw1d9yfna T custom_config_01.sql

NB: EDB360 and SQLD360 reports generated in this way will contain a mixture of data local database and the database from which the AWR export was taken.

• Some reports are based on v$ and x$ views or data dictionary views (DBA_TABLES etc.).  These will report on the database in the virtual machine.
• Only the reports based on DBA_HIST% views will report on the database from which the AWR data was exported.

It is important to keep this distinction in mind.
Nearly every section of the EDB360 report contains reports based on the AWR data, so it is not worth trying to exclude any sections when running EDB360 imported data.

Extracting Execution Plans from AWR

We would normally extract the execution plan of a particular statement from the AWR for the local database using the DISPLAY_AWR function in DBMS_XPLAN package as follows.

SELECT * FROM table(dbms_xplan.display_awr('burz0q5qjtgmv'));

You can specify the database ID to DISPLAY_AWR(), but if it is not the DBID of the current database, then it returns nothing.

SELECT * FROM table(dbms_xplan.display_awr('5dcbnw1d9yfna',db_id=>3743596248));

The various DISPLAY% functions in DBMS_XPLAN are formatted report generators for data held in a particular data structure.  DISPLAY_CURSOR() references V$SQL_PLAN, DISPLAY_AWR() references DBA_HIST_SQL_PLAN, and you can pass a table name to DISPLAY so that it can reference any table or view.  That is useful if you have your own private plan table.
In this case, instead of DISPLAY_AWR(), we can use the DISPLAY() function directly on the DBA_HIST_SQLPLAN view, thus:

set long 5000 lines 150 pages 999
column sql_text format a150
SELECT sql_text from dba_hist_sqltext
where dbid=3743596248 AND sql_id = '5dcbnw1d9yfna'
/
SELECT * FROM table(dbms_xplan.display(
  table_name=>'DBA_HIST_SQL_PLAN'
, format=>'ADVANCED +ADAPTIVE'
, filter_preds=>'dbid=3743596248 AND sql_id = ''5dcbnw1d9yfna'''))
/

Note that:

• The DBID and SQL_ID are passed as quoted literals to the function.
• DBMS_XPLAN.DISPLAY() does not report the SQL statement, so that must be queried separately.

PeopleTools Table Reference Generator

Like many other PeopleSoft professionals, I spend a lot of time looking at the PeopleTools tables because they contain meta-data about the PeopleSoft application. Much of the application is stored in PeopleTools tables. Some provide information about the Data Model. Many of my utility scripts reference the PeopleTools tables, some of them update them too. Therefore, it is very helpful to be able to understand what is in these tables. In PeopleSoft for the Oracle DBA, I discussed some tables that are of regular interest. I included the tables that correspond to the database catalogue and that are used during the PeopleSoft login procedure. The tables that are maintained by the process scheduler are valuable because they contain information about who ran what process when, and how long they ran for.
I am not the only person to have started to document the PeopleTools tables on their website or blog, most people have picked a few tables that are of particular interest. I wanted to produce a complete reference.  However, with over 4000 PeopleTools tables and views, it is simply not viable to do even a significant number of them manually.  So, I wrote some SQL and PL/SQL to dynamically generate a page for each PeopleTools table and views and put the generated pages on my website.  If you use Google to search for a PeopleTools name you will probably find the reference page.
I have now revisited that code and made a number of changes and improvements.

• I have used a later version of PeopleTools to generate the reference on my website.  The list of PeopleTools tables is no longer defined in PeopleTools by object security, so I have used an independent and somewhat broader definition: Table or View records that are either owned by PPT or whose SQLTABLENAME is the same as the record name.
• There is much less reliance on static pages.  There are now only 3 such pages everything else is generated.  Instead, additional data is loaded from static scripts into the PLAN_TABLE that should always be present in an Oracle database and so it doesn't have to be created.  It should be a global temporary table so there is no problem with debris being left behind or interference with other processes.  That data is then combined with data in the PeopleTools tables during the generation process.
• The utility has been rewritten as a PL/SQL package that should be created in the PeopleSoft Owner ID schema (usually SYSADM).
• The generated HTML is simply tidier, and more consistent.

The source is available on GitHub at https://github.com/davidkurtz/PTRef, so you can download and generate your own reference on your own current version of PeopleTools. An example of the generated output can be found on my website.
The idea of the original PeopleTools tables reference was the people could contribute additional descriptions and information that were not in the PeopleTools tables.  That can still happen, and indeed should be easier, by making changes to the scripts that load the additional data and uploading new versions to GitHub.

PS360 enhancement: Added report of DDL models

I have written several blogs and presentations recently about how and how not to collect statistics in PeopleSoft.

• Managing Cost-Based Optimizer Statistics for PeopleSoft recommends
• If you are going to continue to use DBMS_STATS in the DDL model then
• Do not specify ESTIMATE_PERCENT because it disables the hash-based number-of-distinct-values calculation, forcing it to go back to the COUNT(DISTINCT …) method that requires a sort, and may not produce accurate values because it only samples data.
• Do not specify METHOD_OPT='FOR ALL INDEXED COLUMNS SIZE 1' because this will not collect histograms on indexed columns, and will not update column statistics on unindexed columns.
• Do specify FORCE=>TRUE so that you can lock and delete statistics on temporary records.
• However, the alternative is to use GFCSTATS11 package to collects these statistics.  This package is controlled by a metadata table so you can defined statistics collection behaviour for specific records.
• How Not to Collect Optimizer Statistics in an Application Engine Program
• This blog explains why you should not explicitly code DBMS_STATS calls into Application Engine programs.

This has prompted me to add a new report to the PS360 utility that simply reports the various DDL models for Oracle.  Thus during a health check, I can see how statistics are collected during batch processes.

PS360 can be download from https://github.com/davidkurtz/ps360.