Monday, April 17, 2023

Investigating Unfamiliar PL/SQL with the Hierarchical Profiler

The PL/SQL profiler can tell you how much time you spend where in your PL/SQL code (see my presentation Performance Tuning with the PL/SQL Performance Profilers).  Therefore it also tells you which code blocks were executed, and in which test run.  If you are debugging code with which you are unfamiliar, this can provide insight into where to focus attention and determine what is going on.

Example Problem

I was looking at a third-party application that uses the database job scheduler to run multiple concurrent batch jobs.  I needed to work out why they were not balancing properly between the database instances.  This application has its own scheduler package.  It is driven by metadata to define the jobs to be submitted.  This package then calls the delivered DBMS_SCHEDULER package.  However, this third-party package is quite complicated, there are lots of similar calls, and it is difficult to work out what was executed by just reading the code.

I ran the application having enabled the hierarchical profiler, DBMS_HPROF.  I was able to query the profiler tables to find the calls to DBMS_SCHEDULER that were executed.

Querying the Profiler Tables

Each time DBMSHP is run, the data is tagged with a separate run ID, so if I do different tests I can easily separate them.  However, in this example, I have run only one test.

SELECT * 
FROM   dbmshp_runs
ORDER BY runid;

 Run
  ID RUN_TIMESTAMP                  TOTAL_ELAPSED_TIME RUN_COMMENT                                          TRACE_ID
---- ------------------------------ ------------------ -------------------------------------------------- ----------
   3 28-MAR-23 19.23.20.891595000             78254498                                                             2
Usually, I am interested in improving performance, so I look for the code that took the most time and profile the code blocks by elapsed time.  However, this time, I have sorted them by module and line number so I can see which code blocks were executed.
BREAK ON OWNER ON TYPE ON module skip 1
SELECT fi.symbolid, fi.owner, fi.type, fi.module, fi.function, fi.line#, fi.namespace, fi.calls, fi.function_elapsed_time, fi.sql_id
FROM   dbmshp_function_info fi
WHERE  fi.runid = 3
ORDER BY fi.owner, fi.module, fi.line#;
The profile includes the application code and also the Oracle packages owned by SYS Function
 Symbol                                                                                                        Line Name             Elapsed
     ID OWNER              TYPE            MODULE                    FUNCTION                                     # Space   CALLS       Time SQL_ID
------- ------------------ --------------- ------------------------- ---------------------------------------- ----- ----- ------- ---------- -------------
      8 XXXXX_CUST         PACKAGE BODY    CUST_PARALLEL_JOBS        ISJOBSRUNNING                                6 PLSQL      38        708
      9                                                              ISJOBSRUNNING.C_RUNNING_JOBS_CNT            12 PLSQL      38        266
    137                                                              __static_sql_exec_line13                    13 SQL        38       9681 2y7y7t8bf4ykw
    133                                                              __sql_fetch_line23                          23 SQL        38    2026632 2y7y7t8bf4ykw
      6                                                              GETJOBSSTATUS                               42 PLSQL       7        150
      7                                                              GETJOBSSTATUS.C_JOB_STATUS                  48 PLSQL       7         59
    138                                                              __static_sql_exec_line49                    49 SQL         7        565 d5g73bnmxjuqd
    134                                                              __sql_fetch_line59                          59 SQL         7        238 d5g73bnmxjuqd
      3                                                              CUST_SIMULATE_SURRENDER                    105 PLSQL       1       1232
      5                                                              CUST_SIMULATE_SURRENDER.C_JOB_GROUPS       110 PLSQL       1         16
    135                                                              __static_sql_exec_line111                  111 SQL         1        159 1xrrajz8mgbhs
      4                                                              CUST_SIMULATE_SURRENDER.C_CFG              119 PLSQL       1         12
    136                                                              __static_sql_exec_line120                  120 SQL         1         90 9ytv0rhjjp3mr
    131                                                              __sql_fetch_line160                        160 SQL         1        118 1xrrajz8mgbhs
    132                                                              __sql_fetch_line165                        165 SQL         1        135 9ytv0rhjjp3mr
…
     54 XXXXX_SCHEDULER    PACKAGE BODY    SCHEDULER_ENGINE          __pkg_init                                   0 PLSQL       1          5
     55 XXXXX_SCHEDULER    PACKAGE SPEC    SCHEDULER_ENGINE          __pkg_init                                   0 PLSQL       1          5
     52 XXXXX_SCHEDULER    PACKAGE BODY    SCHEDULER_ENGINE          RUN_JOB                                    770 PLSQL       7        176
     53                                                              SET_JOB_ARGUMENT                          1317 PLSQL      21        202
    178                                                              __static_sql_exec_line1355                1355 SQL        21       4733 3h8uatusjv84c
…
    118 SYS                PACKAGE BODY    DBMS_SCHEDULER            CREATE_PROGRAM                              15 PLSQL       1         24
    121                                                              DROP_PROGRAM                                43 PLSQL       2         98
    119                                                              DEFINE_PROGRAM_ARGUMENT                    112 PLSQL       3        186
    122                                                              DROP_PROGRAM_ARGUMENT                      211 PLSQL       6        363
    117                                                              CREATE_JOB                                 432 PLSQL       7        428
    124                                                              RUN_JOB                                    546 PLSQL       7        239
    120                                                              DROP_JOB                                   696 PLSQL      14       7484
    123                                                              ENABLE                                    2992 PLSQL       1         87
    125                                                              SET_ATTRIBUTE                             3063 PLSQL      14        957
    126                                                              SET_ATTRIBUTE                             3157 PLSQL      14       2923
    127                                                              SET_ATTRIBUTE_NULL                        3274 PLSQL       7         42
    116                                                              CHECK_SYS_PRIVS                           3641 PLSQL      69     153470
…
The hierarchical profiler tracks which code blocks call which code blocks, so I can perform a hierarchical query starting where the parent is null.
SELECT symbolid, parentsymid,
       RPAD(' ', (level-1)*2, ' ') || a.name AS name, 
       a.line#, a.calls,
       a.subtree_elapsed_time, 
       a.function_elapsed_time       
FROM   (SELECT fi.symbolid,
               pci.parentsymid,
               RTRIM(fi.owner || '.' || fi.module || '.' || NULLIF(fi.function, fi.module), '.') AS name,
               fi.line#,
               NVL(pci.subtree_elapsed_time, fi.subtree_elapsed_time) AS subtree_elapsed_time, 
               NVL(pci.function_elapsed_time, fi.function_elapsed_time) AS function_elapsed_time, 
               NVL(pci.calls, fi.calls) AS calls
        FROM   dbmshp_function_info fi
               LEFT JOIN dbmshp_parent_child_info pci ON fi.runid = pci.runid AND fi.symbolid = pci.childsymid
        WHERE  fi.runid = 3
        AND    NOT fi.module LIKE 'DBMS_HPROF%'
        ) a
CONNECT BY a.parentsymid = PRIOR a.symbolid
START WITH a.parentsymid IS NULL;
I can see that CUST_PARALLEL_JOBS.CUST_SIMULATE_SURRENDER calls XXXXX_SCHEDULER.SCHEDULER_ENGINE.RUN_JOB and that calls DBMS_SCHEDULER.RUN_JOB.

Symbol  Parent                                                                                                       Line            Elapsed    Elapsed
     ID  Sym ID NAME                                                                                                     #   CALLS       Time       Time
------- ------- ---------------------------------------------------------------------------------------------------- ----- ------- ---------- ----------
     18         XXXXX_CUST_ADDON.CUST_SCHED_SIMSURRENDERS                                                                1       1   78254334        570
      3      18   XXXXX_CUST.CUST_PARALLEL_JOBS.CUST_SIMULATE_SURRENDER                                                105       1   77139478       1232
      4       3     XXXXX_CUST.CUST_PARALLEL_JOBS.CUST_SIMULATE_SURRENDER.C_CFG                                        119       1        102         12
    136       4       XXXXX_CUST.CUST_PARALLEL_JOBS.__static_sql_exec_line120                                          120       1         90         90
…
     52       3     XXXXX_SCHEDULER.SCHEDULER_ENGINE.RUN_JOB                                                           770       7      58708        176
     56      52       XXXXX_SCHEDULER.SCHEDULER_UTILS.LOG_AUDIT_EVENT                                                  173       7         45         40
    115      56         SYS.DBMS_OUTPUT.PUT_LINE                                                                       109      41         43         43
     57      52       XXXXX_SCHEDULER.SCHEDULER_UTILS.SCHEMA_OWNER                                                     238       7         24         24
    124      52       SYS.DBMS_SCHEDULER.RUN_JOB                                                                       546       7      58463        239
    104     124         SYS.DBMS_ISCHED.CHECK_COMPAT                                                                  3509       7         11         11
    112     124         SYS.DBMS_ISCHED.RUN_JOB                                                                        242       7      44391      44391
…
Now I know which code to examine. This query outer joins the profiler data to the source code. NB. Any wrapped code will not be available in the ALL_SOURCE view. You might want to unwrap it, at least in a test environment (see Philipp Salisberg's PL/SQL Unwrapper for SQL Developer).
break on owner on name skip 1 on type
SELECT s.owner, s.type, s.name, h.function, s.line, 
       h.function_elapsed_time/1e6 function_elapsed_time, h.calls, s.text
FROM   all_source s
  LEFT OUTER JOIN dbmshp_function_info h
    ON s.owner = h.owner and s.name = h.module and s.type = h.type and s.line = h.line# and h.runid = 3
WHERE ((         s.owner = 'XXXXX_CUST'
             AND s.name = 'CUST_PARALLEL_JOBS'
             AND s.type = 'PACKAGE BODY'
             AND s.line between 100 and 300
       ) OR (    s.owner = 'XXXXX_SCHEDULER'
             AND s.name = 'SCHEDULER_ENGINE'
             AND s.type = 'PACKAGE BODY'
             AND s.line between 770 and 858
      ))
ORDER BY s.owner, s.name, s.type, s.line
/
Now, I can scan through the code and see how the code blocks were called.
                                                                                     Function
                                                                                      Elapsed
OWNER           TYPE         NAME                 FUNCTION                   LINE        Time   CALLS TEXT
--------------- ------------ -------------------- ------------------------- ----- ----------- ------- -------------------------------------------------------------------------------------------------------------
XXXXX_CUST      PACKAGE BODY CUST_PARALLEL_JOBS   CUST_SIMULATE_SURRENDER     105     .001232       1 PROCEDURE Cust_Simulate_Surrender (pi_bus_in IN SrvContext, pio_err  IN OUT SrvErr)
                                                                              106                     IS
…
                                                                              213                                     -- run current job when it is not started yet
                                                                              214                                     IF l_cfg_tbl(indx_job).allowed = 'Y' -- flag Y - to be started
                                                                              215                                     THEN
                                                                              216                                         -- run current job
                                                                              217                                         XXXXX_scheduler.scheduler_engine.Run_Job (l_cfg_tbl(indx_job).XXXXX_job_name);
                                                                              218                                         --XXXXX_scheduler.scheduler_engine.enable_Job (l_cfg_tbl(indx_job).XXXXX_job_name);
…
XXXXX_SCHEDULER PACKAGE BODY SCHEDULER_ENGINE     RUN_JOB                     770     .000176       7 PROCEDURE RUN_JOB( PI_JOB_NAME SCHEDULER_JOBS.JOB_NAME%TYPE )
…
                                                                              778                     IS
                                                                              779                     BEGIN
                                                                              780                         DBMS_SCHEDULER.RUN_JOB(
                                                                              781                             SCHEDULER_UTILS.SCHEMA_OWNER || '."' || PI_JOB_NAME || '"', USE_CURRENT_SESSION=>FALSE );
                                                                              782
                                                                              783
                                                                              784
                                                                              785                         SCHEDULER_UTILS.LOG_AUDIT_EVENT( 'RunJob', TRUE, PI_OBJECT_NAME => PI_JOB_NAME );
                                                                              786                     EXCEPTION
                                                                              787                       WHEN OTHERS THEN
…
                                                                              798                     END;
                                                                              799
                                                                              800
                                                                              801                     --dmk 29.3.2023 added
                                                                              802                     PROCEDURE ENABLE_JOB( PI_JOB_NAME SCHEDULER_JOBS.JOB_NAME%TYPE )
…
                                                                              810                     IS
                                                                              811                     BEGIN
                                                                              812                         DBMS_SCHEDULER.Enable(
                                                                              813                             SCHEDULER_UTILS.SCHEMA_OWNER || '."' || PI_JOB_NAME || '"');
                                                                              814
                                                                              815
                                                                              816
                                                                              817                         SCHEDULER_UTILS.LOG_AUDIT_EVENT( 'Enable_Job', TRUE, PI_OBJECT_NAME => PI_JOB_NAME );
                                                                              818                     EXCEPTION
                                                                              819                       WHEN OTHERS THEN
…
                                                                              830                     END;
By following the profiler data, I have found that DBMS_SCHEDULER.RUN_JOB was used.  I was then able to add an alternative procedure that calls DBMS_SCHEDULER.ENABLE and call that from the custom application code.

Thursday, April 13, 2023

Using SQL Profiles to Tackle High Parse Time and CPU Consumption

The Challenge of Dynamic SQL with Literals   

The following example is taken from a PeopleSoft General Ledger system. The SQL was generated by the nVision reporting tool (some literal values have been obfuscated).
SELECT L4.TREE_NODE_NUM,SUM(A.POSTED_TOTAL_AMT) 
FROM PS_XX_SUM_XXXXX_VW A, PSTREESELECT10 L4, PSTREESELECT10 L2 
WHERE A.LEDGER='X_UKMGT' 
AND A.FISCAL_YEAR=2022 AND A.ACCOUNTING_PERIOD=1 
AND L4.SELECTOR_NUM=415 AND A.CHARTFIELD3=L4.RANGE_FROM_10 
AND L2.SELECTOR_NUM=416 AND A.ACCOUNT=L2.RANGE_FROM_10 
AND (A.DEPTID BETWEEN '10000' AND '18999' OR
A.DEPTID BETWEEN '20000' AND '29149' OR A.DEPTID='29156' OR
A.DEPTID='29158' OR A.DEPTID BETWEEN '29165' AND '29999' OR A.DEPTID
BETWEEN '30000' AND '39022' OR A.DEPTID BETWEEN '39023' AND '39999' OR
A.DEPTID BETWEEN '40000' AND '49999' OR A.DEPTID BETWEEN '50000' AND
'59999' OR A.DEPTID BETWEEN '60000' AND '69999' OR A.DEPTID BETWEEN
'70000' AND '79999' OR A.DEPTID BETWEEN '80000' AND '89999' OR
A.DEPTID='29150' OR A.DEPTID=' ') 
AND A.CHARTFIELD1='0120413' 
AND A.CURRENCY_CD='GBP' 
GROUP BY L4.TREE_NODE_NUM

Plan hash value: 1653134809

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                                                   | Name               | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                                            |                    |       |       |    27 (100)|          |       |       |        |      |            |
|   1 |  PX COORDINATOR                                             |                    |       |       |            |          |       |       |        |      |            |
|   2 |   PX SEND QC (RANDOM)                                       | :TQ10006           |     1 |    29 |    27  (63)| 00:00:01 |       |       |  Q1,06 | P->S | QC (RAND)  |
|   3 |    HASH GROUP BY                                            |                    |     1 |    29 |    27  (63)| 00:00:01 |       |       |  Q1,06 | PCWP |            |
|   4 |     PX RECEIVE                                              |                    |     1 |    29 |    27  (63)| 00:00:01 |       |       |  Q1,06 | PCWP |            |
|   5 |      PX SEND HASH                                           | :TQ10005           |     1 |    29 |    27  (63)| 00:00:01 |       |       |  Q1,05 | P->P | HASH       |
|   6 |       HASH GROUP BY                                         |                    |     1 |    29 |    27  (63)| 00:00:01 |       |       |  Q1,05 | PCWP |            |
|   7 |        HASH JOIN                                            |                    |     1 |    29 |    27  (63)| 00:00:01 |       |       |  Q1,05 | PCWP |            |
|   8 |         JOIN FILTER CREATE                                  | :BF0000            |     1 |    16 |    25  (68)| 00:00:01 |       |       |  Q1,05 | PCWP |            |
|   9 |          PX RECEIVE                                         |                    |     1 |    16 |    25  (68)| 00:00:01 |       |       |  Q1,05 | PCWP |            |
|  10 |           PX SEND HYBRID HASH                               | :TQ10003           |     1 |    16 |    25  (68)| 00:00:01 |       |       |  Q1,03 | P->P | HYBRID HASH|
|  11 |            STATISTICS COLLECTOR                             |                    |       |       |            |          |       |       |  Q1,03 | PCWC |            |
|  12 |             VIEW                                            | VW_GBC_10          |     1 |    16 |    25  (68)| 00:00:01 |       |       |  Q1,03 | PCWP |            |
|  13 |              HASH GROUP BY                                  |                    |     1 |    67 |    25  (68)| 00:00:01 |       |       |  Q1,03 | PCWP |            |
|  14 |               PX RECEIVE                                    |                    |     1 |    67 |    25  (68)| 00:00:01 |       |       |  Q1,03 | PCWP |            |
|  15 |                PX SEND HASH                                 | :TQ10002           |     1 |    67 |    25  (68)| 00:00:01 |       |       |  Q1,02 | P->P | HASH       |
|  16 |                 HASH GROUP BY                               |                    |     1 |    67 |    25  (68)| 00:00:01 |       |       |  Q1,02 | PCWP |            |
|  17 |                  HASH JOIN                                  |                    |    60 |  4020 |    24  (67)| 00:00:01 |       |       |  Q1,02 | PCWP |            |
|  18 |                   JOIN FILTER CREATE                        | :BF0001            |    60 |  3120 |    22  (73)| 00:00:01 |       |       |  Q1,02 | PCWP |            |
|  19 |                    PX RECEIVE                               |                    |    60 |  3120 |    22  (73)| 00:00:01 |       |       |  Q1,02 | PCWP |            |
|  20 |                     PX SEND HYBRID HASH                     | :TQ10000           |    60 |  3120 |    22  (73)| 00:00:01 |       |       |  Q1,00 | P->P | HYBRID HASH|
|  21 |                      STATISTICS COLLECTOR                   |                    |       |       |            |          |       |       |  Q1,00 | PCWC |            |
|  22 |                       PX BLOCK ITERATOR                     |                    |    60 |  3120 |    22  (73)| 00:00:01 |    29 |    29 |  Q1,00 | PCWC |            |
|  23 |                        MAT_VIEW REWRITE ACCESS INMEMORY FULL| PS_XX_SUM_XXXXX_MV |    60 |  3120 |    22  (73)| 00:00:01 |    29 |    29 |  Q1,00 | PCWP |            |
|  24 |                   PX RECEIVE                                |                    |   306 |  4590 |     2   (0)| 00:00:01 |       |       |  Q1,02 | PCWP |            |
|  25 |                    PX SEND HYBRID HASH                      | :TQ10001           |   306 |  4590 |     2   (0)| 00:00:01 |       |       |  Q1,01 | P->P | HYBRID HASH|
|  26 |                     JOIN FILTER USE                         | :BF0001            |   306 |  4590 |     2   (0)| 00:00:01 |       |       |  Q1,01 | PCWP |            |
|  27 |                      PX BLOCK ITERATOR                      |                    |   306 |  4590 |     2   (0)| 00:00:01 |   416 |   416 |  Q1,01 | PCWC |            |
|  28 |                       TABLE ACCESS STORAGE FULL             | PSTREESELECT10     |   306 |  4590 |     2   (0)| 00:00:01 |   416 |   416 |  Q1,01 | PCWP |            |
|  29 |         PX RECEIVE                                          |                    |   202 |  2626 |     2   (0)| 00:00:01 |       |       |  Q1,05 | PCWP |            |
|  30 |          PX SEND HYBRID HASH                                | :TQ10004           |   202 |  2626 |     2   (0)| 00:00:01 |       |       |  Q1,04 | P->P | HYBRID HASH|
|  31 |           JOIN FILTER USE                                   | :BF0000            |   202 |  2626 |     2   (0)| 00:00:01 |       |       |  Q1,04 | PCWP |            |
|  32 |            PX BLOCK ITERATOR                                |                    |   202 |  2626 |     2   (0)| 00:00:01 |   415 |   415 |  Q1,04 | PCWC |            |
|  33 |             TABLE ACCESS STORAGE FULL                       | PSTREESELECT10     |   202 |  2626 |     2   (0)| 00:00:01 |   415 |   415 |  Q1,04 | PCWP |            |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------

   1 - SEL$6240F0FF
  12 - SEL$B80655F7 / VW_GBC_10@SEL$9C8D6CC0
  13 - SEL$B80655F7
  23 - SEL$B80655F7 / PS_XX_SUM_XXXXX_MV@SEL$CAD4EEF6
  28 - SEL$B80655F7 / L2@SEL$1
  33 - SEL$6240F0FF / L4@SEL$1
… 
In my example, ASH sampled 276 different SQL IDs.  Each one was only executed once.  There may have been more statements, but ASH only persists one sample every 10s. Cumulatively, they consumed 2843 seconds of DB time in SQL hard parse.
                                                                                                                                          Plan
                                                                                          SQL Plan        Force Matching    SQL   Plan   Parse
 # OPRID        RUNCNTLID              ACTION                           SQL_ID          Hash Value             Signature    IDs  Execs    Secs Table Name
-- ------------ ---------------------- -------------------------------- ------------- ------------ --------------------- ------ ------ ------- ------------------ 
 1 NVISION      NVS_RPTBOOK_99         PI=9984520:UKGL999I:12345        01g5hvs91k4hn   1653134809   1995330195085985689    276    276    2843 PS_XX_SUM_XXXXX_MV 
… 
This is one of at least 276 different SQL statements that all have the same force-matching signature. The statements are essentially the same but differ in some of their literal values. That means that the database has to treat each one as a different SQL statement that must be fully parsed separately. 
SQL Parse involves checking the statement is syntactically correct, and that the user has permission to access the objects, then during the SQL optimization stage the optimizer decides how to execute the statement before it moves to row source generation. 
If the statement has been parsed previously and is still in the shared pool, Oracle can skip the optimization and row source generation stages. This is often called soft parse. 

SQL Optimization 

During the optimization stage, the optimizer calculates the 'cost' of different possible execution plans. Depending upon the SQL, the optimizer considers different table join orders, different table join methods, and different SQL transformations. The optimizer cost is an estimation of the time that it will take to execute a particular plan. The unit of cost is roughly equivalent to the duration of a single block read. More expensive plans are abandoned as they become more expensive than the cheapest known plan so far. Thus the 'cost-based' optimizer produces the cheapest plan. However, the process of optimization consumes time and CPU. 
If I write SQL that is executed many times with bind variables rather than literals, then I should avoid some hard parses and the associated CPU consumption. Oracle has always recommended using bind variables rather than literals to improve performance as well as protect against SQL injection. However, there are many applications that still use literals, particularly in dynamically generated SQL. Every statement has to be hard parsed, and the cumulative CPU consumption can start to become significant. PeopleSoft is one such application that does this in some areas of the product, but it is by no means an isolated example. 
Oracle produced a feature called Cursor Sharing. Literals in statements are automatically converted to bind variables. It can be very effective. It does reduce SQL parse, but can sometimes also produce undesirable side effects where the execution plan may not change as the bind variable values change. 

Hints 

Hints are directives to the optimizer. They tell it to do something or more generally not to do something else. If I were to add some optimizer hints to a statement that will produce the same, or a similar, execution plan, then the optimizer should do less work, consume less CPU, and less time coming to the same or similar conclusion. 
 For example, if I add a LEADING hint to force the optimizer to start with a particular object, that will reduce the number of join orders to be considered. 
  • A two-table query has 2 possible join orders; a LEADING hint will reduce it to 1. 
  • A three-table query has 6 possible join orders; a LEADING hint on a single table will reduce it to 2. 
Often, it is not possible to add hints directly to the code in the application because it is all dynamically generated inside a package, or it may not be desirable to alter third-party code. In my example, the SQL was generated by compiled code within the nVision reporting tool that I cannot alter. I can't use a SQL Patch because I would need a patch for every SQL_ID and I can't predicate the SQL_IDs. Instead, I can create a force-matching SQL profile that will match every statement with the same force-matching signature. 
N.B. SQL Profiles require the SQL Tuning pack licence. 

Example SQL Profile 

I don't have to use the full outline of hints from the execution plan, I have chosen to apply just a few. 
  • LEADING(L2): I want the query to start with the dimension table PSTREESELECT10. This will result in a change to the execution plan 
  • REWRITE: PS_XX_SUM_XXXXX_MV is a materialized view built on the view PS_XX_SUM_XXXXX_VW of an underlying summary ledger. Rewriting the SQL to use the materialized view is a cost-based decision. Oracle usually decides to rewrite it to use the materialized view, but I want to ensure that this always happens with this hint. 
  • NO_PARALLEL: This query selects only a single accounting period, so it is only scanning a single partition, therefore I don't want to invoke a parallel query. 
  • PX_JOIN_FILTER(PS_XX_SUM_XXXXX_MV@SEL$CAD4EEF6): The dimension table is equijoined to the fact table. Therefore, it is a good candidate for using a Bloom filter on the look-up fact table. This doesn't always happen naturally on this statement. I have had to use the query block name taken from the execution plan of the rewritten statement.  The query block name is stable, it is a hash value based on the object name and the operation.
The profile is then created with DBMS_SQLTUNE.IMPORT_SQL_PROFILE.
set serveroutput on
DECLARE
  l_sql_text CLOB;
  l_signature NUMBER;
  h       SYS.SQLPROF_ATTR;
…
BEGIN
…
h := SYS.SQLPROF_ATTR(
q'[BEGIN_OUTLINE_DATA]',
q'[NO_PARALLEL]',
q'[LEADING(L2)]',
q'[PX_JOIN_FILTER(PS_XX_SUM_XXXXX_MV@SEL$CAD4EEF6)]',
q'[REWRITE]',
q'[END_OUTLINE_DATA]');

l_signature := DBMS_SQLTUNE.SQLTEXT_TO_SIGNATURE(l_sql_text);

DBMS_SQLTUNE.IMPORT_SQL_PROFILE (
sql_text    => l_sql_text,
profile     => h,
name        => 'NVS_UKGL999I_FUNC_ACEXP1',
category    => 'DEFAULT',
validate    => TRUE,
replace     => TRUE,
force_match => TRUE);
…
END;
/
This is the execution plan with the SQL Profile. The note confirms that a SQL profile was used. The hint report shows the hints from the SQL Profile.  
Note that the SELECTOR_NUM and CHARTFIELD1 predicates have changed.
SELECT L4.TREE_NODE_NUM,SUM(A.POSTED_TOTAL_AMT) 
FROM PS_XX_SUM_XXXXX_VW A, PSTREESELECT10 L4, PSTREESELECT10 L2 
WHERE A.LEDGER='X_UKMGT' 
AND A.FISCAL_YEAR=2023 AND A.ACCOUNTING_PERIOD=1 
AND L4.SELECTOR_NUM=433 AND A.CHARTFIELD3=L4.RANGE_FROM_10 
AND L2.SELECTOR_NUM=434 AND A.ACCOUNT=L2.RANGE_FROM_10 
AND (A.DEPTID BETWEEN '10000' AND '18999' OR
A.DEPTID BETWEEN '20000' AND '29149' OR A.DEPTID='29156' OR
A.DEPTID='29158' OR A.DEPTID BETWEEN '29165' AND '29999' OR A.DEPTID
BETWEEN '30000' AND '39022' OR A.DEPTID BETWEEN '39023' AND '39999' OR
A.DEPTID BETWEEN '40000' AND '49999' OR A.DEPTID BETWEEN '50000' AND
'59999' OR A.DEPTID BETWEEN '60000' AND '69999' OR A.DEPTID BETWEEN
'70000' AND '79999' OR A.DEPTID BETWEEN '80000' AND '89999' OR
A.DEPTID='29150' OR A.DEPTID=' ') 
AND A.CHARTFIELD1='0051001' 
AND A.CURRENCY_CD='GBP' 
GROUP BY L4.TREE_NODE_NUM

Plan hash value: 3033847137

---------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                                  | Name               | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
---------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                           |                    |       |       |   214 (100)|          |       |       |
|   1 |  SORT GROUP BY                             |                    |     5 |   400 |   214  (62)| 00:00:01 |       |       |
|   2 |   HASH JOIN                                |                    |  2347 |   183K|   213  (62)| 00:00:01 |       |       |
|   3 |    HASH JOIN                               |                    |  2347 |   153K|   210  (63)| 00:00:01 |       |       |
|   4 |     JOIN FILTER CREATE                     | :BF0000            |   306 |  4590 |     3   (0)| 00:00:01 |       |       |
|   5 |      PARTITION RANGE SINGLE                |                    |   306 |  4590 |     3   (0)| 00:00:01 |   434 |   434 |
|   6 |       TABLE ACCESS STORAGE FULL            | PSTREESELECT10     |   306 |  4590 |     3   (0)| 00:00:01 |   434 |   434 |
|   7 |     JOIN FILTER USE                        | :BF0000            | 26468 |  1344K|   206  (64)| 00:00:01 |       |       |
|   8 |      PARTITION RANGE SINGLE                |                    | 26468 |  1344K|   206  (64)| 00:00:01 |    42 |    42 |
|   9 |       MAT_VIEW REWRITE ACCESS INMEMORY FULL| PS_XX_SUM_XXXXX_MV | 26468 |  1344K|   206  (64)| 00:00:01 |    42 |    42 |
|  10 |    PARTITION RANGE SINGLE                  |                    |   202 |  2626 |     3   (0)| 00:00:01 |   433 |   433 |
|  11 |     TABLE ACCESS STORAGE FULL              | PSTREESELECT10     |   202 |  2626 |     3   (0)| 00:00:01 |   433 |   433 |
---------------------------------------------------------------------------------------------------------------------------------

Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------

   1 - SEL$38F8C49D
   6 - SEL$38F8C49D / L2@SEL$1
   9 - SEL$38F8C49D / PS_XX_SUM_XXXXX_MV@SEL$CAD4EEF6
  11 - SEL$38F8C49D / L4@SEL$1

Hint Report (identified by operation id / Query Block Name / Object Alias):
Total hints for statement: 4 (U - Unused (1))
---------------------------------------------------------------------------

   0 -  STATEMENT
         U -  NO_PARALLEL
…
   1 -  SEL$38F8C49D
           -  LEADING(L2)
           -  REWRITE

   9 -  SEL$38F8C49D / PS_XX_SUM_XXXXX_MV@SEL$CAD4EEF6
           -  PX_JOIN_FILTER(PS_XX_SUM_XXXXX_MV@SEL$CAD4EEF6)

Note
-----
…
   - SQL profile "NVS_UKGL999I_FUNC_ACEXP1" used for this statement
  • The new execution plan does indeed start with the dimension table 
  • The query was rewritten to use the materialized view 
  • A Bloom filter was used on the materialized view that is now the fact table
  • The NO_PARALLEL hint wasn't used because Oracle chose not to parallelise this statement anyway.

                                                                                 Plan
                                                                                          SQL Plan        Force Matching    SQL   Plan   Parse
 # OPRID        RUNCNTLID              ACTION                           SQL_ID          Hash Value             Signature    IDs  Execs    Secs Table Name
-- ------------ ---------------------- -------------------------------- ------------- ------------ --------------------- ------ ------ ------- ------------------ 
…
 1 NVISION      NVS_RPTBOOK_99         PI=9984933:UKGL278I:12345        03nwc4yy1r1r7   3033847137   1995330195085985689    138    138    1428 PS_XX_SUM_XXXXX_MV
Now just 1428s is spent on parse time. We only found 138 SQL IDs, but that is just because there are fewer ASH samples because it is taking less time. 
In this case, adding these hints with a SQL Profile has halved the time spent parsing this set of SQL statements.

Tuesday, April 11, 2023

Reading Trace files with SQL

Oracle 12.2 provided some new views that enable trace files to be read via SQL. Previously, it had been possible to do this by creating external tables, but the new views make it much easier. You can simply query what trace files exist with SQL, and then access them without need for server access. 

This is particularly useful on some cloud platforms such as autonomous database, where there is no server access, even for the DBA. However, this technique is applicable to all Oracle databases.  Now, not just the DBA, but developers can easily obtain trace files.

Lots of other people have blogged about this, but Chris Antognini makes the point extremely well:
In a post on my PeopleSoft blog, I demonstrated enabling trace on an application server process.  I also specified that as a trace file identifier. Now I can query the trace files that exist, and restrict the query by filename or date.
set pages 99
select * from gv$diag_trace_file f
where 1=1
and f.modify_time > trunc(sysdate)-1
and f.trace_filename like 'finprod%ora%.trc'
order by modify_time desc
/

   INST_ID ADR_HOME                                                     TRACE_FILENAME                 CHANGE_TIME                          MODIFY_TIME                              CON_ID
---------- ------------------------------------------------------------ ------------------------------ ------------------------------------ ------------------------------------ ----------
         1 /u02/app/oracle/diag/rdbms/finprod/finprod1                  finprod1_ora_306641.trc        23/03/2023 21.25.41.000000000 -05:00 23/03/2023 21.25.41.000000000 -05:00          0
Then I can also query the trace file contents, and even just spool it to a local file.
clear screen 
set head off pages 0 feedback off
with x as (
select /*+LEADING(F)*/ f.trace_filename, c.line_number, c.payload
--, max(c.line_number) over (partition by c.trace_filename) max_line_number
from gv$diag_trace_file f, gv$diag_trace_File_contents c
where c.adr_home = f.adr_home
and c.trace_filename = f.trace_filename
and f.modify_time > trunc(sysdate)-1
and f.trace_filename like 'finprod%ora%306641.trc'
)
select payload from x
ORDER BY line_number
/
The contents of the spool file looks just like the trace file.  I can profile it with tkprof or another trace profiler.
Trace file /u02/app/oracle/diag/rdbms/finprod/finprod1/trace/finprod1_ora_306641.trc
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.16.0.0.0
Build label:    RDBMS_19.16.0.0.0DBRU_LINUX.X64_220701
ORACLE_HOME:    /u02/app/oracle/product/19.0.0.0/dbhome_1
System name:      Linux 
Node name:  naukp-aora101
Release:    4.14.35-2047.514.5.1.2.el7uek.x86_64 
Version:    #2 SMP Thu Jul 28 15:33:31 PDT 2022 
Machine:    x86_64 
Storage:    Exadata 
Instance name: finprod1
Redo thread mounted by this instance: 1
Oracle process number: 225
Unix process pid: 306641, image: oracle@xxxxp-aora102

*** 2023-03-23T21:46:34.632063-04:00
*** SESSION ID:(2337.13457) 2023-03-23T21:46:34.632080-04:00
*** CLIENT ID:(NVRUNCNTL) 2023-03-23T21:46:34.632086-04:00
*** SERVICE NAME:(finprod.acme.com) 2023-03-23T21:46:34.632161-04:00
*** MODULE NAME:(RPTBOOK) 2023-03-23T21:46:34.632166-04:00
*** ACTION NAME:(PI=9980346:NVGL0042:42001) 2023-03-23T21:46:34.632171-04:00
*** CLIENT DRIVER:() 2023-03-23T21:46:34.632177-04:00

IPCLW:[0.0]{-}[RDMA]:RC: [1679622394631549]Connection 0x7f83ee131550 not formed (2). Returning retry.
IPCLW:[0.1]{E}[RDMA]:PUB: [1679622394631549]RDMA lport 0x400012c62778 dst 100.107.2.7:40056 bid 0x1805ea7b58 rval 2