Tuesday, March 04, 2025

SQL Quarantine Behaviour When the Same SQL Executes in Different Resource Manager Consumer Groups With Different CPU Time Limits

What happens if I have two consumer groups with different CPU time limits, but the same long-running SQLs can be run in either group?

There is a use case for this question.  In my sample resource manager plan for PeopleSoft, there are consumer groups with different maximum CPU time limits.  4 hours for scheduled queries, and 2 hours for SQL*Plus/SQL Developer sessions.  

The tests in this article are based on the examples in Tim Hall's article about SQL Quarantine in 19c.

N.B. SQL Quarantine is an Exadata-only feature.

I have created a simple plan with two consumer groups that have CPU time-outs of 30 and 60 seconds.  There are no group mappings; I will switch groups manually.

BEGIN 
  DBMS_RESOURCE_MANAGER.CLEAR_PENDING_AREA;
  DBMS_RESOURCE_MANAGER.CREATE_PENDING_AREA;

  DBMS_RESOURCE_MANAGER.CREATE_PLAN('GFC_TEST_PLAN');
  DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP('LOW_LIMITED30_GROUP');
  DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP('LOW_LIMITED60_GROUP');

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE(
    'GFC_TEST_PLAN', 'LOW_LIMITED30_GROUP',
    mgmt_p8 => 1,
    switch_group => 'CANCEL_SQL', 
    switch_time => 30, 
    switch_for_call => TRUE);

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE(
    'GFC_TEST_PLAN', 'LOW_LIMITED60_GROUP', 
    mgmt_p8 => 1,
    switch_group => 'CANCEL_SQL', 
    switch_time => 60, 
    switch_for_call => TRUE);

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE(
    'GFC_TEST_PLAN', 'OTHER_GROUPS');

  DBMS_RESOURCE_MANAGER.VALIDATE_PENDING_AREA;
  DBMS_RESOURCE_MANAGER.SUBMIT_PENDING_AREA;

  DBMS_RESOURCE_MANAGER_PRIVS.grant_switch_consumer_group(
    grantee_name   => user,
    consumer_group => 'LOW_LIMITED30_GROUP',
    grant_option   => FALSE);

  DBMS_RESOURCE_MANAGER_PRIVS.grant_switch_consumer_group(
    grantee_name   => user,
    consumer_group => 'LOW_LIMITED60_GROUP',
    grant_option   => FALSE);
END;
/

ALTER SYSTEM SET resource_manager_plan = 'GFC_TEST_PLAN';
I am going to create a PL/SQL function that will burn CPU as it goes around a loop.  It is just like Tim's, except that I specify the number of seconds for which I want mine to run, and it will return the number of times it has looped.
set serveroutput on timi on
CREATE OR REPLACE FUNCTION burn_cpu (p_secs IN NUMBER)
  RETURN NUMBER
AS
  l_start_time DATE;
  l_number     NUMBER := 1;
BEGIN
  l_start_time := SYSDATE;
  LOOP
    EXIT WHEN SYSDATE - l_start_time > (p_secs/86400);
    l_number := l_number + 1;
  END LOOP;
  RETURN l_number;
END;
/
I will start by switching to the consumer group LOW_LIMITED30_GROUP
DECLARE
  l_session v$session%ROWTYPE;
  l_consumer_group VARCHAR2(30):= 'LOW_LIMITED30_GROUP';
--l_consumer_group VARCHAR2(30):= 'LOW_LIMITED60_GROUP';
BEGIN 
  SELECT * INTO l_session FROM v$session WHERE sid = Sys_Context('USERENV', 'SID');
  dbms_output.put_line(l_session.sid||'.'||l_session.serial#||':'||l_session.module||':'||l_session.action||':'||l_session.resource_Consumer_Group);
  DBMS_RESOURCE_MANAGER.SWITCH_CONSUMER_GROUP_FOR_SESS(l_session.sid,l_session.serial#,consumer_group=>l_consumer_group);
  SELECT * INTO l_session FROM v$session WHERE sid = Sys_Context('USERENV', 'SID');
  dbms_output.put_line(l_session.sid||'.'||l_session.serial#||':'||l_session.module||':'||l_session.action||':'||l_session.resource_Consumer_Group);
END;
/
I have patch 30104721 installed on 19c to backport the new parameters in 23c, so I need to enable quarantine capture (see Oracle Doc ID 2635030.1: 19c New Feature SQL Quarantine - How To Stop Automatic SQL Quarantine).
ALTER SESSION SET OPTIMIZER_CAPTURE_SQL_QUARANTINE = TRUE;
ALTER SESSION SET OPTIMIZER_USE_SQL_QUARANTINE = TRUE;
I can run queries that usually run for 45 and 75 seconds, but they will be stopped when they have consumed 30 seconds of CPU time.
SELECT burn_cpu (45) FROM dual
Error report -
ORA-12777: A non-continuable error encountered.  Check the error stack for additional information [40], [pfrruncae_check_async_events], [], [].
ORA-00040: active time limit exceeded - call aborted

Elapsed: 00:00:33.574
SELECT burn_cpu (75) FROM dual
Error report -
ORA-12777: A non-continuable error encountered.  Check the error stack for additional information [40], [pfrruncae_check_async_events], [], [].
ORA-00040: active time limit exceeded - call aborted

Elapsed: 00:00:34.023
After a short period, I have 2 quarantine directives

select signature, name, sql_text, plan_hash_value, cpu_time, origin
 from dba_sql_quarantine
where sql_text like '%burn_cpu%'


             SIGNATURE NAME                                     SQL_TEXT                       PLAN_HASH_VALUE CPU_TIME   ORIGIN           
---------------------- ---------------------------------------- ------------------------------ --------------- ---------- ----------------
  12281544607895693562 SQL_QUARANTINE_anw6gs0gbta7u125daea2     SELECT burn_cpu (75) FROM dual       308129442 30         RESOURCE-MANAGER
   3516266497211383477 SQL_QUARANTINE_31m29rz6y8gpp125daea2     SELECT burn_cpu (45) FROM dual       308129442 30         RESOURCE-MANAGER
Now, the statements are prevented from running immediately.

SELECT burn_cpu (45) FROM dual
Error report -
ORA-56955: quarantined plan used

Elapsed: 00:00:00.414
SELECT burn_cpu (75) FROM dual
Error report -
ORA-56955: quarantined plan used

Elapsed: 00:00:00.416

Next, I will switch the consumer group to LOW_LIMITED60_GROUP.  The CPU limit is now 60 seconds.  Now, neither command returns ORA-56955.

The 45-second process runs to a successful conclusion.

SELECT burn_cpu (45) FROM dual;

BURN_CPU(45)
------------
    63264226

Elapsed: 00:00:45.846

The 75-second process runs for 65 seconds and is then terminated with ORA-00040 because it has reached the time limit, but not because the execution plan was quarantined.

SELECT burn_cpu (75) FROM dual
Error report -
ORA-12777: A non-continuable error encountered.  Check the error stack for additional information [40], [pfrruncae_check_async_events], [], [].
ORA-00040: active time limit exceeded - call aborted

Elapsed: 00:01:05.952

A quarantine directive with a lower CPU_TIME limit than that of the current consumer group is not applied because the statement may run to completion in a time between the lower limit and the higher limit.  Oracle allows the query to execute; it will be aborted when it reaches the higher CPU time.  

After a while, as the documentation indicates, quarantine capture will update the CPU time limit on the existing quarantine definition limit to the higher limit in the current consumer group.

The query in our example runs for 20 minutes only once, and then never again—unless the resource limit increases or the plan changes. If the limit is increased to 25 minutes, then the Resource Manager permits the statement to run again with the quarantined plan. If the statement runs for 23 minutes, which is below the new threshold, then the Resource Manager removes the plan from quarantine. If the statement runs for 26 minutes, which is above the new threshold, the plan remains in quarantine unless the limit is increased.

However, for the statement that runs successfully (for 45 seconds) in the consumer group with the higher limit (60 seconds), I have not seen the database remove the quarantine directive, even if I remove LOW_LIMITED30_GROUP consumer group from all resource plans and drop it from the database.

SIGNATURE           NAME                                     SQL_TEXT                       PLAN_HASH_VALUE CPU_TIME   ORIGIN          
------------------- ---------------------------------------- ------------------------------ --------------- ---------- ----------------
  12281544607895693562 SQL_QUARANTINE_anw6gs0gbta7u125daea2  SELECT burn_cpu (75) FROM dual       308129442 60         RESOURCE-MANAGER
   3516266497211383477 SQL_QUARANTINE_31m29rz6y8gpp125daea2  SELECT burn_cpu (45) FROM dual       308129442 30         RESOURCE-MANAGER

The next time an attempt is made to execute the 75-second statement in either consumer group, it is quarantined as before.

SELECT burn_cpu (75) FROM dual
Error report -
ORA-56955: quarantined plan used

Conclusion

  • SQL Quarantines are not tied to any particular resource plan or consumer group.
  • If the current consumer group doesn't have any timeout set, then any matching SQL quarantine directive is not applied
  • If a statement has a quarantine directive with a higher CPU time limit than the current consumer group, then it is applied immediately.
  • If a statement has a quarantine directive with a lower CPU time limit than the current consumer group, then it will be allowed to execute.  If the runtime then exceeds the consumer group CPU limit, then it will be timed out with ORA-00040.  The CPU time limit on the quarantine directive will be increased to the limit in the current consumer group.  In future, the statement will immediately be prevented from executing with ORA-56955 in both consumer groups.

Therefore, it is safe to allow different consumer groups in which the same long-running SQL statements may be executed to have different timeouts.  The lower quarantine timeout will not apply to executions in consumer groups with higher timeouts.  The SQL quarantine directives will evolve to have higher timeouts as required.

Monday, March 03, 2025

New Parameters In 21c To Control Automatic SQL Quarantine Can Be Backported To 19c

SQL Quarantine is only available on Exadata.  In 19c, automatic quarantine generation and subsequent use are enabled by default.  

In Oracle 21c, two new parameters have been introduced to control SQL Quarantine.

  • OPTIMIZER_CAPTURE_SQL_QUARANTINE enables or disables the automatic creation of SQL Quarantine configurations.  The default value is FALSE.  If enabled, when the Resource Manager terminates a SQL statement because the statement has exceeded resource limits, the database automatically creates a SQL Quarantine configuration for the execution plan used by the terminated SQL statement. 
  • OPTIMIZER_USE_SQL_QUARANTINE determines whether the optimizer considers SQL Quarantine configurations when choosing an execution plan for a SQL statement.  The default value is TRUE.

Thus, these parameters allow a system to use the SQL quarantine functionality but disable the automatic creation of quarantine configuration or restrict it to particular sessions.

This is a change in default behaviour, or at least a change to the original behaviour in 19c.  Quarantine directives will not be generated by default, but if they exist, the optimiser will apply them.

The parameters can be backported to Oracle 19.3 or later by applying patch 30104721 (see Oracle Doc ID 2635030.1: 19c New Feature SQL Quarantine - How To Stop Automatic SQL Quarantine.

These parameters can both be set at system level and session level.

If SQL quarantine configurations are not created, or set not to be used, or if you are not on Exadata, then SQLs will run to the CPU limit in the current consumer group before they are cancelled.

Monday, February 24, 2025

Quarantined SQL Plans for PS/Queries

This follows on from my previous post, Management of Long Running PS/Queries Cancelled by Resource Manager.

From 19c, on Engineered Systems only (such as Exadata and Exadata Cloud Service) the 'Oracle Database automatically quarantines the plans for SQL statements terminated by … the Resource Manager for exceeding resource limits.

The Resource Manager can set a maximum estimated execution time for a SQL statement, for example, 20 minutes. If a statement execution exceeds this limit, then the Resource Manager terminates the statement. However, the statement may run repeatedly before being terminated, wasting 20 minutes of resources each time it is executed.

Starting in Oracle Database 19c, if a statement exceeds the specified resource limit, then the Resource Manager terminates the execution and “quarantines” the plan. To quarantine the plan means to put it on a blacklist of plans that the database will not execute. Note that the plan is quarantined, not the statement itself.'

[Oracle SQL Tuning Guide: 4.7 About Quarantined SQL Plans]

When an attempt is made to execute a quarantined execution plan an error is produced: ORA-56955: quarantined plan used.

Oracle does not log timed-out or quarantined queries.  On V$SQL and V$SQLSTAT, AVOIDED_EXECUTIONS records the number of times a SQL query has been prevented from running.  However, this will not stay in the library cache long on a PeopleSoft system, due to the continual parse of dynamically generated SQL statements.  As of Oracle 19.20, it is not recorded in AWR on DBA_HIST_SQLSTAT.  

If an error condition occurs during a PSQUERY process run on the process scheduler, the process terminates with an error.  The SQL statement and the error message are recorded in the Application Engine message log.  As demonstrated in the previous blog, we can detect such failures by inspecting the message log of the PSQUERY process that did not finish successfully (ie. it has an end time, but the run status does not indicate success).

Matching Quarantine Directives to Cancelled Queries

Quarantine directives are visible via DBA_SQL_QUARANTINE, including SQL text and execution plan hash value.

It would be useful to know which quarantine directive relates to which query.  However, it is not easy to match the SQL in the PeopleSoft message log entries with that in the quarantine entries. The SQL text in the message log can have multiple spaces. These are stripped out in the DBA_SQL_QUARANTINE view where the normalised SQL statement is visible.  

The timestamp of creation and last execution of the quarantine directive is stored on it, but matching these to when the query was running can result in false positives.

Also, you cannot tell which quarantine directive was created by which consumer group.  The maximum CPU timeout is recorded on DBA_SQL_QUARANTINE.  In my example, it is only possible to distinguish the originating consumer group because the two consumer groups happen to have different timeouts.

A method that matches exactly, but only returns partial rows is to:

  • Obtain ASH data for queries terminated by the resource manager.  It can be matched by timestamp, MODULE, and ACTION (provided that EnableAEMonitoring is enabled).
    • Profile the ASH to find the statement that took the longest during each PSQUERY process, and that is almost certain to be the SQL query.  Thus obtaining the SQL_ID, SQL Plan Hash Value and consumer group ID.  It is also possible to determine the total database time for the query, and the database time spent on CPU.
    • The consumer group name can then be obtained from DBA_HIST_RSRC_CONSUMER_GROUP
  • Obtain the SQL text for the long-running query.  It would also have to be captured by an AWR snapshot.  This does often occur because it was a long-running SQL, but it is not certain.
  • The signature for the SQL statement (not the force-matching signature) can be derived using the SQLTEXT_TO_SIGNATURE function in DBMS_SQLTUNE.  This can be matched to the signature recorded in DBA_SQL_QUARANTINE.
  • You can have multiple quarantine directives for the same signature (i.e. the same SQL statement), each with a different plan hash value.  
    • NB: The plan hash value on DBA_SQL_QUARANTINE is the adaptive plan hash value (with all of its possible plan alternatives), and therefore it matches SQL_FULL_PLAN_HASH_VALUE in the ASH data, and not SQL_PLAN_HASH_VALUE (the plan that actually executed).
Note that
  • When a query executes until timed-out, producing ORA-00040, you usually can find the SQL statement in the AWR repository and so generate the signature to exactly match the quarantine record.
  • When an attempt is made to run a quarantined statement and execution plan, you usually cannot find the SQL statement because it hasn't run for long enough to produce an ASH sample.  Even when it has, you also have to rely on the statement having been captured previously by AWR.  Those conditions only come together occasionally.
This matching process is done by this query: message_log_checker-psquery2.sql.  Below is a sample output.  
  • We can see the quarantine directives that were created when the resource manager cancelled a query, raising error ORA-00040: active time limit exceeded - call aborted.  
  • However, where quarantine directives have prevented SQL from executing, raising error ORA-56955: quarantined plan used, the ASH data from the event that originally created the directive has since been purged, so we cannot use it to match directives.
Mon Feb 24                                                                                                                                                           page    1
                                                     PS/Queries terminated by Resource Manager/quarantined Execution Plan

                                                          Public/                                                              ASH
                                                          Private                                Ru Oracle      Exec    ASH    CPU Message Log
     P.I. DBNAME  OPRID    RUNCNTLID                      Query   QRYNAME                        St Err. #      Secs   Secs   Secs Date/Time Stamp              SQL_ID
--------- ------- -------- ------------------------------ ------- ------------------------------ -- --------- ------ ------ ------ ---------------------------- -------------
   SQL Plan   Full Plan                                                                                      CPU
 Hash Value  Hash Value Consumer Group Name                   SIGNATURE Quarantine Name                      Time  Quarantine Created           Quarantine Last Executed
----------- ----------- ------------------------- --------------------- ------------------------------------ ----- ---------------------------- ----------------------------
…
 31452465 FSPROD  USR0001  GBR_JRNL_LN_DETAIL_ACCT        Public  GBR_JRNL_LN_DETAIL_ACCT        10 ORA-56955     36     10     10 20-FEB-25 06.28.03.578218 PM 0wm9g6xkys12h
 4009529842   653370716 PSQUERY_BATCH_GROUP         5584654620166156419

 31451318 FSPROD  USR0002  GBR_JRNL_LN_DETAIL_ACCT        Public  GBR_JRNL_LN_DETAIL_ACCT        10 ORA-56955     36               20-FEB-25 02.36.38.590841 PM


 31451292 FSPROD  USR0002  GBR_JRNL_LN_DETAIL_ACCT        Public  GBR_JRNL_LN_DETAIL_ACCT        10 ORA-56955     36               20-FEB-25 02.30.51.777351 PM


 31438602 FSPROD  USR0003  1                              Private DK_GBR_GL_DETAIL_NEW           10 ORA-00040  28316  28275  14203 18-FEB-25 11.39.19.502412 PM 5qrbrf775whky
 3446094907  3491308607 PSQUERY_BATCH_GROUP        16266909742923016361 SQL_QUARANTINE_f3gxc76u48u59d019243f 14400 18-FEB-25 11.49.33.091081 PM

 31437925 FSPROD  USR0004  16                             Private TB_TEST2                       10 ORA-00040  17684  17654  17541 18-FEB-25 06.09.14.060615 PM 06xqrgj18wp05
 4256462904  2062199471 PSQUERY_BATCH_GROUP         6341132966559464532 SQL_QUARANTINE_5h01uuscnrg2n7aeaaaaf 14400 18-FEB-25 06.17.20.679769 PM

 31437907 FSPROD  USR0004  16                             Private TB_TEST2                       10 ORA-00040  17694  17695  17592 18-FEB-25 06.04.05.942470 PM 4yurn75y2p0t2
 3232504707   121066138 PSQUERY_BATCH_GROUP         4966087806133732884 SQL_QUARANTINE_49usqjjc001hn0737529a 14400 18-FEB-25 06.17.24.869185 PM
 …

Thursday, February 20, 2025

Management of Long Running PS/Queries Cancelled by Resource Manager CPU Limit

I have written previously about using the Oracle database resource manager to prioritise the allocation of CPU to different processes in a PeopleSoft system.  I proposed a sample resource plan that can be used as a starting point to build a resource plan that meets a system's specific objectives and requirements.

This post looks at 

  • How to configure the resource manager to cancel long-running queries,
  • What happens when it does and what PeopleSoft users experience,
  • How the system administrators can monitor such queries,
  • What action could they take?

Configuring SQL Cancellation in a Resource Manager Consumer Group

The sample resource plan, PSFT_PLAN, contains various server consumer groups.  It relies upon MODULE and ACTION being set by enabling PeopleSoft instrumentation (EnableAEMonitoring=1) and/or the psftapi_store_prcsinstance trigger on PSPRCSRQST (see Effective PeopleSoft Performance Monitoring),
  • PSQUERY_BATCH_GROUP
    • Applies to scheduled PSQUERY Application Engine Programs
    • Limited to 4 hours on CPU (or estimated at >= 4 hours)
  • PSQUERY_ONLINE
    • Applies to queries run online via the PeopleSoft Internet Architecture (PIA).
    • There is no resource manager limit for this consumer group.  
      • The PIA has a session timeout (default 20 minutes).  
      • The ICQuery Tuxedo service that runs the queries also has a timeout (default 20 minutes)
    • When the resource manager cancels a SQL call, it simply raises an Oracle error that appears in a PIA message box without further explanation.  It is better to let the PIA timeouts handle online queries in a more controlled fashion.
    • To prevent queries continuing to run on the database after the client has been terminated
  • LOW_LIMITED_GROUP
  • Applies to SQL*Plus, SQL Developer and Toad.
  • Limited to 2 hours on CPU (or estimated at >= 2 hours)

  • Recommendations: 

    • Users should generally be encouraged to schedule queries that will take more than a few minutes to run on the process scheduler.  
    • Resist the temptation to increase either the PIA inactivity timeout or ICQuery service timeout from the delivered setting of 20 minutes. 

    Plan Directives

    Plan directives are created with DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE
      DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE(
        'PSFT_PLAN', 'PSQUERY_BATCH_GROUP'
        ,mgmt_p6 => 1
        ,switch_group => 'CANCEL_SQL'
        ,switch_time => 14400
        ,switch_estimate => TRUE 
        ,switch_for_call => TRUE
      );
    
      DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE(
        'PSFT_PLAN', 'PSQUERY_ONLINE_GROUP'
        ,mgmt_p6 => 90
      );
    
      DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE(
        'PSFT_PLAN', 'LOW_LIMITED_GROUP'
        ,mgmt_p8 => 1
        ,switch_group => 'CANCEL_SQL'
        ,switch_time => 7200
        ,switch_elapsed_time => 7200
        ,switch_estimate => TRUE 
        ,switch_for_call => TRUE
      );
    Four parameters control cancellation behaviour.
    • SWITCH_GROUP specifies the consumer group to which the session is switched when a switch condition is met.  If the group switches to CANCEL_SQL the current call is cancelled, raising error ORA-00400.
    • SWITCH_TIME specified the number of seconds on CPU (not elapsed time).
    • If SWITCH_ESTIMATE is true, the resource manager also switches group if the estimated run time is greater than the switch time
    • SWITCH_FOR_CALL is set to true so that if the consumer group is switched, it is then restored to the original consumer group at the end of the top call.  Thus a persistent session is not permanently switched. This is important if switching an application engine server (PSAESRV) session.

    Cancellation Behaviour

    The resource manager can cancel long-running queries in these consumer groups raising ORA-00040: active time limit exceeded - call aborted 
    • The query may be cancelled immediately because the estimated execution time is greater than the limit.  
    • Otherwise, it is quite likely to run for an elapsed time that is greater than the CPU time limit. The database session is only on CPU if the event reported on the session is NULL.  Otherwise, it is doing something else.
    • Some time will be consumed in the client process, during which the database session will be idle waiting for the next fetch request from the client.  This is usually reported as event SQL*Net message from client.
    • Some of the database time may not be on CPU because it may be doing something else, such as physical IO.
    • The database session may be held back by the resource manager allocating CPU to higher priority processes,  in which case again the session will not be on CPU, and will report being on event resmgr: cpu quantum.

    Querying the PeopleSoft Message Log

    The full message text is stored in multiple pieces in PS_MESSAGE_LOGPARM and must be reconstructed so that it can be searched for the error code.  I demonstrated this technique in another blog post: Querying the PeopleSoft Message Log with SQL.

    For this analysis, I have made some alterations to the message log query (see message_log_checker-psquery.sql).

    • This query is restricted to messages generated by PSQUERY processes that did not run to success (not run status 9).
    • PeopleSoft messages are typically defined with up to 9 substitution variables, but long SQL statements can have many more entries in PS_MESSAGE_LOGPARM.  So the PL/SQL function in this query simply appends any additional log parameter rows beyond the 9 substitution variables to the end of the generated string.
    • Once the message has been generated we can look for one of the error messages associated with the resource manager terminating a query:
      • ORA-00040: active time limit exceeded - call aborted
      • ORA-56955: quarantined plan used

    It is necessary to filter by message number because even in PS/Query users can write invalid SQL that produces other error messages.  However, all this text processing for each row retrieved makes the query quite slow.

    Here is an example output.  User USR001 ran a private query MY_TEST2 with run control 42.  It ran for 20772s (5h 46m) until terminated by the resource manager.  As explained above, the 4-hour limit is on CPU time that will be less than the elapsed time.
                                                             Public/
                                                              Private                                Ru   Exec                              Msg Msg  Msg
         P.I. DBNAME  OPRID    RUNCNTLID                      Query   QRYNAME                        St   Secs DTTM_STAMP_SEC               Seq Set  Nbr
    --------- ------- -------- ------------------------------ ------- ------------------------------ -- ------ ---------------------------- --- --- ----
    MSG
    ---------------------------------------------------------------------------------------------------------------------------------------------------- 
     12395311 FSPROD  USR001   42                             Private MY_TEST2                       10  20772 10-FEB-25 04.41.47.384694 PM   1  65   30
    File: C:\PT860P13B_2403250500-retail\peopletools\src\pssys\qpm.cppSQL error. Stmt #: 8526  Error Position: 189  Return: 40 - ORA-00040: active time
     limit exceeded - call abortedFailed SQL stmt: SELECT A.LEDGER, A.FISCAL_YEAR, A.BUSINESS_UNIT, …
                                                                                                               10-FEB-25 04.41.47.421800 PM   2  50  380
    Error in running query because of SQL Error, Code=40, Message=ORA-00040: active time limit exceeded - call aborted
    …
    From Oracle 19c on Exadata, timed-out statements are automatically quarantined.  If a quarantined statement is run and a quarantined execution plan is generated, then error ORA-56955 is generated immediately.  Therefore, it can also be detected in the logs.  The query searches for both messages.
                                                              Public/
                                                              Private                                Ru   Exec                              Msg Msg  Msg
         P.I. DBNAME  OPRID    RUNCNTLID                      Query   QRYNAME                        St   Secs DTTM_STAMP_SEC               Seq Set  Nbr
    --------- ------- -------- ------------------------------ ------- ------------------------------ -- ------ ---------------------------- --- --- ----
    MSG
    ----------------------------------------------------------------------------------------------------------------------------------------------------
     12319513 FSPROD  USR002   Transactions                   Public  GBR_JRNL_LINE_DTL_ACCT         10     25 13-FEB-25 11.13.35.746644 PM   1  65   30
    File: C:\PT860P13B_2403250500-retail\peopletools\src\pssys\qpm.cppSQL error. Stmt #: 8526  Error Position: 2783  Return: -8581 - ORA-56955: quarant
    ined plan usedFailed SQL stmt: SELECT A.FISCAL_YEAR, A.ACCOUNTING_PERIOD, A.JRNL_HDR_STATUS, B.LED
    …
                                                                                                               13-FEB-25 11.13.35.814112 PM   2  50  380
    Error in running query because of SQL Error, Code=-8581, Message=ORA-56955: quarantined plan used
    I discuss automatic SQL quarantine for PS/Query in a subsequent blog.

    Opinion

    So far, I have explained how to set a maximum CPU time limit for PS/Queries in a resource manager consumer group and how to detect cancelled PS/Queries by examining the message log.

    The final stage is to close the feedback loop and go back to the users producing the queries, find out what they are trying to do, and why the queries are running for such a long time.

    Tuesday, January 28, 2025

    SQL Profiles for Application Engine Steps that Reference Temporary Records via the %Table() Meta-SQL

    Oracle generally advises using SQL Profiles/Patches/Baselines rather than introducing hints into application code.  Using one of these forms of plan stability saves you from having to alter the code, and then having to test and verify that the change is functionally neutral and release it to production. It also saves repeating that whole process if you ever choose to remove or change the hint.  

    I generally use SQL profiles with PeopleSoft, because they are the only plan stability mechanism that can force match a statement.  That is to say, a force-matching SQL profile will match other SQL statements that differ only in embedded literal values.  PeopleSoft code makes extensive use of literal values. SQL patches and baselines only exactly match statements (so they match the SQL ID and not the force matching signature). Note that SQL Profiles require that the Tuning Pack is licenced, and that is only available on Enterprise Edition or Oracle.

    I have written previously about using SQL Profiles to inject hints into dynamically generated code.  If any part of the SQL (other than literal values) changes, then a different SQL profile is needed for each variation.  

    When generating SQL profiles for dynamically generated code, you have to be able to predict every possible variation in the code and generate a SQL profile for every combination of every variation.  You end up writing code that mimics the dynamic code generation in the application

    The same is true of Application Engine steps that reference temporary records via the %Table() meta-SQL but are otherwise static.  This variation is a special case because it is a part of PeopleTools' behaviour.  It is easy to determine which tables could be referenced by querying some of the PeopleTools tables.  Although, the table name itself can be set dynamically!

    Let's take an example statement.  I identified it as performing poorly during an analysis of ASH data.  I have a SQL_ID, a plan hash value, and ACTION indicates the name of the Application Engine step that generated it (because Application Engine instrumentation is enabled by setting EnableAEMonitoring - see https://blog.go-faster.co.uk/2015/03/undocumented-application-engine.html).

    • SQL ID = bk98x60cspttj
    • SQL Plan Hash Value = 113493817
    • Action = XX_TREESEL.iDetVal.iDetVal.S

    This is the statement and execution plan report generated by dbms_xplan.display_workload_repository (formatted and edited slightly).  I have the statement, the execution plan and the outline of hints that describes the plan.

    SELECT * FROM table(dbms_xplan.display_workload_repository('bk98x60cspttj',113493817,'ADVANCED +ADAPTIVE'));
    
    SQL_ID
    -------------
    INSERT INTO PS_TSEL_P_TAO12 (CHARTFIELD, CHARTFIELD_VALUE,
    PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
    FROM PS_TSEL_R30_TAO12 A, PS_GL_ACCOUNT_TBL DV 
    WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 
    AND DV.SETID = '12345' 
    AND DV.EFFDT = (SELECT MAX(EFFDT) 
     	FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT 
     	AND EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) 
    AND DV.ACCOUNT >= A.RANGE_FROM_30
    AND DV.ACCOUNT <= A.RANGE_TO_30
    
    Plan hash value: 113493817
    
    --------------------------------------------------------------------------------------------------------
    | Id  | Operation                          | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
    --------------------------------------------------------------------------------------------------------
    |   0 | INSERT STATEMENT                   |                   |       |       | 38754 (100)|          |
    |   1 |  LOAD TABLE CONVENTIONAL           | PS_TSEL_P_TAO17   |       |       |            |          |
    |   2 |   SORT UNIQUE                      |                   |     1 |   161 | 38754   (1)| 00:00:02 |
    |*  3 |    FILTER                          |                   |       |       |            |          |
    |   4 |     MERGE JOIN                     |                   | 24219 |  3807K|     8  (50)| 00:00:01 |
    |   5 |      SORT JOIN                     |                   |  1138 |   151K|     3  (34)| 00:00:01 |
    |*  6 |       INDEX STORAGE FAST FULL SCAN | PSATSEL_R30_TAO17 |  1138 |   151K|     2   (0)| 00:00:01 |
    |*  7 |      FILTER                        |                   |       |       |            |          |
    |*  8 |       SORT JOIN                    |                   |  8513 |   207K|     3  (34)| 00:00:01 |
    |*  9 |        INDEX STORAGE FAST FULL SCAN| PS_GL_ACCOUNT_TBL |  8513 |   207K|     2   (0)| 00:00:01 |
    |  10 |     SORT AGGREGATE                 |                   |     1 |    25 |            |          |
    |  11 |      FIRST ROW                     |                   |     1 |    25 |     2   (0)| 00:00:01 |
    |* 12 |       INDEX RANGE SCAN (MIN/MAX)   | PS_GL_ACCOUNT_TBL |     1 |    25 |     2   (0)| 00:00:01 |
    --------------------------------------------------------------------------------------------------------
    
    Query Block Name / Object Alias (identified by operation id):
    -------------------------------------------------------------
       1 - SEL$1
       6 - SEL$1 / A@SEL$1
       9 - SEL$1 / DV@SEL$1
      10 - SEL$2
      12 - SEL$2 / B@SEL$2
    
    Outline Data
    -------------
      /*+
          BEGIN_OUTLINE_DATA
          IGNORE_OPTIM_EMBEDDED_HINTS
          OPTIMIZER_FEATURES_ENABLE('19.1.0')
          DB_VERSION('19.1.0')
          ALL_ROWS
          OUTLINE_LEAF(@"SEL$2")
          OUTLINE_LEAF(@"SEL$1")
          OUTLINE_LEAF(@"INS$1")
          FULL(@"INS$1" "PS_TSEL_P_TAO17"@"INS$1")
          INDEX_FFS(@"SEL$1" "A"@"SEL$1" ("PS_TSEL_R30_TAO17"."PROCESS_INSTANCE"
                  "PS_TSEL_R30_TAO17"."CHARTFIELD" "PS_TSEL_R30_TAO17"."RANGE_FROM_30"
                  "PS_TSEL_R30_TAO17"."RANGE_TO_30"))
          INDEX_FFS(@"SEL$1" "DV"@"SEL$1" ("PS_GL_ACCOUNT_TBL"."SETID" "PS_GL_ACCOUNT_TBL"."ACCOUNT"
                  "PS_GL_ACCOUNT_TBL"."EFFDT"))
          LEADING(@"SEL$1" "A"@"SEL$1" "DV"@"SEL$1")
          USE_MERGE(@"SEL$1" "DV"@"SEL$1")
          PQ_FILTER(@"SEL$1" SERIAL)
          INDEX(@"SEL$2" "B"@"SEL$2" ("PS_GL_ACCOUNT_TBL"."SETID" "PS_GL_ACCOUNT_TBL"."ACCOUNT"
                  "PS_GL_ACCOUNT_TBL"."EFFDT"))
          END_OUTLINE_DATA
      */
    …

    Below is the underlying Application Engine step that generated the SQL.  There is other dynamic code in this step that is driven by configuration data as well as the %Table() meta-SQLs.  Other reasons not to introduce hints into the statement include:

    • different dynamic variations of the code might require different hints, and then I would have to write more code to generate the hint dynamically.  However, in this case, I am only going to deal with a single variation, and I am only going to produce one set of SQL Profiles,
    • this step and the code generation are delivered by PeopleSoft, any change in these areas would be considered as a customisation.

    %InsertSelect(DISTINCT
    , %Bind(FT_TSEL_AET.RECNAME_SEL_TBL, NoQuotes)
    , %Bind(FT_TSEL_STR_AET.DTL_RECNAME, NoQuotes
    ) DV
    , %Bind(FT_TSEL_AET.FIELDNAME_CF, NoQuotes) = %Bind(FT_TSEL_STR_AET.DTL_FIELDNAME)
    , %Bind(FT_TSEL_AET.FIELDNAME_VALUE, NoQuotes) = DV.%Bind(FT_TSEL_STR_AET.DTL_FIELDNAME, NoQuotes)
    , PROCESS_INSTANCE = %Bind(PROCESS_INSTANCE)
    , TREE_NAME = %Bind(FT_TSEL_AET.TREE_NAME)
    , TREE_NODE = A.TREE_NODE
    , TREE_NODE_NUM = A.TREE_NODE_NUM
    , TREE_LEVEL_NUM = A.TREE_LEVEL_NUM, SETCNTRLVALUE =%Bind(FT_TSEL_AET.SETCNTRLVALUE)
    , %Bind(FT_TSEL_GEN_AET.FIELDNAME_FROM, NoQuotes) = A.%Bind(FT_TSEL_GEN_AET.FIELDNAME_FROM, NoQuotes)
    , %Bind(FT_TSEL_GEN_AET.FIELDNAME_TO, NoQuotes) = A.%Bind(FT_TSEL_GEN_AET.FIELDNAME_TO, NoQuotes)
    , SETID_TREE = %Bind(FT_TSEL_GEN_AET.SETID_TREE)
    , EFFDT = %Bind(FT_TSEL_STR_AET.EFFDT)
    , CFV_SET = %Bind(FT_TSEL_AET.CFV_SET)) 
    FROM %Table(%Bind(FT_TSEL_GEN_AET.RECNAME_SEL_TBL, NoQuotes)) A
    , %Table(%Bind(FT_TSEL_STR_AET.DTL_RECNAME, NoQuotes)) DV 
    WHERE A.CHARTFIELD = %Bind(FT_TSEL_AET.FIELDNAME) 
    AND A.PROCESS_INSTANCE = %Bind(PROCESS_INSTANCE) 
    %Bind(FT_TSEL_WRK_AET.WHERE_TXT_LONG, NoQuotes)
    This step references two temporary records.  It generates data in one temporary table TSEL_P_TAO based on data in another, TSEL_R30_TAO.  It happens to use instance 12 of both records.  This is really just a coincidence.  Temporary table instances are allocated and deallocated in a group when an application engine starts and ends, so it is common for one step to use the same instance of different tables, but there is nothing to stop different table instances from being used.  That can occur when concurrently executing multiple instances of a program or different programs that allocate just some of the same tables.

    I now need to produce SQL profiles for each permutation.  I will start by generating a script to create a single profile for a single SQL statement using Oracle's coe_xfr_sql_profile.sql script that is delivered as a part of the SQLT utility.  It is available from the Oracle support website (All About the SQLT Diagnostic Tool (Doc ID 215187.1)).  
    It produces a SQL script such as the one below (I have removed the comments).  The SQL text is in a CLOB variable, and all the hints required to reproduce the same execution plan are in an array.  The script generates a signature for the statement and then creates a SQL profile from that information.
    I have already customised my version of coe_xfr_sql_profile.sql so that FORCE_MATCH => TRUE in the generated scripts.
    WHENEVER SQLERROR EXIT SQL.SQLCODE;
    VAR signature NUMBER;
    DECLARE
    sql_txt CLOB;
    h       SYS.SQLPROF_ATTR;
    BEGIN
    sql_txt := q'[
    INSERT INTO PS_TSEL_P_TAO17 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678
    FROM PS_TSEL_R30_TAO17 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
    = 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
    EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30
    ]';
    h := SYS.SQLPROF_ATTR(
    q'[BEGIN_OUTLINE_DATA]',
    q'[IGNORE_OPTIM_EMBEDDED_HINTS]',
    q'[OPTIMIZER_FEATURES_ENABLE('19.1.0')]',
    q'[DB_VERSION('19.1.0')]',
    q'[ALL_ROWS]',
    q'[OUTLINE_LEAF(@"SEL$683B0107")]',
    q'[OUTLINE_LEAF(@"SEL$C772B8D1")]',
    q'[UNNEST(@"SEL$2")]',
    q'[OUTLINE_LEAF(@"INS$1")]',
    q'[OUTLINE(@"SEL$2")]',
    q'[OUTLINE(@"SEL$7511BFD2")]',
    q'[OUTLINE(@"SEL$1")]',
    q'[FULL(@"INS$1" "PS_TSEL_P_TAO17"@"INS$1")]',
    q'[NO_ACCESS(@"SEL$C772B8D1" "VW_SQ_1"@"SEL$7511BFD2")]',
    q'[INDEX(@"SEL$C772B8D1" "DV"@"SEL$1" ("PS_GL_ACCOUNT_TBL"."SETID" "PS_GL_ACCOUNT_TBL"."ACCOUNT" "PS_GL_ACCOUNT_TBL"."EFFDT"))]',
    q'[FULL(@"SEL$C772B8D1" "A"@"SEL$1")]',
    q'[LEADING(@"SEL$C772B8D1" "VW_SQ_1"@"SEL$7511BFD2" "DV"@"SEL$1" "A"@"SEL$1")]',
    q'[USE_NL(@"SEL$C772B8D1" "DV"@"SEL$1")]',
    q'[USE_MERGE(@"SEL$C772B8D1" "A"@"SEL$1")]',
    q'[PARTIAL_JOIN(@"SEL$C772B8D1" "A"@"SEL$1")]',
    q'[INDEX_FFS(@"SEL$683B0107" "B"@"SEL$2" ("PS_GL_ACCOUNT_TBL"."SETID" "PS_GL_ACCOUNT_TBL"."ACCOUNT" "PS_GL_ACCOUNT_TBL"."EFFDT"))]',
    q'[END_OUTLINE_DATA]');
    
    :signature := DBMS_SQLTUNE.SQLTEXT_TO_SIGNATURE(sql_txt);
    
    DBMS_SQLTUNE.IMPORT_SQL_PROFILE (
    sql_text    => sql_txt,
    profile     => h,
    name        => 'coe_g7wz6ctquwjcy_2476903986',
    description => 'coe g7wz6ctquwjcy 2476903986 '||:signature||'',
    category    => 'DEFAULT',
    validate    => TRUE,
    replace     => TRUE,
    force_match => TRUE /* TRUE:FORCE (match even when different literals in SQL). FALSE:EXACT (similar to CURSOR_SHARING) */ );
    END;
    /
    WHENEVER SQLERROR CONTINUE
    SET ECHO OFF;
    PRINT signature
    PRO
    PRO ... manual custom SQL Profile has been created
    PRO
    SET TERM ON ECHO OFF LIN 80 TRIMS OFF NUMF "";
    SPO OFF;
    PRO
    PRO COE_XFR_SQL_PROFILE_g7wz6ctquwjcy_2476903986 completed
    In Application Engine, different tables will be used if multiple programs referencing the same table run concurrently, or if a process crashes and holds its allocation to a particular table instance.  Therefore, I need to create a set of similar SQL profiles, one for each statement that could be generated by this Application Designer step.  I need to iterate through all possibilities, so I have added some additional pieces to the script.  They are explained in the footnotes below.
    REM coe_xfr_sql_profile_XX_TREESEL.iDetVal.iDetVal.sql
    SPO coe_xfr_sql_profile_XX_TREESEL.iDetVal.iDetVal.log;
    WHENEVER SQLERROR CONTINUE
    SET serveroutput on ECHO ON TERM ON LIN 2000 TRIMS ON NUMF 99999999999999999999;
    clear screen
    ALTER SESSION SET CURRENT_SCHEMA=SYSADM;1
    …
    WHENEVER SQLERROR EXIT SQL.SQLCODE;
    VAR signature NUMBER;
    
    DECLARE
      l_recname1 VARCHAR2(15) :=    'TSEL_R30_TAO';2
      l_table1   VARCHAR2(18) := 'PS_TSEL_R30_TAO17';
      l_recname2 VARCHAR2(15) :=    'TSEL_P_TAO';
      l_table2   VARCHAR2(18) := 'PS_TSEL_P_TAO17';
      l_name     VARCHAR2(30);
      sql_txt CLOB;
      h       SYS.SQLPROF_ATTR;
    
      e_no_sql_profile EXCEPTION;
      PRAGMA EXCEPTION_INIT(e_no_sql_profile, -13833);
    BEGIN
    FOR i IN (3
      WITH v4  AS (SELECT rownum-1 row_number FROM dual CONNECT BY LEVEL <= 100) 
      SELECT DISTINCT 
             v1.row_number id1, r1.recname recname1, t1.table_name table_name1
      ,      v2.row_number id2, r2.recname recname2, t2.table_name table_name2
      ,      o.TEMPTBLINSTANCES
      FROM   psrecdefn r1
      ,      psrecdefn r2
      ,      pstemptblcntvw5 i1
      ,      pstemptblcntvw i2
      ,      all_tables t1
      ,      all_tables t2
      ,      psoptions o 
      ,      ps.psdbowner p 
      ,      v v1
      ,      v v2
      WHERE  r1.rectype = 7 AND r1.recname = i1.recname AND r1.recname = l_recname1
      AND    r2.rectype = 7 AND r2.recname = i2.recname AND r2.recname = l_recname2
      AND    v1.row_number <= i1.temptblinstances + o.temptblinstances6 --up to total number of instances
      AND    v2.row_number <= i2.temptblinstances + o.temptblinstances  --up to total number of instances
      AND    (v1.row_number = 0 OR v1.row_number > o.temptblinstances)7 --omit online temp tables
      AND    (v2.row_number = 0 OR v2.row_number > o.temptblinstances)  --omit online temp tables
      and    t1.owner = p.ownerid AND t1.table_name 
             = DECODE(r1.sqltablename,' ','PS_'||r1.recname,r1.sqltablename) 
             ||DECODE(v1.row_number,0,'',LTRIM(TO_NUMBER(v1.row_number)))8 --derive table table
      and    t2.owner = p.ownerid AND t2.table_name 
             = DECODE(r2.sqltablename,' ','PS_'||r2.recname,r2.sqltablename) 
             ||DECODE(v2.row_number,0,'',LTRIM(TO_NUMBER(v2.row_number)))  --derive table table
      AND    ABS(v1.row_number-v2.row_number)<=19  --max variation in instance numbers
      ORDER BY id1, id2
    ) LOOP
      -----------123456789012345678901234567890
      l_name := 'XX_TREESEL.iDetVal.iDetVal'||i.id1||i.id2;10
      
    sql_txt := q'[
    INSERT INTO PS_TSEL_P_TAO17 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678
    FROM PS_TSEL_R30_TAO17 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
    = 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
    EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30
    ]';
    
    h := SYS.SQLPROF_ATTR(
    q'[BEGIN_OUTLINE_DATA]',
    q'[IGNORE_OPTIM_EMBEDDED_HINTS]',
    q'[ALL_ROWS]',
    q'[LEADING(@"SEL$1" "A"@"SEL$1")]',11
    q'[END_OUTLINE_DATA]');
    
    sql_txt := REPLACE(sql_txt, l_table1,i.table_name1);12
    sql_txt := REPLACE(sql_txt, l_table2,i.table_name2);
    :signature := DBMS_SQLTUNE.SQLTEXT_TO_SIGNATURE(sql_txt);
    dbms_output.put_line(l_name||':'||:signature||':'||sql_txt);
    
    for n in 1 .. h.count LOOP13
      if h(n) LIKE '%'||l_table1||'%' THEN
        h(n) := REPLACE(h(n), l_table1,i.table_name1);
        dbms_output.put_line(n||':'||h(n));
      end if;
      if h(n) LIKE '%'||l_table2||'%' THEN
        h(n) := REPLACE(h(n), l_table2,i.table_name2);
        dbms_output.put_line(n||':'||h(n));
      end if;
    end loop;
    
    BEGIN --drop profile if already exists
      DBMS_SQLTUNE.drop_SQL_PROFILE(name => l_name);
      EXCEPTION WHEN e_no_sql_profile THEN NULL;
    END;
    
    DBMS_SQLTUNE.IMPORT_SQL_PROFILE ( --create new profile
    sql_text    => sql_txt,
    profile     => h,
    name        => l_name,
    description => 'coe XX_TREESEL.iDetVal.iDetVal '||l_name||' '||:signature||'',
    category    => 'DEFAULT',
    validate    => TRUE,
    replace     => TRUE,
    force_match => TRUE /* TRUE:FORCE (match even when different literals in SQL). FALSE:EXACT (similar to CURSOR_SHARING) */ );
      END LOOP;
    END;
    /
    WHENEVER SQLERROR CONTINUE
    …
    SPO OFF;
    PRO
    PRO coe_xfr_sql_profile_XX_TREESEL.iDetVal.iDetVal completed

    1. Set CURRENT_SCHEMA to specify PeopleSoft owning schema, SYSADM.
    2. For each temporary record in the original SQL statement, add pairs of variables to specify the name of PeopleSoft record and Oracle temporary table instance referenced in the statement.  I could derive the record name from the table name, but it is easier just to hard-code it.
    3. The SQL statement produces all combinations of temporary records that could appear in the SQL statement.  I will put it in an implicit cursor loop, and then for each row returned, the script will create a SQL profile.
    4. Common table expression V returns 100 rows, numbered 0 to 99.  Irrespective of the number of temporary table instances specified in each Application Engine program, there can only be non-shared 99 table instances for each PeopleTools record, plus the shared instance (that doesn't have a suffix number).  
    5. PSTEMPTBLCNTVW returns the number of non-shared batch (i.e. not online) instances of each temporary record that needs to be built.  This is in addition to the number of online temporary table instances.
    6. The query will return a row for each instance of each temporary table up to the number of instances required by the application engines plus the number of online table instances, but not exceeding the 99 rows returned by CTE V.
    7. Most Application Engines do not run online in the component processor, therefore there is no need to build SQL profiles on these instances.  There are exceptions, such as some as the journal and voucher edit and post processes in Financials, in which case these criteria should be removed.
    8. This expression joins the record and instance number to the table in the database.  Instance 0 will be used to refer to the shared instance.
    9. The instance numbers of the temporary records is permitted to vary by up to one in either direction.  Thus the script generates profiles with instance 12 of one table, and instances 11 to 13 of the other.  It is rare, but possible, for there to be any difference in instance numbers between tables.  It is possible, but rarer for the difference to be greater than one.
    10. SQL Profile names are limited to 30 characters.  I have specified a meaningful name based on the Application Engine step, up to 26 characters, and then the two IDs (which can be up to 2 digits each).
    11. In this example, I am not going to use the full set of hints in the captured profile.  I just want to introduce a single leading hint.
    12. I will substitute each table name in the SQL text with the specific table instance name.
    13. Tables are generally referenced in hints via the row source alias.  However, sometimes the table name appears in the hints, and must also be replaced with the specific table name.  So I also work through all the hints in array h and substitute any table names that may be there.  Indexes are not referenced by their names but by a list of indexed columns.
    The result is a set of profiles for each possible SQL statement.
    XX_TREESEL.iDetVal.iDetVal00:11943215885789639839:
    INSERT INTO PS_TSEL_P_TAO (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
    FROM PS_TSEL_R30_TAO A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678AND DV.SETID 
    = 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
    EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30
    
    XX_TREESEL.iDetVal.iDetVal1717:15747497907378648788:
    INSERT INTO PS_TSEL_P_TAO17 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
    FROM PS_TSEL_R30_TAO17 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
    = 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
    EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30
    
    XX_TREESEL.iDetVal.iDetVal1718:12015611546030583918:
    INSERT INTO PS_TSEL_P_TAO18 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
    FROM PS_TSEL_R30_TAO17 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
    = 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
    EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30
    
    XX_TREESEL.iDetVal.iDetVal1817:14883898515022367531:
    INSERT INTO PS_TSEL_P_TAO17 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
    FROM PS_TSEL_R30_TAO18 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
    = 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
    EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30
    …
    Note
    • If the number of non-shared instances of a table in an Application Engine is increased, you need to build add table instances with Application Designer.
    • If the number of online temporary instances is increased, you need to build additional instances for every temporary record in the database.  If the number is reduced some tables will cease to be used, and they ought to be dropped.  
    • If either the number of application engine table instances or online temporary table instances are changed, then you will need to rerun the script to create additional SQL profiles.