Thursday, June 25, 2009

Controlling How %UpdateStats Collects Optimizer Statistics

I have written a number of entries on this blog about updating database statistics on tables during Application Engine processes.
I proposed a PL/SQL package (wrapper.sql) called from the DDL model to intercept the call from the %UpdateStats macro in Application Engine. By default it
  • Gathers statistics on regular tables.
  • Refreshed only stale statistics on partitioned tables.
  • Does not gather statistics on Global Temporary Records.
I have now published an enhanced version: wrapper848meta.sql.
  • A table PS_GFC_STATS_OVRD holds meta-data to override the default behaviour of the script for certain records. The meta-data can also specify the size of the sample, and the options to control the collection of histograms.
  • If a private instance of a Temporary Records is a Global Temporary Tables, the wrapper may still collect statistics (normally this would be suppressed because of the risk of one session using the statistics collected by another session, but this will not happen for these tables).
Now, it is possible to specify the few tables where statistics must still be explicitly gathered, or whether to do this only if the current statistics on the table are stale. The DBA is probably the person best placed to decide whether and how to collect statistics on which tables, and these decisions can be implemented with the meta-data, but without code change.

Thus, it is possible to
  • Reduce the number of calls to dbms_stats,
  • to reduce the overhead of the remaining calls
  • and at least preserve, if not improve performance of batch processes without making any code changes.

Oracle Statistics History Retention in PeopleSoft

I have been working on a system where many Application Engine programs are running throughout the day, and are frequently collecting Optimizer statistics with the %UpdateStats macro on many working storage tables. Concurrent calls to dbms_stats are typical.

There are two new behaviours in Oracle 10g RDBMS that can in extreme cases, in combination with a system that calls dbms_stats very frequently, create a significant performance overhead.

From Oracle 10g, histograms may, by default, be collected automatically. That means that rows are concurrently deleted from and inserted into histgrm$ and hist_head$, leading to contention and consistent read.
  • Also from Oracle 10g, every time you collect statistics on a table the old statistics are retained in the SYS.WRI$_OPTSTAT%HISTORY tables. If histograms have previously been collected, these are also copied. DBMS_STATS has the additional overhead of writing this history. I found in excess of 10,000 versions of previous statistics for some tables, because the batch processes have updated statistics on working storage tables that many times.
  • dbms_stats also appears to be responsible for purging history older than the retention limit. The default retention period is 31 days. I have seen concurrent calls to dbms_stats blocked on row level locks on the statistics history tables. For me, this occurred 31 days after the system went live on a significantly increased volume.
  • SELECT dbms_stats.get_stats_history_retention FROM dual;
    GET_STATS_HISTORY_RETENTION 
    --------------------------- 
                             31
Statistics history was designed to work in conjunction with schema wide statistics jobs that only refreshed stale statistics. There is an option on gather_schema_stats to collect only statistics on tables where the current statistics are stale. However, there is no such option on gather_table_stats. If you have decided to call this procedure for a particular table, then it is assumed you know you need to refresh the statistics. However, by calling dbms_stats from a batch program you can end up calling it much more frequently than is really necessary.

Recommendations
  • Disable statistics history by using dbms_stats.alter_stats_history_retention to set the retention period to zero. Unfortunately this can only be set at database level. The statistics history is there in case you want to revert to a previous version of the statistics should a new set of statistics produce a problem, but it is only used rarely, and I think this is a necessary sacrifice.
  • EXECUTE dbms_stats.alter_stats_history_retention(retention=>0);
  • Use Oracle Optimizer Dynamic Sampling. However, I suggest increasing the level from the default of 2 to 4 to increase the situations in which it is used.
  • Introduce the new version of the PL/SQL wrapper package for dbms_stats so that you can specify the records for which statistics will be explicitly collected, and whether histograms are to be collect. Thus you can reduce the number of calls to dbms_stats.
  • If you have allowed the statistics history to grow before you disable history retention, then you might like to read John Hallas' posting on Purging statistics from the SYSAUX tablespace.

    Tuesday, May 19, 2009

    Manually Booting Tuxedo Application Server Processes in Parallel

    Normally when an Application Server is booted, initialisation of each process completes before the next one is started. The ability to boot Application Server processes in parallel was added to the psadmin utility in PeopleTools 8.48. However, psadmin is merely a wrapper for the BEA Tuxedo tmadmin command line utility, and it has always been possible to do this manually in previous versions of PeopleTools via the tmadmin utility as follows.

    1. Boot the Tuxedo Bulletin Board Liaison process.

    #boot the Tuxedo administrative processes
    boot -A

    2. Boot the PeopleSoft Application Server processes and but specify the -w parameter so that they don't wait as they start

    boot -g APPSRV -w

    If you are running PUBSUB or other servers in other groups then you would also boot them here.

    3. Boot the JREPSRV process (which maps Java Classes to Tuxedo Services).

    boot -g JREPGRP

    4. List the servers with print server so you know that the PeopleSoft servers are booted.

    psr

    5. When all the other processes have booted, boot the WSL and JSL processes.

    boot -g BASE
    boot -g JSLGRP

    Wednesday, May 13, 2009

    Wednesday, April 22, 2009

    Reducing Unnecessary Instances of Temporary Records

    In a previous posting, I recommended moving temporary records, that are very frequently truncated by Application Engine programs, to a tablespace with a 32Kb block size, and using a larger uniform extent size (I chose 1Mb).

    However, continuing the example for my last posting on this subject, TL_TIMEADMIN has 150 temporary records, that have 173 indexes (in HCM8.9). So you get 323 segments for every instance set in the Application Engine properties, and that would consume at least 323Mb of the 32Kb tablespace. If that space consumption is not a problem, then stop reading now.

    However, I noticed that some temporary records are used by several Application Engine programs. This is usually because one program calls another and the temporary records are referenced in both. However, if both programs have a number of instances of temporary records defined in their properties, then temporary tables will be built for both.

    Lets take an example TL_PUB_TM_AE calls TL_PUB_TM1. They are both delivered with 5 instances.

    AE_APPLID    TEMPTBLINSTANCES
    ------------ ----------------
    TL_PUB_TM1                  5
    TL_PUB_TM_AE                5

    They share 8 temporary records in common.

    SELECT a.recname, a.ae_applid, b.ae_applid
    FROM  psaeappltemptbl a
    FULL OUTER JOIN psaeappltemptbl b
    ON  a.recname = b.recname
    AND b.ae_applid = 'TL_PUB_TM_AE'
    WHERE a.ae_applid = 'TL_PUB_TM1'
    ORDER BY 1
    /
    
    RECNAME         AE_APPLID    AE_APPLID
    --------------- ------------ ------------
    TL_PROF_LIST    TL_PUB_TM1
    TL_PROF_WRK     TL_PUB_TM1
    WRK_PROJ1_TAO   TL_PUB_TM1   TL_PUB_TM_AE
    WRK_PROJ2_TAO   TL_PUB_TM1   TL_PUB_TM_AE
    WRK_PROJ3_TAO   TL_PUB_TM1   TL_PUB_TM_AE
    WRK_PROJ4_TAO   TL_PUB_TM1   TL_PUB_TM_AE
    WRK_PROJ5_TAO   TL_PUB_TM1   TL_PUB_TM_AE
    WRK_PROJ6_TAO   TL_PUB_TM1   TL_PUB_TM_AE
    WRK_PROJ7_TAO   TL_PUB_TM1   TL_PUB_TM_AE
    WRK_PROJ_TAO    TL_PUB_TM1   TL_PUB_TM_AE

    5 temporary records are built by Application Designer for each Application Engine program. But TL_PUB_TM1 is never run on its own. So do you need the extra instances of those 8 temporary records? The temporary records defined on TL_PUB_TM_AE are a subset of TL_PUB_TM1. If you reduced the number of instances on TL_PUB_TM_AE to 0, you would still have the 5 instances defined on TL_PUB_TM_TM1. But that would enable you to drop 40 tables and their indexes.

    So, I started to wonder if there was a general principle here. If the temporary tables on an Application Engine program are a subset of those on another program, then providing you make ensure the number of instances on the superset is not less than those of the subset, you could reduce the number of instances on the subset to 0.

    This view reports Application Engine programs whose temporary records are a subset of those on another program, and also counts the number of records in the subset.

    CREATE OR REPLACE VIEW gfc_aetemptbl_hier AS
    SELECT
    sup.ae_applid sup_applid, supa.temptblinstances sup_instances
    ,  sub.ae_applid sub_applid, suba.temptblinstances sub_instances
    , (SELECT COUNT(*)
    FROM   psaeappltemptbl supc, psaeappltemptbl subc
    WHERE  supc.ae_applid = sup.ae_applid
    AND    subc.ae_applid = sub.ae_applid
    AND    subc.recname = supc.recname) num_records
    FROM (SELECT DISTINCT ae_applid FROM psaeappltemptbl) sup
    , (SELECT DISTINCT ae_applid FROM psaeappltemptbl) sub
    , psaeappldefn supa
    , psaeappldefn suba
    WHERE sup.ae_applid != sub.ae_applid
    AND supa.ae_applid = sup.ae_applid
    AND suba.ae_applid = sub.ae_applid
    AND EXISTS( /*a temporary record in common*/
    SELECT 'x'
    FROM psaeappltemptbl sup1
    , psaeappltemptbl sub1
    WHERE sub1.ae_applid = sub.ae_applid
    AND sup1.ae_applid = sup.ae_applid
    AND sup1.recname = sub1.recname
    AND ROWNUM = 1)
    /*there is no record in the subset that is not in the superset*/
    AND NOT EXISTS(
    SELECT  'x'
    FROM psaeappltemptbl sub2
    WHERE   sub2.ae_applid = sub.ae_applid
    AND NOT EXISTS(
    SELECT  'x'
    FROM psaeappltemptbl sup2
    WHERE   sup2.ae_applid = sup.ae_applid
    AND sub2.recname = sup2.recname
    AND ROWNUM = 1)
    AND ROWNUM = 1)
    /*there is a record in the subset that is not in the subset - so there is a difference*/
    AND EXISTS(
    SELECT  'x'
    FROM psaeappltemptbl sup2
    WHERE   sup2.ae_applid = sup.ae_applid
    AND NOT EXISTS(
    SELECT  'x'
    FROM psaeappltemptbl sub2
    WHERE   sub2.ae_applid = sub.ae_applid
    AND sup2.recname = sub2.recname
    AND ROWNUM = 1)
    AND ROWNUM = 1)
    ORDER BY 1,2;

    This is the output from the view for the Application Engine programs in the example.

    SUP_APPLID   SUP_INSTANCES SUB_APPLID   SUB_INSTANCES NUM_RECORDS
    ------------ ------------- ------------ ------------- -----------
    …
    TL_PUB_TM1               5 TL_PUB_TM_AE             5           8
    TL_PUB_TM1               5 TL_PY_PUB_TM             5           5
    TL_PUB_TM_AE             5 TL_PY_PUB_TM             5           5
    …

    I found that some Application Engine programs have identical sets of temporary records. This can happen when a program is cloned, which some customers do when they want to customise a vanilla program. This view reports on them.

    CREATE OR REPLACE VIEW gfc_aetemptbl_eq AS
    SELECT sup.ae_applid sup_applid, supa.temptblinstances sup_instances
    , sub.ae_applid sub_applid, suba.temptblinstances sub_instances
    , (SELECT COUNT(*)
    FROM   psaeappltemptbl supc, psaeappltemptbl subc
    WHERE  supc.ae_applid = sup.ae_applid
    AND    subc.ae_applid = sub.ae_applid
    AND    subc.recname = supc.recname) num_records
    FROM (SELECT DISTINCT ae_applid FROM psaeappltemptbl) sup
    , (SELECT DISTINCT ae_applid FROM psaeappltemptbl) sub
    , psaeappldefn supa
    , psaeappldefn suba
    WHERE sup.ae_applid < ae_applid =" sup.ae_applid" ae_applid =" sub.ae_applid" ae_applid =" sub.ae_applid" ae_applid =" sup.ae_applid" recname =" sub1.recname" rownum =" 1)" ae_applid =" sub.ae_applid" ae_applid =" sup.ae_applid" recname =" sup2.recname" rownum =" 1)" rownum =" 1)" ae_applid =" sup.ae_applid" ae_applid =" sub.ae_applid" recname =" sub2.recname" rownum =" 1)" rownum =" 1)">

    Here, three programs share the same set of temporary records.

    SUP_APPLID   SUP_INSTANCES SUB_APPLID   SUB_INSTANCES NUM_RECORDS
    ------------ ------------- ------------ ------------- -----------
    …
    ELEC_TSCRPT             20 E_TSCRPT_BAT            20           2
    ELEC_TSCRPT             20 E_TSCRPT_LIB            20           2
    E_TSCRPT_BAT            20 E_TSCRPT_LIB            20           2
    …

    I can use these view to set the instances on the subsets to 0 and increase the instances on the supersets as necessary. There are examples of both subsets within subsets and more than two Application Engine programs that share the same set of temporary tables. So I do this repeatedly until all the subsets have zero instances.

    This PL/SQL script makes the updates to the Application Engine programs (including maintaining PeopleSoft version numbers), and also creates an Application Designer project called GFC_TTI with all the programs and records. This project can then be used to migrate the Application Engine programs to another environment.

    DECLARE
    l_any BOOLEAN;
    l_projectname VARCHAR2(30 CHAR) := 'GFC_TTI';
    l_version_aem INTEGER;
    l_version_pjm INTEGER;
    
    PROCEDURE projitem(objecttype   INTEGER
    ,objectid1    INTEGER
    ,objectvalue1 VARCHAR2) IS
    BEGIN
    INSERT INTO psprojectitem
    (projectname ,objecttype
    ,objectid1 ,objectvalue1 ,objectid2 ,objectvalue2
    ,objectid3 ,objectvalue3 ,objectid4 ,objectvalue4
    ,nodetype ,sourcestatus ,targetstatus ,upgradeaction ,takeaction ,copydone)
    VALUES
    (l_projectname,objecttype
    ,objectid1, objectvalue1, 0, ' '
    , 0, ' ', 0, ' '
    ,0,0,0,0,1,0);
    EXCEPTION WHEN dup_val_on_index THEN NULL;
    END;
    
    BEGIN
    UPDATE psversion
    SET version = version+1
    WHERE objecttypename IN('SYS','AEM','PJM');
    
    UPDATE pslock
    SET version = version+1
    WHERE objecttypename IN('SYS','AEM','PJM');
    
    SELECT version
    INTO   l_version_aem
    FROM   psversion
    WHERE  objecttypename = 'AEM';
    
    SELECT version
    INTO   l_version_pjm
    FROM   psversion
    WHERE  objecttypename = 'PJM';
    
    l_any := TRUE;
    WHILE l_any LOOP
    l_any := FALSE;
    FOR i IN(
    SELECT *
    FROM gfc_aetemptbl_hier a
    WHERE a.sub_instances > 0
    AND NOT EXISTS(
    SELECT 'x'
    FROM   gfc_aetemptbl_hier b
    WHERE  b.sup_applid = a.sub_applid
    AND    b.sub_instances > 0
    AND    ROWNUM = 1)
    ORDER BY 1
    ) LOOP
    UPDATE psaeappldefn x
    SET    temptblinstances =     
    GREATEST(x.temptblinstances
    ,i.sub_instances,i.sup_instances)
    ,      version = l_version_aem
    ,      lastupddttm = SYSDATE
    WHERE  ae_applid = i.sup_applid; 
    
    projitem(33,66,i.sup_applid);
    
    UPDATE psaeappldefn x
    SET    temptblinstances = 0
    ,    version = l_version_aem
    ,      lastupddttm = SYSDATE
    WHERE  ae_applid = i.sub_applid;
    
    projitem(33,66,i.sub_applid);
    l_any := TRUE;
    END LOOP;
    END LOOP;
    
    l_any := TRUE;
    WHILE l_any LOOP
    l_any := FALSE;
    FOR i IN(
    SELECT *
    FROM  gfc_aetemptbl_eq a
    WHERE a.sub_instances > 0
    AND NOT EXISTS(
    SELECT 'x'
    FROM   gfc_aetemptbl_eq b
    WHERE  b.sup_applid = a.sub_applid
    AND    b.sub_instances > 0
    AND    ROWNUM = 1)
    ORDER BY 1
    ) LOOP
    UPDATE psaeappldefn x
    SET    temptblinstances =     
    GREATEST(x.temptblinstances
    ,i.sub_instances,i.sup_instances)
    ,      version = l_version_aem
    ,      lastupddttm = SYSDATE
    WHERE  ae_applid = i.sup_applid;
    
    projitem(33,66,i.sub_applid);
    
    UPDATE psaeappldefn x
    SET    temptblinstances = 0
    ,      version = l_version_aem
    ,      lastupddttm = SYSDATE
    WHERE  ae_applid = i.sub_applid;
    
    projitem(33,66,i.sub_applid);
    l_any := TRUE;
    END LOOP;
    END LOOP;
    END;
    
    INSERT INTO psprojectitem
    (projectname ,objecttype
    ,objectid1 ,objectvalue1 ,objectid2 ,objectvalue2
    ,objectid3 ,objectvalue3 ,objectid4 ,objectvalue4
    ,nodetype ,sourcestatus ,targetstatus ,upgradeaction ,takeaction ,copydone)
    SELECT DISTINCT
    l_projectname,0
    , 1, recname, 0, ' '
    , 0, ' ', 0, ' '
    , 0,0,0,0,1,0
    FROM   psaeappltemptbl t
    ,      psprojectitem i
    WHERE  i.projectname  = l_projectname
    AND    i.objecttype   = 33
    AND    i.objectid1    = 66
    AND i.objectvalue1 = t.ae_applid
    AND NOT EXISTS(
    SELECT 'x'
    FROM   psprojectitem i1
    WHERE  i1.projectname = l_projectname
    AND    i1.objecttype  = 0
    AND    i1.objectid1   = 1
    AND    i1.objectvalue1 = t.recname
    );
    
    BEGIN
    INSERT INTO psprojectdefn
    (projectname,version,projectdescr,tgtservername,tgtdbname
    ,tgtoprid,tgtopracct,comprelease,srccompreldttm,tgtcompreldttm
    ,compreldttm,keeptgt,tgtorientation,comparetype,commitlimit
    ,reportfilter,maintproj,lastupddttm,lastupdoprid,releaselabel
    ,releasedttm,objectownerid,descrlong)
    VALUES
    (l_projectname,l_version_pjm,'Temporary Table Instances',' ',' '
    ,' ',' ',' ',NULL,NULL
    ,NULL,31,0,1,50
    ,16232832,0,SYSDATE,'PS',' '
    ,NULL,' ','Application Engine programs, and related Temporary Records, '
    ||'whose number of temporary table instances have been changed');
    EXCEPTION WHEN dup_val_on_index THEN
    UPDATE psprojectdefn
    SET    version = (SELECT version FROM psversion
    WHERE  objecttypename = 'PJM')
    ,      lastupddttm = SYSDATE
    WHERE  projectname = l_projectname;
    END;
    END;


    Conclusion

    The effect on my demo HCM8.9 system was to reduce the total number of temporary table instances from 5942 to 5106, a 14% reduction. However, when I tried this on an HCM9.0 system, I got a reduction of only 7%. This shows that PeopleSoft has been more careful about specifying the number of temporary tables in the later version.

    Then you can use the script in an earlier posting to remove the excess tables.