Tuesday, March 18, 2025

Locally Partitioned Unique Indexes on Reference Partitioned Tables

Normally, if you want to locally partition a unique index, you must include the partitioning key in the index key. Otherwise, you get will error ORA-14039: partitioning columns must form a subset of key columns of a UNIQUE index
CREATE TABLE PS_JRNL_HEADER 
(BUSINESS_UNIT VARCHAR2(5 CHAR) NOT NULL
,JOURNAL_ID VARCHAR2(10 CHAR) NOT NULL
,JOURNAL_DATE DATE NOT NULL
,UNPOST_SEQ NUMBER NOT NULL
…
) 
PARTITION BY RANGE (fiscal_year) INTERVAL (1)
(PARTITION FISCAL_YEAR_2016 VALUES LESS THAN (2017))
…
/

CREATE UNIQUE INDEX PS_JRNL_HEADER 
ON PS_JRNL_HEADER (BUSINESS_UNIT, JOURNAL_ID, JOURNAL_DATE, UNPOST_SEQ)
LOCAL
/

ORA-14039: partitioning columns must form a subset of key columns of a UNIQUE index
This rule also applies to indexes on reference partitioned tables but in a slightly different way. The unique key on the child table cannot contain the partitioning key because those columns are only on the parent table. However, it can still be locally partitioned if it includes the parent table's primary key. 
Here is an example taken from PeopleSoft General Ledger. We can't add foreign keys to the PeopleSoft database, but we can add them to an archive database to support queries of archived data.
CREATE TABLE PS_JRNL_HEADER 
(BUSINESS_UNIT VARCHAR2(5 CHAR) NOT NULL
,JOURNAL_ID VARCHAR2(10 CHAR) NOT NULL
,JOURNAL_DATE DATE NOT NULL
,UNPOST_SEQ NUMBER NOT NULL
…
,CONSTRAINT PS_JRNL_HEADER PRIMARY KEY (BUSINESS_UNIT, JOURNAL_ID, JOURNAL_DATE, UNPOST_SEQ)
) 
PARTITION BY RANGE (fiscal_year) INTERVAL (1)
SUBPARTITION BY RANGE (accounting_period) 
SUBPARTITION TEMPLATE 
(SUBPARTITION accounting_period_00 VALUES LESS THAN (1)
…
,SUBPARTITION accounting_period_12 VALUES LESS THAN (13)
,SUBPARTITION accounting_period_max VALUES LESS THAN (maxvalue)
)
(PARTITION FISCAL_YEAR_2016 VALUES LESS THAN (2017))
COMPRESS FOR QUERY LOW 
/

CREATE TABLE PS_JRNL_LN 
(BUSINESS_UNIT VARCHAR2(5 CHAR) NOT NULL
,JOURNAL_ID VARCHAR2(10 CHAR) NOT NULL
,JOURNAL_DATE DATE NOT NULL
,UNPOST_SEQ NUMBER NOT NULL 
,JOURNAL_LINE NUMBER(9,0) NOT NULL
,LEDGER VARCHAR2(10 CHAR) NOT NULL
…
,CONSTRAINT PS_JRNL_LN PRIMARY KEY (BUSINESS_UNIT, JOURNAL_ID, JOURNAL_DATE, UNPOST_SEQ, JOURNAL_LINE, LEDGER)
,CONSTRAINT PS_JRNL_LN_FK FOREIGN KEY (BUSINESS_UNIT, JOURNAL_ID, JOURNAL_DATE, UNPOST_SEQ) REFERENCES PS_JRNL_HEADER 
)
PARTITION BY REFERENCE(PS_JRNL_LN_FK)
COMPRESS FOR ARCHIVE LOW 
/
If I try to locally partition a unique index without one of the parent table's unique key columns, I get ORA-14039, which is exactly as I might expect.
CREATE UNIQUE INDEX PS_JRNL_LN2 
ON PS_JRNL_LN (BUSINESS_UNIT, JOURNAL_ID, JOURNAL_DATE, /*UNPOST_SEQ,*/ JOURNAL_LINE, LEDGER) 
LOCAL COMPRESS 3
/

ORA-14039: partitioning columns must form a subset of key columns of a UNIQUE index
14039. 00000 -  "partitioning columns must form a subset of key columns of a UNIQUE index"
*Cause:    User attempted to create a UNIQUE partitioned index whose
           partitioning columns do not form a subset of its key columns
           which is illegal
*Action:   If the user, indeed, desired to create an index whose
           partitioning columns do not form a subset of its key columns,
           it must be created as non-UNIQUE; otherwise, correct the
           list of key and/or partitioning columns to ensure that the index'
           partitioning columns form a subset of its key columns

What is going on here? 

  • JRNL_HEADER is partitioned on FISCAL_YEAR and sub-partitioned on ACCOUNTING_PERIOD. 
  • JRNL_LN is reference-partitioned. Reference partitioning requires an enforced foreign key constraint. JRNL_LN has a foreign key constraint that references JRNL_HEADER. Thus, there is a 1:1 relationship of partitions on JRNL_LN (the child table), to partitions or in this case, sub-partitions on JRNL_HEADER (the parent table).
SELECT table_name, composite, partition_name, subpartition_count, partition_position, high_value
FROM   user_tab_partitions
WHERE  table_name LIKE 'PS_JRNL%' ORDER BY 1 DESC
/

                                                                 SubP Part
TABLE_NAME         COM PARTITION_NAME                           Count  Pos HIGH_VALUE
------------------ --- ---------------------------------------- ----- ---- --------------------
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_00        0    1
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_01        0    2
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_02        0    3
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_03        0    4
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_04        0    5
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_05        0    6
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_06        0    7
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_07        0    8
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_08        0    9
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_09        0   10
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_10        0   11
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_11        0   12
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_12        0   13
PS_JRNL_LN         NO  FISCAL_YEAR_2016_ACCOUNTING_PERIOD_MAX       0   14
PS_JRNL_HEADER     YES FISCAL_YEAR_2016                            14    1 2017

15 rows selected.
The partitioning keys recorded for the reference partitioned table JRNL_LN are the primary key columns on the parent table, although it is partitioned and sub-partitioned by different columns (FISCAL_YEAR and ACCOUNTING_PERIOD).
SELECT * FROM user_part_key_columns WHERE name LIKE 'PS_JRNL%' ORDER BY 1,2 desc, 4
/

NAME                 OBJEC COLUMN_NAME          COLUMN_POSITION COLLATED_COLUMN_ID
-------------------- ----- -------------------- --------------- ------------------
PS_JRNL_HEADER       TABLE FISCAL_YEAR                        1                   
PS_JRNL_LN           TABLE BUSINESS_UNIT                      1                   
PS_JRNL_LN           TABLE JOURNAL_ID                         2                   
PS_JRNL_LN           TABLE JOURNAL_DATE                       3                   
PS_JRNL_LN           TABLE UNPOST_SEQ                         4
The parent table in a foreign key relationship must have a primary key, and the child table must reference it. The primary key of the parent table is a proxy for the partitioning key. The two tables effectively share the partitioning key through the 1:1 relationship of partitions. Each primary key on the parent table can only appear in a single sub-partition, and therefore, corresponding child rows can only appear in the corresponding partition in the child table. Therefore, the uniqueness of the locally partitioned index on the child table can be assured by inspecting just the local partition. 

Natural -v- Surrogate Keys

The example above uses natural keys. The key on the child table leads with the key columns of the parent table, followed by one or more additional key columns.  Thus it is possible to locally partition the primary or unique key index on the child table.  
However, if a data model uses surrogate keys then the key on the child table is a completely different column to the key on the parent table, and it would not be possible to locally partition an index unless it also contained the parent surrogate key, in which case it could not be used to enforce uniqueness.

TL;DR

It appears that the rule that partitioning columns must form a subset of key columns of a unique index has been relaxed. It hasn't really. Instead, the reference partition uses the primary key on the parent of the foreign key as a proxy for the partitioning key.

Tuesday, March 04, 2025

SQL Quarantine Behaviour When the Same SQL Executes in Different Resource Manager Consumer Groups With Different CPU Time Limits

What happens if I have two consumer groups with different CPU time limits, but the same long-running SQLs can be run in either group?

There is a use case for this question.  In my sample resource manager plan for PeopleSoft, there are consumer groups with different maximum CPU time limits.  4 hours for scheduled queries, and 2 hours for SQL*Plus/SQL Developer sessions.  

The tests in this article are based on the examples in Tim Hall's article about SQL Quarantine in 19c.

N.B. SQL Quarantine is an Exadata-only feature.

I have created a simple plan with two consumer groups that have CPU time-outs of 30 and 60 seconds.  There are no group mappings; I will switch groups manually.

BEGIN 
  DBMS_RESOURCE_MANAGER.CLEAR_PENDING_AREA;
  DBMS_RESOURCE_MANAGER.CREATE_PENDING_AREA;

  DBMS_RESOURCE_MANAGER.CREATE_PLAN('GFC_TEST_PLAN');
  DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP('LOW_LIMITED30_GROUP');
  DBMS_RESOURCE_MANAGER.CREATE_CONSUMER_GROUP('LOW_LIMITED60_GROUP');

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE(
    'GFC_TEST_PLAN', 'LOW_LIMITED30_GROUP',
    mgmt_p8 => 1,
    switch_group => 'CANCEL_SQL', 
    switch_time => 30, 
    switch_for_call => TRUE);

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE(
    'GFC_TEST_PLAN', 'LOW_LIMITED60_GROUP', 
    mgmt_p8 => 1,
    switch_group => 'CANCEL_SQL', 
    switch_time => 60, 
    switch_for_call => TRUE);

  DBMS_RESOURCE_MANAGER.CREATE_PLAN_DIRECTIVE(
    'GFC_TEST_PLAN', 'OTHER_GROUPS');

  DBMS_RESOURCE_MANAGER.VALIDATE_PENDING_AREA;
  DBMS_RESOURCE_MANAGER.SUBMIT_PENDING_AREA;

  DBMS_RESOURCE_MANAGER_PRIVS.grant_switch_consumer_group(
    grantee_name   => user,
    consumer_group => 'LOW_LIMITED30_GROUP',
    grant_option   => FALSE);

  DBMS_RESOURCE_MANAGER_PRIVS.grant_switch_consumer_group(
    grantee_name   => user,
    consumer_group => 'LOW_LIMITED60_GROUP',
    grant_option   => FALSE);
END;
/

ALTER SYSTEM SET resource_manager_plan = 'GFC_TEST_PLAN';
I am going to create a PL/SQL function that will burn CPU as it goes around a loop.  It is just like Tim's, except that I specify the number of seconds for which I want mine to run, and it will return the number of times it has looped.
set serveroutput on timi on
CREATE OR REPLACE FUNCTION burn_cpu (p_secs IN NUMBER)
  RETURN NUMBER
AS
  l_start_time DATE;
  l_number     NUMBER := 1;
BEGIN
  l_start_time := SYSDATE;
  LOOP
    EXIT WHEN SYSDATE - l_start_time > (p_secs/86400);
    l_number := l_number + 1;
  END LOOP;
  RETURN l_number;
END;
/
I will start by switching to the consumer group LOW_LIMITED30_GROUP
DECLARE
  l_session v$session%ROWTYPE;
  l_consumer_group VARCHAR2(30):= 'LOW_LIMITED30_GROUP';
--l_consumer_group VARCHAR2(30):= 'LOW_LIMITED60_GROUP';
BEGIN 
  SELECT * INTO l_session FROM v$session WHERE sid = Sys_Context('USERENV', 'SID');
  dbms_output.put_line(l_session.sid||'.'||l_session.serial#||':'||l_session.module||':'||l_session.action||':'||l_session.resource_Consumer_Group);
  DBMS_RESOURCE_MANAGER.SWITCH_CONSUMER_GROUP_FOR_SESS(l_session.sid,l_session.serial#,consumer_group=>l_consumer_group);
  SELECT * INTO l_session FROM v$session WHERE sid = Sys_Context('USERENV', 'SID');
  dbms_output.put_line(l_session.sid||'.'||l_session.serial#||':'||l_session.module||':'||l_session.action||':'||l_session.resource_Consumer_Group);
END;
/
I have patch 30104721 installed on 19c to backport the new parameters in 23c, so I need to enable quarantine capture (see Oracle Doc ID 2635030.1: 19c New Feature SQL Quarantine - How To Stop Automatic SQL Quarantine).
ALTER SESSION SET OPTIMIZER_CAPTURE_SQL_QUARANTINE = TRUE;
ALTER SESSION SET OPTIMIZER_USE_SQL_QUARANTINE = TRUE;
I can run queries that usually run for 45 and 75 seconds, but they will be stopped when they have consumed 30 seconds of CPU time.
SELECT burn_cpu (45) FROM dual
Error report -
ORA-12777: A non-continuable error encountered.  Check the error stack for additional information [40], [pfrruncae_check_async_events], [], [].
ORA-00040: active time limit exceeded - call aborted

Elapsed: 00:00:33.574
SELECT burn_cpu (75) FROM dual
Error report -
ORA-12777: A non-continuable error encountered.  Check the error stack for additional information [40], [pfrruncae_check_async_events], [], [].
ORA-00040: active time limit exceeded - call aborted

Elapsed: 00:00:34.023
After a short period, I have 2 quarantine directives

select signature, name, sql_text, plan_hash_value, cpu_time, origin
 from dba_sql_quarantine
where sql_text like '%burn_cpu%'


             SIGNATURE NAME                                     SQL_TEXT                       PLAN_HASH_VALUE CPU_TIME   ORIGIN           
---------------------- ---------------------------------------- ------------------------------ --------------- ---------- ----------------
  12281544607895693562 SQL_QUARANTINE_anw6gs0gbta7u125daea2     SELECT burn_cpu (75) FROM dual       308129442 30         RESOURCE-MANAGER
   3516266497211383477 SQL_QUARANTINE_31m29rz6y8gpp125daea2     SELECT burn_cpu (45) FROM dual       308129442 30         RESOURCE-MANAGER
Now, the statements are prevented from running immediately.

SELECT burn_cpu (45) FROM dual
Error report -
ORA-56955: quarantined plan used

Elapsed: 00:00:00.414
SELECT burn_cpu (75) FROM dual
Error report -
ORA-56955: quarantined plan used

Elapsed: 00:00:00.416

Next, I will switch the consumer group to LOW_LIMITED60_GROUP.  The CPU limit is now 60 seconds.  Now, neither command returns ORA-56955.

The 45-second process runs to a successful conclusion.

SELECT burn_cpu (45) FROM dual;

BURN_CPU(45)
------------
    63264226

Elapsed: 00:00:45.846

The 75-second process runs for 65 seconds and is then terminated with ORA-00040 because it has reached the time limit, but not because the execution plan was quarantined.

SELECT burn_cpu (75) FROM dual
Error report -
ORA-12777: A non-continuable error encountered.  Check the error stack for additional information [40], [pfrruncae_check_async_events], [], [].
ORA-00040: active time limit exceeded - call aborted

Elapsed: 00:01:05.952

A quarantine directive with a lower CPU_TIME limit than that of the current consumer group is not applied because the statement may run to completion in a time between the lower limit and the higher limit.  Oracle allows the query to execute; it will be aborted when it reaches the higher CPU time.  

After a while, as the documentation indicates, quarantine capture will update the CPU time limit on the existing quarantine definition limit to the higher limit in the current consumer group.

The query in our example runs for 20 minutes only once, and then never again—unless the resource limit increases or the plan changes. If the limit is increased to 25 minutes, then the Resource Manager permits the statement to run again with the quarantined plan. If the statement runs for 23 minutes, which is below the new threshold, then the Resource Manager removes the plan from quarantine. If the statement runs for 26 minutes, which is above the new threshold, the plan remains in quarantine unless the limit is increased.

However, for the statement that runs successfully (for 45 seconds) in the consumer group with the higher limit (60 seconds), I have not seen the database remove the quarantine directive, even if I remove LOW_LIMITED30_GROUP consumer group from all resource plans and drop it from the database.

SIGNATURE           NAME                                     SQL_TEXT                       PLAN_HASH_VALUE CPU_TIME   ORIGIN          
------------------- ---------------------------------------- ------------------------------ --------------- ---------- ----------------
  12281544607895693562 SQL_QUARANTINE_anw6gs0gbta7u125daea2  SELECT burn_cpu (75) FROM dual       308129442 60         RESOURCE-MANAGER
   3516266497211383477 SQL_QUARANTINE_31m29rz6y8gpp125daea2  SELECT burn_cpu (45) FROM dual       308129442 30         RESOURCE-MANAGER

The next time an attempt is made to execute the 75-second statement in either consumer group, it is quarantined as before.

SELECT burn_cpu (75) FROM dual
Error report -
ORA-56955: quarantined plan used

Conclusion

  • SQL Quarantines are not tied to any particular resource plan or consumer group.
  • If the current consumer group doesn't have any timeout set, then any matching SQL quarantine directive is not applied
  • If a statement has a quarantine directive with a higher CPU time limit than the current consumer group, then it is applied immediately.
  • If a statement has a quarantine directive with a lower CPU time limit than the current consumer group, then it will be allowed to execute.  If the runtime then exceeds the consumer group CPU limit, then it will be timed out with ORA-00040.  The CPU time limit on the quarantine directive will be increased to the limit in the current consumer group.  In future, the statement will immediately be prevented from executing with ORA-56955 in both consumer groups.

Therefore, it is safe to allow different consumer groups in which the same long-running SQL statements may be executed to have different timeouts.  The lower quarantine timeout will not apply to executions in consumer groups with higher timeouts.  The SQL quarantine directives will evolve to have higher timeouts as required.

Monday, March 03, 2025

New Parameters In 21c To Control Automatic SQL Quarantine Can Be Backported To 19c

SQL Quarantine is only available on Exadata.  In 19c, automatic quarantine generation and subsequent use are enabled by default.  

In Oracle 21c, two new parameters have been introduced to control SQL Quarantine.

  • OPTIMIZER_CAPTURE_SQL_QUARANTINE enables or disables the automatic creation of SQL Quarantine configurations.  The default value is FALSE.  If enabled, when the Resource Manager terminates a SQL statement because the statement has exceeded resource limits, the database automatically creates a SQL Quarantine configuration for the execution plan used by the terminated SQL statement. 
  • OPTIMIZER_USE_SQL_QUARANTINE determines whether the optimizer considers SQL Quarantine configurations when choosing an execution plan for a SQL statement.  The default value is TRUE.

Thus, these parameters allow a system to use the SQL quarantine functionality but disable the automatic creation of quarantine configuration or restrict it to particular sessions.

This is a change in default behaviour, or at least a change to the original behaviour in 19c.  Quarantine directives will not be generated by default, but if they exist, the optimiser will apply them.

The parameters can be backported to Oracle 19.3 or later by applying patch 30104721 (see Oracle Doc ID 2635030.1: 19c New Feature SQL Quarantine - How To Stop Automatic SQL Quarantine.

These parameters can both be set at system and session level.

If SQL quarantine configurations are not created, or set not to be used, or if you are not on Exadata, then SQLs will run to the CPU limit in the current consumer group before they are cancelled.

Friday, November 29, 2024

SQL Developer Tip: Exporting SQL Results to Excel by Default

I frequently use SQL queries to extract performance data from Active Session History (ASH).  I often then want to graph that data.  Excel is a very effective charting tool, it is nearly always available, the charts can be copied into other applications, and it can be useful to keep historical data in spreadsheets.  

SQL Developer is very effective at extracting the results of a SQL query into various formats, including as an Excel workbook file.  

Tip: Make Excel the Default Export Format

  • From the main menu, go to Tools 🠊 Preferences  
  • In the preferences window, go to Database 🠊 Utilities 🠊 Export
  • Set Excel 2003+ (.xlsx) as the default export format, and a directory of your choice as the default location.  It is also useful to have a copy of the SQL query in the workbook.
  • When you have run your query and the first array of results has been fetched (50 rows by default), right-click in the query result grid and select 'Export'.
  • Then when the Export Wizard opens, the defaults will be as you set them in the preferences and you just have to click next, or hit return. The file name defaults to export…
  • I find it helpful to pin export.xlsx file in the recent file list in Excel.

Friday, September 13, 2024

Purging Standard and Unified Audit Data

Introduction

It is not uncommon for businesses to have policies to retain audit data for several years, but then it ought to follow that that data should be purged at the end of the retention period.  A regular purge process should be put in place, otherwise, audit data will accumulate.  However, it is all too common for the purge to be deferred, often indefinitely, until it becomes a problem.

I have been working on a system that has built up 12 years of audit data.  It was starting to consume significant space in the SYSAUX tablespace, and queries on the standard audit data had become slow.  We noticed that Oracle Enterprise Manager was regularly querying this data looking for recent failed logins.  This was performing so poorly that it had become a never-ending process.

This company has a stated requirement that audit data be retained for 7 years, but some can be purged after 2 years.  However, within the last 7 years, this production database has had different database IDs due to its upgrade history.  

Oracle delivers a PL/SQL package to manage audit purges.  However, this utility will only purge the current database ID, and it will only purge all of that data by a date.  It cannot selectively purge some data by other criteria.  Therefore, a custom purge utility that extends that package was required.  This post discusses that extension. 

We introduced

  • Regular weekly purge jobs for all forms of audit, for all DBIDs.  They run every weekend when the system is quieter.
  • This includes a custom purge job to delete some data after 2 years

General Recommendations

  • Standard audit is deprecated from Oracle 19c and will be desupported in a future release.  Therefore, Oracle recommends that customers migrate from standard audit to Unified Audit.  Oracle provides a Traditional to Unified Audit Syntax Converter - Generate Unified Audit Policies from Current Traditional Audit Configuration (Doc ID 2909718.1). Note, that this will only migrate the audit configuration.  The standard audit data that has built up will still need to be purged according to the retention policy until it is all gone.
  • Should any future database maintenance/upgrade activity cause the database ID to change, a new purge timestamp will be required for the new database ID, and an additional purge job will be required for the new legacy database ID.

Overview

Standard audit and Unified Audit (introduced in 12c) can be configured to monitor certain operations by database users.  Audit data is generally written to tables in the SYSAUX tablespace, but unified audit can be written to XML files outside the database.

  • Standard audit data is written to SYS.AUD$.  It is exposed via DBA_AUDIT_TRAIL.  Oracle does not support partitioning this table, and users who have tried have reported errors.
  • Unified audit data is written to AUDSYS.UNIFIED$AUD. It is exposed via UNIFIED_AUDIT_TRAIL. Unified Audit is enabled by default, so if you think you are only using standard audit, you are probably also writing some Unified Audit.  This table is interval range partitioned.  The default interval is calendar monthly, but can be altered.

Purging Audit

Audit Timestamps

Oracle's delivered audit management package DBMS_AUDIT_MGMT can purge various types of audit.  Each audit purge works on an individual database ID, the current database ID by default.  An audit purge timestamp is created for each type of audit, and for each DBID to be purged.  This is defined as a fixed date and not a retention period.  Audit data that is before the date is purged.  The audit timestamps must be updated before each purge to maintain the desired retention period.

The first step is to identify the DBIDs.  In my case, most of the data is standard audit.  I will simply query the number of rows and the range of dates for each DBID in the audit data.

select dbid, count(*), min(timestamp), max(timestamp)
from dba_audit_trail
group by dbid
order by 3
/

      DBID   COUNT(*) MIN(TIMESTAMP)      MAX(TIMESTAMP)
---------- ---------- ------------------- -------------------
1000000002   33606824 11/09/2012 00:06:42 20/08/2016 00:46:22
1000000003  241327475 20/08/2016 03:36:36 13/05/2023 04:47:20
1000000005   23320615 12/05/2023 23:54:18 06/01/2024 00:56:49
1000000006     134188 06/01/2024 01:05:29 04/03/2024 22:24:53
2000000008     282780 05/03/2024 03:39:39 02/05/2024 15:05:41

However, some unified audit is enabled by default, so both types of audit have to be purged.

select dbid, count(*), min(event_timestamp), max(event_timestamp)
from unified_audit_trail
group by dbid
order by 3
/

      DBID   COUNT(*) MIN(EVENT_TIMESTAMP MAX(EVENT_TIMESTAMP
---------- ---------- ------------------- -------------------
1000000004      28323 21/04/2023 10:09:39 21/04/2023 12:00:51
1000000005    1434181 12/05/2023 22:44:47 06/01/2024 00:56:49
1000000006    1700032 06/01/2024 01:05:29 04/03/2024 22:02:29
2000000008     263038 05/03/2024 03:12:27 07/05/2024 16:39:41

The following PL/SQL creates an audit timestamp for each database ID, for each type of audit.  The list of database IDs obtained above was then hard-coded in the following PL/SQL block.  Doing this dynamically can involve a long query on the audit data, so it is easier to hard code them.

If a database ID is not specified to DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP, a control record for the current database ID is created. The audit timestamp is set to 7 years ago because our policy is to purge any audit data older than 7 years.  Later, a database scheduler job will be created for each timestamp defined.

At the time of writing, it will be several years until the Unified Audit data is old enough to be purged, but if the purge jobs are not created now, nobody will remember to add them later!

declare
  k_standard_retention_years CONSTANT INTEGER NOT NULL := 7;
begin
  FOR i IN(
    with d (dbid) as (
          select TO_NUMBER(NULL) from dual
    union select 1000000001 from dual 
    union select 1000000002 from dual 
    union select 1000000003 from dual 
    union select 1000000004 from dual 
    union select 1000000005 from dual 
    union select 1000000006 from dual 
    )
    select dbid from d
  ) LOOP
    DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP
    (audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD
    ,last_archive_time => ADD_MONTHS(sysdate,-12*k_standard_retention_years)
    ,database_id => i.dbid
    );
  END LOOP;

  FOR i IN(
    with d (dbid) as (
          select TO_NUMBER(NULL) from dual
    union select 1000000004 from dual --2023 unified only
    union select 1000000005 from dual --2023-24 --current prod +unified
    union select 1000000006 from dual --2024 +unified
    --union select 2000000008 from dual --2024 +unified --current PRT
    )
    select dbid from d
  ) LOOP
    DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP
    (audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_UNIFIED
    ,last_archive_time => ADD_MONTHS(sysdate,-12*k_standard_retention_years)
    ,database_id => i.dbid
    );
  end loop;
end;
/

As old database IDs are completely purged their last archive timestamps can be removed from the audit purge configuration in the database with DBMS_AUDIT_MGMT.CLEAR_LAST_ARCHIVE_TIMESTAMP and the corresponding database jobs can be dropped.

Should any future database maintenance/migration/upgrade activity cause the database ID to change again, new timestamps must be set up for the new current database ID, and corresponding scheduler purge jobs must be created.

Custom Audit Purge Management Package

A custom package procedure XX_CUSTOM_AUDIT_PURGE has been created.  It is available on GitHub.  The package is deliberately created to be owned by the AUDSYS user that is used to run the default audit purge jobs.  That account is delivered by Oracle and is locked. It should not be unlocked.  Therefore, a SYSDBA privileged account must deploy the package.  

This package contains various procedures

  • INIT_AUDIT_PURGE
    • initialises the standard and fine-grained audit purges,
    • creates a job resource constraint so the scheduler only runs one purge process at a time, and 
    • creates a job class to assign the job to a resource plan consumer group so that the purge job runs at low priority.
  • UPDATE_AUDIT_PURGE_TS updates the audit purge timestamps to 7 years ago, thus maintaining a rolling 7-year purge.
  • CUSTOM_PURGE performs a limited purge of some audit data after 2 years.  Exactly which audit actions are to be purged is hard-coded in this procedure.
  • CREATE_AUDIT_PURGE_JOBS uses DBMS_AUDIT_MGMT.CREATE_PURGE_JOB to create the various scheduler jobs to run the Oracle-delivered purge procedure DBMS_AUDIT_MGMT.CLEAN_AUDIT_TRAIL for the various audits to be purged. 
    • However, this procedure alters the jobs to control their scheduler, concurrency and resource manager consumer group.
  • CREATE_CUSTOM_PURGE_JOB creates a scheduler job to execute the CUSTOM_PURGE procedure.
  • CREATE_UPDATE_AUDIT_PURGE_TS_JOB creates a job on the database scheduler to run UPDATE_AUDIT_PURGE_TS.

Some constants and defaults have been hard-coded in the package including

  • The 2-year and 7-year purge rules.  
  • Resource consumer group of the purge jobs
  • Job Class for purge jobs to which resource consumer group is assigned.
  • Number of rows to be deleted between commits during custom delete (2-year purge).

The audit purge requirements are unlikely to change, and if they do a code change is required to implement the change that must be installed by the core DBA.  I felt that this approach was easier and more secure than creating and controlling access to metadata.

After an initial large purge to clear the backlog, we find that these jobs only run for a few minutes each.  By scheduling them to run regularly, we maintain a relatively steady data volume.

Reclaiming Space after Initial Large Purge

The audit purge is no more than a simple delete.  Once deleted (and committed) the space freed up can be used by other rows of audit data, but space is not released from the audit tables back to the SYSAUX tablespace.

The only supported way to recover space from the purged audit tables is to rebuild them by moving them to a different tablespace, and if necessary, moving them back using the DBMS_AUDIT_MGMT.SET_AUDIT_TRAIL_LOC procedure again.

Create a tablespace SYSAUX_AUD.  It should be created as a BIGFILE tablespace as SYSAUX is also a BIGFILE tablespace.  Otherwise, in our case, it would have needed at least 3 data files (so it could expand to 96G).

The following PL/SQL block creates a database scheduler job for each call to this procedure so that it can run in the background.

  • This time the jobs are one-time execution jobs that will be auto-dropped on completion.
  • The same resource constraint XX_PURGE_RESOURCE, as used by the other purge jobs, ensures that it will not run concurrently with any of those jobs (nor vice versa).
  • If the job start date is not set, the job will run as soon as it is enabled and no other job using the same resource is running.

DECLARE
  TYPE t_aud_types IS TABLE OF VARCHAR2(8);
  a_aud_types t_aud_types;
  l_job_name VARCHAR2(30);
  l_tablespace_name VARCHAR2(30) := 'SYSAUX_AUD';
  e_job_does_not_exist EXCEPTION;
  PRAGMA EXCEPTION_INIT(e_job_does_not_exist, -27475);
BEGIN
  a_aud_types := t_aud_types('AUD_STD','UNIFIED') /*FGA_STD deliberately excluded*/;
  FOR i IN a_aud_types.FIRST..a_aud_types.LAST
  LOOP
    l_job_name := 'XX_AUDIT_TRAIL_LOC_'||a_aud_types(i);
    BEGIN
      dbms_scheduler.drop_job(job_name => l_job_name);
    EXCEPTION WHEN e_job_does_not_exist THEN NULL;
    END;
    dbms_scheduler.create_job
    (job_name => l_job_name
    ,job_type => 'PLSQL_BLOCK'
    ,job_action => 'BEGIN dbms_audit_mgmt.set_audit_trail_location(audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_'||a_aud_types(i)||',audit_trail_location_value => '''||l_tablespace_name||''');  END;'
    ,start_date => SYSTIMESTAMP at time zone ('US/Eastern') 
    ,job_class => 'XX_PURGE_CLASS'
    ,auto_drop => TRUE
    ,enabled => FALSE);
    DBMS_SCHEDULER.set_resource_constraint --Only one purge job can run at any one time
    (object_name   => l_job_name
    ,resource_name => 'AUDSYS.XX_PURGE_RESOURCE'
    ,units         => 1);     
    dbms_scheduler.enable(l_job_name);
  END LOOP;     
END;
/

  • NB: I specified the time zone for the database job because I am working in a different time zone to the database!

We have found that due to a bug, the unified audit table is not relocated to the new tablespace.  It is expected that this will be fixed in version 23ai. 

Validate/Reinstate Privileges

Using SET_AUDIT_TRAIL_LOCATION will strip off any explicitly granted privileges on the audit tables. Ensure privileges on the standard audit tables are intact.  Otherwise, the custom purge will fail.

column grantee format a8
column owner format a8
column table_name format a20
column privilege format a20
column grantor format a8
select * from dba_tab_privs
where (owner,table_name) IN(('SYS','FGA_LOG$'),('SYS','AUD$'),('AUDSYS','AUD$UNIFIED'))
/

Minimum expected output

GRANTEE  OWNER    TABLE_NAME           GRANTOR  PRIVILEGE            GRA HIE COM TYPE                     INH
-------- -------- -------------------- -------- -------------------- --- --- --- ------------------------ ---
AUDSYS   SYS      AUD$                 SYS      DELETE               NO  NO  NO  TABLE                    NO 
AUDSYS   SYS      AUD$                 SYS      SELECT               NO  NO  NO  TABLE                    NO

TL;DR

If you collect audit data, then you should create a retention policy and schedule purge jobs to delete it after the retention period expires.  It is not unreasonable to keep audit data for several years.  The database ID may change in that time due to upgrade or migration.  Purge processes are also required for each legacy DBID.

After an initial large purge reclaims space by relocating the Standard Audit table, but remember to reinstate any explicitly granted privileges.  

Unified Audit cannot be relocated until a bug is resolved in 23ai.  

The performance of purge and query of standard audit will improve after doing this.