Showing posts with label 26ai. Show all posts
Showing posts with label 26ai. Show all posts

Thursday, April 09, 2026

Consuming Inbound HTTP Requests with REST Services

This blog is part of a series about aspects and features of Oracle 26 and Autonomous Transaction Processing Database.

I have created a demo project that integrates Strava (an application that tracks athletes' activities – in my case, cycling) with an Oracle Autonomous database that then performs some spatial data processing and sends the results back to Strava.  
I have described how to call the Strava API in HTTP requests from the database to pull data from Strava or push it back.  To complete the integration, the application must receive and process inbound HTTP requests from Strava when activities are added, updated or deleted.  
The database has to handle two kinds of messages from Strava.  
  • I have to create a subscription in Strava to receive notifications of new activities.  Part of the authentication of that request includes responding promptly to an HTTP GET request.
  • Then, when I have created my subscription, I receive HTTP POST requests to tell me that an activity has been created, updated or deleted in Strava that I then process.
I have created PL/SQL packaged procedures and REST services for each of these requests.

ORDS URL

Oracle REST Data Services (ORDS) are configured by default for Autonomous Database.  You can find the public access URL for the database on the OCI console under Tool Configuration

Enabling REST

All my code exists within a database schema called STRAVA.  Access to ORDS must be granted to this schema.
ORDS.ENABLE_SCHEMA(
    p_enabled             => TRUE,
    p_schema              => 'STRAVA',
    p_url_mapping_type    => 'BASE_PATH',
    p_url_mapping_pattern => 'strava',
    p_auto_rest_auth      => FALSE
  );

Defining REST services

My application runs entirely within the database schema STRAVA. To set up my Strava subscription, I had to give my API Application a name in Strava - so I called it PlaceCloud.  Therefore, my REST services are in a module also called PlaceCloud
  ORDS.DEFINE_MDULE(
    p_module_name    => 'placecloud',
    p_base_path      => 'placecloud/'
  );
Finally, I have to create a template
  ORDS.DEFINE_TEMPLATE(
    p_module_name => 'placecloud',
    p_pattern     => 'event'
  );
Schema, Module and Template combine to define the path of the REST Service.  
  • The schema is STRAVA.
  • The module is placecloud.
  • The template pattern is event.
That is appended to the public access URL that I got from the OCI console above. Thus, the complete URL of my REST service is 
https://GE************9-GOFASTER1.adb.uk-london1.oraclecloudapps.com/ords/strava/placecloud/event.

When I request to create a Strava Webhook event subscription, Strava sends a callback to a URL I specify to validate the request.  It will be the URL of my REST service.  I can simulate the callback for testing with curl.

curl -i -S -X GET -H "Content-Type: application/json" "https://GE************9-GOFASTER1.adb.uk-london-1.oraclecloudapps
.com/ords/strava/placecloud/event?hub.verify_token=MyPlaceCloud&hub.challenge=abc123&hub.mode=subscribe"
NB: Double quotes, especially around HTTPS URLs, are important; otherwise, the & will be interpreted by the command line, and you will get an error from the REST service!

Get Handler

The REST service for GET calls an anonymous PL/SQL block that calls my packaged procedure, passing parameters from the query string.  A status code and JSON response are returned.
  ORDS.DEFINE_HANDLER(
    p_module_name => 'placecloud',
    p_pattern     => 'event',
    p_method      => 'GET',
    p_source_type => ORDS.SOURCE_TYPE_PLSQL,  
    p_source      => q'[
      DECLARE
        l_status_code NUMBER;
        l_response    VARCHAR2(200 CHAR);
        l_message     CLOB;
      BEGIN 
        strava.webhook_pkg.handle_get(:hub_challenge,:hub_verify_token, l_response, l_status_code, l_message); 
        owa_util.status_line(l_status_code, l_message, FALSE);
        owa_util.mime_header('application/json', FALSE);
        :status_code := l_status_code;
        owa_util.http_header_close;
        htp.p(l_response);
        :response := l_response;
      END;
      ]',
    p_mimes_allowed => 'application/json',
    p_items_per_page => 0
  );

Handling Parameters

Strava puts three parameters in the callback.  Their names have a dot (".") in them.  I have to map them to a bind variable in the PL/SQL block named without the dot.
  ORDS.DEFINE_PARAMETER(
    p_module_name        => 'placecloud',
    p_pattern            => 'event',
    p_method             => 'GET',
    p_name               => 'hub.challenge',
    p_bind_variable_name => 'hub_challenge',
    p_source_type        => 'URI',
    p_param_type         => 'STRING',
    p_access_method      => 'IN'
  );
  ORDS.DEFINE_PARAMETER(
    p_module_name        => 'placecloud',
    p_pattern            => 'event',
    p_method             => 'GET', 
    p_name               => 'hub.verify_token',
    p_bind_variable_name => 'hub_verify_token',
    p_source_type        => 'URI',
    p_param_type         => 'STRING',
    p_access_method      => 'IN'
  );
  ORDS.DEFINE_PARAMETER(
    p_module_name        => 'placecloud',
    p_pattern            => 'event',
    p_method             => 'GET',
    p_name               => 'hub.mode',
    p_bind_variable_name => 'hub_mode',
    p_source_type        => 'URI',
    p_param_type         => 'STRING',
    p_access_method      => 'IN'
  );

Post Handler

Having created the subscription, Strava sends an HTTP POST request to the same URL with a JSON body every time an activity is created, updated or deleted.
  ORDS.DEFINE_HANDLER(
    p_module_name => 'placecloud',
    p_pattern     => 'event',
    p_method      => 'POST',
    p_source_type => ORDS.SOURCE_TYPE_PLSQL,
    p_source      => q'[
      BEGIN strava.webhook_pkg.handle_post(:body_text,:status_code); END;
      ]'
  );
I simply pass the message body to my procedure and return a status code.  See Loading and Processing JSON with PL/SQL.  Again, I can test this with curl.  I can put a sample JSON document in a file,
{
"object_type":"activity",
"object_id":123,
"aspect_type":"create",
"owner_id":999,
"event_time":1700000000
}
and post the file with curl.
curl -i -S -X POST --data-ascii @C:\temp\resttest.json -H "Content-Type: application/json" \
"https://GE************9-GOFASTER1.adb.uk-london-1.oraclecloudapps.com/ords/strava/placecloud/event"

Further Reading

ThatJeffSmith has several very helpful posts: REST APIs for Oracle Database, everything you need to know.

Wednesday, April 08, 2026

Loading GeoJSON Format GeoSpatial Data into Oracle Autonomous Database

This blog is part of a series about aspects and features of Oracle 26 and Autonomous Transaction Processing Database.

As a training exercise, I have created a database application that loads GPS data from activities logged on Strava and compares them with geospatial data to identify the named places visited by the activity. This geospatial data is publicly available in many places, often from government, and is usually freely available, at least for non-commercial purposes.

ESRI Shapefiles

When I first wrote the spatial processing in 2020, I used spatial data formatted as ESRI shapefiles.  These were then converted to Oracle spatial geometries and loaded into a database using the oracle.spatial.util: Class SampleShapefileToJGeomFeature Java conversion.  
See also

It reads attributes from a dBase file and geometries from a Shapefile, and then writes them to a database table.  I was able to run it on the database server.

export clpath=$ORACLE_HOME/suptools/tfa/release/tfa_home/jlib/ojdbc5.jar:$ORACLE_HOME/md/jlib/sdoutl.jar:$ORACLE_HOME/md/jlib/sdoapi.jar
java -cp $clpath oracle.spatial.util.SampleShapefileToJGeomFeature -h <dbhost> -p 1521 -sn oracle_pdb -u strava -d strava \
     -t $table -f $base -r 4326 -g geom
However, this approach is not suitable for an autonomous database.  ADB cannot access the local file system, there is no host execution access, and it does not allow external libraries (JARs) required by the Shapefile utility.  It would have to be run on another host.  It only makes a simple JDBC connection and cannot handle the ADB wallet out of the box.

GeoJSON

I have found that many spatial data sets are now available as GeoJSON, often as well as a shapefile.  GeoJSON is much easier to handle, especially in the Autonomous database, because it can be processed entirely within the database.

Often, GeoJSON can be downloaded directly into a database with an HTTP call.  For example, the definitions of Areas of Outstanding Natural Beauty (AONBs) in England can be downloaded as GeoJSON file from the UK Government Planning Data website.  I can make the HTTP call to download the file directly from PL/SQL.  I usually load the JSON into a staging table before trying to convert it to a spatial geometry.  The HTTP request is made by a packaged function strava_http.http_request.
TRUNCATE TABLE strava.stage_geo_data;
DECLARE 
  l_url VARCHAR2(1000) := 'https://files.planning.data.gov.uk/dataset/area-of-outstanding-natural-beauty.geojson';
  l_clob CLOB;
BEGIN
  DBMS_LOB.createtemporary(l_clob, TRUE);
  l_clob:=strava_http.http_request(l_url,99);    
  INSERT INTO stage_geo_data (name, geo_json) VALUES ('Natural England', l_clob);
  DBMS_LOB.freetemporary(l_clob);
END;
/
SELECT x.*, length(geo_json) FROM stage_geo_data x;
However, sometimes the GeoJSON must be downloaded as a file, uploaded to OCI Bucket storage, and then read into the database from there.
DECLARE
  l_blob BLOB;
BEGIN
  -- Read the file from Object Storage into CLOB
  l_blob := DBMS_CLOUD.GET_OBJECT
            (credential_name => 'OBJECT_STORE_CRED'
            ,object_uri      => 'https://objectstorage.uk-london-1.oraclecloud.com/n/l**********a/b/bucket-gofaster1/o/ch0.json'
            );
  INSERT INTO strava.stage_geo_data(name, geo_json) VALUES ('Switzerland', l_blob);
  COMMIT;
END;
/

Converting Shapefiles to GeoJSON 

Some providers still prefer to make only Shapefiles available and not GeoJSON (for example, the Swiss Federal Office of Topography - swisstopo).  However, it is simple to convert the shapefiles to GeoJSON with the ogr2ogr utility available as a part of GDAL (a translator library for various geospatial data formats).  On Windows, I use OSGeo4W and run org2ogr within that.

Many countries have their own geoid and coordinate systems.  It can be more accurate over a limited region.

  • In the UK, the Ordnance Survey uses EPSG 27700 – British National Grid for Great Britain.
  • The Republic of Ireland and Northern Ireland both use EPSG 2157 - Irish Transverse Mercator
  • In Switzerland, the Federal Office of Topography uses EPSG 2056 (Swiss CH1903+ / LV95) and 5728 (LN02 Height)
  • Etc.
I can convert a shapefile to a GeoJSON, simultaneously changing the spatial reference identifier (SRID) to WGS84 (also known as EPSG 4326).
ogr2ogr -f GeoJSON swissBOUNDARIES3D_1_5_TLM_LANDESGEBIET.geojson swissBOUNDARIES3D_1_5_TLM_LANDESGEBIET.shp -t_srs EPSG:4326
Then I can proceed with the physical JSON file as previously described.

Converting GeoJSON to an Oracle Spatial Geometry

GeoJSON is just a JSON document, but it is structured in a particular way.  
  • It supports Point, LineString, Polygon, MultiPoint, MultiLineString, and MultiPolygon geometries. 
  • Geometric objects with additional properties are Feature objects. 
  • Sets of features are contained by FeatureCollection objects.
  • The specific fields in properties can vary.  I usually create a PL/SQL script specific to each GeoJSON to be loaded.
The sample below is taken from a GeoJSON of French departments.  The properties contain the department code, the name and the region number they are within.  There is no SRID; the data is already in WGS84.
{"type":"FeatureCollection"
,"features":
  [
    {"type":"Feature"
    ,"properties":
      {"code":"01","nom":"Ain","region":"84"}
    ,"geometry":
      {"type":"Polygon"
      ,"coordinates":[[[5.825,45.939],...,[5.825,45.939]]]
      }
    }
,   {"type":"Feature"
    ,"properties":
      {"code":"02","nom":"Aisne","region":"32"}
    ,"geometry":
      {"type":"Polygon"
      ,"coordinates":[[[3.987,49.379],...,[3.987,49.379]]]
      }
    }
…
  ]
}
This PL/SQL parses the GeoJSON, extracting the descriptive data.  The geometry data is converted with Oracle's SDO_UTIL.FROM_GEOJSON function.
DECLARE 
  l_clob        CLOB;
  j_root        JSON_OBJECT_T;
  j_crs         JSON_OBJECT_T;
  j_features    JSON_ARRAY_T;
  j_feature     JSON_OBJECT_T;
  j_properties  JSON_OBJECT_T;
  j_geometry    JSON_OBJECT_T;

  l_geom        MDSYS.SDO_GEOMETRY;

  l_id         INTEGER;
  l_srid       VARCHAR2(10 char);
  l_name       VARCHAR2(100 char);
  
  e_json_syntax_error  EXCEPTION;
  PRAGMA exception_init(e_json_syntax_error,-40441);
BEGIN 
  SELECT geo_json INTO l_clob FROM strava.stage_geo_data WHERE name = 'France Regions';
  l_clob := strava_http.clean_clob(l_clob);
  --strava_http.pretty_json(l_clob);
  j_root := JSON_OBJECT_T.parse(l_clob);

  j_crs := j_root.get_object('crs');
	  
  j_features := j_root.get_array('features');
  --l_srid     := REGEXP_SUBSTR(j_crs.get_object('properties').get_string('name'),'[^:]+',1,2);
  l_srid := 4326;
  FOR i IN 0 .. j_features.get_size - 1 LOOP
    j_feature := TREAT(j_features.get(i) AS JSON_OBJECT_T);
    j_properties := j_feature.get_object('properties');
    IF j_properties IS NULL THEN 
      dbms_output.put_line('J_PROPERTIES is null');
      l_id := i;
    ELSE
      l_clob := j_properties.to_clob;
      strava_http.print_clob(l_clob); --use this to understand new GeoJSON file properties, but comment it out later
      l_id               := j_properties.get_number('code');
      l_name             := j_properties.get_string('nom');
      dbms_output.put_line(l_id||', '||l_srid||', '||l_name);
    END IF;
	
    j_geometry := j_feature.get_object('geometry');
    IF j_geometry IS NULL THEN
      dbms_output.put_line(l_name||': j_geometry is null');
      l_geom := NULL;
    ELSE
      l_clob := j_geometry.to_clob /*Get coordinates array*/; 
      
      BEGIN
        l_geom := sdo_util.from_geojson(l_clob);
      EXCEPTION 
        WHEN e_json_syntax_error THEN
          dbms_output.put_line(sqlerrm||' during parse of '||l_name||' ('||l_id||'). Switch to own function.' );
          l_geom := strava_sdo.build_sdo_geometry_from_geojson(j_geometry, l_srid);
      END;

      l_geom.SDO_SRID := TO_NUMBER(l_srid);
      IF l_srid != 4326 THEN
        l_geom := SDO_CS.TRANSFORM(l_geom, 4326);
        l_geom.SDO_SRID := 4326;
      END IF;
      l_geom := sdo_util.rectify_Geometry(SDO_CS.MAKE_2D(l_geom),0.001);
    END IF;
    
    dbms_output.put_line(i||','||l_id||','||l_name);
    INSERT INTO stage_my_areas (area_code, area_number, name, geom)
    VALUES ('REG', l_id, l_name, l_geom);
  END LOOP;
  COMMIT;
END;
/
The resulting spatial geometry is written to a staging table.  From there, I can move it to where I finally want it.

Working Around Errors in SDO_UTIL.FROM_GEOJSON

The SDO_UTIL.FROM_GEOJSON function converts a GeoJSON object (or more specifically, a geometry object in GeoJSON format) to a Spatial geometry object.  It was introduced in Oracle 12.2.  However, even in Oracle 26, I have occasionally experienced errors with this function with certain data sets. 
I am working with publicly available data sets.  Therefore, I will not easily get any problem resolved, even if I could identify the exact cause.  My workaround has been to use my own function  (build_sdo_geometry_from_geojson).  It loads the array of coordinates from the JSON and passes it to Oracle's sdo_geometry constructor to create the geometry. Though I have found that my function always succeeds when from_geojson fails, my function is much slower.  Therefore, I only use it in an exception handler when I get an error from Oracle's function.

For example, I experienced ORA-40441 when loading data for Co. Carlow from the Irish government's Open Data Unit (I don't know why), but not for any of the other 25 counties.

REM ireland_counties_load.sql 
…
DECLARE 
…
  e_json_syntax_error  EXCEPTION;
  PRAGMA exception_init(e_json_syntax_error,-40441);
BEGIN 
…
  BEGIN
    l_clob := j_geometry.to_clob /*Get coordinates array*/; 
…
    l_geom := sdo_util.from_geojson(l_clob);
  EXCEPTION 
    WHEN e_json_syntax_error THEN
      dbms_output.put_line(sqlerrm||' during parse of '||l_name||' ('||l_id||'). Switch to own function.');
      l_geom := strava_sdo.build_sdo_geometry_from_geojson(j_geometry, l_srid);
  END;
…
END;
/
In the log below, you can see that the error was reported, but then the alternative function was successful.
{"OBJECTID":16,"CO_ID":"10000","ENGLISH":"CARLOW","GAEILGE":"Ceatharlach","LOGAINM_ID":"100004","GUID":"2ae19629-143d-13a3-e055-000000000001"
,"CONTAE":"Ceatharlach","COUNTY":"CARLOW","PROVINCE":"Leinster","CENTROID_X":680448.23,"CENTROID_Y":660624.58,"AREA":896306186.01}
10000, 2157, Carlow, Co. Carlow, Province:Leinster, Centre:-6.80998032929769,52.690783228944
ORA-40441: JSON syntax error during parse of Carlow. Switch to own function.
ID:10000, Province:Leinster, County:Carlow, Council:Carlow, Centre:-6.82336146940769,52.7293094021289, 14445 points

Thursday, March 26, 2026

Loading and Processing JSON with PL/SQL

This blog is part of a series about aspects and features of Oracle 26 and Autonomous Transaction Processing Database.

"JSON (JavaScript Object Notation) is a text-based format for storing and exchanging data in a way that’s both human-readable and machine-parsable. … it has grown into a very capable data format that simplifies data interchange across diverse platforms and programming languages."

I have created a demo project that integrates Strava (an application that tracks athletes' activities – in my case, cycling) with an Oracle Autonomous Database, that then performs some spatial data processing.  The Strava APIs all return data in JSON.  My application, written in PL/SQL, reads and processes those messages.

Mostly, I want to hold that data in a regular database table, structured conventionally in reasonably named columns.  In different places, I have loaded that JSON data in different ways, depending on requirements and circumstances.  There are three options to choose from:
  1. Directly through a JSON Duality View
  2. Convert each name-value pair in explicit code
  3. Extract Values from JSON in Virtual Columns

Directly through a JSON Duality View

Oracle introduced JSON Relational Duality in Oracle 23ai.  A JSON duality view is a mapping between table data and JSON documents.  It is possible to extract data from a table as a JSON document simply by querying a duality view based on that table.  It is also possible to insert data into the table through duality views.

For example, Strava tracks my gear (the bike I ride, or the shoes I wear).  I can extract details of each item with the Strava API, and I get a simple JSON document in return.  This is what I get for one of my bikes.
{
  "id" : "b4922223",
  "primary" : false,
  "name" : "Saracen",
  "nickname" : "Saracen",
  "resource_state" : 3,
  "retired" : false,
  "distance" : 1321925,
  "converted_distance" : 1321.9,
  "brand_name" : null,
  "model_name" : null,
  "frame_type" : 3,
  "description" : "",
  "weight" : 14
}
I want to import that into a table in my database that corresponds to that document.
CREATE TABLE gear 
(gear_id            VARCHAR2(20) NOT NULL
,primary            BOOLEAN
,name               VARCHAR2(60) 
,nickname           VARCHAR2(60) 
,resource_state     INTEGER
,retired            BOOLEAN
,distance_m         INTEGER      
,distance_km        NUMBER       
,brand_name         VARCHAR2(60) 
,model_name         VARCHAR2(60)
--frame_type
,description        CLOB
,weight             NUMBER       
,last_updated       TIMESTAMP DEFAULT SYSTIMESTAMP
,CONSTRAINT gear_pk PRIMARY KEY(gear_id)
);
I can use a JSON duality view to make the JSON document correspond to my table structure, rather than code it explicitly.
My table uses the column GEAR_ID as the primary key, but the Strava JSON document just has an ID. A JSON duality view must have a key value called '_id'.  I will map that to the primary key column in this table.  
All the other name-value pairs each map to the corresponding column in the table.  However, I am not bothering to import 'frame type'
CREATE OR REPLACE JSON RELATIONAL DUALITY VIEW gear_dv AS
SELECT JSON {'_id'    : g.gear_id
,'primary'            : g.primary
,'name'               : g.name
,'nickname'           : g.nickname
,'resource_state'     : g.resource_state
,'retired'            : g.retired
,'distance'           : g.distance_m
,'converted_distance' : g.distance_km
,'brand_name'         : g.brand_name
,'model_name'         : g.model_name
--frame_type
,'description'        : g.description
,'weight'             : g.weight
}
FROM gear g
WITH INSERT UPDATE
/
The JSON document arrives as an HTTP response and is held in a CLOB variable.  That has to be parsed into a JSON object with JSON_OBJECT_T.PARSE(). Then I either update an existing record or insert a new one.  In either case, that is done via the duality view.  Note that
  • The Strava id name is updated to _id to match the duality view, and it has to be _id.  
  • I have removed the frame_type and notification_distance name-value pairs.
  j_obj := JSON_OBJECT_T.parse(l_clob);
  l_id  := j_obj.get_string('id');

  BEGIN
    SELECT * INTO r_gear FROM gear WHERE gear_id = p_gear_id FOR UPDATE;
  EXCEPTION
    WHEN no_data_found THEN null;
  END;
  
  IF r_gear.gear_id = p_gear_id THEN
    UPDATE gear_dv d
    SET    d.data = JSON_TRANSFORM
           (value
           ,RENAME '$.id' = '_id'
           ,REMOVE '$.frame_type'
           ,REMOVE '$.notification_distance'
           )
    FROM JSON_TABLE(
           l_clob,
           '$[*]'
           COLUMNS (
             value CLOB FORMAT JSON PATH '$'
           ))
    WHERE d.data."_id" = l_id;
  ELSE
    INSERT INTO gear_dv
    SELECT JSON_TRANSFORM
           (value
           ,RENAME '$.id' = '_id'
           ,REMOVE '$.frame_type'
           ,REMOVE '$.notification_distance'
           )
    FROM JSON_TABLE(
           l_clob,
           '$[*]'
           COLUMNS (
             value CLOB FORMAT JSON PATH '$'
           )
     );
  END IF;
This approach works well where the JSON document structure closely matches the database table structure, and where I don't have to convert the inbound data with any function.  
However, if, for example, the name of the gear had to be upper case, I might put that into the definition of the duality view, thus
CREATE OR REPLACE JSON RELATIONAL DUALITY VIEW gear_dv AS
SELECT JSON {'_id'    : g.gear_id
,'name'               : UPPER(g.name)
…
}
FROM gear g WITH INSERT UPDATE
/
But then I would not be able to insert name via the duality view.  I wouldn't get an error, but it simply wouldn't process the name column.   It would be null after the insert, or would not be updated.
While the duality view is a very elegant way to map the data, it has limitations.  As soon as you need to transform data during the import, you probably have to go back to coding the mapping for each name-value pair.

See also 

Convert each name-value pair

The more conventional approach is to copy each name-value pair to a column using one of the get functions, sometimes passing the value through a function, and possibly with logic to determine whether to copy the data.
Here, I have used a row type variable and selected the whole current row from the database before updating data values 
BEGIN
    SELECT * INTO r_activities 
    FROM   activities 
    WHERE  activity_id = p_activity_id 
    FOR UPDATE;
  EXCEPTION
    WHEN no_data_found THEN r_activities.activity_id := p_activity_id;
  END;
…
  j_obj := JSON_OBJECT_T.parse(l_clob);

  r_activities.activity_id       := j_obj.get_number('id');
  r_activities.athlete_id        := j_obj.get_object('athlete').get_number('id');
  r_activities.start_date_utc    := iso8601_utc(j_obj.get_string('start_date'));
  r_activities.start_date_local  := iso8601_tz(j_obj.get_string('start_date_local'), j_obj.get_string('timezone'));
…
  r_activities.distance_km       := j_obj.get_number('distance')/1000;

  r_activities.gear_id           := j_obj.get_string('gear_id');
  IF r_activities.type IN('Ride','Walk','Hike','VirtualRide','Run') THEN
    j_subobj                     := j_obj.get_object('gear');
    IF j_subobj IS NOT NULL THEN
      r_activities.gear_name     := j_subobj.get_string('name');
    END IF;
  END IF;
…
  r_activities.photo_count       := j_obj.get_object('photos').get_number('count');
…
Then the entire row can be inserted or updated at the end from the row-type variable.
  BEGIN  
    INSERT INTO activities VALUES p_activities;
    dbms_output.put_line(sql%rowcount||' activity inserted');
    COMMIT;

  EXCEPTION 
    WHEN DUP_VAL_ON_INDEX THEN
      UPDATE activities
      SET ROW = p_activities
      WHERE  activity_id = p_activities.activity_id;
      dbms_output.put_line(sql%rowcount||' activity updated');   
      COMMIT;
  END;

Extract Values from JSON in Virtual Columns

The other option is to store the JSON in a CLOB column in the database and convert it on demand via virtual columns.  Whenever I log a new Strava activity or update, or delete an existing activity, I have subscribed to receive a message from Strava. That message is received by the database using a REST service.  
The message from Strava just tells me that an activity has been created, updated or deleted.  Then I have to process it.  Sometimes, I get multiple messages for the same activity in quick succession.
{
    "aspect_type": "update",
    "event_time": 1516126040,
    "object_id": 1360128428,
    "object_type": "activity",
    "owner_id": 134815,
    "subscription_id": 120475,
    "updates": {
        "title": "Messy"
    }
}
Strava requires that the REST service respond within 2 seconds, so any processing in it must be kept light.  I want to avoid:
  • spending time converting the JSON data while the REST service is running,
  • any malformed or unexpected variation in JSON causing an error in the REST service,
  • concurrent processing of different requests relating to the same activity causing one REST service handler to block another.  
Therefore, my REST service just stores the JSON in a CLOB column on a table and then triggers a scheduler job to process the message.  The subsequent processing needs to access the name-values in the JSON, so I have created virtual columns on the queue table that will only be evaluated on demand.
CREATE TABLE webhook_events
(ID                NUMBER GENERATED ALWAYS AS IDENTITY
,PAYLOAD           CLOB
,processing_status NUMBER DEFAULT 0 NOT NULL
…
,CONSTRAINT webhook_events_pk PRIMARY KEY (id)
);

ALTER TABLE webhook_events ADD aspect_type      GENERATED ALWAYS AS (JSON_VALUE(payload, '$."aspect_type"')) VIRTUAL;
ALTER TABLE webhook_events ADD object_type      GENERATED ALWAYS AS (JSON_VALUE(payload, '$."object_type"')) VIRTUAL;
ALTER TABLE webhook_events ADD object_id NUMBER GENERATED ALWAYS AS (JSON_VALUE(payload, '$."object_id"'  )) VIRTUAL;
In Strava, times are held in Unix 'Epoch Time' (the number of non-leap seconds since midnight UTC on 1st Jan 1970).  I have created a deterministic PL/SQL function to convert it to an Oracle timestamp and have referenced it in my virtual column definition.  
One virtual column cannot reference another.  So, I could not reference the virtual column EVENT_TIME in another virtual column EVENT_TIMESTAMP.  Instead, I had to reference the event_time name-value pair in both column definitions.
CREATE OR REPLACE FUNCTION strava.epoch_to_tstz 
(p_epoch_seconds IN NUMBER
) RETURN TIMESTAMP DETERMINISTIC IS
BEGIN
  RETURN TO_TIMESTAMP_TZ('1970-01-01 00:00:00 UTC', 'YYYY-MM-DD HH24:MI:SS TZR') 
       + NUMTODSINTERVAL(p_epoch_seconds, 'SECOND');
END epoch_to_tstz;
/

ALTER TABLE webhook_events ADD event_time NUMBER 
   GENERATED ALWAYS AS (JSON_VALUE(payload, '$."event_time"' )) VIRTUAL;
ALTER TABLE webhook_events ADD event_timestamp TIMESTAMP WITH TIME ZONE 
   GENERATED ALWAYS AS (epoch_to_tstz(JSON_VALUE(payload, '$."event_time"'))) VIRTUAL;
…
If the message is an update, it contains a JSON object listing the updated items and their new values.  The column updates contains this JSON document.
ALTER TABLE webhook_events ADD updates GENERATED ALWAYS AS (JSON_QUERY(payload, '$."updates"' )) VIRTUAL;
I can now reference the virtual columns in SQL in the queue handler without converting and storing the values in regular columns.
…
  FOR i IN ( --interate requests
    SELECT h.*, a.activity_id
    FROM webhook_events h
      LEFT OUTER JOIN activities a ON a.activity_id = h.object_id 
    WHERE h.processing_status = 0
    AND   h.object_type = 'activity'
    ORDER BY h.id
    FOR UPDATE OF h.processing_status 
  ) LOOP
…

Wednesday, March 25, 2026

Oracle 23ai/26ai: The New RETURNING Clause for the MERGE Statement

This blog is part of a series about aspects and features of Oracle 26 and Autonomous Database. 

The SQL MERGE statement was introduced in Oracle version 9i, allowing what is sometimes called UPSERT logic: a single SQL statement that conditionally inserts or updates rows.  However, one limitation remained.  Unlike INSERT, UPDATE, and DELETE, the MERGE statement did not support the RETURNING clause.  Oracle 23ai/26ai removes this restriction. Developers can now use the RETURNING clause directly in MERGE statements to retrieve values of affected rows. 

The Problem Before Oracle 23

Before Oracle 23, I would have to code a query loop capturing the values that were going to be updated and then update them in separate statements within the loop with additional exception handling as required.
…
  l_rows_processed := FALSE;

  FOR s IN (
    SELECT a.activity_id
    ,      listagg(DISTINCT ma.name,', ') WITHIN GROUP (ORDER BY ma.area_level, ma.name) area_list
    FROM   activities a
      INNER JOIN activity_areas aa ON a.activity_id = aa.activity_id
      INNER JOIN my_areas ma ON ma.area_code = aa.area_code and ma.area_number = aa.area_number
    WHERE a.activity_id = p_activity_id
    AND a.processing_status = 4
    And ma.matchable = 1
    GROUP BY a.activity_id;
  ) LOOP
    l_rows_processed := TRUE;

    UPDATE activities u
    SET    u.area_list = s.area_list
    WHERE  u.activity_id = s.activity_id

    update_activity_description(l_new_area_list,l_description);
  END LOOP;

  IF NOT l_rows_processed THEN 
    RAISE e_activity_not_found;
  END IF;
…

New Syntax in Oracle 23/26

Alternatively, I can use the MERGE statement to write a single SQL statement to generate the new value for a column and then update it in one go.  Now, the return clause also captures that new value in a variable that can be passed to another procedure.
MERGE INTO activities u
  USING (
    SELECT a.activity_id
    ,      listagg(DISTINCT ma.name,', ') WITHIN GROUP (ORDER BY ma.area_level, ma.name) area_list
    FROM   activities a
      INNER JOIN activity_areas aa on a.activity_id = aa.activity_id
      INNER JOIN my_areas ma on ma.area_code = aa.area_code and ma.area_number = aa.area_number
    WHERE a.activity_id = p_activity_id
    AND a.processing_status = 4
    AND ma.matchable = 1
    GROUP BY a.activity_id
  ) S 
  ON (s.activity_id = u.activity_id)
  WHEN MATCHED THEN UPDATE 
  SET u.area_list = s.area_list
  RETURNING new area_list INTO l_new_area_list; --new in Oracle 23
  
  IF SQL%ROWCOUNT = 0 THEN 
    RAISE e_activity_not_found;
  ELSE
    update_activity_description(l_new_area_list,l_description);
  END IF;

The benefits are
  • Less and simpler code, which ought therefore to be easier to test and maintain, requiring less additional logic and exception handling.
  • Fewer SQL statements and therefore fewer context switches between PL/SQL and SQL.
Just like the return clause on UPDATE and DELETE, it is also possible to
  • Reference new and/or old column values,
  • Single values into a scalar (single value) variable
  • Bulk collect multiple rows into an array variable
  • Aggregate multiple rows into a scalar variable
I am far from the first to blog about this feature, but it deserves to be better known.

See also:

Tuesday, March 24, 2026

ChatGPT & Oracle Development

This blog is part of a series about aspects and features of Oracle 26 and Autonomous Database.

TL;DR

This is an opinion piece about the impact of AI on developers and administrators.  I'll tell you my opinion here at the start:  

AI won't be replacing us, at least not yet, but we may be replaced by someone who is more productive because they are using AI!  I certainly advocate using it, but do so thoughtfully.  Consider whether the answers are sensible, and then test them carefully.

Introduction

To learn more about Oracle 26ai Autonomous Database, I returned to a project I created in 2021 to explore spatial data.  I had exported my activity data from Strava as flat files and then imported them into an Oracle database.  

Now, I have migrated that project to an Autonomous database on OCI and integrated it directly with Strava through their API.  Notifications of new activities are received via a REST service, some processing is done in the Oracle database, and results are written back to the Strava activity description.  All quite simple, but it made me use techniques and technologies that I have never used before.

ChatGPT

When I created the original project in 2021, I had the Oracle documentation and Google.  I had to design and write every bit of code myself.  It all took time.

Now, I have been able to use ChatGPT (other AI Chatbots are available, but this is where I started), and the effect has been remarkable.  I have been pointed at features and techniques that are new to me, and often I have been given a concrete example to start work on, and therefore I have learned about them.

I asked ChatGPT questions in plain language about the details of both Oracle 26 and the Strava API, and it gave me sensible answers in plain language that were generally sensible.  In some cases, it designed complete processing flows; sometimes it just illustrated the answer with code examples.  I could ask follow-up questions, and it would answer them in the context of the earlier question, refining its response.  It became a genuine conversation.  Though it is not going to pass the Turing Test!

ChatGPT's answers were mostly accurate, though some of its generated code was not always completely correct.  On some subjects, such as character set, we went round in circles.  Sometimes, I would point out mistakes, and it would say 'Yes, you are right!' or 'Well spotted!'.  I am not convinced it learnt anything from that.  Over time, I learnt that I needed to ask quite precise questions, otherwise it would go off in other directions.  Nevertheless, I found I got very quickly from a first draft of code to debugging almost working code.  I have no doubt that using ChatGPT increased my productivity.  If I had to quantify the effect, I would estimate that it improved my productivity by a factor of about 3.

These are some of my early questions to ChatGPT:

  • "How can Strava notify my Oracle database, using only PL/SQL, that an activity has been added, deleted or updated?" 
    • The result included a complete design for creating a Strava webhook to send an HTTP message to a REST service, including a database data model design and how to process it by calling the Strava API to extract the activity data

  • "How would I load GeoJSON … into an Oracle spatial data object geometry in an Oracle autonomous database using just PL/SQL"
    • I got a complete PL/SQL procedure to extract the GeoJSON from the data.gov.ie website, and then how to read the GeoJSON into an Oracle spatial geometry.
      • I was able to ask follow-up questions.   When one particular public data set produced errors from sdo_util.from_geojson, after a few other suggestions, ChatGPT provided a complete alternative PL/SQL procedure to create a spatial geometry from just the array of coordinates.  It is slower, but it works reliably.  I use it as an alternative when I get an error from the Oracle function.

There were some notable examples of code that ChatGPT produced correctly the first time, and much faster than I could have.  In particular, extracting all the data in a Strava activity (see strava_http.get_activity_stream) as both an Oracle spatial geometry and a GPX file, including heart monitor, cadence and power meter data if also present (that must conform to the Topographix and Garmin XML schemas).  My code is on GitHub, so you can judge the result for yourself!

Nullius in Verba

This motto (it can be translated as "Take Nobody's Word for It!") is at the heart of the scientific principle.  It can usefully be applied to many things, and certainly to ChatGPT.  

ChatGPT is a hugely powerful tool that seems to be capable of answering any reasonable query.  I would encourage anyone to use it to help develop code faster.  However, every response should be treated with healthy scepticism and be tested carefully.   Whether code compiles and executes is a straightforward question with an essentially binary answer.  Whether that code then does what it is supposed to do requires thorough testing, but then so does human-written code!

Nonetheless, I am hugely impressed by ChatGPT.  I have no doubt that I got further and got there much faster than I ever would otherwise! 

Monday, March 23, 2026

Job Classes on Autonomous Database

This blog is part of a series about aspects and features of Oracle 26 and Autonomous Database.

I have written about using Job Classes with the database scheduler.  It is essentially the same on Autonomous database, but some configuration is delivered by Oracle.  You may choose to use it directly as delivered.  However, I suggest using it as the basis for a custom configuration.

The Autonomous Transaction Processing (ATP) database is delivered with 5 consumer groups and 5 corresponding job classes that map to them.  
OWNER JOB_CLASS_NAME RESOURCE_CONSUMER_GROUP SERVICE
----- -------------- ----------------------- ------------------------------------------------------
LOGGING_LEVEL LOG_HISTORY COMMENTS                                
------------- ----------- ----------------------------------------
SYS    TPURGENT      TPURGENT                GE***********09_GOFASTER1_tpurgent.adb.oraclecloud.com 
RUNS                      Urgent transaction processing jobs     

SYS    TP            TP                      GE***********09_GOFASTER1_tp.adb.oraclecloud.com     
RUNS                      Transaction processing jobs            

SYS    HIGH          HIGH                    GE***********09_GOFASTER1_high.adb.oraclecloud.com   
RUNS                      High priority jobs                     

SYS    MEDIUM        MEDIUM                  GE***********09_GOFASTER1_medium.adb.oraclecloud.com 
RUNS                      Medium priority jobs                   

SYS    LOW           LOW                     GE***********09_GOFASTER1_low.adb.oraclecloud.com    
RUNS                      Low priority jobs                      
It is easy and perfectly reasonable to allocate these delivered job classes to scheduler jobs.  However, these job classes cannot be changed, even by the ADMIN user.
BEGIN dbms_Scheduler.set_attribute('SYS.TPURGENT', 'comments', 'A Comment'); END;
*
ERROR at line 1:
ORA-01031: insufficient privileges
ORA-06512: at "SYS.DBMS_ISCHED", line 3513
ORA-06512: at "SYS.DBMS_SCHEDULER", line 3460
ORA-06512: at line 1

Note that the service names are different and unique to every autonomous database.  I have been careful to avoid hard-coding this anywhere within my scripts and code.  Instead, I duplicate the delivered job classes and then alter as necessary.  Thus, each job or group of jobs has its own job class.  I prefer to manage the job and the job scheduler, as far as possible, within a packaged procedure.  This has several advantages.

  • The right version is always available because it has been installed into the database and can be migrated like any other version-controlled source code.  This also covers when a database has been cloned, restored or flashed back.  This saves looking for the right version of the right script.  
  • Jobs can be created and managed by a user who does not have access to manage the job scheduler, but who can execute procedures in the package.
  • I have created a procedure to clone a job class and adjust attributes as required.
  • The code is available on GitHub.
CREATE OR REPLACE PACKAGE BODY strava.strava_job AS
...
e_job_already_exists EXCEPTION;
PRAGMA EXCEPTION_INIT(e_job_already_exists,-27477);
...
PROCEDURE create_job_class
(p_job_class_name          all_scheduler_job_classes.job_class_name%TYPE
,p_based_on_job_class      all_scheduler_job_classes.job_class_name%TYPE
,p_resource_consumer_group all_scheduler_job_classes.resource_consumer_group%TYPE DEFAULT NULL
,p_service                 all_scheduler_job_classes.service%TYPE                 DEFAULT NULL
,p_logging_level           all_scheduler_job_classes.logging_level%TYPE           DEFAULT NULL
,p_log_history             all_scheduler_job_classes.log_history%TYPE             DEFAULT NULL
,p_comments                all_scheduler_job_classes.comments%TYPE                DEFAULT NULL)
IS
  r_job_class all_scheduler_job_classes%ROWTYPE;
...
BEGIN
...  
  SELECT * INTO r_job_class FROM all_scheduler_job_classes
  WHERE owner = 'SYS' AND job_class_name = p_based_on_job_class;
  
  BEGIN
    DBMS_SCHEDULER.CREATE_JOB_CLASS(p_job_class_name); 
  EXCEPTION WHEN e_job_already_exists THEN NULL;
  END;
  
  IF p_resource_consumer_group IS NOT NULL THEN r_job_class.resource_consumer_group := p_resource_consumer_group; END IF;
  IF p_service                 IS NOT NULL THEN r_job_class.service := p_service; END IF;
  IF p_logging_level           IS NOT NULL THEN r_job_class.logging_level := p_logging_level; END IF;
  IF p_log_history             IS NOT NULL THEN r_job_class.log_history := p_log_history; END IF;
  IF p_comments                IS NOT NULL THEN r_job_class.comments := p_comments; END IF;
  
  dbms_Scheduler.set_attribute(p_job_class_name, 'resource_consumer_group', r_job_class.resource_consumer_group);
  dbms_Scheduler.set_attribute(p_job_class_name, 'service'                , r_job_class.service);
  IF    r_job_class.logging_level = 'OFF'         THEN dbms_Scheduler.set_attribute(p_job_class_name, 'logging_level', DBMS_SCHEDULER.LOGGING_OFF);
  ELSIF r_job_class.logging_level = 'RUNS'        THEN dbms_Scheduler.set_attribute(p_job_class_name, 'logging_level', DBMS_SCHEDULER.LOGGING_RUNS);
  ELSIF r_job_class.logging_level = 'FAILED RUNS' THEN dbms_Scheduler.set_attribute(p_job_class_name, 'logging_level', DBMS_SCHEDULER.LOGGING_FAILED_RUNS);
  ELSIF r_job_class.logging_level = 'FULL'        THEN dbms_Scheduler.set_attribute(p_job_class_name, 'logging_level', DBMS_SCHEDULER.LOGGING_FULL);
  END IF;
  dbms_Scheduler.set_attribute(p_job_class_name, 'log_history'            , r_job_class.log_history);
  dbms_Scheduler.set_attribute(p_job_class_name, 'comments'               , r_job_class.comments);
...
EXCEPTION 
  WHEN no_data_found THEN
...
    RAISE;
END create_job_class;
This new procedure is called from the procedures that create jobs.  In the example below, the LOW job class is cloned into a new PURGE_API_LOG_CLASS that is used by the PURGE_API_LOG job.  I have set the log history retention to 7 days, but all other settings remain the same.  
PROCEDURE create_purge_api_log_job
IS
  k_job_name  CONSTANT VARCHAR2(128 CHAR) := 'STRAVA.PURGE_API_LOG';
  k_job_class CONSTANT VARCHAR2(128 CHAR) :=    'SYS.PURGE_API_LOG_CLASS';
BEGIN
...
  create_job_class(k_job_class,'LOW', p_log_history=>7);
  BEGIN
    dbms_scheduler.create_job(
    (job_name => k_job_name
    ,job_type => 'STORED_PROCEDURE'
    ,job_action => 'STRAVA.STRAVA_HTTP.PURGE_API_LOG'
    ,enabled => FALSE
    );
  EXCEPTION WHEN e_job_already_exists THEN NULL;
  END;
...
  dbms_scheduler.set_attribute(name => k_job_name, attribute => 'JOB_CLASS', value => k_job_class);
...
  dbms_scheduler.enable(name => k_job_name);
...
END create_purge_api_log_job;
...
END strava_job;
/

Now I have several job classes 

OWNER JOB_CLASS_NAME                        RESOURCE_CON SERVICE
----- ------------------------------------- ------------ ------------------------------------------------------
LOGGING_LEVEL LOG_HISTORY COMMENTS                      
------------- ----------- ------------------------------
SYS  CREATE_ACTIVITY_HSEARCH_UPD_ALL_CLASS  LOW          GE***********09_GOFASTER1_low.adb.oraclecloud.com    
RUNS                   7 Low priority jobs            

SYS  ACTIVITY_AREA_LIST_UPD_ALL_CLASS        LOW         GE***********09_GOFASTER1_low.adb.oraclecloud.com    
RUNS                   7 Low priority jobs            

SYS  PURGE_API_LOG_CLASS                     LOW         GE***********09_GOFASTER1_low.adb.oraclecloud.com    
RUNS                   7 Low priority jobs            

SYS  PURGE_EVENT_QUEUE_CLASS                 LOW         GE***********09_GOFASTER1_low.adb.oraclecloud.com    
RUNS                   7 Low priority jobs            

SYS  BATCH_LOAD_ACTIVITIES_CLASS             MEDIUM      GE***********09_GOFASTER1_medium.adb.oraclecloud.com 
RUNS                   7 Medium priority jobs         

SYS  UPDATE_STRAVA_ACTIVTY_CLASS             MEDIUM      GE***********09_GOFASTER1_medium.adb.oraclecloud.com 
RUNS                   7 Medium priority jobs         

SYS  PROCESS_WEBHOOK_QUEUE_CLASS             MEDIUM      GE***********09_GOFASTER1_medium.adb.oraclecloud.com 
RUNS                   7 Medium priority jobs         

SYS  RENEW_STRAVA_TOKENS_CLASS               HIGH        GE***********09_GOFASTER1_high.adb.oraclecloud.com   
RUNS                   7 High priority jobs
Each job has been allocated to a different job class.  In future, I can control the behaviour of each job by adjusting the job class.
OWNER  JOB_NAME                             JOB_CLASS                            
------ ------------------------------------ -------------------------------------
STRAVA ACTIVITY_AREA_LIST_UPD_ALL_JOB      ACTIVITY_AREA_LIST_UPD_ALL_CLASS 
STRAVA BATCH_LOAD_ACTIVITIES_JOB           BATCH_LOAD_ACTIVITIES_CLASS 
STRAVA CREATE_ACTIVITY_HSEARCH_UPD_ALL_JOB CREATE_ACTIVITY_HSEARCH_UPD_ALL_CLASS
STRAVA PROCESS_WEBHOOK_QUEUE_JOB           PROCESS_WEBHOOK_QUEUE_CLASS 
STRAVA PURGE_API_LOG                       PURGE_API_LOG_CLASS 
STRAVA PURGE_EVENT_QUEUE                   PURGE_EVENT_QUEUE_CLASS 
STRAVA RENEW_STRAVA_TOKENS_JOB             RENEW_STRAVA_TOKENS_CLASS 
STRAVA UPDATE_STRAVA_ACTIVTY_JOB           UPDATE_STRAVA_ACTIVTY_CLASS