You're in the middle of a critical Oracle database patching process using OPatch, and suddenly, it grinds to a halt. The error logs are cryptic, hinting at memory issues, yet your server's resources appear to be plentiful. This was the exact scenario faced by me attempting to apply patch 37642901 to their Oracle 19c database. The solution, surprisingly, had less to do with Oracle's memory management and more with the underlying Linux operating system's file system caching.
This blog post will dissect the problem, walk through the unexpected resolution, and explain the commands that saved the day.
The Problem: A Halted Patch and Misleading Clues
I have initiated the opatch apply command for patch 37642901. The process started as expected, verifying the environment and proceeding with the initial checks. However, during the "Applying interim patch" phase, the process stopped.
A look at the OPatch log revealed the following message:
[oracle@EQVLA360RDBDR1 ~]$ $ORACLE_HOME/OPatch/opatch apply /oracle_home/patches/37642901 Oracle Interim Patch Installer version 12.2.0.1.46 Copyright (c) 2025, Oracle Corporation. All rights reserved. Oracle Home : /oracle_home/app/19.0.0.0/dbhome_1 Central Inventory : /oracle_home/app/oraInventory from : /oracle_home/app/19.0.0.0/dbhome_1/oraInst.loc OPatch version : 12.2.0.1.46 OUI version : 12.2.0.7.0 Log file location : /oracle_home/app/19.0.0.0/dbhome_1/cfgtoollogs/opatch/opatch2025-05-28_00-24-55AM_1.log Verifying environment and performing prerequisite checks... OPatch continues with these patches: 37642901 Do you want to proceed? [y|n] y User Responded with: Y All checks passed. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/oracle_home/app/19.0.0.0/dbhome_1') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files... Applying interim patch '37642901' to OH '/oracle_home/app/19.0.0.0/dbhome_1' ApplySession: Optional component(s) [ oracle.network.gsm, 19.0.0.0.0 ] , [ oracle.assistants.asm, 19.0.0.0.0 ] , [ oracle.crypto.rsf, 19.0.0.0.0 ] , [ oracle.pg4appc, 19.0.0.0.0 ] , [ oracle.pg4mq, 19.0.0.0.0 ] , [ oracle.oraolap.mgmt, 19.0.0.0.0 ] , [ oracle.precomp.companion, 19.0.0.0.0 ] , [ oracle.rdbms.ic, 19.0.0.0.0 ] , [ oracle.rdbms.tg4db2, 19.0.0.0.0 ] , [ oracle.sdo.companion, 19.0.0.0.0 ] , [ oracle.tfa, 19.0.0.0.0 ] , [ oracle.ons.cclient, 19.0.0.0.0 ] , [ oracle.options.olap, 19.0.0.0.0 ] , [ oracle.ons.eons.bwcompat, 19.0.0.0.0 ] , [ oracle.options.olap.api, 19.0.0.0.0 ] , [ oracle.network.cman, 19.0.0.0.0 ] , [ oracle.rdbms.tg4sybs, 19.0.0.0.0 ] , [ oracle.rdbms.tg4tera, 19.0.0.0.0 ] , [ oracle.oid.client, 19.0.0.0.0 ] , [ oracle.ldap.ztk, 19.0.0.0.0 ] , [ oracle.java.sqlj.sqljruntime, 19.0.0.0.0 ] , [ oracle.net.cman, 19.0.0.0.0 ] , [ oracle.xdk.companion, 19.0.0.0.0 ] , [ oracle.rdbms.tg4msql, 19.0.0.0.0 ] , [ oracle.rdbms.tg4ifmx, 19.0.0.0.0 ] , [ oracle.jdk, 1.8.0.191.0 ] , [ oracle.jdk, 1.8.0.391.11 ] not present in the Oracle Home or a higher version is found. ^Z [1]+ Stopped $ORACLE_HOME/OPatch/opatch apply /oracle_home/patches/37642901
While this message might seem alarming, it can often be a red herring. Oracle's documentation and community forums suggest that this message can frequently be ignored, as it simply indicates that the patch contains components for features that are not installed in the target Oracle Home.
[May 28, 2025 12:35:31 AM] [INFO] Oracle Home : /oracle_home/app/19.0.0.0/dbhome_1
Central Inventory : /oracle_home/app/oraInventory
from : /oracle_home/app/19.0.0.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.46
OUI version : 12.2.0.7.0
OUI location : /oracle_home/app/19.0.0.0/dbhome_1/oui
Log file location : /oracle_home/app/19.0.0.0/dbhome_1/cfgtoollogs/opatch/opatch2025-05-28_00-35-30AM_1.log
[May 28, 2025 12:35:31 AM] [INFO] Patch history file: /oracle_home/app/19.0.0.0/dbhome_1/cfgtoollogs/opatch/opatch_history.txt
[May 28, 2025 12:35:35 AM] [INFO] [OPSR-TIME] Loading raw inventory
[May 28, 2025 12:35:35 AM] [INFO] [OPSR-MEMORY] Loaded all components from inventory. Heap memory in use: 27 (MB)
[May 28, 2025 12:35:35 AM] [INFO] [OPSR-MEMORY] Loaded all one offs from inventory. Heap memory in use: 18 (MB)
[May 28, 2025 12:35:35 AM] [INFO] [OPSR-TIME] Raw inventory loaded successfully
[May 28, 2025 12:35:35 AM] [INFO] NApply::no CAS enabled, OPatch runs with legacy process.
[May 28, 2025 12:35:35 AM] [INFO] Verifying environment and performing prerequisite checks...
[May 28, 2025 12:35:35 AM] [INFO] [OPSR-TIME] Running prerequisite checks
[May 28, 2025 12:35:35 AM] [INFO] opatch-external.jar is in /oracle_home/app/19.0.0.0/dbhome_1/OPatch/jlib/opatch-external.jar
[May 28, 2025 12:35:57 AM] [INFO] [OPSR-TIME] Loading cooked inventory
[May 28, 2025 12:35:57 AM] [INFO] [OPSR-MEMORY] : Loading cooked one offs. Heap memory used 1103 (MB)
[May 28, 2025 12:36:15 AM] [INFO] [OPSR-MEMORY] : Loaded cooked oneoffs. Heap memory used : 1247 (MB)
The more perplexing part of the issue was the mention of "heap memory" in the accompanying log files, which led the user down a path of investigating Java heap size settings for OPatch. However, a closer look at the detailed log (opatch
The Unexpected Solution: Clearing the System's Cache
After much troubleshooting, the issue was resolved by executing two simple Linux commands:
sync; echo 3 > /proc/sys/vm/drop_caches
Let's break down what these commands do:
- sync: This command is used to flush the file system buffers. In simpler terms, it ensures that all the data that the operating system has been holding in its memory cache is written to the physical disk. This is a crucial first step to ensure data integrity before clearing the caches.
- echo 3 > /proc/sys/vm/drop_caches: This command instructs the Linux kernel to drop clean caches. The number 3 specifically tells the kernel to free all page cache, dentries, and inodes. This is a non-destructive operation, meaning it only clears data that has already been saved to disk (which the sync command just ensured).
Why Did This Work? A Look at the Underlying Mechanics
At first glance, it might seem counterintuitive that clearing the operating system's file system cache would resolve an issue seemingly related to Oracle's Java heap memory. The key to understanding this lies in the interaction between the OPatch utility and the underlying file system.
The OPatch process involves extensive reading and writing of files within the Oracle Home directory. During this process, the Linux kernel, in its effort to optimize performance, will cache a significant amount of this file data in memory. While this is generally beneficial, it can, in some scenarios, lead to a large portion of the system's RAM being allocated to the file system cache.
Although the kernel is supposed to release this cache when an application demands more memory, there can be situations where this process is not as immediate or efficient as needed. It's plausible that the large number of files being accessed by OPatch caused the file system cache to grow to a point where it interfered with the memory allocation for the OPatch Java process, even if there was technically "free" memory available. The "heap memory" messages in the log may have been a symptom of this underlying resource contention rather than a direct issue with the Java heap configuration itself.
By forcing a synchronization of the file system buffers to disk and then clearing these caches, the user effectively freed up a significant amount of system RAM. This provided the OPatch process with the necessary memory resources to complete the patching operation without contention.
This experience serves as a valuable reminder that when troubleshooting complex software like Oracle, it's essential to consider the entire technology stack, from the application down to the operating system. Misleading error messages can often send us down the wrong path. In this case, what appeared to be an application-level memory issue was, in fact, a resource contention problem at the operating system level.
Toufique Khan
