Australian cloud for business software — built and run in Sydney since 2003

NetLeverage
NetLeverage

vCenter appliance partition full: finding what filled /storage/log, /storage/db or /storage/seat

Technical article
VMware

Direct answer: a vCenter Server Appliance keeps its logs, its database and its events/tasks/statistics data on separate logical volumes, each on its own virtual disk. When one fills, the symptoms depend on which one: a full /storage/log produces disk-exhaustion alarms, slow sign-in and sometimes 503 errors; a full /storage/seat stops the database from extending its files, so vCenter services can fail to start with "no healthy upstream". Start by confirming which mount point is full with df -h and df -i, then find the largest files or file counts inside that one mount point, then match what you find to a known Broadcom cause. The supported remedies are cause-specific cleanup documented by Broadcom, or growing the matching virtual disk and running the appliance's own expansion script. Hand-deleting PostgreSQL data files, tablespace folders or files a running service still has open is not a fix.

Why it matters

vCenter is the control plane for the cluster. A full partition rarely affects running virtual machines directly, but it can stop the services that schedule backups, apply DRS and HA configuration, serve the vSphere Client and record events. Broadcom states the risk on both sides plainly: deleting critical files may stop the appliance from working, and resizing appliance virtual disks carries a risk of data corruption. That is why both a recent backup and correct identification of the disk come before any change.

The pressure to "just free some space" is highest exactly when the appliance is least able to tolerate a mistake. Read-only checks first often point to a single runaway file or directory.

How it works

The appliance uses Logical Volume Management. Each major mount point under /storage is a logical volume in its own volume group, backed by one virtual disk attached to the vCenter VM. Broadcom KB 316602 shows the pattern in its example output: /storage/log is log_vg-log on one disk, /storage/db is db_vg-db on another, and /storage/core on a third. The exact disk number for each mount point differs between versions and deployment sizes, so Broadcom maintains separate "List of VMDKs/Partitions" articles for 6.7, 7.0 and 8.0, and its resize procedure maps the mount point to the SCSI node from inside the appliance rather than relying on a fixed table.

Broadly:

Mount point What writes there Typical effect when full
/storage/log Service logs, support bundles, some heap dumps, PostgreSQL logs under /var/log/vmware/vpostgres Health alarms ("Log disk exhaustion"), slow sign-in, possible 503 errors, services unable to log
/storage/db The vPostgres database and its configuration Database operations fail; services depending on it degrade
/storage/seat Statistics, events and tasks tablespaces (/storage/seat/vpostgres/...) PostgreSQL refuses to extend files; vpxd cannot insert events; services fail to start, "no healthy upstream"

Under default settings, Broadcom KB 313077 describes yellow status when /storage/log stays at 75% for at least 10 minutes, and red critical alarms at 85%. The VAMI health status reads "File system /storage/log is low on storage space. Increase the size of disk /storage/log."

KB 410118 explains why a full /storage/seat is so disruptive: PostgreSQL deliberately stops creating or extending data files when space runs out, to avoid corruption. The article records that in this state maintenance commands such as vacuumdb, REINDEX and TRUNCATE also fail. These commands work in different ways, but the documented outcome is the same: the usual "clean up the database" tools cannot be relied on at the moment you want them.

On 8.0 U3h and later there is an additional sizing change. KB 409413 states that from 8.0 U3h, /storage/log is 50 GB by default for fresh deployments, major upgrades and Reduced Downtime Update patching, and that existing appliances with smaller log partitions (typically 10 GB on Tiny and 25 GB on other sizes) receive a pre-check warning. Broadcom asks for exactly 50 GB.

Read-only checks

None of these was run for this article. Run them from the appliance Bash shell as root (from the appliance Command> prompt, shell switches to Bash). They read state only.

  1. Which mount point is full, by space and by inodes.

df -h df -i

KB 313077 notes that /storage/log can look fine in df -h but be full by inode count in df -i. A partition full of small files needs a different explanation from one holding a few huge ones.

  1. VAMI view. In the vCenter Server Management Interface (port 5480), Monitor > Disks and Summary > Health Status show the same condition with the alarm level. Record the percentages before you change anything.

  2. Largest files inside the full mount point only. Based on Broadcom KB 313077, with -xdev added so find does not cross into another mounted filesystem:

cd /storage/log find . -xdev -type f -print0 | xargs -0 du -h | sort -rh | head -n 10

and, for a directory-by-directory walk that stays on one filesystem:

du -xh -d 1 | sort -rh

Repeat du in the largest subdirectory until you reach the files responsible. Recursive scans add disk I/O to an appliance that is already struggling, so run them one at a time and stop once the cause is clear.

  1. File counts, when inodes are the problem. KB 313077 counts entries per top-level directory with a loop that starts a separate find in each one. A separate start point is a problem here: if a top-level entry is itself a mount point, that find starts on the other filesystem, and -xdev cannot keep it out. Instead, walk once from the mount point itself, so -xdev keeps every nested mount out, and count entries by top-level directory:

cd /storage/log find . -xdev | cut -d/ -f2 | sort | uniq -c | sort -n

A nested mount point shows up as a single entry, and nothing beneath it is counted. File names that contain a newline make the counts a little inaccurate, so treat the result as a screen that shows where the files are. Like the scans in step 3, this reads every directory entry on the partition, so run it once.

  1. For /storage/seat. Check df -h /storage/seat and the size of the event and task tablespace directories under /storage/seat/vpostgres/. In /var/log/vmware/vpxd/vpxd.log, KB 410118 shows the signature could not extend file "pg_tblspc/..." : No space left on device on event inserts.

  2. Snapshots and backups. Note whether the vCenter VM has any snapshots (they block disk extension and are a separate risk) and when the last successful file-based or image-level backup completed.

Interpreting the results

Finding Likely meaning Where to look next
/storage/log high in df -h, a few very large *.stderr or service logs A service log not rotating or a known runaway logger Match against the known-issue table in KB 313077 by version
/storage/log normal in df -h, full in df -i Very many small files (state files, access logs) KB 313077 links an rsyslog imfile-state issue and access-log accumulation issues
Large support bundle archives under /storage/log Exported log bundles not removed KB 313077 lists bundle export as a known cause in all versions
Large postgresql*.log under /var/log/vmware/vpostgres auto_explain logging slow query plans (7.0 U3g and later) KB 323922
.hprof heap dump files A Java service wrote heap dumps KB 313077 lists analytics and infraprofile heap dump issues
/storage/seat at 100%, services fail to start, could not extend file Events/tasks/statistics data filled the tablespace KB 410118; often a single high-volume event type
8.0 U3h+ pre-check warns about log partition below 50 GB Sizing requirement, not a fault KB 409413

KB 410118 gives one illustration worth remembering: tens of millions of guest-operation events from one account filled the events table. A tool that calls guest operations in a tight loop can fill /storage/seat without anything being "broken" in vCenter itself, so identify the event source before planning only for disk growth.

Decision guidance

Before any change. Take or confirm a recent VAMI file-based backup or an image-level backup of the appliance, as KB 313077 and KB 409413 both require. If the vCenter is in Enhanced Linked Mode, Broadcom's cleanup articles call for offline (powered-off) snapshots of all linked nodes, not just the one being changed, and KB 323922 asks for vCenter HA to be removed before making the change. If the vCenter runs vSphere Supervisor or VKS, KB 313886 warns that vCenter snapshot rollbacks will not work with the Supervisor Control Plane VMs, so the rollback point is a file-based backup taken as part of the Supervisor backup and restore process. Record the df -h, df -i and VAMI values as the baseline you will compare against.

Known cause with a Broadcom article. Follow that article for your version. They are narrow on purpose. For example, KB 323922 empties the current PostgreSQL log in place (because the service holds it open) and then either raises auto_explain.log_min_duration above 10 seconds or comments out the auto_explain settings, after copying postgresql.conf aside, with a configuration reload rather than a restart. Rollback for that change is restoring the copied postgresql.conf and reloading. Do not generalise the steps from one article to other files.

/storage/seat full. KB 410118 offers three routes in order: a manual, support-grade procedure that temporarily relocates a tablespace directory to make room for truncation; a scripted purge of old tasks, events and statistics using the SQL script from Broadcom's "Delete old tasks, events and statistics data" article (which resets the event ID sequence); and finally growing the /storage/seat disk. The first two change database contents and are best done with Broadcom Support or under a change record that names the retention you will keep. Also fix whatever produced the event volume, or the partition fills again.

Growing a disk. Per KB 316602 and KB 409413:

  1. Remove any snapshots on the vCenter VM. If the extend option is greyed out, a snapshot or datastore maximum is usually the reason. KB 410118 specifically says not to take a vCenter snapshot before growing /storage/seat.
  2. Identify the virtual disk from inside the appliance: lsblk -o NAME,HCTL,MOUNTPOINT,SIZE and lsscsi map the logical volume to a SCSI node such as (0:4), which you match to the hard disk's virtual device node in Edit Settings. Do not rely on disk letters alone.
  3. Increase only that virtual disk (to exactly 50 GB for /storage/log on 8.0 U3h+).
  4. As root, run /usr/lib/applmgmt/support/scripts/autogrow.sh, then confirm with df -h or lsblk. KB 316602 also documents the Appliance Management CLI storage commands, which run only from the Command> prompt, not inside Bash.
  5. If services were stopped (especially vpostgres), KB 410118 restarts all services after the script completes.

Growing a disk cannot be undone by shrinking it. KB 409413 adds a backup consequence: a backup taken before resizing must be restored before the resize, and one taken after must be restored after it. Record which side of the change your last backup is on.

Root partition. KB 316602 states that resizing the root partition is not supported on 7.0 and later. For a root free-space shortfall in an upgrade pre-check, Broadcom's documented option is vacuuming old journal entries, not growing the disk.

What not to do

  • Do not delete files under /storage/db or /storage/seat/vpostgres. These are live database files. PostgreSQL's refusal to extend them is a protection, not a fault to work around.
  • Do not delete a log file a running service has open. Space is not returned until the process closes it, and the service may misbehave. Broadcom articles that reclaim this space empty the file in place for a specific, named log.
  • Do not run find ... -exec rm patterns copied from one Broadcom article against other directories or other file patterns.
  • Do not grow a disk while the VM has snapshots, and do not guess which disk to grow from its size or letter.
  • Do not take a snapshot of one linked-mode node and revert it alone. Broadcom's linked-mode guidance is all nodes offline together, or none. With Supervisor or VKS, do not rely on snapshot rollback at all.
  • Do not treat the alarm clearing as the end. If the cause is a non-rotating log or an event storm, schedule a recheck.

Limitations

This article does not reproduce the per-version partition tables or every known-issue article Broadcom links from KB 313077; check those for your exact build. Default event, task and statistics retention values were not verified for this article and are deliberately not stated. Broadcom article dates were not displayed when accessed, so confirm that each cited article still applies to your build. External Platform Services Controller deployments are mentioned in KB 316602 but are not covered here.

Acceptance checklist

  • [ ] Full mount point identified by both df -h and df -i, with values recorded
  • [ ] Largest files or highest file counts found within that one mount point
  • [ ] Cause matched to a specific Broadcom article for the installed build, or escalated
  • [ ] Recent file-based or image backup confirmed; linked-mode and vCenter HA preconditions met; with Supervisor or VKS, a file-based backup is the rollback point, not a snapshot
  • [ ] Any change applied only to the named file, setting or disk, with a copy or backup for rollback
  • [ ] After a disk change: no snapshots beforehand, SCSI node mapped from inside the appliance, autogrow.sh run as root, new size confirmed
  • [ ] VAMI health returns to normal and services start; sign-in and task history work
  • [ ] Source of growth addressed (rotation, logging setting or event producer) and a recheck date set

References

All accessed 2 November 2026. Broadcom article update dates were not displayed.

  • Broadcom KB 313077, "vCenter log disk exhaustion or /storage/log full": https://knowledge.broadcom.com/external/article/313077
  • Broadcom KB 316602, "Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7, 7.0 and 8.0": https://knowledge.broadcom.com/external/article/316602/increasing-the-disk-space-for-the-vcente.html
  • Broadcom KB 410118, "vCenter service fails to start due to /storage/seat full": https://knowledge.broadcom.com/external/article/410118/vcenter-service-fails-to-start-due-to-st.html
  • Broadcom KB 409413, /storage/log 50 GB recommendation for vCenter 8.0 U3h and later: https://knowledge.broadcom.com/external/article/409413
  • Broadcom KB 323922, "VCSA /storage/log partition full due to large postgres logs": https://knowledge.broadcom.com/external/article/323922/vcsa-storagelog-partition-full-due-to-la.html
  • Broadcom KB 313886, snapshot best practices for vCenter in Enhanced Linked Mode (offline snapshots of all nodes): https://knowledge.broadcom.com/external/article/313886

Related articles

Learning notes

  1. Each /storage mount point is its own logical volume on its own virtual disk; identify the full one before acting.
  2. Check inodes as well as bytes; a partition can be "full" with plenty of gigabytes free.
  3. A full /storage/seat disables the database tools you would use to clean it, which is why Broadcom's routes are staged.
  4. Map the logical volume to a SCSI node from inside the appliance before growing any disk, and never with snapshots present.
  5. On 8.0 U3h and later, a log partition under 50 GB is a sizing requirement to plan for, not a fault.