Australian cloud for business software — built and run in Sydney since 2003

NetLeverage
NetLeverage

Files Deleted but Linux Disk Space Is Still Used: Find Open File References

Technical article
Troubleshooting

Deleting a Linux filename does not necessarily release its storage immediately. If a process still has the last unlinked file open, the file remains until the remaining open references close. Identify the owning process and file before choosing the service's supported reopen or restart procedure.1

This is a specific explanation for space retained after deletion or log rotation. It is not the only reason filesystem usage differs from a directory total. The examples are source-checked and unexecuted; no process was signalled, restarted or inspected on a live service.

A name and an open file are different things

Linux unlink() removes a directory name. The manual distinguishes removing the last link when no process has the file open from removing it while a process still has an open descriptor. In the latter case, the file continues to exist until the remaining descriptor references close.1

A logging process can therefore keep writing through an already open descriptor after an operator removes the visible filename. Creating a new file at the old path does not automatically redirect that descriptor to the new file. The process's logging lifecycle determines when it closes and reopens its output.

That explains a common incident shape: a large log disappears from the directory listing, but filesystem free space barely changes. The operator sees an empty-looking directory while the service continues to hold the old object.

Do not conclude that every deleted-open file is abnormal. Applications can deliberately use unlinked temporary files. The important questions are whether the retained allocation explains the capacity problem and whether its lifetime is correct for the workload.

Reconcile measurements on the same filesystem

Identify the exact filesystem and mount namespace first. Keep the target path, device identity, type and sample time. A directory total from one view and a filesystem report from another cannot support a meaningful comparison.5

GNU df reports filesystem usage. GNU du measures usage reachable through the names it traverses, with options that control filesystem boundaries and apparent versus allocated size.4, 6 An unlinked open file can contribute to filesystem allocation without being reachable by its former directory name.

An illustrative bounded directory report is:

du -x -h --max-depth=1 -- '/approved/application/directory'

-x avoids crossing into different filesystems. --max-depth limits displayed depth, not necessarily the traversal work needed to calculate totals.4 A large directory can still be expensive to scan, so authorise the scope and prefer existing monitoring when it already answers the question.

Do not use apparent sizes as if they were allocated blocks. Sparse files, shared storage features, inaccessible paths, metadata and other filesystem-specific factors can also affect the comparison. If deleted-open files do not explain the gap, retain those other possibilities rather than forcing the initial theory.

Find unlinked open files

The lsof manual hosted by man7.org documents +L1 as selecting open files with link counts below one, which identifies unlinked open files. Its combined filesystem form can restrict the inspection to the relevant filesystem.2

lsof +aL1 '/approved/mountpoint'

This command lists metadata; it does not close or truncate the files. It is an untested example. Use existing authorised process visibility, because an ordinary account may not see other users' processes. Do not assume an empty restricted result proves that no such references exist.

For a known process, the manual's -a option combines selection criteria with logical AND. A scoped example is:

lsof -a -p 12345 +L1

Here 12345 is an illustrative process ID to replace only after identifying the actual process. Without understanding selection semantics, combining filters can accidentally broaden the result. Do not copy a historical process ID from a ticket into an intervention.

Capture process name/ID, descriptor, device, inode, link count and reported size/offset. Keep raw output restricted because paths and process information can be sensitive. The SIZE/OFF column may represent size or offset depending on the file and options; read it according to the manual, not as a universal allocated-byte counter.2

Attribute retained storage accurately

Several descriptors or processes can reference the same file. Do not sum every displayed row as independent storage. Reconcile device and inode identity and distinguish unique objects from multiple references.

Likewise, logical file size is not necessarily physical allocation. A large sparse file can have a much smaller block footprint. Use filesystem-aware allocation evidence when quantifying the storage that will actually become reusable after closure.

Match the process to its service, container or job owner. Confirm the process start context and whether it is still the same instance that produced the incident. A familiar executable name does not prove which configured service owns the descriptor.

Ask why the object was unlinked. Was a normal rotation followed by a missing reopen action? Did a cleanup job remove a still-active temporary file? Is an application intentionally holding it until a request finishes? The cause determines the correction.

Choose the supported release mechanism

The logrotate manual warns that copytruncate has a copy-to-truncate window in which log records can be lost. It is not a lossless substitute for an application-supported reopen; agree that trade-off before selecting a rotation method.3

For a service log, find the application's documented reopen or rotation mechanism. Some services respond to a specific signal; others use an API, their own rotation facility or a controlled restart. There is no universal safe signal for every process, so this article supplies none.

Review the rotation configuration and execution result with the service owner. The logrotate manual documents post-rotation scripting and several rotation methods, including methods with different handling of the original file.3 A rotation configuration needs to match the application's actual file-handling contract.

Do not edit a generic rotation rule merely because a filename ends in .log. Database logs, journals and queue files may have different consistency requirements from plain append-only text. The owning application's supported maintenance route takes precedence.

If a restart is required, assess shared users, pending requests, dependencies and recovery access. A process retaining space is not automatically safe to stop. Record what will happen to in-flight work and how the application will be checked after restart.

Avoid destructive shortcuts

Do not truncate a file through /proc just because its former name was deleted. The open object may contain active database or application state, and changing its length bypasses the owner's lifecycle. A capacity symptom is not authority to discard its contents.

Do not kill every process in a list of deleted-open files. Some may be using legitimate temporary files or mapped code, and multiple references may need coordinated handling. Select the confirmed object and service through a reviewed plan.

Also avoid deleting more visible files merely because the first deletion did not free enough space. That can remove evidence or business data without addressing the reference that retains the allocation. Capacity containment should be based on measured reserve and a known-safe path.

Verify closure and future rotation

After an approved reopen or restart, inspect the specific reference again and confirm the service now writes to the intended current file. Compare filesystem allocation before and after, accounting for other activity during the interval. A changed inode in a directory is not proof that every old reference closed.

Then verify the application itself: healthy requests, required logs arriving, correct ownership/permissions and no missed or duplicate work. Storage release alone is not service acceptance.

Test the corrected rotation lifecycle in an authorised disposable environment using synthetic log output. It should rotate, preserve the required history, reopen correctly and keep retained allocation within the expected reserve. No such test was run here.

Keep the previous service/rotation configuration for a scoped return, but understand that reverting configuration does not recreate content already discarded. If the incident involved lost logs, record that evidence gap instead of treating the recovered disk space as a complete resolution.

Learning notes

  • Removing a filename does not necessarily close the file held by a process.
  • A directory total and a filesystem total measure different views.
  • Correct rotation must coordinate file lifetime with the owning application.

Acceptance checklist

  • [ ] The filesystem and unlinked open file are matched to the owning process.
  • [ ] The selected reopen/rotation route and any data-loss trade-off are approved.
  • [ ] The old reference closes and the service writes to the intended current file.
  • [ ] Capacity, logging and application behaviour are verified through a representative rotation cycle.

References

  1. unlink(2) - Linux manual page. Accessed 2026-11-01.
  2. lsof(8) - Linux manual page. Accessed 2026-11-01.
  3. logrotate(8) - Linux manual page. Accessed 2026-11-01.
  4. du invocation (GNU Coreutils 9.12). Accessed 2026-11-01.
  5. findmnt(8) - Linux manual page. Accessed 2026-11-01.
  6. df invocation (GNU Coreutils 9.12). Accessed 2026-11-01.

Related articles