Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

Elasticsearch and OpenSearch snapshots: repositories, what is included, and the version rules for restore

Technical article
Backup & Recovery

Originally published:

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Search clusters are often described as rebuildable. The data came from somewhere else, so if the cluster is lost, it can be reindexed. That holds until the source system has already deleted older records, the reindex turns out to take days, or the cluster's own configuration (index templates, pipelines, lifecycle policies, dashboards, users and roles) proves to exist nowhere else.

Elasticsearch and OpenSearch both provide a snapshot mechanism, and both say it is the only reliable way to back up a cluster. The products share several snapshot concepts, but their documented contents, security controls and version rules differ. This guide covers what both say about repositories, contents, compatibility and restore, and notes where they differ.

Why it matters

The Elastic snapshot overview makes the configuration point directly: snapshots "preserve more than your data", and without backing up system indices and cluster state, "a disaster recovery scenario can result in the loss of your stack configuration and feature states, even if the underlying data can be restored."

It also rules out the approach many server backup routines take by default. Copying each node's data folder is not a backup, as the next section explains.

How it works

Why node data folders are not a backup

The Elastic overview is unambiguous: "Taking a snapshot is the only reliable and supported way to back up a cluster. You cannot back up an Elasticsearch cluster by making copies of the data directories of its nodes." Restoring from such a copy "may fail with reports of corruption or missing files or other data inconsistencies, or it may appear to have succeeded having silently lost some of your data."

It closes the obvious workarounds too: "You cannot fix this by shutting down nodes while making the copies, nor by taking atomic filesystem-level snapshots, because Elasticsearch has consistency requirements that span the whole cluster."

The OpenSearch snapshot documentation says the same: "Don't back up a cluster by copying the data directories of its nodes. A copy of the data directories is not a consistent backup, and restoring from such a copy can fail or silently lose data. Snapshots are the only reliable way to back up a cluster."

For anyone backing up search nodes as ordinary Linux servers, this means the image or file backup of a node protects its operating system and configuration, not its index data.

Repositories

Snapshots are stored in a repository. Elastic describes it as "an off-cluster storage location". The repository must be registered with the cluster before any snapshot can be taken or restored. OpenSearch lists the types as including "a shared file system, Amazon Simple Storage Service (Amazon S3), Hadoop Distributed File System" and others.

For a shared file system repository, Elastic says to "mount the file system to the same location on all master and data nodes" and add it to the path.repo setting on each, which "requires a rolling restart of each node."

Three rules about repositories matter for recovery:

  • One writer. Elastic: "If you register the same snapshot repository with multiple clusters, only one cluster should have write access to the repository. On other clusters, register the repository as read-only." The reason given: to prevent "multiple clusters from writing to the repository at the same time and corrupting the repository's contents." OpenSearch says the same.
  • Hands off the contents. Elastic: "Don't modify anything within the repository or run processes that might interfere with its contents." If anything else modifies it, later operations "may fail, reporting corruption or other data inconsistencies, or may appear to succeed having silently lost some of your data."
  • One repository per major version (OpenSearch). "Because the snapshot format can change between major versions, we recommend using a separate repository for each major version of OpenSearch."

The rule against touching the contents has a documented exception that matters for backing up the repository itself. Elastic says you "may however safely restore a repository from a backup as long as" the repository "is not registered with Elasticsearch while you are restoring its contents", and, once finished, "its contents are exactly as they were when you took the backup."

A repository backup also needs a coherent capture. A file-by-file copy while snapshots or retention change the repository can mix states and miss shared objects; unregistering only at restore time cannot fix that. Follow the repository-backup guidance, controlling writes during capture or using a supported atomic storage snapshot, and preserve the entire captured repository. That makes it reasonable to protect the repository with storage the cluster cannot alter, so that a compromised cluster cannot delete its own history through the snapshot interface.

How snapshots are stored

Elastic explains that snapshots are deduplicated: "a snapshot makes a copy of the index's segments and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to copy any new segments created since the repository's last snapshot." Each snapshot is still "logically independent": deleting one removes "only... the segments used exclusively by that snapshot."

Two consequences. First, frequent snapshots are cheaper than they sound, because unchanged segments are not copied again. Second, the snapshots in a repository share files, which is why deletions must go through the snapshot interface and never through the file system.

What a snapshot represents

A snapshot is not an instant. Elastic: "A snapshot doesn't represent a cluster at a precise point in time. Instead, each snapshot includes a start and end time. The snapshot represents a view of each shard's data at some point between these two times."

It copies from primary shards, and "if one or more primary shards aren't available, the snapshot attempt fails." A request with partial set to true skips unavailable shards; the result "is marked as partial and does not contain data for the skipped shard." Our recommendation: treat any partial snapshot as a failed backup for alerting purposes.

There is also a gap that catches teams out. Elastic notes: "Snapshots back up only open indices. If you close an index, it is not included in snapshots and you will not be able to restore the data." OpenSearch similarly says that if indices are not specified, "the snapshot includes all open indexes".

What is included, and what is not

By default, Elastic says, a snapshot "contains the cluster state, all regular data streams, and all regular indices." The cluster state includes persistent cluster settings, index templates, ingest pipelines, lifecycle policies, stored scripts and, for snapshots after 7.12.0, feature states.

Equally important is what is left out. Snapshots do not contain:

  • "Transient cluster settings"
  • "Registered snapshot repositories"
  • "Node configuration files"
  • "Security configuration files"

The create snapshots guide therefore recommends that self-managed clusters "take regular backups of the files in each node's $ES_PATH_CONF directory using the file backup software of your choice", noting that "these files will differ on each node, so each node's files should be backed up individually." It warns that the keystore, TLS keys and some realm key files "contain sensitive information. Consider encrypting your backups of these files."

OpenSearch's list differs slightly. It notes that "the global state does not include snapshot repository registrations" and that plugin configuration in system indexes, such as Index State Management policies, "is backed up only if those indexes are included in the snapshot."

Feature states and system indices

In Elasticsearch 8.0 and later, the Elastic overview says, "feature states are the only way to back up and restore system indices and system data streams." Feature states cover features such as security, Kibana and Watcher.

The create guide flags a security issue. The security feature state "includes system indices that may contain user names and encrypted password hashes." disclosure does not automatically grant a successful login, but a snapshot "would disclose confidential information, and if a third party can modify snapshots, they could install a back door." The documented suggestion is "a dedicated repository and SLM policy for snapshots of the cluster state", so access can be strictly limited and audited.

OpenSearch's Security plugin takes a different line. The documentation says "You can't restore snapshots that contain a global state or the .opendistro_security index", recommends excluding that index when taking snapshots, and says: "We strongly recommend against restoring .opendistro_security using an admin certificate because doing so can alter the security posture of the entire cluster." It points to a separate process for backing up security configuration. Plan for that separately.

Version rules

Restore compatibility is where many recovery plans fail, especially across upgrades.

For Elasticsearch, the overview states: "You can't restore a snapshot to an earlier version of Elasticsearch." And "any index you restore from a snapshot must also be compatible with the current cluster's version", judged by the version the index was created in. Its compatibility table shows, for example, that an index created in 6.8 can be restored to a cluster in the 9.0.0–9.5.6 range shown in the table at review only "with archive indices" (a read-only form), and that a 7.17 snapshot containing a 6.8-created index will fail on an 8.18 cluster "unless you use the archive functionality."

Elastic adds a caution that goes beyond the index layer: "Elasticsearch compatibility does not guarantee validity across versions" for feature data such as Kibana saved objects, so "a restore that Elasticsearch accepts can still fail or leave data in an unusable state." Its advice: "Do not use it as a substitute for upgrading Elastic Stack features."

For OpenSearch: "Compatibility is determined by the version of OpenSearch that created each index in the snapshot, not by the version that took the snapshot. Each index must have been created by the current major version or the previous major version. If any index in the snapshot is incompatible, you can't restore the snapshot." Also, "Restoring a snapshot to an older version of OpenSearch is not supported."

Our recommendations: record the cluster version and the oldest index creation version with your backup records. Before any major upgrade, check that your oldest snapshots will still restore on the target version. If they will not, decide in advance whether you will keep an older cluster version available for restores or accept that those snapshots become archive-only.

Restoring without damaging what is there

The restore guide sets the basic constraint: "You can only restore an existing index if it's closed and the index in the snapshot has the same number of primary shards. You can't restore an existing open index."

It offers two ways to avoid conflicts: "Delete and restore" and "Rename on restore". Renaming avoids overwriting the existing index, but still creates cluster state, consumes capacity and can restore aliases. Review those effects before allowing application access. Deleting first is simpler but riskier, and the guide warns that with action.destructive_requires_name set to false, a delete using the * or .* wildcard will, with security features enabled, "delete system indices required for authentication."

It also advises that you "temporarily stop all indexing until the restore operation is complete" to prevent the index being re-created. OpenSearch describes a specific case: if an index and its alias are deleted, "a write request to the now-deleted alias creates a new index with the same name as the alias", and then the alias "fails to restore due to a naming conflict with the new index."

Restoring a whole cluster is a larger step. Elastic warns that it "involves deleting important system indices, including those used for authentication. Consider whether you can restore specific indices or data streams instead." It also requires a snapshot "originally created with both the cluster state and feature states included." And it notes that the licence in use when the snapshot was taken is restored too.

Our recommendations: rename on restore wherever possible, compare the restored data, then switch over. Restore whole clusters onto a new, separate cluster, not over a production one. Keep a separate administrator path (Elastic's guide describes a temporary file-realm user for this) before any restore that touches security indices.

Restoring to another cluster

Both products say snapshots are not tied to the cluster that made them. Elastic: "You can create a snapshot in one cluster and restore it in another compatible cluster", and "the topology of the clusters doesn't need to match." If the original cluster still writes to the repository, register it on the new cluster "as read-only". Check capacity first; the guide says to "ensure the new cluster has enough capacity".

This is also the safest way to test backups: a separate, read-only connection from a test cluster to the production repository, used to restore and check data on a schedule.

Automating snapshots

Elastic's snapshot lifecycle management (SLM) "automatically takes snapshots on a preset schedule" and "can also delete snapshots based on retention rules you define." OpenSearch provides Snapshot Management for the same purpose. Its documentation says to use it "to take snapshots on a schedule and delete old snapshots automatically".

Our recommendation: treat automated retention as part of your backup design, not a default. Check that retention keeps snapshots long enough to cover the time it might take to notice a problem, and that the account running the schedule cannot also remove the separate copy of the repository.

What to choose and why

These are our recommendations, based on the documentation above.

  • Use snapshots, not node folder copies, for index data.
  • Back up node configuration separately, per node, encrypted where it holds keys.
  • Include cluster state and feature states, and consider a separate, tightly controlled repository for security-sensitive state.
  • Keep one writing cluster per repository; connect others read-only.
  • Protect the repository with a coherent storage-level copy the cluster cannot change. Control repository writes during capture, preserve all shared objects, and restore the whole captured state while unregistered.
  • Alert on partial snapshots and on closed indices you expect to protect.
  • Record versions and check restore compatibility before upgrades.
  • Restore by renaming or onto a separate cluster; avoid wildcard deletes.
  • Test restores to a separate cluster on a schedule.

Risks and limits

  • Node data folder copies are not restorable backups.
  • Snapshots are not a single instant; each shard is captured within a start-to-end window.
  • Closed indices are skipped.
  • Version rules can make old snapshots unrestorable after upgrades.
  • Whole-cluster restores remove system indices, including authentication.
  • Security data in snapshots is sensitive and, in OpenSearch, needs its own process.
  • Managed services behave differently; this guide covers self-managed clusters.
  • Nothing here was run. Behaviour is quoted from the documentation for planning.

Sources

This article was originally published on .