feat(roles/opensearch)!: support OpenSearch 3.x, drop 1.x

CHANGELOG.md

@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Breaking Changes
 
+* **role:opensearch**: OpenSearch 1.x is no longer supported, and the run aborts on a host that runs it. Upgrade such hosts to OpenSearch 2.19 first, following the [OpenSearch upgrade documentation](https://docs.opensearch.org/latest/install-and-configure/upgrade-opensearch/index/). OpenSearch 2.x cannot be installed on RHEL 10 and Debian 13, whose crypto policy rejects the SHA-1 signature in the OpenSearch 2.x signing key; use OpenSearch 3.x there.
 * **role:mongodb**: MongoDB 4.2, 4.4, 5.0 and 6.0 are no longer supported, since they are end of life; the role aborts on a host that runs one of them. Upgrade such hosts to MongoDB 7.0 or 8.0 first, one major release at a time, as described in the MongoDB upgrade documentation, and set `repo_mongodb__version` accordingly.
 * **role:chrony**: Without `chrony__ntp_pools` or `chrony__ntp_servers` in the inventory, chronyd synchronises with `ntp.linuxfabrik.ch` instead of `ntp.metas.ch`. Allow outgoing NTP to `ntp.linuxfabrik.ch` (195.15.242.254) where a firewall only lets the METAS servers through, or set `chrony__ntp_servers: ['ntp.metas.ch']`.
 * **role:exoscale_vm**: The role scales an existing instance and grows its disk when `exoscale_vm__service_offering` or `exoscale_vm__disk_size` differ from it, and aborts on a running instance instead of stopping it, unless `exoscale_vm__force: true` is set. Before running the role against existing instances, set both variables to their current values, or set `exoscale_vm__force: true` in a maintenance window. `exoscale_vm__state` only accepts `absent`, `present`, `restarted`, `started` and `stopped`.
@@ -54,6 +55,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+* **role:opensearch**: Supports OpenSearch 3.x. The role README describes the upgrade from 2.x.
 * **playbook:setup_basic**: Deploys the Linuxfabrik package repository on Debian, Ubuntu and RHEL 8 to 10 and their derivatives, right after EPEL; on hosts whose mirror does not serve it, dnf skips it with an error message and apt stops the run (`setup_basic__skip_repo_linuxfabrik`).
 * **role:repo_linuxfabrik, playbook:repo_linuxfabrik**: Add a role and playbook that deploy the Linuxfabrik package repository repo.linuxfabrik.ch on Debian, Fedora, the Red Hat family, SUSE and Ubuntu, with an optional testing channel.
 * **role:chrony**: chronyd can serve as a public NTP server, for example in the NTP Pool, with response rate limiting (`chrony__ratelimit`) and room for many clients (`chrony__clientloglimit`).
@@ -105,6 +107,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+* **role:opensearch**: `/etc/sysconfig/opensearch` adds the native library path of the Neural Search plugin, as the OpenSearch package does.
 * **role:acme_sh, playbooks with acme_sh**: acme.sh comes as a package from the Linuxfabrik repository, which these playbooks now deploy, and is updated by the package manager instead of updating itself from GitHub on every renewal run; existing installations below `/opt/acme.sh` are migrated on the next run, certificates are kept.
 * **role:postfix**: On hosts without an IPv6 default route, Postfix uses IPv4 only, so it no longer logs `Network is unreachable` for the IPv6 address of the next hop on part of the deliveries.
 * **role:nextcloud**: The `support` app is enabled by default, as Nextcloud ships it, so an Enterprise instance keeps its subscription record current.
@@ -160,6 +163,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+* **role:repo_opensearch**: A host switched from OpenSearch 2.x to 3.x gets the signing key of 3.x, instead of the key the controller downloaded first, which made `dnf` fail with `repomd.xml GPG signature verification error: Bad GPG signature`.
+* **role:repo_opensearch**: The repository works on Ubuntu 26.04, where `apt` ignored the signing key because of its file name.
+* **role:opensearch**: The transport layer uses the key of the transport certificate (`node-transport.key`) instead of the one of the HTTP certificate, so cluster nodes whose two certificates have different keys can talk to each other.
 * **role:repo_proxysql, playbook:proxysql**: ProxySQL 3.0 installs on RHEL 8, and ProxySQL 2.4 and 2.5 on RHEL 9, instead of failing with a 404 on the repository metadata. The `proxysql` playbook deploys EPEL (and CRB), which ProxySQL 3.0 on RHEL 8 needs.
 * **role:repo_proxysql**: With `repo_proxysql__mirror_url`, the repository is expected below the same path as on repo.proxysql.com (`/ProxySQL/proxysql-<version>.x/...`), instead of a path that never matched it.
 * **role:logstash**: `/etc/default/logstash` is no longer world-readable and no longer reported as changed after every Logstash package update, which restarted Logstash. The first run after updating LFOps restarts Logstash once.

COMPATIBILITY.md

@@ -119,7 +119,7 @@ Which Ansible role is proven to run on which OS?
 | nfs_server                            |  (x)   |  (x)   |   x    |  (x)   |   (x)   |    (x)    |    (x)    |    (x)    |                                              |
 | nodejs                                |  (x)   |  (x)   |   x    |   x    |   (x)   |    (x)    |    (x)    |    (x)    |                                              |
 | open_vm_tools                         |        |        |   x    |   x    |   (x)   |           |           |           |                                              |
-| opensearch                            |        |        |   x    |   x    |   (x)   |           |           |           |                                              |
+| opensearch                            |        |        |   x    |   x    |    x    |           |           |           | OpenSearch 2.x not on RHEL 10                |
 | openvpn_server                        |   -    |   -    |   x    |   x    |    x    |     -     |     -     |     -     |                                              |
 | php                                   |   x    |   x    |   x    |   x    |    x    |    (x)    |    (x)    |    (x)    |                                              |
 | podman_containers                     |        |        |  (x)   |   x    |   (x)   |           |           |           |                                              |
@@ -150,7 +150,7 @@ Which Ansible role is proven to run on which OS?
 | repo_mongodb                          |   x    |   x    |   x    |   x    |    x    |     x     |     x     |     x     |                                              |
 | repo_monitoring_plugins               |   x    |   x    |   x    |   x    |    x    |     x     |     x     |     x     | SLES 15, SLES 16                             |
 | repo_mydumper                         |   x    |   x    |   x    |   x    |    x    |     x     |     x     |     x     |                                              |
-| repo_opensearch                       |   x    |   x    |   x    |  (x)   |   (x)   |     x     |     x     |    (x)    |                                              |
+| repo_opensearch                       |   x    |   x    |   x    |   x    |    x    |     x     |     x     |     x     | OpenSearch 2.x not on Deb 13 and RHEL 10     |
 | repo_openvpn                          |   -    |   -    |   x    |   x    |    x    |     -     |     -     |     -     |                                              |
 | repo_postgresql                       |        |        |   x    |   x    |    x    |           |           |           |                                              |
 | repo_proxysql                         |  (x)   |  (x)   |   x    |   x    |    x    |    (x)    |    (x)    |    (x)    | RHEL 10: ProxySQL 3.0 only                   |

roles/opensearch/README.md

@@ -6,8 +6,10 @@ Note that this role does NOT let you specify a particular OpenSearch version. It
 
 Supported versions:
 
-* 1.x
 * 2.x
+* 3.x
+
+OpenSearch 2.x cannot be installed on RHEL 10 and Debian 13, see the Known Limitations of [linuxfabrik.lfops.repo_opensearch](https://github.com/Linuxfabrik/lfops/tree/main/roles/repo_opensearch).
 
 Hints for configuring TLS:
 
@@ -27,7 +29,7 @@ Any [LFOps playbook](https://github.com/Linuxfabrik/lfops/blob/main/playbooks/RE
 
 ## Single-Node Setup
 
-For a single-node setup, no special configuration is needed beyond the mandatory variables. When `opensearch__discovery_seed_hosts` is not set, OpenSearch 2.x automatically runs in single-node mode (`discovery.type: single-node`). After installation, verify that OpenSearch is running and set the default replica count to 0, otherwise the cluster health stays `yellow` (see Post-Installation Steps below).
+For a single-node setup, no special configuration is needed beyond the mandatory variables. When `opensearch__discovery_seed_hosts` is not set, the role configures OpenSearch in single-node mode (`discovery.type: single-node`). After installation, verify that OpenSearch is running and set the default replica count to 0, otherwise the cluster health stays `yellow` (see Post-Installation Steps below).
 
 
 ## Cluster Setup
@@ -192,6 +194,17 @@ ansible-playbook --inventory inventory linuxfabrik.lfops.opensearch --tags opens
 ```
 
 
+## Upgrading from OpenSearch 2.x to 3.x
+
+The role installs OpenSearch, but does not upgrade an installed one to the next major version. Upgrade a cluster one node at a time:
+
+1. Upgrade the node to the latest OpenSearch 2.19 release first. OpenSearch 3.x only joins a cluster with 2.19 nodes and cannot read indices created by OpenSearch 1.x; reindex those before the upgrade.
+2. Set `repo_opensearch__version__host_var: '3.x'` (or the `__group_var`) and adjust `opensearch__version__host_var` / `opensearch__version__group_var` if set, then deploy the repository with `--tags repo_opensearch`.
+3. Upgrade the package, for example with `dnf upgrade opensearch`.
+4. Replace `/etc/opensearch/jvm.options` with the file shipped by the package: `mv /etc/opensearch/jvm.options.rpmnew /etc/opensearch/jvm.options`. The 2.x file enables the Java Security Manager, which the JDK of OpenSearch 3.x refuses (see Troubleshooting).
+5. Run the playbook, which deploys the 3.x configuration and starts OpenSearch.
+
+
 ## Post-Installation Steps
 
 After setting up a single node or cluster, verify that OpenSearch is running:
@@ -379,7 +392,7 @@ Only optional if `opensearch__plugins_security_disabled` is `true`.
 
 `opensearch__plugins_security_disabled`
 
-* Enables or disables the OpenSearch [security plugin](https://opensearch.org/docs/1.3/security-plugin/index/), which offers encryption, authentication, access control as well as audit logging and compliance.
+* Enables or disables the OpenSearch [security plugin](https://docs.opensearch.org/latest/security/), which offers encryption, authentication, access control as well as audit logging and compliance.
 * Type: Bool.
 * Default: `false`
 
@@ -588,6 +601,11 @@ opensearch__generate_certs_nodes:
 **The run aborts with `OpenSearch X.x is not supported by this role`**
 
 * The enabled repositories offer an OpenSearch major version this role has no configuration template for. Either pin the host to a supported version, or add the matching `roles/opensearch/templates/etc/opensearch/<major>.x-opensearch.yml.j2` template and list the major in `roles/opensearch/vars/main.yml`.
+* On a host that still runs OpenSearch 1.x, upgrade to OpenSearch 2.19 by hand first, see the [OpenSearch upgrade documentation](https://docs.opensearch.org/latest/install-and-configure/upgrade-opensearch/index/).
+
+**OpenSearch fails to start after an upgrade to 3.x with `A command line option has attempted to allow or enable the Security Manager`**
+
+* `/etc/opensearch/jvm.options` is still the file of OpenSearch 2.x. Replace it with the one of the package, see "Upgrading from OpenSearch 2.x to 3.x".
 
 
 ## License

roles/opensearch/templates/etc/opensearch/2.x-opensearch.yml.j2

@@ -1,6 +1,7 @@
 # {{ ansible_managed }}
-# 2026070801
+# 2026102701
 # 2.x
+{# Based on opensearch 2.19.7 #}
 # ======================== OpenSearch Configuration =========================
 #
 # NOTE: OpenSearch comes with reasonable defaults for most settings.
@@ -140,6 +141,14 @@ cluster.initial_cluster_manager_nodes: {{ opensearch__cluster_initial_cluster_ma
 # Once there is no observed impact on performance, this feature flag can be removed.
 #
 #opensearch.experimental.optimization.datetime_formatter_caching.enabled: false
+#
+# Gates the functionality of enabling Opensearch to use pluggable caches with respective store names via setting.
+#
+#opensearch.experimental.feature.pluggable.caching.enabled: false
+#
+# Gates the functionality of star tree index, which improves the performance of search aggregations.
+#
+#opensearch.experimental.feature.composite_index.star_tree.enabled: false
 action.auto_create_index: {{ opensearch__action_auto_create_index__combined_var | default(true) | string | lower }}
 plugins.security.disabled: {{ opensearch__plugins_security_disabled | default(true) | string | lower }}
 
@@ -159,6 +168,6 @@ plugins.security.ssl.http.pemkey_filepath: 'node-http.key'
 plugins.security.ssl.http.pemtrustedcas_filepath: 'root-ca.pem'
 plugins.security.ssl.transport.enforce_hostname_verification: {{ opensearch__plugins_security_transport_enforce_hostname_verification | d(false) | string | lower }}
 plugins.security.ssl.transport.pemcert_filepath: 'node-transport.pem'
-plugins.security.ssl.transport.pemkey_filepath: 'node-http.key'
+plugins.security.ssl.transport.pemkey_filepath: 'node-transport.key'
 plugins.security.ssl.transport.pemtrustedcas_filepath: 'root-ca.pem'
 plugins.security.ssl.transport.resolve_hostname: {{ opensearch__plugins_security_transport_resolve_hostname | d(true) | string | lower }}

roles/opensearch/templates/etc/opensearch/1.x-opensearch.yml.j2 → roles/opensearch/templates/etc/opensearch/3.x-opensearch.yml.j2

@@ -1,6 +1,7 @@
 # {{ ansible_managed }}
-# 2026070801
-# 1.x
+# 2026102701
+# 3.x
+{# Based on opensearch 3.9.0 #}
 # ======================== OpenSearch Configuration =========================
 #
 # NOTE: OpenSearch comes with reasonable defaults for most settings.
@@ -79,13 +80,15 @@ network.host: {{ opensearch__network_host }}
 #discovery.seed_hosts: ["host1", "host2"]
 {% if opensearch__discovery_seed_hosts is defined and opensearch__discovery_seed_hosts | length %}
 discovery.seed_hosts: {{ opensearch__discovery_seed_hosts | ansible.builtin.to_yaml }}
+{% else %}
+discovery.type: single-node
 {% endif %}
 #
-# Bootstrap the cluster using an initial set of master-eligible nodes:
+# Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:
 #
-#cluster.initial_master_nodes: ["node-1", "node-2"]
+#cluster.initial_cluster_manager_nodes: ["node-1", "node-2"]
 {% if opensearch__cluster_initial_cluster_manager_nodes is defined and opensearch__cluster_initial_cluster_manager_nodes | length %}
-cluster.initial_master_nodes: {{ opensearch__cluster_initial_cluster_manager_nodes | ansible.builtin.to_yaml }}
+cluster.initial_cluster_manager_nodes: {{ opensearch__cluster_initial_cluster_manager_nodes | ansible.builtin.to_yaml }}
 {% endif %}
 #
 # For more information, consult the discovery and cluster formation module documentation.
@@ -94,7 +97,7 @@ cluster.initial_master_nodes: {{ opensearch__cluster_initial_cluster_manager_nod
 #
 # Block initial recovery after a full cluster restart until N nodes are started:
 #
-#gateway.recover_after_nodes: 3
+#gateway.recover_after_data_nodes: 3
 #
 # For more information, consult the gateway module documentation.
 #
@@ -103,6 +106,50 @@ cluster.initial_master_nodes: {{ opensearch__cluster_initial_cluster_manager_nod
 # Require explicit names when deleting indices:
 #
 #action.destructive_requires_name: true
+#
+# ---------------------------------- Remote Store -----------------------------------
+# Controls whether cluster imposes index creation only with remote store enabled
+# cluster.remote_store.enabled: true
+#
+# Repository to use for segment upload while enforcing remote store for an index
+# node.attr.remote_store.segment.repository: my-repo-1
+#
+# Repository to use for translog upload while enforcing remote store for an index
+# node.attr.remote_store.translog.repository: my-repo-1
+#
+# ---------------------------------- Experimental Features -----------------------------------
+# Gates the visibility of the experimental segment replication features until they are production ready.
+#
+#opensearch.experimental.feature.segment_replication_experimental.enabled: false
+#
+# Gates the functionality of a new parameter to the snapshot restore API
+# that allows for creation of a new index type that searches a snapshot
+# directly in a remote repository without restoring all index data to disk
+# ahead of time.
+#
+#opensearch.experimental.feature.searchable_snapshot.enabled: false
+#
+#
+# Gates the functionality of enabling extensions to work with OpenSearch.
+# This feature enables applications to extend features of OpenSearch outside of
+# the core.
+#
+#opensearch.experimental.feature.extensions.enabled: false
+#
+#
+# Gates the optimization of datetime formatters caching along with change in default datetime formatter
+# Once there is no observed impact on performance, this feature flag can be removed.
+#
+#opensearch.experimental.optimization.datetime_formatter_caching.enabled: false
+#
+# Path to the spill directory used by the analytics-backend-datafusion plugin. When
+# unset (default), spill is disabled and queries that exceed
+# datafusion.memory_pool_limit_bytes will fail with a "DiskManager is disabled" error.
+# Set this to a directory on a dedicated filesystem to enable spill — the directory
+# should not contend with the data path. NodeScope, Final — DataFusion's DiskManager
+# is built once at runtime startup.
+#
+#datafusion.spill_directory:
 action.auto_create_index: {{ opensearch__action_auto_create_index__combined_var | default(true) | string | lower }}
 plugins.security.disabled: {{ opensearch__plugins_security_disabled | default(true) | string | lower }}
 
@@ -122,6 +169,6 @@ plugins.security.ssl.http.pemkey_filepath: 'node-http.key'
 plugins.security.ssl.http.pemtrustedcas_filepath: 'root-ca.pem'
 plugins.security.ssl.transport.enforce_hostname_verification: {{ opensearch__plugins_security_transport_enforce_hostname_verification | d(false) | string | lower }}
 plugins.security.ssl.transport.pemcert_filepath: 'node-transport.pem'
-plugins.security.ssl.transport.pemkey_filepath: 'node-http.key'
+plugins.security.ssl.transport.pemkey_filepath: 'node-transport.key'
 plugins.security.ssl.transport.pemtrustedcas_filepath: 'root-ca.pem'
 plugins.security.ssl.transport.resolve_hostname: {{ opensearch__plugins_security_transport_resolve_hostname | d(true) | string | lower }}

roles/opensearch/templates/etc/sysconfig/opensearch.j2

@@ -1,5 +1,6 @@
 # {{ ansible_managed }}
-# 2024110601
+# 2026102701
+{# Based on opensearch 3.9.0 #}
 
 ################################
 # OpenSearch
@@ -8,9 +9,6 @@
 # OpenSearch home directory
 OPENSEARCH_HOME=/usr/share/opensearch
 
-# k-NN Lib Path
-LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/share/opensearch/plugins/opensearch-knn/lib
-
 # OpenSearch Java path
 #OPENSEARCH_JAVA_HOME=/usr/lib/jvm/java-11-amazon-corretto
 
@@ -57,3 +55,10 @@ OPENSEARCH_SD_NOTIFY=true
 # When using Systemd, this setting is ignored and the 'vm.max_map_count'
 # property is set at boot time in /usr/lib/sysctl.d/opensearch.conf
 #MAX_MAP_COUNT=262144
+
+################################
+# Plugin properties
+################################
+
+# k-NN / Neural Sparse Native Lib Path
+LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/share/opensearch/plugins/opensearch-knn/lib:/usr/share/opensearch/plugins/opensearch-neural-search/lib

roles/opensearch/vars/main.yml

@@ -3,5 +3,5 @@
 # matched against this list so an unsupported one aborts with a clear message instead
 # of a missing-file error. Extend it whenever a template for a new major is added.
 __opensearch__supported_versions:
-  - '1.x'
   - '2.x'
+  - '3.x'