Storage v1
Storage is the most critical component in a database workload. Storage must always be available, scale, perform well, and guarantee consistency and durability. The same expectations and requirements that apply to traditional environments, such as virtual machines and bare metal, are also valid in container contexts managed by Kubernetes.
Important
When it comes to dynamically provisioned storage, Kubernetes has its own specifics. These include storage classes, persistent volumes, and Persistent Volume Claims (PVCs). You need to own these concepts, on top of all the valuable knowledge you've built over the years in terms of storage for database workloads on VMs and physical servers.
There are two primary methods of access to storage:
- Network – Either directly or indirectly. (Think of an NFS volume locally mounted on a host running Kubernetes.)
- Local – Directly attached to the node where a pod is running. This also includes directly attached disks on bare metal installations of Kubernetes.
Network storage, which is the most common usage pattern in Kubernetes, presents the same issues of throughput and latency that you can experience in a traditional environment. These issues can be accentuated in a shared environment, where I/O contention with several applications increases the variability of performance results.
Local storage enables shared-nothing architectures, which is more suitable for high transactional and very large database (VLDB) workloads, as it guarantees higher and more predictable performance.
Warning
Info
EDB Postgres for Kubernetes doesn't use StatefulSet
for managing data persistence.
Rather, it manages PVCs directly. If you want
to know more, see
Custom pod controller.
Backup and recovery
Since EDB Postgres for Kubernetes supports volume snapshots for both backup and recovery, we recommend that you also consider this aspect when you choose your storage solution, especially if you manage very large databases.
Important
See the Kubernetes documentation for a list of all the supported container storage interface (CSI) drivers that provide snapshot capabilities.
Benchmarking EDB Postgres for Kubernetes
Before deploying the database in production, we recommend that you benchmark EDB Postgres for Kubernetes in a controlled Kubernetes environment. Follow the guidelines in Benchmarking.
Briefly, we recommend operating at two levels:
- Measuring the performance of the underlying storage using fio, with relevant metrics for database workloads such as throughput for sequential reads, sequential writes, random reads, and random writes
- Measuring the performance of the database using pgbench, the default benchmarking tool distributed with PostgreSQL
Important
You must measure both the storage and database performance before putting the database into production. These results are extremely valuable not just in the planning phase (for example, capacity planning). They are also valuable in the production lifecycle, particularly in emergency situations when you don't have time to run this kind of test. Databases change and evolve over time, and so does the distribution of data, potentially affecting performance. Knowing the theoretical maximum throughput of sequential reads or writes is extremely useful in those situations. This is true especially in shared-nothing contexts, where results don't vary due to the influence of external workloads.
Know your system: benchmark it.
Encryption at rest
Encryption at rest is possible with EDB Postgres for Kubernetes. The operator delegates that to the underlying storage class. See the storage class for information about this important security feature.
Persistent Volume Claim (PVC)
The operator creates a PVC for each PostgreSQL instance, with the goal of
storing the PGDATA
. It then mounts it into each pod.
Additionally, it supports creating clusters with:
- A separate PVC on which to store PostgreSQL WAL, as explained in Volume for WAL
- Additional separate volumes reserved for PostgreSQL tablespaces, as explained in Tablespaces
In EDB Postgres for Kubernetes, the volumes attached to a single PostgreSQL instance are defined as a PVC group.
Configuration via a storage class
Important
EDB Postgres for Kubernetes was designed to work interchangeably with all storage classes. As usual, we recommend properly benchmarking the storage class in a controlled environment before deploying to production.
The easiest way to configure the storage for a PostgreSQL class is to request storage of a certain size, like in the following example:
Using the previous configuration, the generated PVCs are satisfied by the default storage class. If the target Kubernetes cluster has no default storage class, or even if you need your PVCs to be satisfied by a known storage class, you can set it into the custom resource:
Configuration via a PVC template
To further customize the generated PVCs, you can provide a PVC template inside the custom resource, like in the following example:
Volume for WAL
By default, PostgreSQL stores all its data in the so-called PGDATA
(a
directory). One of the core directories inside PGDATA
is pg_wal
, which
contains the log of transactional changes that occurred in the database, in the
form of segment files. (pg_wal
is historically known as pg_xlog
in
PostgreSQL.)
Info
Normally, each segment is 16MB in size, but you can configure the size
using the walSegmentSize
option. This option is applied at cluster
initialization time, as described in
Bootstrap an empty cluster.
In most cases, having pg_wal
on the same volume where PGDATA
resides is fine. However, having WALs stored in a separate
volume has a few benefits:
I/O performance – By storing WAL files on different storage from
PGDATA
, PostgreSQL can exploit parallel I/O for WAL operations (normally sequential writes) and for data files (tables and indexes for example), thus improving vertical scalability.More reliability – By reserving dedicated disk space to WAL files, you can be sure that exhausting space on the
PGDATA
volume never interferes with WAL writing. This behavior ensures that your PostgreSQL primary is correctly shut down.Finer control – You can define the amount of space dedicated to both
PGDATA
andpg_wal
, fine tune WAL configuration and checkpoints, and even use a different storage class for cost optimization.Better I/O monitoring – You can constantly monitor the load and disk usage on both
PGDATA
andpg_wal
. You can also set alerts that notify you in case, for example,PGDATA
requires resizing.
Write-Ahead Log (WAL)
See Reliability and the Write-Ahead Log in the PostgreSQL documentation for more information.
You can add a separate volume for WAL using the .spec.walStorage
option.
It follows the same rules described for the storage
field and provisions a
dedicated PVC. For example:
Important
Removing walStorage
isn't supported. Once added, a separate volume for
WALs can't be removed from an existing Postgres cluster.
Volumes for tablespaces
EDB Postgres for Kubernetes supports declarative tablespaces. You can add one or more volumes, each dedicated to a single PostgreSQL tablespace. See Tablespaces for details.
Volume expansion
Kubernetes exposes an API allowing expanding PVCs
that's enabled by default. However, it needs to be supported by the underlying StorageClass
.
To check if a certain StorageClass
supports volume expansion, you can read the allowVolumeExpansion
field for your storage class:
Using the volume expansion Kubernetes feature
Given the storage class supports volume expansion, you can change the size
requirement of the Cluster
, and the operator applies the change to every PVC.
If the StorageClass
supports online volume resizing,
the change is immediately applied to the pods. If the underlying storage class
doesn't support that, you must delete the pod to trigger the resize.
The best way to proceed is to delete one pod at a time, starting from replicas and waiting for each pod to be back up.
Expanding PVC volumes on AKS
Currently, Azure can resize the PVC's volume without restarting the pod only on specific regions.
EDB Postgres for Kubernetes has overcome this limitation through the
ENABLE_AZURE_PVC_UPDATES
environment variable in the
operator configuration.
When set to true
, EDB Postgres for Kubernetes triggers a rolling update of the
Postgres cluster.
Alternatively, you can use the following workaround to manually resize the volume in AKS.
Workaround for volume expansion on AKS
You can manually resize a PVC on AKS. As an example, suppose you have a cluster with three replicas:
An Azure disk can be expanded only while in "unattached" state, as described in the
Kubernetes documentation.
This means that, to resize a disk used by a PostgreSQL cluster, you need to
perform a manual rollout, first cordoning the node that hosts the pod using the
PVC bound to the disk. This prevents the operator from re-creating the pod and
immediately reattaching it to its PVC before the background disk resizing is
complete.
First, edit the cluster definition, applying the new size. In this example, the
new size is 2Gi
.
Assuming the cluster-example-1
pod is the cluster's primary, you can proceed
with the replicas first. For example, start with cordoning the Kubernetes node
that hosts the cluster-example-3
pod:
Then delete the cluster-example-3
pod:
Run the following command:
Wait until you see the following output:
Then, you can uncordon the node:
Wait for the pod to be re-created correctly and get in a "Running and Ready" state:
Verify the PVC expansion by running the following command, which returns 2Gi
as configured:
You can repeat these steps for the remaining pods.
Important
Leave the resizing of the disk associated with the primary instance as the
last disk, after promoting through a switchover a new resized pod, using
kubectl cnp promote
. For example, use kubectl cnp promote cluster-example 3
to promote cluster-example-3
to primary.
Re-creating storage
If the storage class doesn't support volume expansion, you can still regenerate your cluster on different PVCs. Allocate new PVCs with increased storage and then move the database there. This operation is feasible only when the cluster contains more than one node.
While you do that, you need to prevent the operator from changing the existing
PVC by disabling the resizeInUseVolumes
flag, like in the following example:
To move the entire cluster to a different storage area, you need to re-create all the PVCs and all the pods. Suppose you have a cluster with three replicas, like in the following example:
To re-create the cluster using different PVCs, you can edit the cluster
definition to disable resizeInUseVolumes
. Then re-create every instance in a
different PVC.
For example, re-create the storage for cluster-example-3
:
Important
If you created a dedicated WAL volume, both PVCs must be deleted during
this process. The same procedure applies if you want to regenerate the WAL
volume PVC. You can do this by also disabling resizeInUseVolumes
for the
.spec.walStorage
section.
For example, if a PVC dedicated to WAL storage is present:
Having done that, the operator orchestrates creating another replica with a resized PVC:
Static provisioning of persistent volumes
EDB Postgres for Kubernetes was designed to work with dynamic volume provisioning. This capability allows storage volumes to be created on demand when requested by users by way of storage classes and PVC templates. See Re-creating storage.
However, in some cases, Kubernetes administrators prefer to manually create
storage volumes and then create the related PersistentVolume
objects for
their representation inside the Kubernetes cluster. This is also known as
pre-provisioning of volumes.
Important
We recommend that you avoid pre-provisioning volumes, as it has an effect on the high availability and self-healing capabilities of the operator. It breaks the fully declarative model on which EDB Postgres for Kubernetes was built.
To use a pre-provisioned volume in EDB Postgres for Kubernetes:
- Manually create the volume outside Kubernetes.
- Create the
PersistentVolume
object to match this volume using the correct parameters as required by the actual CSI driver (that is,volumeHandle
,fsType
,storageClassName
, and so on). - Create the Postgres
Cluster
using, for each storage section, a coherentpvcTemplate
section that can help Kubernetes match thePersistentVolume
and enable EDB Postgres for Kubernetes to create the neededPersistentVolumeClaim
.
Warning
With static provisioning, it's your responsibility to ensure that Postgres
pods can be correctly scheduled by Kubernetes where a pre-provisioned volume
exists. (The scheduling configuration is based on the affinity rules of your
cluster.) Make sure you check for any pods stuck in Pending
after you deploy
the cluster. If the condition persists, investigate why it's happening.
Block storage considerations (Ceph/ Longhorn)
Most block storage solutions in Kubernetes recommend having multiple replicas
of a volume to improve resiliency. This works well for workloads that don't
have resiliency built into the application. However, EDB Postgres for Kubernetes has this
resiliency built directly into the Postgres Cluster
through the number of
instances and the persistent volumes that are attached to them.
In these cases, it makes sense to define the storage class used by the Postgres clusters as one replica. By having additional replicas defined in the storage solution (like Longhorn and Ceph), you might incur what's known as write amplification, unnecessarily increasing disk I/O and space used.
- On this page
- Backup and recovery
- Benchmarking EDB Postgres for Kubernetes
- Encryption at rest
- Persistent Volume Claim (PVC)
- Configuration via a storage class
- Configuration via a PVC template
- Volume for WAL
- Volumes for tablespaces
- Volume expansion
- Static provisioning of persistent volumes
- Block storage considerations (Ceph/ Longhorn)